When does it make sense to use an LLM?

As we build out the Inscribe product, we often find ourselves solving a series of smaller tasks. These tasks might be something like "Is this document a bank statement" or "Is this address real?" or "Does this person work at this company?"

Each decision about which technology to use—whether a Large Language Model (LLM), an API, a machine learning (ML) model, or simple hardcoded logic—can impact the efficiency, cost, and long-term maintainability of our product.

The question then becomes: How do we choose the right tool for each task?

Matching the tool to the task

Hardcoded approaches

For many tasks, especially those that are well-defined and straightforward, starting with a "low tech" hardcoded solution often makes the most sense. These approaches—such as using regular expressions (regex), switch statements, or simple rule-based algorithms—are the quickest and most resource-efficient way to get a solution up and running.

Benefits of hardcoded logic:

• Speed of implementation: Hardcoded logic is quick to develop, making it ideal for getting a basic solution up and running with minimal time investment.
• Cost efficiency: These approaches are cost-free from a resource standpoint, as they do not require external dependencies or computational resources beyond the initial development and runtime.
• Independence: Since they rely solely on internal logic, they are immune to external factors such as API changes or third-party service outages.

However, these advantages come with significant trade-offs:

Drawbacks of hardcoded logic:

• Limited utility: Hardcoded logic is inherently limited in scope. It works well for specific, well-understood scenarios but struggles with edge cases or variability in inputs.
• Deterministic nature: Such approaches are rigid, executing the same way every time, regardless of context or nuance.
• Inflexibility: As the requirements evolve, maintaining and updating hardcoded logic can become cumbersome.

Hardcoded logic is an excellent starting point for simple tasks, but it quickly reaches its limits as complexity increases.

The role of APIs

When a task demands more flexibility or functionality than hardcoded logic can provide, and when a third-party solution already exists, using an API can be a highly effective next step. APIs are particularly useful for tasks that involve data retrieval, verification, or integration with outside services.

• Speed: Many APIs process requests in tens or hundreds of milliseconds, offering near-instantaneous results.
• Cost-effectiveness: In many cases — but not always! — APIs are more cost-efficient than deploying an LLM or building a custom solution.
• Ease of integration: APIs are typically straightforward to integrate, requiring only simple HTTP requests, which makes them an attractive option for quickly adding new capabilities to a product.

However, relying on APIs also introduces several potential risks and challenges:

• Business continuity risks: APIs provided by third parties can go out of business, or their services can be discontinued.
• Versioning: Although most APIs are strict with versioning, occasionally they can introduce breaking changes that cause errors in our product.
• Data management complications: Since using an API can involve sending sensitive data, any vendor must pass our rigorous security requirements.

If an API exists to solve the exact problem we're tackling, and it's secure, cheap, and fast, it could make sense to use.

Leveraging ML models

As tasks become more complex and require a higher degree of precision, machine learning models offer a robust solution. ML models excel in scenarios where hardcoded logic and APIs fall short, particularly in cases that involve pattern recognition, prediction, or decision-making based on large datasets.

Strengths of ML models:

• High precision: Once properly trained, ML models can deliver high accuracy for specific tasks, often outperforming general-purpose solutions like LLMs.
• Speed at scale: After deployment, ML models can offer fast inference times, making them suitable for high-throughput applications.

However, the power of ML models comes with development and operational overhead:

• Development complexity: Building an ML model requires significant expertise and resources. The process involves data collection, labeling, model training, validation, and fine-tuning.
• Data dependency: The effectiveness of an ML model depends on the quality and quantity of training data, which is not always straightforward to obtain
• Limited generalization: ML models are designed for specific tasks and may struggle to generalize beyond their training data.

ML models are best reserved for tasks where their high precision provides a clear advantage over other approaches, and where the high development and maintenance costs can be justified.

But… maybe start with an LLM

Large Language Models (LLMs) represent a significant advancement in AI, offering generative capabilities that extend beyond the scope of traditional ML models.

The latest frontier LLMs have been trained on so much data that they can often get you 80% of the way there performance-wise (or beyond) compared to a specialized ML model — of course, depending on use case. And the newest ones like Claude or GPT-4o are multimodal so they work for images as well as text.

Experimenting with an LLM approach requires no training data and no infrastructure, so it's a great place to start. You can even finetune at runtime by using in-context learning examples in the prompt.

LLMs can be slow and expensive per request, and providers often impose rate limits. But it's worth it in terms of iteration speed when building new features.

And LLMs can do things that ML models can't…

Like reasoning on arbitrary inputs. This is the big capability that is enabling Inscribe's AI analysts. It allows us to provide an equivalent of human-like reasoning which can analyze complex and subtle scenarios. And it's getting better fast.

State-of-the-art solutions embrace a strategic, nuanced approach

Selecting the right tool for a given task is not just about solving the problem at hand—it's about doing so in a way that aligns with your broader goals of efficiency, scalability, and maintainability.

At Inscribe, we've found that a balanced, strategic approach works best:

1. Start with the basics: For straightforward tasks, begin with hardcoded logic. It's quick, cost-effective, and easy to implement.
2. Leverage existing APIs: If a reliable and cost-effective API exists, use it to save development time and resources.
3. Try an LLM: When facing a new or complex task, LLMs offer a quick and flexible starting point. Their ability to handle a wide range of inputs makes them a valuable tool for rapid prototyping as well as production.
4. Consider ML for precision: For tasks that require high precision or customization, and where the investment in development can be justified, ML models offer a powerful solution.
5. Embrace hybrid approaches: In many cases, the best solution involves a combination of these approaches. By leveraging the strengths of different tools, you can create a more robust and adaptable product.

At Inscribe, we continually explore and refine our approach to technology selection, ensuring that we continue to deliver the best possible solutions for our customers.

We're committed to helping you navigate these decisions and implement the best solutions for your specific needs. Our team has deep expertise in deploying the right mix of technologies to solve even the most complex problems.

Ready to see how Inscribe can help you? Request a Demo today to speak with one of our experts. We'll walk you through our approach, discuss your unique challenges, and show you how our AI-driven solutions can elevate your business. Whether you're looking to streamline operations, enhance accuracy, or innovate with cutting-edge AI, we're here to partner with you every step of the way.