4  Techniques for LLM Deployment

Context matters

Since every new token generated is conditioned on the previously generated tokens, the response you get for a given prompt will be heavily influenced by the content of the promp, down to the exact phrasing you use. Though LLMs are notoriously good at capturing the underlying semantics of text, the actual words, the tone, the style, and even the exact order in which you construct the prompt may determine the quality of the response.

Current research suggests, for example, that LLMs tend to focus more on the beginning and final parts of the prompt and less on the middle, although this may change rapidly as novel architectures are invented. But, regardless of the idisincracies of specific models, the critical insight here is that whatever you put in the prompt will heavily influence the response, so everything important should be explicitly mentioned.

Focus matters

For the same reason, it is notoriously hard for LLMs to perform many tasks at once. The more focused the instructions are, the better–and more robust–output you’ll get. As we have seen in Chapter 13, LLMs are weak reasoners, and struggle with complicated conditional instructions. Some bigger models may be able to deal with a larger degree of flexibility, but in general you should avoid writing prompts with conditions, and instead make them as straightforward as possible.

For example, you can try prompting “if the user says X then reply Y, otherwise reply Z”, hoping the LLM will correctly classify the input and choose the right response in the same API call. This might work in many cases, depending on how good the LLM is. But the problem here is that you’re asking the LLM to solve two problems at once. Instead, you can solve the same problem more robustly with two sequential instructions: first ask the LLM to classify the user query, and then pick the corresponding prompt for that category, thus the LLM never has to explicity choose.

Reasoning requires verbosity

As we say in ?sec-training, LLMs perform a fixed amount of computation per token. This includes input and generated tokens. Thus, intuitively, a larger, more detailed prompt will tend to produce a better response, especially when some complicated reasoning is involved.

But crucially, this also applies to output generated with the LLM. The more it talks, the more computation is performing in total. For this reason, asking for very terse output is often not optimal. Instead, your prompt should incite the LLM to be verbose, explain its reasoning, summarize its key points before reaching a conclusion, etc.

But more is not always better

However, keep in mind that just stuffing the prompt with redundant information or instructions is not always better. The information density in the context matters more than its raw length.

You should strive for a minimum valuable prompt: the shortest prompt that contains the necessary amount of information to produce a succesful response. To achieve it, consider making your instructions more intentional, using more precise wording and avoiding vague terms. Provide informative examples where necessary.

But crucially, do not add irrelevant instructions, as LLMs are lousy at ignoring things. A typical issue is adding negative examples to correct some behavior and discover the LLM doubles down. The reason is simple, everything in the context will influence what the LLM generates, so negative examples are still examples that will bias the sampling process towards similar content.

Experimentation rules

All of the above being said, prompt engineering is still mostly artisanal, and far from a established science. For this reason, no amount of theory can replace good old experimentation. You should try different orderings of the instructions, different output formats, different writing styles, and see which gives you better results.

While you should write prompts that are, in general, agnostic to the specific LLM you’re using, keep in mind that the optimally-tuned prompt for one model, say GPT-4, might not be the universally best prompt. Different LLMs trained on different datasets and tuned with different strategies might have subtle differences that make one, for example, perform better with terse instructions while the other prefers verbosity. This may go down to the actual selection of words: a single word changed by an appropriate synonym may very well improve results significantly.

These principles are high-level insights that should inform how you approach prompt engineering. But keep in mind everything we know about large language models is changing very rapidly at the moment, and many of their current limitations could be fixed or at least reduced considerably with newer models, making some of these principles less relevant in the near future.

Zero-shot instructions

Zero-shot means providing a model with a single instruction and asking it to solve a problem without any additional training data or example. This should be the baseline for any new application, and is useful for complex or novel tasks where there is no existing data to draw from.

Zero-shot learning works by leveraging the model’s ability to generalize from a single instruction. By providing a clear and specific prompt, the model can use its internal knowledge and understanding to generate a solution without needing additional training data.

Some exaplmes of zero-shot learning are:

• Generating product descriptions for new or unique products.
• Translating text between languages without parallel data.
• Summarizing long documents or articles into concise overviews.

Few-shot learning

Few-shot learning involves adding a small set of examples to the model’s input to help it generalize better. This technique is particularly useful for rare or ambiguous queries, as it allows the model to learn from a small number of examples.

Few-shot learning works by providing the model with a few examples of similar queries and their corresponding answers. This helps the model learn from these examples and apply the knowledge to new, unseen queries. The few-shot approach also reinforces the output format, thus improving fidelity.

Some examples of few-shot learning are:

• Solving tasks that aren’t easy to explain concisely.
• Reinforcing an output format or response style.
• Generating recomendations of products, activities, etc., based on examples.

Role playing

Role playing involves informing the model about the audience, tone, role, and other context-specific details to bias it towards a specific complexity level, extension, or style. This technique is useful for generating responses that are tailored to a specific audience or context.

Role playing works by providing the model with context-specific information that helps it generate responses that are more relevant and engaging to the target audience. By understanding the audience and context, the model can tailor its responses to meet their needs and expectations.

Examples of role playing include:

• Writing dialogue for characters with distinct personalities and speaking styles.
• Generating social media posts tailored to different demographics and platforms.
• Drafting emails with appropriate tone and formality for different recipients.

Chain of thought

Chain of thought involves asking the model to output a detailed reasoning process before providing the final answer. This technique is useful for complex queries that require multi-step reasoning or problem-solving.

Chain of thought works by forcing the model to explicitly demonstrate its thought process. This helps ensure that the model’s responses are based on sound reasoning and logic, making them more accurate and trustworthy.

Examples where chain of thought is useful include:

• Solving logic puzzles and brain teasers by breaking down the steps.
• Providing step-by-step instructions for complex procedures or recipes.
• Analyzing data to draw insights and conclusions.

Structured output

Structured output involves instructing the model to produce the output in a structured format, such as JSON. This technique is useful for applications that require structured data, such as database queries or data analysis.

Structured output works by simplifying the parsing of the response and allowing for easier integration with downstream applications. By providing a structured format, the model can generate responses that are easily consumable and actionable.

Examples where structured ouput is useful:

• Generating tabular data like schedules, calendars, or price lists.
• Producing API responses in a standardized JSON format.
• Extracting structured information like addresses, dates, or product details from text.

Self-reflection

Self-reflection involves asking the model to evaluate its own response and determine if, given new context, it would change it. This technique is useful for identifying and correcting errors or inconsistencies in the model’s original output.

Self-reflection works by allowing the model to assess its own responses and identify potential errors or inconsistencies. By reflecting on its own output, the model can refine its responses and improve their accuracy and fidelity.

Examples of using self-reflection include:

• Identifying biased or unethical statements in the model’s own outputs.
• Detecting logical inconsistencies or contradictions in generated text.
• Refining responses based on feedback or additional context provided.

Ensembling

Ensembling involves combining the output of several models and asking a final model to produce a consensus answer. This technique is useful for improving the overall accuracy and fidelity of the response.

Ensembling works by leveraging the strengths of multiple models to generate a more accurate and reliable response. By combining the output of multiple models, ensembling can reduce the impact of individual errors and improve the overall quality of the response.

Examples of ensembling include:

• Combining outputs from models with different specialties or training data.
• Aggregating responses from models with different decoding strategies.
• Leveraging models with different strengths to produce well-rounded outputs.

Retrieval strategies

These are the most common retrieval strategies.

Query strategies

The following are ways to use the user query for generating a suitable search query.

How function calling works

First, you define a set of “functions”, which can be anything from real code functions (e.g., Python methods) to API calls. It doesn’t matter what is the underlying implementation, as the LLM will never directly interact with the function. It will just tell you when and how it should be invoked.

For that reason, you need to provide the LLM with a natural language description as well as an structured definition of the arguments of every function. This is usually all encapsulated in a standarized JSON schema, such as the following:

{
    "functions": [
        {
            "name": "get_user_info",
            "description": "get information about what a user has bought.",
            "arguments": [
                {
                    "name": "user_id",
                    "description": "The unique user identifier",
                    "type": "string",
                    "mandatory": true,
                }
            ]
        },
        {
            "name": "get_item_info",
            "description": "get information about an item's status and location.",
            "arguments": [
                {
                    "name": "item_id",
                    "description": "The unique item identifier",
                    "type": "string",
                    "mandatory": true,
                }
            ]
        }
    ]
}

Then, at inference time, a special system prompt instructs the LLM to either respond as normal or to produce a function call. The function call is a special type of structured response in which the LLM provides just a JSON object with the identifier of the function to call and the values for all mandatory arguments. An onversimplified example might be:

The following is a set of API functions you can invoke to obtain
relevant information to answer a user query.

{functions}

Given the following user query, determine whether an API call is appropriate.
If any arguments are missing from the conversation, ask the user.
If all arguments are available, output your response in JSON
format with the corresponding function call.
Otherwise, answer to the user in natural language.

{query}
```##
##
##
##

Given a prompt like the above, an well-tuned LLM should be able to determine whether, given a specific query, it needs to call an API function or not. The developer must capture these function calling replies and, instead of outputting them to the user, call the appropriate function and inject the result back into the LLM context. Then, the LLM will produce an appropriate natural language response.

An example of a possible conversation in this fictional setting would be as follows.

First, the user asks for a specific information.

```txt
USER: Hey, please show me my latest purchases.

Given this query, and an appropriate prompt like the one shown above, the LLM might recognize it needs to call the get_user_info function but it’s missing the user_id argument.

ASSISTANT: Sure, I will need your user ID for that.

The user replies back.

USER: Of course, my user ID is 12345.

Since the LLM is receiving the whole conversation story, the second time it’s called it will recognize that it has all the appropriate arguments, and produce a function call.

ASSISTANT: {"function": "get_user_info", "arguments": {"user_id": {"12345"}}}

This time, instead of showing this message to the user, the developer intercepts the function call, invokes the API, and outputs the return value, presumably a list of purchases.

TOOL: {"function": "get_user_info", "result": [ ... ]}

Given this new information, the LLM can now answer back to the user.

ASSISTANT: You have bought 3 items in the last month...

This process can occur as many times as necessary in a conversation. With a suitable prompt, the LLM can even detect when some argument value is missing and produce the corresponding natural language question for the user. This way, we can naturally weave a conversation in which the user supplies the necessary arguments for a given function call in any order. The LLM can also call more than one function in the same conversation, giving a lot more flexibility than a rigid RAG cycle.

Use cases for function calling

Function calling is particularly useful for integrating an LLM with an external tool that can be consumed as an API. A typical use case (which we will see in ?sec-shopping) is building a shopping assistant for an online store that can suggest products, add or remove items from the shopping cart, provide information on delivery status, etc.

An interesting trick is to use function calling for structured generation. When you want an LLM to produce a JSON-formatted output, it’s typically hard to guarantee you always get the exact schema you need–except maybe when using the best models. However, even some of the smaller models, once fine-tuned for function calling, are extremely robust in generating the exact argument names and types for any function. Thus, if you can frame your generation prompt as an API function call, you get all this robustness for free.

But the possibilities don’t end here. Whatever service you can encapsulate behind a reasonably well-structured and fine-grained API, you can now stamp an LLM upfront and make your API queryable in natural language. Here are some typical examples:

• Customer support: Integrate an LLM with a company’s knowledge base, product information, and customer data to create an intelligent virtual agent for customer support. The LLM can handle common queries, provide product recommetions, look up order status, and escalate complex issues to human agents.

• Information systems: Connect an LLM to a query API that provides realtime information about some specific domain, from weather to stocks. Use it for internal tools connected to a company dashboard and integrate a conversational-style interface with a traditional graphical user interface.

• Workflow automation: Connect an LLM to APIs for various business tools like CRM, project management, HR systems etc. Allow users to automate common workflows by querying the LLM in natural language, e.g. “Create a new Salesforce lead for this email”, “Schedule a meeting with the team next week”, “Approve this time off request”.

• Collaborative writing: Integrate an LLM with document editing and collaboration tools to assist with writing tasks. The LLM can help brainstorm ideas, provide feedback on tone and structure, check for grammar and spelling, and even generate content based on prompts. We will see an example of this use case in ?sec-writer.

• Software development: When combined with the powerful code generation skills of language models, another possibility opens up: connecting an LLM to code repositories, documentation, and APIs to create an AI programming assistant. Developers can ask the LLM to explain code, debug issues, suggest improvements, and even generate new code based on high-level requirements. We will see an example of this use case in ?sec-coder.

The key is to identify areas where humans currently interact with APIs and information systems, and see how an LLM can make those interactions more natural, efficient and productive.

Some caveats and limitations

As usual with LLMs, there are significant caveats and limitations to any integration. Although in general you can mitigate hallucinations considerably, the LLM can still hallucinate a wrong function call by, e.g., passing the wrong arguments. In the simplest case, maybe you can catch that error when arguments have the wrong type or are out of range. However, subtle hallucinations might result in a function call that succeeds but wasn’t the user intention.

For this reason, in all critical systems it is crucial that you don’t simply call an API blindly on behalf of the user, specially when doing so can have irreversible effects. For example, in a banking app, your LLM might hallucinate an incorrect destination in a transference, effectively sending the user money to an arbitrary third party. Furthermore, hackers might find a way to mess with your prompt and trigger the hallucination.

In these cases, you should always make the user explicitely trigger the final action, and make sure they have reviewed and understood the implications of such action. This enhances reliability at a small cost in usefulness, turning the LLM into an assistant that fills in the data for you, but doesn’t click the red button.

Another possible source of concern is when the LLM hallucinates the response, even though it made the right call and received the right data. This is the same problem we had with RAG: even if the context contains the right answer, there is no guarantee the LLM will pick it. One easy fix in many cases is to display the function result next to the LLM interpretation, so the user can double check the response.

One final caveat that may be relevant in many cases is regarding privacy. If you are interacting with a private API–say, a banking app–using a commercial LLM, you are effectively sending to OpenAI (or any other provider) you users’ information as part of the prompts, and this may include user IDs, addresses, financial details, etc. This underscores the need for powerfull open source LLMs that companies can self-host for added privacy and security.

RAG vs Function Calling

Function calling can be seen at the same time a special case and a generalization of retrieval augmented generation. It is a special case because it involves injecting external information in the prompt to enhance the capabilities of an LLM. It is a generalization because you can implement RAG with function calling, simply by encapsulating your search functionality in a function call specification.

This pattern is extremely flexible, but at the same time it’s very repeatible. However, to make it work, it is crucial to get the prompt right. Since prompts are, in general, not entirely portable across different models, implementing this workflow from scratch every single time is a chore.

For this reason, most LLM services provide a native way to perform function calling, basically abstracting away the fragile prompt engineering component. Moreover, the LLM provider might have fine-tuned their model to a specific function-calling prompt and formatting. And since most LLM providers implement the OpenAI API specification, porting function calling between different providers is way easier.

How code generation works

In the simplest case, you can think of code generation as a subset instance of text generation. If your model is trained on mixed natural language and code input, and/or fine-tuned with coding instructions, it will naturally learn to answer to some prompts with actual code. For example, you can train an LLM on programming contests, where the input is a problem statement, and the output is the code of the solution in some programming language.

It is at least somewhat surprising that vanilla LLMs, trained on code, can learn to write code at all. The reason to be skeptic is that programming languages have very strict syntax rules which make it hard, at least in principle, for a purely statistical language model to produce something that is just parseable code, not to mention semantically correct. Failing to produce a single semicolon in the exact location can make incorrect an otherwise perfect piece of code. Yet, LLMs learn to code, almost without additional effort.

In fact, most general-purpose models now available have at least some general capabilities for code generation, if only because they are trained of vast datasets that contain, among the many types of text modalities, lots and lots of code. And even if you don’t want an LLM explicitely for code generation, training on code and text (rather than just text) has shown to improve the general reasoning capabilities of a model, even for text-only tasks! But although you can get reasonably good code generation almost for free, the best coding LLMs are fine-tuned on precise text-and-code datasets.

There are many reasons to prefer a model fine-tuned for coding to a general one. The simplest argument is that, contrary to natural language, highly plausible code can still be incorrect. Fine-tuning a model specifically on code reinforces the syntax rules and makes it much less likely to generate almost-correct but still syntactically wrong code.

In the same vein, since programming languages are much more rigid in terms of syntax than natural language, fine-tuning can make a smaller model as good or even better than larger, general models, if focused on a specific language. Likewise, even if your general LLM can code Python, it may not know the specific framework you’re interested in, or code with the exact style you want.

Code generation use cases

In this section we’ll look at code generation from a high-level perspective, to understand what are the most interesting use cases it unlocks. We won’t go in-depth into technical details of how to make these use cases work in practice, as we will have plenty of time in Part 3 to see concrete examples in action.

Prompting tips for code generation

In many common cases, you can make an LLM write code simply by asking. A prompt like “Generate a Python function to find the third maximum element from a list” will work almost flawlessly in any sufficiently capable language model you can find today. And this works fine for many use cases where the code is all you need. At least is no worst than searching for a similar snippet of code online.

However, there are several drawbacks with this KISS approach. First, most LLMs you’ll find online are fine-tuned for chat, so they are… chatty. Instead of the raw code, they might answer with something like “Sure, here is a function in Python to do ….” and then the code. This makes it hard to integrate them with external tools that need just the code, because then you have to parse the response.

In many cases, you can get away by adding an explicit instruction like “Please answer just with the source code”, but still, some models may refuse. And even if they comply, different models output code in different formats. Some will enclose the code in markdown-style code block annotations while other models might indent the code. It depends heavily on their training data.

Another problem you may face is when asking for one-liners, i.e., single instructions or expressions that you want to evaluate with, e.g., the eval function in Python. If you ask for a single pandas expression to, say, group and filter a dataset, the model may sometimes produce a propper expression—e.g., df.groupby(...).agg(...)—and other times an instruction—e.g., df = df.groupby(...). You may work around these issues by doing some checking and post-processing of the response, like removing anything before the last = sign, but this is a very brittle approach.

In these cases, some of our well-known prompt techniques also apply. Be very intentional with the prompt and provide positive examples of the exact response format you expect. While none of this will 100% guarantee you’ll get a response in the format you need, when paired with a try-and-repeat strategy, you can often get away with the performance you need. For example, if the model makes a mistake 10% of the time, you’ll need to redo one in ten queries on average, which is not that terrible all things considered.

In many cases, when retrying the same code generation task, it helps including in the prompt the previous answer and the error. This can often be automated simply by trying to run the code, capturing any exceptions, and feeding the model back with the exception message asking it to try and fix it.

Finally, with some tricks, we can force the LLM to produce syntactically-correct code—even if not guaranteed to be semantically valid. The trick is to restrict the sampling step to only select among the tokens that would be syntactically valid.

Some open-source LLM inference engines, like llama.cpp, support passing a formal grammar that defines the syntax of the programming language. During sampling, the engine will select among the top-k tokens only those that are valid according to the production rules of the grammar. This can be done efficiently with a linearly bound automaton that can be constructed automatically from the formal grammar definition. While this is a relatively novel and, arguably, rather advanced feature, some commercial APIs, like <fireworks.ai> are starting to support it.

Limitations and caveats

Needless to say, code generation is full of subtle and not-so-subtle problems. For starters, some hallucinations are going to happen, and this might result in several different types of problems. The simplest case is getting code that is not syntactically-correct, that is, code that doesn’t parse. If this is your main problem, then you’re lucky because this is simple to check. Just run a linter for your target language and retry if you find any syntax errors.

A more complicated issue is when your model generates syntactically-correct code that throws an exception. This is still not so terrible because you can run the code and check for exceptions. However, running code generated by an LLM is a bad idea if you don’t have some guardrails in place. For all you know, the code may have an instruction to wipe out your hard drive. So you must always run LLM-generated code in a sandboxed environment. This is especially true when you’re running code generated by a user-facing LLM. Someone will hack that LLM to generate some system-breaking instruction.

The third level of problem is when your LLM generates code that runs without exceptions but doesn’t do the right thing. This is, in general, impossible to detect beforehand. However, depending on your specific use case, you may be able to check the result is what you expect, and even roll-back any potential side effects if that isn’t the case. For example, you can have your code work on a copy of the relevant data, and check for any unexpected changes before merging that data back.

This is, however, the most important open problem in program synthesis from natural language, and one that, I believe, will require a new paradigm that goes beyond statistical language modeling to fully solve it.

Grammar-restricted output is is one of the most effective ways to make code generation more robust. Still, this process is ad-hoc, not baked into the training process. Thus, it is entirely possible for the LLM to get stuck simply because it doesn’t give a high probability to any valid tokens. If the LLM wouldn’t naturally produce, at least with some non-trivial probability, the right response, there is no ad-hoc filtering mechanism that can force it to generate correct code.

This means adequate prompting and possibly fine-tuning for specific domains will remain relevant strategies in the near term.

Adding natural language

So far we haven’t used the fact our agent is LLM-powered, and for good reason. The basic agent architecture we just defined is agnostic to implementation details. Adding an LLM to the agent architecture just means some, or all of the modules are implemented using language models: that is, with prompts, RAG, function calling, and everything else we’ve seen so far.

First, perception may be LLM-based: your agent can receive information from the environment in the form of natural language documents. Likewise, the actuators can be LLM-based, e.g., via function calling. But more importantly, the reasoning process and the internal representation may involve language models as well. The internal representation can be a straightforward collection natural language claims about the environment, which the agent updates and refines via reasoning. And the reasoning itself can be guided with some clever prompting and thus occur entirely in a linguistic framework. Finally, the environment itself can contain other language models, or otherwise be implemented also as a language-based simulation with clever prompts.