Fine-tuning Large Language Models (LLMs)

Fine-tuning Large Language Models (LLMs) refers to the specialized process of adapting pre-trained, general-purpose language models to excel in specific tasks or domains. This is achieved by further training these models on smaller, tailored datasets that consist of <input, output> pairs, which are representative examples of the desired behavior or output.

By updating the model’s parameters during this process, fine-tuning effectively narrows the gap between the broad capabilities of generic pre-trained models and the nuanced needs of specific applications. This ensures that the model’s performance is significantly improved and more closely aligned with human expectations for the task at hand.

Fine-tuning, on the other hand, allows for further customization by adding new knowledge to the model itself so that it learns – or adapts its learned knowledge – to master specific tasks. It is a supervised learning task based on labeled datasets to update the model’s weights. The demonstration datasets are usually prompt-response pairs that indicate the kind of refined knowledge needed for a particular task.

Dataset preparation is a costly process. Automating parts of this process is a vital step towards offering a scalable solution for fine-tuning LLMs for enterprise use cases.

Here’s an example: Suppose you want to customize a model to be able to generate social media posts following your company’s marketing strategy and tone.

As an organization you might already have a sufficient record of such posts which you can use as golden outputs. These outputs form a Knowledge Base from which you can generate key content points using RAG. The generated content points paired with the corresponding outputs can form your dataset to fine-tune the model to master this new skill.

Fine-tuning and RAG are not mutually exclusive. On the contrary, combining both in a hybrid approach may improve the model’s accuracy and it is definitely a direction worth investigating.

A recent study by Microsoft shows how capturing geographic-specific knowledge in an agricultural dataset generated using RAG resulted in a significant increase in the accuracy of the model that was fine-tuned on that dataset.

Making fine-tuning LLMs less of a black box and more accessible to the enterprise involves making each step in the workflow as simple and transparent as possible. A high-level workflow involves the following steps:

1. Experimenting with prompting different LLMs and selecting a baseline model that fits one’s needs.
2. Defining a precise use case for which a fine-tuned model is needed.
3. Applying automation techniques to the data preparation process.
4. Training a model preferably with some default values for the model’s hyperparameters.
5. Evaluating and comparing different fine-tuned models against a number of metrics.
6. Customizing the values for the model’s hyperparameters based on feedback from the evaluation step.
7. Testing the adapted model before deciding that it’s good enough to be used in actual applications.

There are several key factors that need to be considered before deciding on how to adapt a generic model to specific business needs. Fine-tuning usually comes into play when instructing the model to perform a specific task fails or does not produce the desired outputs consistently. Experimenting with prompts and setting the baseline for the Small Language Models performance is a first step toward understanding the problem or task.

In order to do so, one has to understand the exact use case that needs to be addressed and decide on the best way to ground the model’s responses to match the business expectations reliably. There are various ways to give context to a general-purpose generative model. Fine-tuning and RAG (Retrieval Augmented Generation) are two popular approaches.

In cases where an out-of-the-box model is missing knowledge of domain or organization-specific terminology, a custom fine-tuned model also called a domain-specific LLM might be an option for performing standard tasks in that domain or micro-domain.

BloombergGPT and Med-PaLM 2 are examples of LLMs that were trained from scratch to have a better understanding of highly specialized vocabulary found in financial and medical texts, respectively. In the same sense, domain-specificity can be addressed through fine-tuning at a smaller scale.

Fine-tuning is finally effective when cost or latency needs to be reduced during inference. A fine-tuned model may be able to achieve high-quality results in specific tasks with shorter instruction prompts. However, you have to be aware that interpreting or debugging the predictions of a fine-tuned model is not trivial. Different factors like data quality, data ordering, and the model’s hyperparameters may impact the performance.

Fine-tuning heavily relies on accurate, targeted datasets. Before fine-tuning a model, one should first make sure that enough representative data is available so that the model does not overfit on limited data. Overfitting describes a model’s limited ability to generalize to new data.

Leveraging a full-stack Generative AI platform to address the workflow above essentially means connecting the dots between:

• Prompt engineering and evaluating prompts for different models;
• Using knowledge retrieval to generate demonstration data;
• Fine-tuning a model with generated data.

This is the future for operationalizing LLMs for the enterprise and reducing time, effort, and cost for maximizing their adaptability to specific skills. Book an AI demo with Aisera to find out more about our Generative AI capabilities.