LLM Fine-Tuning End to End | Part 1: Methods, Data, and Training Optimization

A product team fine-tunes a 7B model on 500 support tickets, watches the loss drop, and ships it. A week later the model is worse than the base model at everything except parroting the exact phrasing of those 500 tickets. It forgot how to reason, picked up the support team's typos as gospel, and confidently hallucinates ticket numbers. The training loss looked perfect the whole time.

That failure is not bad luck. It is what happens when fine-tuning is treated as a single button instead of an engineering discipline with a dozen real decisions. Fine-tuning is one of the highest-leverage and most misunderstood techniques in applied AI. Done well, it gives you a small, fast, specialized model that beats a giant general one on your task at a fraction of the serving cost. Done carelessly, it produces a model that is expensive to train, worse than where you started, and impossible to debug.

This is a two-part series. Part 1 covers everything up to a trained model: what fine-tuning is, when it is the right tool, the full spectrum of methods, how to build the dataset, and the optimization strategies that make training actually fit on the GPUs you can afford. Part 2 covers serving that model to billions of requests efficiently and monitoring it in production.

What fine-tuning actually is#

A pretrained LLM has already learned language, facts, and reasoning from trillions of tokens. Fine-tuning continues that training on a much smaller, targeted dataset so the model adapts its behavior to your specific task or domain. Mechanically it is the same gradient descent that produced the base model. You are just nudging the existing weights rather than learning from scratch.

You are editing a model that already knows almost everything it needs. Pretraining is the expensive part: millions of dollars and months of compute. Fine-tuning rides on top of that investment, which is why it can cost a few dollars and an afternoon. You are not teaching the model English. You are teaching it your format, your domain vocabulary, your tone, or your judgment.

When fine-tuning is the right tool (and when it is not)#

Fine-tuning competes with two cheaper techniques, and reaching for it first is the most common mistake. Prompting changes behavior with zero training by putting instructions and examples in the context. RAG gives the model new knowledge at query time by retrieving documents, covered in depth in our RAG series. Fine-tuning changes the weights themselves.

The decision rule is about what kind of change you need. Fine-tuning is the right tool for behavior, format, and style: consistent JSON output, a specific tone, a narrow classification task, or a skill the base model is weak at. It is the wrong tool for knowledge that changes. Use RAG there, because re-training every time a document updates is absurd. We unpack this tradeoff fully in Fine-Tuning vs RAG for AI Products.

The method spectrum#

"Fine-tuning" is an umbrella over several distinct techniques that happen at different stages and optimize different things. Understanding where each fits is the difference between picking the right method and cargo-culting whatever a tutorial used.

Continued pretraining#

Before any instruction tuning, you can keep doing plain next-token prediction on a large corpus of unlabeled domain text. This is how you teach a general model the vocabulary and patterns of medicine, law, or a low-resource language it barely saw in pretraining. It is the most data-hungry option (you need a lot of raw text), and it is usually a first stage, followed by SFT to make the model usable.

Supervised fine-tuning (SFT)#

SFT is what most people mean by fine-tuning. You train on pairs of input and ideal output, the (prompt, response) pair, and the model learns to produce that response style. This is instruction tuning when the data is instructions and answers, and it is the workhorse for teaching format (always return valid JSON), task behavior (classify this ticket), or a specialized skill.

The critical detail is loss masking: you compute the training loss only on the response tokens, not the prompt tokens, so the model learns to generate the answer rather than to predict the question. Modern training libraries handle this, but getting it wrong silently wrecks results.

Preference alignment: RLHF and DPO#

Some qualities cannot be captured in a single gold answer: "be helpful but not sycophantic," "refuse unsafe requests gracefully," "prefer concise answers." For these you train on comparisons. Given a prompt and two responses, a human (or a model) labels which is better, and training pushes the model toward the preferred style.

The original approach, RLHF (Reinforcement Learning from Human Feedback, Ouyang et al., arXiv:2203.02155), trains a separate reward model and optimizes against it with reinforcement learning. It is powerful but complex and unstable. DPO (Direct Preference Optimization, Rafailov et al., arXiv:2305.18290) achieves similar results by optimizing the preference objective directly on the model, with no separate reward model and no RL loop. By 2026 DPO has displaced RLHF for alignment in most production teams. It is cheaper, more stable, and comparable in quality. SFT first, then DPO to polish, remains a reliable recipe.

DPO itself now has a family of variants, each fixing a specific weakness, worth knowing by name so you can reach for the right one. ORPO (Odds Ratio Preference Optimization) folds preference learning into a single SFT-style stage, removing the separate reference model and the two-step pipeline. KTO (Kahneman-Tversky Optimization) needs only a binary "good/bad" label per response instead of paired comparisons, which makes data collection far easier. You can use thumbs up/down from production. SimPO drops the reference model entirely and uses length-normalized rewards to curb the verbosity bias DPO can introduce. Start with DPO; switch to KTO when you only have binary feedback, ORPO when you want a one-stage pipeline, or SimPO when outputs are drifting long. A 2024 DPO survey catalogs the full landscape.

Reinforcement learning for reasoning: GRPO and RLVR#

The biggest shift in fine-tuning since the original DPO work is the rise of reinforcement learning for reasoning, popularized by DeepSeek-R1. The idea is RLVR, Reinforcement Learning with Verifiable Rewards. Instead of a learned reward model or human preferences, you reward the model based on whether its output is objectively correct. A math answer either equals the solution or it does not; code either passes the tests or it does not. That verifiable signal is cheap, unhackable, and exactly what reasoning tasks need.

The algorithm of choice is GRPO (Group Relative Policy Optimization), introduced by DeepSeek to train their R1 models. GRPO strips away the machinery of classic RLHF: there is no separate value model and no learned reward model. For each prompt, it samples a group of responses (typically 8), scores each with your reward function, and standardizes those scores within the group (a z-score) to produce the advantage that drives the update. Removing both the value and reward networks makes GRPO memory-efficient enough to run with QLoRA, so modern implementations (Unsloth, TRL) can do GRPO on an 8B model on a single GPU.

The skill in RLVR is designing the reward function. Favor several small, composable scoring components over one monolithic reward: for an email task you might give +1 for including the required keyword, +1 for naming the recipient, and −1 for excessive length, rather than a single opaque score. Reward shaping that is too coarse or gameable can actively degrade the model, so this is where the engineering effort goes.

# A GRPO reward function: composable, verifiable signals
def reward_fn(prompt, completion, answer):
    score = 0.0
    score += 3.0 if extract_final_answer(completion) == answer else -3.0  # correctness
    score += 0.5 if "<reasoning>" in completion else 0.0                   # used the format
    score -= 0.5 if len(completion) > 2000 else 0.0                        # not too long
    return score

Use GRPO/RLVR when correctness is automatically checkable: math, code generation, symbolic tasks, structured extraction. For open-ended quality (tone, helpfulness) where there is no ground truth to verify against, stick with SFT and DPO. The two families are complementary, not competing.

Choosing a base model in 2026#

The method matters less if you start from the wrong base. The open-weight landscape in 2026 is crowded and strong, and the right choice balances capability, size, license, and ecosystem maturity.

• Qwen3 (Alibaba) is the most versatile family for fine-tuning: Apache 2.0, sizes from a 0.5B edge model to a 235B MoE flagship, strong multilingual support, and the deepest ecosystem of existing fine-tunes and quantizations.
• Llama 4 (Meta) offers competitive performance with full weight access; mature tooling, though check the license terms for your use case.
• DeepSeek models (MIT-licensed) are the reasoning and coding specialists, and the reference point for GRPO-style training.
• Mistral and GLM round out the options with mature fine-tuning ecosystems and permissive licenses (GLM under MIT).

Two practical rules. First, start as small as your task allows. A fine-tuned 7–8B model often beats a general 70B one on a narrow task while costing a fraction to serve (the subject of Part 2). Second, check the license before you invest. Qwen3 (Apache 2.0), DeepSeek and GLM (MIT) are the safest for unrestricted commercial fine-tuning and deployment.

Full fine-tuning vs parameter-efficient fine-tuning (PEFT)#

This is the single most consequential choice for your budget, and it is orthogonal to the method above. You can do SFT or DPO either way.

Full fine-tuning updates every weight in the model. For a 7B model in mixed precision, the optimizer state alone (with Adam) needs roughly 12 bytes per parameter, around 84 GB just for optimizer and gradients, before activations. That requires multiple high-end GPUs for even a small model, and it produces a full-size copy of the model per task.

PEFT freezes the pretrained weights and trains a small number of new parameters instead. You get most of the quality for a tiny fraction of the memory, and the result is a few megabytes of adapter weights rather than a multi-gigabyte model. For the overwhelming majority of use cases, PEFT is the correct default.

LoRA: the technique that changed the economics#

LoRA (Low-Rank Adaptation, Hu et al., arXiv:2106.09685) is the dominant PEFT method. The insight is that the change a fine-tune makes to a weight matrix is low-rank. It can be approximated by the product of two small matrices. So instead of updating a huge weight matrix W, you freeze it and learn two small matrices A and B whose product is added to it. You train maybe 1% of the parameters, and at inference you can merge the adapter back in for zero overhead, or keep it separate to swap adapters per request (the key to multi-tenant serving in Part 2).

The main knobs are rank r (the size of the small matrices, where higher means more capacity and more parameters) and alpha (a scaling factor). The 2026 consensus starting point is r=16 with alpha=16 (a 1.0 scaling factor), a learning rate of 2e-4 with cosine warmup, and lora_dropout=0.05. For high ranks (64+), the older alpha=2r rule and enabling rsLoRA (rank-stabilized scaling) keeps training stable. Push rank to 32–64 only for complex reasoning or code tasks where the extra capacity earns its keep.

Two refinements are now defaults worth adopting. Target all linear layers (target_modules="all-linear", the q/k/v/o attention projections plus the gate/up/down MLP projections) rather than attention only; current benchmarks show it consistently helps with minimal extra VRAM. And consider DoRA (below) as a near-free quality bump. One honest caveat from the research: LoRA "learns less and forgets less" than full fine-tuning (arXiv:2405.09673). It is more resistant to catastrophic forgetting, but on tasks demanding large behavior shifts it can underfit, which is the rare case where full fine-tuning still wins.

QLoRA: fine-tuning a 70B model on one GPU#

QLoRA (Dettmers et al., arXiv:2305.14314) combines LoRA with 4-bit quantization of the frozen base model. The base weights are stored in a 4-bit format (NF4) to slash memory, while the LoRA adapters train in higher precision. The result is dramatic: you can fine-tune a model far larger than would otherwise fit, on a single consumer or workstation GPU, with minimal quality loss. QLoRA is why fine-tuning a capable model is now within reach of a solo developer.

The rest of the PEFT family#

LoRA is not the only option, though it is the default. DoRA (Weight-Decomposed LoRA, arXiv:2402.09353) splits weights into magnitude and direction for a quality bump. Prefix/prompt tuning learns soft prompt vectors instead of modifying weights. IA³ scales activations with learned vectors. Adapters insert small trainable layers between existing ones. In practice, start with LoRA or QLoRA; reach for the others only if you have a specific reason.

The dataset is the product#

Here is the truth that tutorials skip: your dataset matters more than your method, your hyperparameters, and your base model combined. A model fine-tuned on 1,000 excellent, diverse examples beats one trained on 100,000 noisy, repetitive ones. The support-ticket disaster from the intro was a data problem, not a training problem.

Quality, diversity, and quantity, in that order#

Quality means every example is correct, well-formatted, and represents what you actually want the model to do. One wrong example teaches a wrong pattern. Diversity means the examples cover the real distribution of inputs, including edge cases. Train only on easy cases and the model fails on hard ones. Quantity matters least of the three: a few thousand high-quality, diverse examples is often enough for SFT, and more bad data actively hurts.

Formatting and chat templates#

The model must see data in exactly the format it will see at inference, including the special tokens that mark turns. Every instruction-tuned model has a chat template, the precise way system, user, and assistant turns are wrapped. Using the wrong template is a silent, common bug: the model trains on a format it will never see served, and quality quietly collapses.

# Each training example follows the model's chat template
messages = [
    {"role": "system", "content": "You are a support classifier. Return one label."},
    {"role": "user", "content": "My payment failed three times today."},
    {"role": "assistant", "content": "billing_issue"},
]
# The tokenizer applies the model's exact template + special tokens
text = tokenizer.apply_chat_template(messages, tokenize=False)

Avoiding catastrophic forgetting#

When you train hard on a narrow dataset, the model can forget general capabilities, the "worse at everything else" failure. Two defenses: keep the learning rate low and the training short (you are nudging, not retraining), and mix in general-purpose data so the model retains broad ability while learning your task. A fine-tune that destroys the base model's reasoning to memorize your format is a net loss.

A minimal SFT run#

Here is what a parameter-efficient SFT looks like with the Hugging Face stack (transformers, peft, trl). This trains a LoRA adapter on a chat dataset.

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

bnb = BitsAndBytesConfig(
    load_in_4bit=True, bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-8B", quantization_config=bnb, device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")

from peft import LoraConfig, get_peft_model

lora = LoraConfig(
    r=16, lora_alpha=16, lora_dropout=0.05,   # 2026 default: alpha = r
    target_modules="all-linear",              # attention + MLP projections
    use_dora=True,                            # weight-decomposed LoRA, small quality gain
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora)
model.print_trainable_parameters()   # ~0.5% of total params

from trl import SFTTrainer, SFTConfig

trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    args=SFTConfig(
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        learning_rate=2e-4,
        num_train_epochs=2,
        bf16=True,
        gradient_checkpointing=True,
    ),
)
trainer.train()

trainer.model.save_pretrained("./support-classifier-lora")

You rarely write this from scratch in 2026. The standard toolchain: Unsloth for single-GPU training, which claims 2–5× faster training and up to ~90% less memory than a stock Hugging Face loop via custom kernels; Axolotl for YAML-driven, multi-GPU (4+) pipelines; TRL for the training objectives themselves (SFT, DPO, GRPO); and TorchTune as a newer native-PyTorch alternative. Start with Unsloth on one GPU and graduate to Axolotl when you need to scale out.

Optimization strategies: how training fits on real hardware#

Everything above assumes the training run fits in memory and finishes in reasonable time. It only does because of a stack of systems-level optimizations. These are not optional niceties; without them, fine-tuning a useful model is impossible on accessible hardware.

Memory optimizations#

• Mixed precision (bf16/fp16): store and compute in 16-bit instead of 32-bit, halving memory and roughly doubling throughput on modern GPUs. bf16 is the safer default for stability.
• Gradient checkpointing: instead of storing all activations for the backward pass, recompute them on the fly. This trades extra compute for a large memory saving, often the difference between fitting and not.
• Quantization (4-bit / 8-bit): store the frozen base weights in 4-bit (NF4), the core of QLoRA. The model that needed 80 GB now needs ~6 GB.
• Gradient accumulation: simulate a large batch by accumulating gradients over several small forward/backward passes before stepping. You get the stability of a big batch without the memory.

Speed optimizations#

• FlashAttention: an exact attention implementation that avoids materializing the full attention matrix in memory, giving large speedups and memory savings (Dao et al., arXiv:2205.14135). Use it whenever your hardware supports it.
• Sequence packing: concatenate short examples into full-length sequences so no compute is wasted on padding tokens, a big throughput win on datasets with variable lengths.
• Learning-rate schedules: a warmup followed by cosine decay is the reliable default; a constant high rate is a common cause of divergence or forgetting.

Scaling across multiple GPUs#

When one GPU is not enough, you distribute. The strategies stack:

• Data parallelism: replicate the model on each GPU, split the batch. Simple, but every GPU holds the full model.
• FSDP / ZeRO: shard the model, gradients, and optimizer state across GPUs so no single device holds the whole thing (DeepSpeed ZeRO, Rajbhandari et al., arXiv:1910.02054). This is how you train models too big for one GPU's memory.
• Tensor and pipeline parallelism: split individual layers (tensor) or groups of layers (pipeline) across devices, for the largest models. Mostly relevant at the frontier scale.

For most fine-tuning, QLoRA plus gradient checkpointing on a single GPU, or FSDP across a few GPUs, is all you need.

What this costs in 2026#

The payoff of the whole optimization stack is that fine-tuning is now genuinely cheap. Approximate QLoRA figures on rented GPUs, for a typical SFT run on a few thousand examples:

The contrast in the last two rows is the whole argument for PEFT: QLoRA fine-tunes a 70B model for the price of lunch; full fine-tuning of the same model costs 20–30× more and a far larger machine. Note too that modern 4-bit quantization now costs only ~1–2% accuracy versus 16-bit, a trade almost always worth taking. (Figures are order-of-magnitude and vary with dataset size, sequence length, and provider.)

The toolchain: what to use for each phase#

Fine-tuning is not one tool but a stack, and picking the right tool per phase saves far more time than any hyperparameter. Here is the 2026 landscape, mapped to the phase where each earns its place.

The practical decision for the training framework is simple. Solo or one GPU: Unsloth. A team needing multi-GPU and reproducible configs from day one: Axolotl (or LLaMA-Factory if you want a web UI). Want full low-level control: TorchTune. And these increasingly compose rather than compete. Unsloth's kernels work under TRL's trainers, and Axolotl can leverage both, so you are picking a primary entry point, not locking yourself in. If you would rather not manage GPUs at all, Together AI or Modal run the same training for you.

On data, the 2026 shift worth adopting is synthetic generation: write a small set of seed examples, expand them with a strong model, generate DPO preference pairs with an LLM judge, then filter for quality in a tool like Argilla before training. Curated synthetic data routinely beats scraped data and costs less than manual labeling at scale, but the filtering step is what keeps it honest, so do not skip it.

Evaluation: before you trust the loss#

Training loss going down does not mean the model got better. The intro model's loss looked perfect. Hold out around 20% of your data as a test set the model never trains on, and evaluate with metrics that reflect the actual task: exact-match or F1 for classification, pass-rate for code, an LLM-as-judge score for open-ended generation. Watch eval loss every 100–500 steps, not just train loss. When eval loss turns up while train loss keeps falling, you are overfitting.

Critically, run a general-capability benchmark before and after to catch catastrophic forgetting: standard public suites like MMLU (broad knowledge), GSM8K (math reasoning), and HumanEval (code) tell you whether the fine-tune quietly broke abilities outside your task. A fine-tune that gains 10 points on your task but loses 15 on MMLU is usually a bad trade.

Evaluation tools and how to choose#

Evaluation in 2026 splits into two layers, and conflating them is a common mistake. You need both.

Layer 1: academic benchmark runners. These run standardized public benchmarks (MMLU, GSM8K, HumanEval, and hundreds more) to measure raw capability and detect forgetting. The standard is EleutherAI's lm-evaluation-harness. It powers Hugging Face's Open LLM Leaderboard, covers 200+ tasks, and runs against local Hugging Face models, vLLM, or any OpenAI-compatible API. LightEval (Hugging Face) is a newer, model-agnostic alternative in the same family. Use one of these to score your base model and your fine-tune on the same suites and confirm you did not regress.

# Benchmark the fine-tuned model the same way you benchmarked the base
lm_eval --model hf \
  --model_args pretrained=Qwen/Qwen3-8B,peft=./support-classifier-lora \
  --tasks mmlu,gsm8k,hellaswag --batch_size auto

Layer 2: application and LLM-as-judge evaluation. Benchmarks do not measure whether your model does your task well. For that you build a task-specific eval set and score it, usually with an LLM-as-judge. The 2026 options:

A sensible default stack: lm-evaluation-harness for capability and forgetting checks, plus DeepEval or Promptfoo for your task-specific eval, with scores logged to a platform like Langfuse so the same eval set runs in CI before each release and again on production traffic afterward. The principle that ties them together is traceability. Every score should link back to the exact model, prompt, and dataset version that produced it. We cover building real evaluations in Add Evaluation to an AI Feature and Evaluating Small Agent Workflows, and the production-monitoring half of this story in Part 2. Never ship a fine-tune you have only judged by its training curve.

Common mistakes and tradeoffs#

The central tradeoff is specialization versus generality. Every fine-tune makes the model better at your task and risks making it worse at everything else. The art is taking just enough of a step (the right method, a clean dataset, a low learning rate, and an honest evaluation) to gain the specialization without losing the base model's hard-won breadth.

Key Takeaways#

• Fine-tuning adapts behavior, format, and style by editing weights. Use prompting or RAG for knowledge that changes; fine-tune when prompting plateaus.
• The method spectrum runs from continued pretraining (domain) to SFT (format and task, the default) to preference alignment like DPO and its variants ORPO/KTO/SimPO (tone and judgment) to GRPO with verifiable rewards (reasoning on auto-gradable tasks). SFT then DPO is a reliable recipe; add GRPO when correctness is checkable.
• PEFT is the right default. LoRA trains ~1% of parameters; QLoRA adds 4-bit quantization so you can fine-tune large models on a single GPU.
• The dataset is the product. Quality and diversity beat quantity, and the wrong chat template silently destroys results.
• Training fits on real hardware only because of optimizations: mixed precision, gradient checkpointing, quantization, packing, FlashAttention, and FSDP/ZeRO for scale.
• Pick tools per phase: Unsloth (single-GPU speed) or Axolotl/LLaMA-Factory (multi-GPU), TRL for the objectives, Argilla/Distilabel for data, W&B for tracking, and managed options like Together AI or Modal if you skip the infra.
• Evaluate in two layers: lm-evaluation-harness/LightEval for capability and forgetting (MMLU, GSM8K, HumanEval), and DeepEval/Promptfoo/RAGAS plus a tracing platform for your task-specific eval. Never the loss curve alone.

FAQ#

How much data do I need to fine-tune an LLM?

For SFT, a few hundred to a few thousand high-quality, diverse examples is often enough to teach a format or task. Quality and coverage matter far more than volume, and adding more noisy data hurts. Continued pretraining needs much more (large unlabeled corpora) because it is teaching broad domain language.

LoRA or full fine-tuning?

LoRA (or QLoRA) for almost everything. It reaches close to full fine-tuning quality while training ~1% of the parameters, fitting on a single GPU and producing swappable megabyte-scale adapters. Reserve full fine-tuning for cases where you have measured a real quality gap and have the hardware to spare.

What is the difference between SFT, RLHF, and DPO?

SFT trains on ideal (prompt, response) pairs to teach format and behavior. RLHF and DPO train on preferences, which of two responses is better, to align qualities you cannot write as one gold answer, like tone or helpfulness. DPO does this directly without RLHF's separate reward model and RL loop, so it is simpler and the common practical choice. Variants like KTO (binary good/bad labels), ORPO (single-stage), and SimPO (no reference model) trade off data format and stability.

When should I use GRPO / reinforcement learning instead of SFT?

Use GRPO with verifiable rewards (RLVR) when you want to improve reasoning on tasks whose correctness you can check automatically: math, code that runs against tests, structured extraction. GRPO samples several answers per prompt and rewards the correct ones, with no reward model needed, and runs with QLoRA on a single GPU. For open-ended quality with no ground truth to verify, stay with SFT and DPO. The two are complementary: many teams SFT first, then apply GRPO for reasoning.

Will fine-tuning make my model dumber at other things?

It can, and that is catastrophic forgetting. Training too hard on a narrow dataset overwrites general capability. Mitigate it with a low learning rate, few epochs, and a mix of general-purpose data alongside your task data, then verify with a general-capability check in evaluation.

Can I fine-tune to add new knowledge to the model?

You can, but you usually should not. Fine-tuning is a poor and expensive way to inject facts, and it goes stale the moment the facts change. For knowledge, use RAG, which retrieves current documents at query time. Fine-tune for how the model behaves, not what it knows. See Fine-Tuning vs RAG.

Which fine-tuning framework should I use: Unsloth, Axolotl, TRL, or LLaMA-Factory?

For a single GPU, Unsloth is the fastest and most memory-efficient. For multi-GPU and reproducible YAML configs from the start, Axolotl (FSDP/DeepSpeed built in) or LLaMA-Factory (with a web UI). TRL is the underlying trainer library for SFT, DPO, and GRPO that the others build on, and TorchTune is the PyTorch-native option for full control. They increasingly compose, so start with Unsloth on one GPU and move to Axolotl when you scale out. If you do not want to manage GPUs, Together AI and Modal run the training as a service.

How do I evaluate a fine-tuned LLM, and what tools should I use?

Use two layers. Run lm-evaluation-harness (or LightEval) on standard suites (MMLU, GSM8K, HumanEval) before and after, to measure capability and catch catastrophic forgetting. Then build a task-specific eval set and score it with an LLM-as-judge using DeepEval or Promptfoo (or RAGAS if it is a RAG system), logging scores to a platform like Langfuse so the same eval runs in CI and again on production traffic. Keep every score traceable to the exact model, prompt, and dataset version.

Should I generate synthetic training data?

Increasingly yes. In 2026 synthetic generation is the default way to scale a dataset. Write a small seed set, expand it with a strong model, generate preference pairs with an LLM judge, then filter aggressively for quality in a tool like Argilla. Curated synthetic data beats scraped data and costs less than manual labeling, but the quality-filtering step is essential. Unfiltered synthetic data teaches the same bad patterns as any noisy dataset.

What hardware do I need to start?

With QLoRA, a single GPU with 16–24 GB of memory can fine-tune a 7–8B model, and 48 GB reaches into the tens of billions of parameters. Full fine-tuning of the same models needs multiple high-end GPUs with FSDP. Start with QLoRA on one GPU. It is enough for most real tasks.