What you will learn
- Distinguish prompting, retrieval, and parameter adaptation.
- Prepare representative, permissioned training examples.
- Compare a tuned model against a baseline and held-out cases.
- 01Observed failureStable behavior or missing facts?
- 02BaselinePrompt and retrieval checks
- 03Dataset + adaptationReviewed examples and held-out split
- 04Compare + deployMeasured gain and rollback
Diagnose before collecting training data
Prompting changes the instructions and examples supplied at inference. Retrieval supplies external information for the current question. Fine-tuning changes model parameters or an adaptation layer through additional training. These mechanisms can be combined, but they are not interchangeable.
Frequently changing policy facts belong in a maintainable source of truth. Repeated style, classification, or format behavior may be a better adaptation candidate after a sound prompt baseline. Fine-tuning can influence factual behavior, but it is not a dependable document-update mechanism and does not eliminate unsupported answers.
Define the target behavior precisely
Suppose the desired output is a concise support classification with an issue category and a short explanation. Write the label definitions, ambiguous-case policy, and examples of acceptable explanations. Include difficult cases and the expected handling of missing information.
Collect only data you are entitled to use, remove unnecessary personal information, and review labels for consistency. Repeating a weak example many times does not make it strong. Split related conversations and near-duplicates together so evaluation does not benefit from leakage. Keep a held-out set that represents the behavior you actually expect after deployment.
Understand the training choices
Full fine-tuning updates a large set of model weights. Parameter-efficient approaches train a smaller adaptation component; LoRA uses low-rank updates to reduce the number of trainable parameters. Such methods can reduce training requirements, but hardware, compatibility, and serving behavior still depend on the selected model and tooling.
Hosted services have provider-specific supported models, file formats, and job APIs. Self-hosted training needs its own compute and operational setup. This lesson intentionally does not present a universal upload command: verify the current service contract before spending money or transferring a dataset.
Run an experiment with a stopping rule
Compare the baseline and candidate on the same held-out cases. Measure the intended behavior, factual support, safety boundaries, latency, and total operational cost. Check whether improvement on your narrow task damages more general behavior that the product still needs.
Version the dataset, training configuration, base model, and resulting artifact. Keep a rollback path. If a small prompt change or better retrieval fixes the problem, training may add maintenance without enough benefit. If adaptation improves a well-defined behavior consistently, keep the evidence and document the conditions under which you chose it.
PUT IT TO WORK
Your practice task
Write a fine-tuning proposal for the support classifier, including target behavior, example schema, data rights, split strategy, baseline, and success criteria. Then explain why a policy change from 30 to 45 days should normally update the retrieval corpus instead of triggering a new training job.
Checkpoint: compare your reasoning
The classifier proposal targets stable behavior and can be evaluated on labeled cases. The changing return window needs a versioned source update with stale-index removal. Training the number into a model would make correction and source attribution harder.
References and further reading
Use these primary references for deeper study and current API details. Examples in this lesson use fictional Northstar data.