Local models and inference engineering · Module 1: Local models and inference engineering in practice
Adapters, fine-tuning and evaluation debt
Learning outcome: Plan what must be maintained after customisation.
Parameter-efficient adaptation trains a smaller set of parameters than a full model update, depending on the method. It can reduce some training requirements, but does not remove the need for appropriate data and evaluation. An adapter is not a substitute for a current source of factual information.
Keep the base-model version, adapter version, training-data provenance and evaluation set together. Test whether gains on the target task come with losses elsewhere. Re-evaluate when the base model or serving setup changes. Include the cost of preparing data, reviewing errors and maintaining versions when deciding whether customisation is worthwhile.
Worked example
A fictional team trains an adapter for a strict report style. It follows headings more consistently but begins omitting uncertainty statements. The training examples rarely contained unresolved facts. The team revises the data and evaluates both format compliance and uncertainty preservation before considering the adapter usable.
Try it yourself
Write a customisation decision note with the target behaviour, a simpler baseline, training-data requirements, two evaluation criteria and a maintenance owner.
You can do this exercise on paper. If using an assistant, keep to the fictional material and an approved tool.
Compare your approach
The note should compare customisation against prompting and validation first. Preserve a held-out set and an explicit rollback version. A lower training cost does not mean the resulting behaviour is automatically acceptable.
Check your understanding
What must be checked after an adapter improves formatting?
- Nothing if training completed successfully
- Whether other required behaviours have deteriorated
- Only whether the file is smaller than the base model
Show the answer and explanation
Answer: Whether other required behaviours have deteriorated. Customisation can improve one behaviour while weakening another. Evaluate the full set of required behaviours, including preservation of uncertainty, before accepting the change.
Workbook
Download the course workbook to record your answer and compare it after the knowledge check.