What you will learn
- Distinguish pretraining, adaptation, and inference.
- Reason about context, output length, and decoding.
- Design a small model comparison that reflects product needs.
- 01Task casesOrdinary, difficult and absent
- 02Same inputsEvidence and prompt version
- 03MeasureQuality, time and cost
- 04Choose + revisitRecord a bounded decision
Separate how a model learned from how you use it
Pretraining learns broad statistical patterns from a large dataset under a training objective. Later adaptation can include supervised instruction examples or preference-based methods. At inference time, your application supplies instructions and context to the resulting model.
Parameter count describes the scale of learned weights, not a guarantee of quality on every task. Data, architecture, training, inference settings, and the task all matter. An open-weight release also needs its license and deployment requirements checked; downloadable weights do not automatically imply every permission associated with open-source software.
Budget the whole request
The context includes more than the user's sentence: instructions, conversation turns, retrieved passages, tool descriptions, and tool results all consume space. Reserve room for the answer and any model-specific overhead. Exact accounting varies by API and model, so check the selected model's documentation rather than relying on a screenshot's token limit.
Long context can be helpful, but adding every available document also adds irrelevant material and cost. Start with a compact, justified evidence set. If a policy exception sits in a distant appendix, measure whether retrieval and context construction actually bring it into the answer.
Treat decoding settings as experimental controls
Greedy decoding selects a highest-scoring next token. Sampling draws from a distribution, potentially adjusted by settings such as temperature or nucleus sampling. Supported controls vary, and some reasoning models do not expose the same parameters as older chat models.
Use a fixed prompt and record supported settings when comparing candidates. Do not assume a temperature of zero gives perfect determinism or factual accuracy. For creative writing you may value variation; for policy explanations you are more likely to value consistency, evidence use, and correct abstention. Those are outcome criteria, not parameter names.
Run a task-shaped comparison
Prepare ten policy questions: ordinary cases, exceptions, ambiguous questions, and questions without an answer. Give each candidate the same evidence and output requirements. Score factual support, exception handling, source accuracy, response time, and failure behavior. Repeat a few cases to see whether a single good answer was luck.
Record the model identifier, date, prompt version, and dataset version. Avoid declaring a universal winner from ten examples. The result is a shortlist for Northstar's workload. A more capable model may justify its cost on difficult cases, while a smaller one may be sufficient for routine classification. Any routing scheme must itself be evaluated: misrouting hard questions can erase the expected savings.
PUT IT TO WORK
Your practice task
Build a comparison sheet with ten rows and two candidate columns. Before calling any model, fill in the expected evidence and acceptable abstention behavior for each row. Leave price and latency blank until measured or checked against the provider's current documentation.
Checkpoint: compare your reasoning
A model that answers every question is not automatically the winner. An unsupported question should produce a useful limitation or clarification. If your score rewards confident completion regardless of evidence, the evaluation is encouraging the wrong behavior.
References and further reading
Use these primary references for deeper study and current API details. Examples in this lesson use fictional Northstar data.