What you will learn
- Distinguish supervised, unsupervised, and reinforcement learning.
- Explain loss, gradients, and generalization.
- Avoid leakage and choose task-relevant metrics.
- 01TrainingFit parameters
- 02ValidationChoose settings
- 03Held-out testEstimate generalization
- 04Production checksWatch new failure patterns
Understand what is being learned
In supervised learning, examples pair inputs with target labels or values. A ticket classifier learns categories such as delivery or returns; a regression model estimates a numeric quantity. Unsupervised learning searches for structure without the same target labels, such as clusters of recurring complaints. Reinforcement learning learns behavior from interaction and rewards, with a different setup from ordinary classification.
Features are the input representation. Labels are the expected targets. A model maps between them using learned parameters. In a neural network, layers transform representations through weights, biases, and nonlinear activations. ReLU, sigmoid, and tanh are examples of nonlinear functions; their usefulness depends on the architecture and task.
Separate learning from evaluation
Use training data to fit parameters, validation data to choose settings, and a held-out test set for the final estimate. A 70/15/15 split is one teaching example, not a universal recipe. Small datasets, time-dependent events, and repeated customers may require different splitting strategies.
Imagine the same complaint is copied into five tickets. If copies appear in both training and test sets, the model can benefit from near-duplicates instead of learning to handle new issues. Split related records together. For future ticket performance, consider a chronological split so the evaluation reflects the direction in which the product will operate.
Know what loss and optimization tell you
A loss function measures how predictions differ from targets. Mean squared error penalizes numeric errors; cross-entropy is common for classification. Gradient descent adjusts parameters in a direction intended to reduce loss. Backpropagation computes gradients through the network. You do not need to implement a large neural network to understand why these pieces exist.
Lower training loss alone does not establish a better product. Overfitting means the model fits training patterns that do not transfer well. Underfitting means the model fails to capture enough useful structure. Compare training and validation behavior, investigate data quality, and resist repeatedly tuning against the test set until its score stops being an honest estimate.
Work through a metric instead of memorizing it
Suppose 10 of 100 tickets are urgent. A classifier that calls everything non-urgent achieves 90% accuracy while missing every urgent case. Now suppose another model flags 12 tickets: 8 are genuinely urgent and 4 are false alarms. It misses 2 urgent cases.
Precision is 8/12, about 66.7%: how many flagged tickets were urgent? Recall is 8/10, or 80%: how many urgent tickets were found? The acceptable tradeoff depends on the workflow. A review queue may tolerate extra flags; an overloaded team may not. Write the confusion counts before selecting a headline score, and inspect examples from each error category.
PUT IT TO WORK
Your practice task
Create a table with true positives, false positives, false negatives, and true negatives for the example. Calculate precision and recall by hand. Then describe a leakage risk in Northstar's documents and a split that would prevent it.
Checkpoint: compare your reasoning
The counts are TP=8, FP=4, FN=2, TN=86. Accuracy is 94%, but the missed urgent cases still matter. For document experiments, group near-duplicate revisions or related records so evaluation questions do not simply repeat training examples.
References and further reading
Use these primary references for deeper study and current API details. Examples in this lesson use fictional Northstar data.