A worked case study in choosing to fine-tune instead of prompt-engineer or retrieve — LoRA-tuning a small open-weight model for team-and-urgency ticket routing, with a worked cost comparison against a general-purpose LLM API call, per-class precision/recall evaluation, and confidence-based production monitoring.
The project brief: a support team receives thousands of incoming tickets a week that need automatic routing to the right team — billing, technical, account, or feature-request — and tagging with an urgency level. Rather than sending every ticket to a general-purpose LLM with a per-call prompt, this module walks through the case for fine-tuning a small open-weight model instead — a decision this program's fine-tuning framework favors precisely because ticket routing is a stable, high-volume, well-defined classification task with nothing external to retrieve, not a knowledge-lookup problem RAG would be built for.
This module works the whole build the way a real ML engineer would: dataset preparation that treats label consistency as more important than raw ticket volume, a worked cost comparison putting real per-token prices against a fine-tuned small model versus a general-purpose few-shot API call, a train/validation/test split evaluated on per-class precision and recall (not just overall accuracy, since missing an urgent ticket costs more than over-flagging a routine one), and a production monitoring plan built on confidence-based human review rather than blind trust in every prediction. The full worked numbers, complete tech-stack manifest, and finished reasoning are part of the unlocked module below.