Local models and inference engineering · Module 1: Local models and inference engineering in practice
Latency, throughput and the waiting user
Learning outcome: Measure the performance that the task actually needs.
Latency is how long a request takes; throughput is how much work a system completes over time. Time to first token and total completion time capture different parts of an interactive experience. A high tokens-per-second figure can coexist with a long wait before the first useful output.
Measure representative inputs and outputs, including retrieval or tool time where relevant. Report distributions or at least the range, not just one best run. Batching can improve throughput while increasing individual waiting time. Use a task-level success criterion so faster unusable answers do not count as a performance improvement.
Worked example
In a fictional comparison, system A starts after one second and finishes after twelve. System B starts after six seconds and finishes after nine. B finishes sooner, while A feels more immediately responsive. For a background batch job, completion throughput may matter most; for live assistance, the initial delay may be important.
Try it yourself
Write a measurement plan for live help and overnight document processing. Choose two metrics for each and explain why the priorities differ.
You can do this exercise on paper. If using an assistant, keep to the fictional material and an approved tool.
Compare your approach
Live help should include time to first useful output and complete response time. Batch work should include successful tasks per period and total processing time. Both need quality checks and the same workload definition.
Check your understanding
Which statement is justified by the example?
- B completes sooner but A begins responding sooner
- B is better for every use case
- A has higher throughput because it starts sooner
Show the answer and explanation
Answer: B completes sooner but A begins responding sooner. The example gives two separate observations: earlier first output for A and earlier completion for B. Which matters more depends on the task; throughput was not established.
Workbook
Download the course workbook to record your answer and compare it after the knowledge check.