Local models and inference engineering · Module 1: Local models and inference engineering in practice
Quantisation and memory estimates
Learning outcome: Estimate weight storage without confusing it with total memory.
Quantisation stores numerical values at reduced precision to lower memory requirements. Different methods and hardware have different quality and speed trade-offs. A smaller representation does not guarantee faster inference or acceptable accuracy on a particular task.
A rough weight-only estimate multiplies parameter count by bits per parameter and divides by eight. It excludes metadata, runtime buffers, cache and other overhead. Use the estimate as a starting point, then measure the intended configuration. Compare outputs on a fixed test set after changing precision, especially where exact formatting or specialised vocabulary matters.
Worked example
An illustrative seven-billion-parameter model at four bits per weight needs about 3.5 billion bytes for weights alone. That is roughly 3.5 decimal GB, not a promise that it fits in a 4 GB device. The runtime and context cache need additional memory. The same parameter count at sixteen bits gives about 14 billion bytes before overhead.
Try it yourself
Calculate weight-only storage for three billion parameters at eight bits and sixteen bits. List three reasons actual memory use can be higher.
You can do this exercise on paper. If using an assistant, keep to the fictional material and an approved tool.
Compare your approach
The estimates are 3 billion and 6 billion bytes respectively. Cache, runtime buffers and representation metadata add overhead. Use decimal GB consistently and avoid silently treating it as binary GiB.
Check your understanding
What does the simple parameter-times-bits estimate describe?
- Guaranteed response speed
- Approximate weight storage before overhead
- Guaranteed total device memory use
Show the answer and explanation
Answer: Approximate weight storage before overhead. The calculation estimates weight storage. Runtime buffers, cache and other overhead add memory, and storage size alone does not establish response speed.
Workbook
Download the course workbook to record your answer and compare it after the knowledge check.