Inside language models · Module 1: Inside language models in practice
Tokens, spelling and the price of long inputs
Learning outcome: Explain why a word count is not a token count.
A tokenizer converts text into the units a particular model uses. A token might represent a word, part of a word or punctuation. Token boundaries vary by model and language. You cannot infer an exact token count just by counting spaces. This matters when a request has an input limit or usage is measured in tokens.
Keep the distinction between content length and useful context. Removing every space may make a document less readable without giving the saving you expect. Start by removing duplicated passages and irrelevant material. Preserve headings, units and qualifications that the task needs. If you use a token-counting tool, choose the tokenizer for the model being evaluated.
Worked example
A fictional team pastes a 900-word handbook twice and asks for a ten-word answer. The short requested answer does not make the duplicated input short. A better input contains the relevant section once, its title and the actual question. A reference code such as ZX-0417 must remain unchanged even if it splits into several tokens.
Try it yourself
Compare a short email, a list of reference codes and a paragraph in another language. Predict which is hardest to estimate by word count. Remove duplication from one without deleting needed facts.
You can do this exercise on paper. If using an assistant, keep to the fictional material and an approved tool.
Compare your approach
Keep the reference codes exact. Record predicted counts as estimates unless you have used the appropriate tokenizer. The useful improvement is a shorter, complete brief, not an invented percentage saving.
Check your understanding
A 100-word input is guaranteed to contain how many tokens?
- No fixed number without knowing the text and tokenizer
- Exactly 100
- Always fewer than 100
Show the answer and explanation
Answer: No fixed number without knowing the text and tokenizer. Tokens are determined by the exact text and tokenizer. Words can split into multiple units, while spaces and punctuation also affect the representation. A word count is only a rough proxy.
Workbook
Download the course workbook to record your answer and compare it after the knowledge check.