Basic · module 31 of 100
What actually is a token?
Understand how AI splits text into pieces and why that affects length and cost.
After this lesson
You can explain what a token is and why short, clear prompts often work better.
In view · From text to pieces
Text consists of smaller pieces
You type words and punctuation into a prompt.
A token can be a part of a word, a character or a whole word.
Input and output use tokens; this affects limits and costs.
Part 1 of 3 · Your text
Your preference for less movement is on. Choose each part yourself.
Tokens are pieces of text
A token is a word part, a word, or a punctuation mark. “discovery” can consist of multiple tokens. The model calculates in tokens, not in words.
Why it matters
More tokens mean more computation and sometimes higher costs. Long context can also confuse the model. Short and relevant is often stronger.
Practical tip
Write the core first, only add details if they are necessary. Ask the model to answer briefly if you want a concise result.
New practice example
From explanation to a usable result.
A token is a piece of processed text. It is not the same as one word or one letter.
Fictional practice situationFiction: two prompts contain the same number of words. One contains ordinary sentences; the other contains many product codes and unusual names.
View a possible solution
This is how it can be done
Do not assume equal token counts. Check the actual processing with the tools of the model used and reserve space for input and output.
Why this works
Word count is a handy writing limit, but not an exact technical measurement of tokens.
Make it slightly more difficult
Compare ordinary words, a web address, and a long code. What should you measure before budgeting costs?
Extra practice, at your own pace. You do not need to submit anything. This does not count towards an automatic assessment.
Read more: Google AI: processing text as tokens. The situation above is a fictional lesson example of your own.
Your workplace · learning by doing
Make something. See what changes.
Checking the AI connection… You can already view the written example.
Present your assignment to Ende.
Read the answer, change one thing, and compare.
Previously held
New version
You remain in control
What do you want to improve in your next attempt?
On this device only. AI answers remain only in this page memory. This is self-review, not a grade.
Why this approach works
Word count is a handy writing limit, but not an exact technical measurement of tokens.
A step further
Compare ordinary words, a web address, and a long code. What should you measure before budgeting costs?
Try & discover · no obligation
Fill the context window
Make your input longer. Then reserve more space for the answer. Where is the limit?
Blue: inputGreen: answer spaceGrey: free space
The specified input and reserved output fit together in this window.
Practice model, not a real tokenizer or product limit. Input also includes instructions and history. Fictitious rate: € 0,002 per 1.000 input tokens and € 0,006 per 1.000 output tokens. Actual rates and calculation rules differ.
This trial works locally with fictitious data and does not count towards your lesson completion. It starts anew with a new lesson or reload.
A prompt to get you started
Explain in a maximum of 80 words what a token is and give én example where a long prompt works less effectively than a short one.
Apply it
A colleague counts 100 words in a prompt and therefore estimates exactly 100 tokens for the request.
Which approach best fits this situation?
- Always calculate one token per word.
- Use the tokenizer or the usage measurement of the chosen modèle de langage.
- Only count the punctuation marks.
With JavaScript, you can check your approach and receive immediate feedback.
Test your insight
What is a token?
- Always a whole word.
- A piece of text (word part, word, or punctuation mark) that the model calculates with.
- Only punctuation marks.
Extra practice
Write a short and a long version of the same question. Note which one you find clearer and why.
Short: “Give three tips for a clear email.” Long: “I would like you to, taking into account all sorts of factors, explain in detail how I…” The short version is often sharper.
