Deep dive · module 81 of 100
Evaluation: benchmarks and rubrics
Measure the quality of AI output in a consistent manner.
After this lesson
You create a simple rubric to assess AI results.
In view · Along the same yardstick
Assessing with a fixed rubric
Use, for example, accuracy, completeness, and tone.
Describe what 'good enough' means for each criterion.
Use the same test cases and rubric for both versions.
Part 1 of 3 · Choose criteria
Your preference for less movement is on. Choose each part yourself.
What is a rubric?
A fixed set of criteria with scores, for example accuracy, completeness and tone.
Why useful?
This allows you to compare results fairly and see if an adjustment is truly better.
Keep it practical
3–5 criteria are enough. Too many criteria make the assessment slow.
New practice example
From explanation to a usable result.
A rubric describes in advance what you are assessing and what a passing result concretely entails.
Fictional practice situationFictitious: you are assessing summaries of ten note sets. Two contain an important exception.
View a possible solution
This is how it can be done
Score correct facts, necessary information and source basis separately. Set a stopping criterion that a meaning-altering error is unacceptable, even with fluent language.
Why this works
An average can hide important errors. Therefore, also examine the individual failed examples.
Make it slightly more difficult
Add a difficult example that has not yet been used to improve your prompt. What do you learn from that result?
Extra practice, at your own pace. You do not need to submit anything. This does not count towards an automatic assessment.
Your workplace · learning by doing
Make something. See what changes.
Checking the AI connection… You can already view the written example.
Present your assignment to Ende.
Read the answer, change one thing, and compare.
Previously held
New version
You remain in control
What do you want to improve in your next attempt?
On this device only. AI answers remain only in this page memory. This is self-review, not a grade.
Why this approach works
An average can hide important errors. Therefore, also examine the individual failed examples.
A step further
Add a difficult example that has not yet been used to improve your prompt. What do you learn from that result?
Try & discover · no obligation
Compare quality with a fixed yardstick
Give tone a lot of weight. Is a beautiful but incorrect answer still allowed to pass?
Fictitious: an online workshop, Saturday from 10 to 11 o'clock. The price is not yet known.
Smooth but incorrect
Discover our amazing online workshop this Saturday from 10 to 11 o'clock, completely free!
Facts: 0/en/3 · Toon: 3/en/3
Stop: the free price is made up. A high total score does not offset that error.
Factual and sober
Workshop: Saturday, online, 10–11 o'clock. Price to be confirmed.
Facts: 3/en/3 · Toon: 1/en/3
Can proceed to human final check.
The two criterion scores have been pre-assigned for this fictitious case. You change their weight. No free text is automatically assessed.
This trial works locally with fictitious data and does not count towards your lesson completion. It starts anew with a new lesson or reload.
A prompt to get you started
Create a simple rubric with 4 criteria to assess the quality of a AI summary. Provide a short description for each criterion.
Apply it
Two reviewers give the same AI summary very different scores, without being able to explain their criteria.
Which approach best fits this situation?
- Use fixed criteria with score anchors and compare several sample assessments.
- Only take the average of their intuitive scores.
- Let each reviewer choose different criteria per answer.
With JavaScript, you can check your approach and receive immediate feedback.
Test your insight
What is the advantage of a rubric?
- You assess results more consistently and can measure improvements.
- AI will improve because of it.
- It is not necessary.
Extra practice
Create a rubric of 3 criteria for a task that you frequently do with AI.
Criteria: accuracy, completeness, clarity. Each with score 1–3.
