Model evaluation

• Pairwise preference: which of two answers is better, and why, against written criteria.
• Rubric scoring: helpfulness, accuracy, tone and safety on agreed scales.
• Fact-checking of answers against sources, with the source recorded.
• Adversarial testing: writing prompts designed to make a model fail, in English, Kiswahili, Filipino and other languages we work in.
• Vision evaluation: checking a model’s descriptions, detections and judgments of images against what is really there.

Evaluation work is done by senior labellers and reviewers who have passed an additional assessment, most of them graduates, some of them specialists (nurses, accountants, teachers) recruited for particular projects. Every rubric includes an “unsure” option, and reviewers are scored on calibration, not only on agreement: a reviewer who says “unsure” when the gold answer is uncertain scores better than one who guesses.