Case study · Language technology
Evaluating a language model in Kiswahili and Filipino
Pairwise and rubric evaluation of model answers by reviewers who speak the languages as they are really spoken.
- Judgments delivered
- 412,000
- Languages
- 6
- Reviewer agreement
- 87%
The work
The client wanted to know how its model performed for people who write in Kiswahili, Sheng, Filipino and Taglish, not in the textbook forms of those languages. Our senior labellers and reviewers wrote prompts the way they and their families would, compared pairs of answers, and scored them against a rubric for accuracy, helpfulness and tone.
Reviewers were scored on calibration, including their use of “unsure”, and the client used the results to choose between model versions before release.