Multilingual LLM evaluation data by native speakers
Prompts written by native speakers the way people really ask, and model answers judged by native speakers of the same language and place. Ratings cover correctness, cultural accuracy, register and politeness, fluency and invented facts, with a corrected answer wherever the model got it wrong.
Specification
- Languages
- Hindi, Tamil, Bengali, Marathi, French (France and Québec), Mexican Spanish, Haitian Creole and other languages
- Prompts
- Written from scratch by native speakers: everyday questions, local knowledge, advice, writing tasks and reasoning, never translated from English
- Ratings
- Correctness, cultural and local accuracy, register and politeness, fluency, invented facts, and preference between two answers
- Raters
- Native speakers living in the country, matched to the language and region of each prompt
- Annotation
- Scores per answer, errors marked with a short reason, and a corrected answer where the model was wrong
- Quality check
- Several raters per answer, agreement tracked, and hidden reference items catch careless rating
Example of the data
What you see, and the file you get.
| Hindi · answer A | correct and natural | 5 / 5 |
|---|---|---|
| Québec French · answer B | France’s rules, not Québec’s | local fit ✗ |
| Haitian Creole · answer C | invented holiday date | hallucination ✗ |
| item_id | language | prompt | model | cultural_accuracy_1_5 | hallucination | rater_note |
|---|---|---|---|---|---|---|
| ml_0001 | hi-IN | दिवाली पर लक्ष्मी पूजा का शुभ मुहूर्त कैसे तय होता है? | model_a | 5 | no | |
| ml_0002 | fr-CA | C’est quand la date limite pour poser mes pneus d’hiver? | model_b | 2 | no | answers with France’s rules; the Québec deadline is 1 December |
| ml_0003 | ht-HT | Ki dat yo fete Fèt Drapo an Ayiti? | model_a | 1 | yes | wrong date; Flag Day is 18 May |
| ml_0004 | es-MX | ¿Qué documentos necesito para sacar mi RFC? | model_b | 4 | no | correct, but leaves out the CURP |
| ml_0005 | ta-IN | பொங்கல் பண்டிகை எந்த மாதத்தில் கொண்டாடப்படுகிறது? | model_a | 5 | no |
Sample rows. Also as JSON or Excel.
Contact us to get the dataset
Tell us what you need. We walk you through the data, its formats and how it is checked on a short call.
What it is used for
Release evaluation in every language
See where a model is weaker outside English, language by language.
Preference and RLHF data
Train on the answers native speakers prefer, not on translated judgements.
Cultural and local accuracy
Catch answers that are fluent but wrong for the place: holidays, laws, food, names and manners.
Benchmarks
Build test sets in languages where public benchmarks are thin or translated.
Questions
Are the prompts translated from English?
No. Native speakers write them in their own language, about the questions people in that place really ask.
Which languages are included?
Hindi, Tamil, Bengali, Marathi, French from France and Québec, Mexican Spanish, Haitian Creole and more. Tell us the languages you need.
Can I see samples first?
Yes. Request samples with the form on this page and we will send them with the full specification.
Tell us what your model needs.
Request samples and the full specification. We reply within a couple of business days.
Request samples