Text & LLM data · Native speakers

Multilingual LLM evaluation data by native speakers

Prompts written by native speakers the way people really ask, and model answers judged by native speakers of the same language and place. Ratings cover correctness, cultural accuracy, register and politeness, fluency and invented facts, with a corrected answer wherever the model got it wrong.

Specification

Languages
Hindi, Tamil, Bengali, Marathi, French (France and Québec), Mexican Spanish, Haitian Creole and other languages
Prompts
Written from scratch by native speakers: everyday questions, local knowledge, advice, writing tasks and reasoning, never translated from English
Ratings
Correctness, cultural and local accuracy, register and politeness, fluency, invented facts, and preference between two answers
Raters
Native speakers living in the country, matched to the language and region of each prompt
Annotation
Scores per answer, errors marked with a short reason, and a corrected answer where the model was wrong
Quality check
Several raters per answer, agreement tracked, and hidden reference items catch careless rating

Example of the data

What you see, and the file you get.

Preview
Hindi · answer Acorrect and natural5 / 5
Québec French · answer BFrance’s rules, not Québec’slocal fit ✗
Haitian Creole · answer Cinvented holiday datehallucination ✗
ratings.csvCSV · JSON · Excel
item_idlanguagepromptmodelcultural_accuracy_1_5hallucinationrater_note
ml_0001hi-INदिवाली पर लक्ष्मी पूजा का शुभ मुहूर्त कैसे तय होता है?model_a5no
ml_0002fr-CAC’est quand la date limite pour poser mes pneus d’hiver?model_b2noanswers with France’s rules; the Québec deadline is 1 December
ml_0003ht-HTKi dat yo fete Fèt Drapo an Ayiti?model_a1yeswrong date; Flag Day is 18 May
ml_0004es-MX¿Qué documentos necesito para sacar mi RFC?model_b4nocorrect, but leaves out the CURP
ml_0005ta-INபொங்கல் பண்டிகை எந்த மாதத்தில் கொண்டாடப்படுகிறது?model_a5no

Sample rows. Also as JSON or Excel.

Contact us to get the dataset

Tell us what you need. We walk you through the data, its formats and how it is checked on a short call.

Contact us

What it is used for

  • Release evaluation in every language

    See where a model is weaker outside English, language by language.

  • Preference and RLHF data

    Train on the answers native speakers prefer, not on translated judgements.

  • Cultural and local accuracy

    Catch answers that are fluent but wrong for the place: holidays, laws, food, names and manners.

  • Benchmarks

    Build test sets in languages where public benchmarks are thin or translated.

Questions

Are the prompts translated from English?

No. Native speakers write them in their own language, about the questions people in that place really ask.

Which languages are included?

Hindi, Tamil, Bengali, Marathi, French from France and Québec, Mexican Spanish, Haitian Creole and more. Tell us the languages you need.

Can I see samples first?

Yes. Request samples with the form on this page and we will send them with the full specification.

Tell us what your model needs.

Request samples and the full specification. We reply within a couple of business days.

Request samples