Lawyer-reviewed legal AI evaluation data
Legal questions written the way clients really ask them, in English and Arabic, with AI answers rated by practising lawyers whose credentials are checked. Each answer is judged for accuracy under the right jurisdiction, for what it leaves out and for whether it should have sent the person to a lawyer, and every rating comes with the lawyer’s reasons.
Specification
- Questions
- Everyday legal questions from people and small businesses: tenancy, employment, family, consumer, contracts, immigration, debt and company matters
- Jurisdictions
- US states, England and Wales, India, the UAE and Saudi Arabia, each question tagged with its jurisdiction
- Languages
- English and Arabic, as people really write them, typos and dialect included
- Ratings
- Legal accuracy, completeness, jurisdiction fit, cited law real or invented, overall score 1–5, and whether to refer to a lawyer
- Reviewers
- Practising lawyers whose credentials are checked, each rating answers only in the law they practise
- Quality check
- A second lawyer from the same jurisdiction checks every rating
Example of the data
What you see, and the file you get.
| Answer A | accuracy 5 / 5 | right law cited ✓ |
|---|---|---|
| Answer B | wrong jurisdiction | English rule, Texas lease |
| Answer C | invented case | should refer to a lawyer |
| question_id | jurisdiction | language | question | accuracy_1_5 | error_found | lawyer_note |
|---|---|---|---|---|---|---|
| q_0012 | US-TX | en | My landlord kept my whole deposit and gave no reason. What can I do? | 3 | missing_rule | Misses the 30-day refund and itemized-list duties, Tex. Prop. Code §§ 92.103–92.104. |
| q_0047 | GB-ENG | en | My new washing machine broke after two weeks. Does the shop only have to repair it? | 2 | wrong_rule | Within 30 days the buyer may reject it for a full refund: Consumer Rights Act 2015, s. 22. |
| q_0103 | IN | en | A client’s cheque bounced. How long do I have to send a legal notice? | 2 | wrong_period | Notice within 30 days of the bank’s return memo, not 15: Negotiable Instruments Act 1881, s. 138(b). |
| q_0158 | AE | ar | استقلت بعد سنتين من العمل في شركة خاصة، فهل تُخصم مكافأة نهاية خدمتي؟ | 2 | repealed_law | Applies the resignation cut of the repealed 1980 law; Federal Decree-Law No. 33 of 2021 has no such cut. |
| q_0201 | SA | ar | أريد إنهاء عقد عملي غير محدد المدة، فكم يجب أن تكون مدة الإشعار؟ | 4 | Right notice rule under the Saudi Labor Law; should add that notice must be in writing. |
Sample rows. Also as JSON or Excel.
Contact us to get the dataset
Tell us what you need. We walk you through the data, its formats and how it is checked on a short call.
What it is used for
Legal AI benchmarks
Measure how often an assistant gets the law right, jurisdiction by jurisdiction.
Fine-tuning and RLHF
Train on lawyers’ preferences between answers, with their reasons.
Hallucination checks
Catch invented cases, wrong sections and repealed law before users do.
Safe referral
Teach assistants when a question needs a lawyer, not a chatbot.
Questions
Who rates the answers?
Practising lawyers whose credentials are checked, each rating answers only in the law they practise: US state law, the law of England and Wales, Indian law, UAE law or Saudi law.
Are the questions real?
They are written by real people the way clients ask, in plain English or Arabic, with no names or personal details from real cases.
Can I see samples first?
Yes. Request samples with the form on this page and we will send them with the full specification.
Tell us what your model needs.
Request samples and the full specification. We reply within a couple of business days.
Request samples