Text & LLM data · Practising lawyers

Lawyer-reviewed legal AI evaluation data

Legal questions written the way clients really ask them, in English and Arabic, with AI answers rated by practising lawyers whose credentials are checked. Each answer is judged for accuracy under the right jurisdiction, for what it leaves out and for whether it should have sent the person to a lawyer, and every rating comes with the lawyer’s reasons.

Specification

Questions
Everyday legal questions from people and small businesses: tenancy, employment, family, consumer, contracts, immigration, debt and company matters
Jurisdictions
US states, England and Wales, India, the UAE and Saudi Arabia, each question tagged with its jurisdiction
Languages
English and Arabic, as people really write them, typos and dialect included
Ratings
Legal accuracy, completeness, jurisdiction fit, cited law real or invented, overall score 1–5, and whether to refer to a lawyer
Reviewers
Practising lawyers whose credentials are checked, each rating answers only in the law they practise
Quality check
A second lawyer from the same jurisdiction checks every rating

Example of the data

What you see, and the file you get.

Preview
Answer Aaccuracy 5 / 5right law cited ✓
Answer Bwrong jurisdictionEnglish rule, Texas lease
Answer Cinvented caseshould refer to a lawyer
ratings.csvCSV · JSON · Excel
question_idjurisdictionlanguagequestionaccuracy_1_5error_foundlawyer_note
q_0012US-TXenMy landlord kept my whole deposit and gave no reason. What can I do?3missing_ruleMisses the 30-day refund and itemized-list duties, Tex. Prop. Code §§ 92.103–92.104.
q_0047GB-ENGenMy new washing machine broke after two weeks. Does the shop only have to repair it?2wrong_ruleWithin 30 days the buyer may reject it for a full refund: Consumer Rights Act 2015, s. 22.
q_0103INenA client’s cheque bounced. How long do I have to send a legal notice?2wrong_periodNotice within 30 days of the bank’s return memo, not 15: Negotiable Instruments Act 1881, s. 138(b).
q_0158AEarاستقلت بعد سنتين من العمل في شركة خاصة، فهل تُخصم مكافأة نهاية خدمتي؟2repealed_lawApplies the resignation cut of the repealed 1980 law; Federal Decree-Law No. 33 of 2021 has no such cut.
q_0201SAarأريد إنهاء عقد عملي غير محدد المدة، فكم يجب أن تكون مدة الإشعار؟4Right notice rule under the Saudi Labor Law; should add that notice must be in writing.

Sample rows. Also as JSON or Excel.

Contact us to get the dataset

Tell us what you need. We walk you through the data, its formats and how it is checked on a short call.

Contact us

What it is used for

  • Legal AI benchmarks

    Measure how often an assistant gets the law right, jurisdiction by jurisdiction.

  • Fine-tuning and RLHF

    Train on lawyers’ preferences between answers, with their reasons.

  • Hallucination checks

    Catch invented cases, wrong sections and repealed law before users do.

  • Safe referral

    Teach assistants when a question needs a lawyer, not a chatbot.

Questions

Who rates the answers?

Practising lawyers whose credentials are checked, each rating answers only in the law they practise: US state law, the law of England and Wales, Indian law, UAE law or Saudi law.

Are the questions real?

They are written by real people the way clients ask, in plain English or Arabic, with no names or personal details from real cases.

Can I see samples first?

Yes. Request samples with the form on this page and we will send them with the full specification.

Tell us what your model needs.

Request samples and the full specification. We reply within a couple of business days.

Request samples