§ 1
Where frontier models fail Indian law
Four numbers from this edition, each drawn over the graded answers behind it. Read them in order: together they are the gap between an answer that reads well and one an advocate could file.
Edition
A Dewans & Associates initiative, practising since 1919
Dewans Legal Bench
0%
Best score: model
Share of weighted rubric criteria met by the top model.
0%
Answers with every criterion met
The partner's standard: nothing missed, nothing wrong.
0%
Answers citing an authority that does not exist
At least one invented case, section or citation.
§ 2
Leaderboard
Ranked by score, the weighted share of rubric criteria met. Hover or focus a card to see where each model is strong and where it slips. Slice by track, practice area or task type; the cards, the strip and the table all follow.
| # | Model |
|---|
Strength by area
Score per model in each slice. Darker gold is higher. Hatched cells were not tested in this edition.
LowerHigher
§ 3
Anatomy of a task
Every task is built like a partner's brief to an associate. The model sees the left side. It never sees the right side, and neither does anyone outside the bench. Select any part to read what it does.
Illustrative schematic. Placeholder wording only. No real task is shown; the test set is private.
What the model sees
What the model never sees
Select a part. Each one exists because juniors get that part wrong in practice.
§ 4
How an invented citation looks to a judge
An invented authority does not look invented. It has a party name, a year, a volume and a page, and it is cited with confidence. The bench checks every authority in every answer and counts the ones that do not exist.
Illustrative placeholders. These are not real model outputs and not real citations.
§ 5
Method
Four numbers per model, one process for every answer, and rules that keep the board honest.
Score
The weighted share of rubric criteria met, averaged across tasks. Criteria carry weights by how much they matter to the client.
All-pass
The share of tasks where every criterion was met: work that could go out without correction.
Hallucination rate
The share of answers citing at least one authority that does not exist. Citation tasks test this directly.
Critical fail
The share of tasks where a criterion marked critical was missed: the wrong forum, a missed limitation date, a void clause accepted as valid.
The advocate
A senior advocate at the Bar sets the standard: he writes every task and its rubric, checks a sample of grades by hand, and his marks govern how every line is read.
The brief
The model receives the instruction, facts, documents and dates, as a junior would. No hints, no multiple choice.
The answer
The model writes the work product: a note, a draft, a marked list, a timetable.
The cross-check
Flagship models apply the advocate's standard line by line to every answer, a second model family cross-checks, and every authority they flag is verified against the judgment or the Act before it counts.
Private set
No task is ever published, so none can leak into training data. Scores are public; the work behind them is not.
Same test for everyone
Every model and every product meets the same tasks, rubric and judge. No one sees a task in advance.
Consent first
Closed legal AI products are scored only with the vendor's consent, and a vendor may withdraw before publication.
No paid placement
Money never touches a public score. Private work runs on separate task sets and never mixes with the board.
Disclosure
Infee Legal was graded by hand by the author on the same 160 tasks and rubrics; the model cross-check applied to every other row is pending for it. The author holds an interest in Infee.
This edition
By track
§ 6
Editions
The bench is published in editions, each a dated research report. India is live. The next stops take the same method across the borders Indian matters actually cross.
§ 7
How to use this bench
The bench is a published work of legal research. Citing the public scores with attribution is free. Everything else is used under a licence from the author. Pick the reader you are.
Know where a tool can be trusted before you sign.
For general counsel and firms deciding on legal AI. The public board tells you who leads overall. The report tells you whether a tool holds up on the matters your team actually handles.
- Edition report subscription. The full research report for every edition: results by practice area, task type and court level, failure patterns, and an advocate's commentary on what each failure would cost in practice.
- Buyer's report. Your shortlisted tools tested privately on matters like yours, with a written opinion on where each one can be relied on and where it must be checked.
- Briefing. A call with the author to walk your team through the findings.
On what terms
Subscribe to the report
Ask for a buyer's report
Reports are licensed to your organisation for internal use. The fee is a licence fee for the published work.
Be scored on the same tasks as the frontier models.
Builders of legal AI products are invited into the next edition. The terms are identical for every vendor.
Access, not extraction
You give access for the evaluation period and a contact who can confirm we use it as intended. We do not reverse-engineer, scrape, or publish your prompts or outputs beyond the scores.
Same tasks, no hints
Tasks reach your product exactly as they reach the models. Your product may use its own research tools and content.
Same grading, shown to you first
Judge model against the private rubric, with a sample confirmed by hand by a senior advocate at the Bar. You see your grades and reasons before publication.
You may withdraw
From any task or the whole edition, before publication. The board then says "did not participate" or "withdrew after review", and nothing more.
Money never touches a score
Public scoring is free, and no fee can be paid to change a public score. Private evaluation runs on a separate task set.
The test set stays private
You will not receive the tasks in advance or in bulk.
On what terms
Request scoring
Ask about the vendor report
Public scoring carries no fee, and seeing your grades and reasons before publication (term 3) is part of it. What is licensed separately, as a royalty for use of the work, is the deeper task-level report under NDA: an advocate's written failure analysis by task and criterion, with remediation notes. It never changes a public score.
Vendors who decline are listed as "not scored", without comment.
Indian legal work your model has never seen.
For AI labs and data teams. The private set has never been released, so a score on it measures reasoning, not recall.
- Pre-release evaluation. Run an unreleased model on the private set or a held-out set. Grades on every criterion and a failure analysis written by an advocate. Nothing is published without your consent.
- Private sets under licence. Held-out task sets written to your specification by practice area, court level, difficulty and language, with rubrics and reference answers. Exclusive or not.
- Expert grading at scale. Practising advocates grade your outputs against the same rubric method.
On what terms
Enquire about evaluation
Ask about a set licence
Sets are licensed for evaluation. Training use is a separate licence. Fees are royalties for use of the author's work.
A Dewans & Associates initiative, practising since 1919
About the author
The bench is written by people who brief juniors for a living, and it tests models on the mistakes juniors actually make: the wrong court, the missed date, the case that was overruled, the authority that never existed.
Published legal benchmarks cover American, British, European, Australian and Chinese law. India is the largest common-law jurisdiction by population, and until now there was no number for it.
Dewan Vakul Vinod Goyal
Advocate. Author of the Dewans Legal Bench.