Guide
Table of Authorities Benchmark: 25 Supreme Court Briefs Rebuilt from the PDF
On this page7 sections
The short answer
In an internal pre-launch benchmark on 25 U.S. Supreme Court merits briefs filed 2018–2025, the BriefAuthority citation engine, reading only each filed PDF, found 1,313 of the 1,318 cases listed in the briefs' own printed Tables of Authorities (99.6%). For 1,099 of the 1,252 found cases whose pages could be checked (87.8%), its page list was identical to counsel's. Counted page by page, it found 94.5% of counsel's page references, and 93.4% of the page references it produced were in counsel's table. It also flagged 115 case citations that are not in counsel's tables (about 5 per brief), mostly opinions-below citations, subsequent history and briefs cited by Westlaw number. That is why every result goes through a review screen. Cases only; the statutes figure is experimental. Download the data: JSON · CSV.
How the benchmark works
The test asks one question: given only the final PDF, can the engine rebuild the cases part of the Table of Authorities that counsel filed? The printed table is the answer key. It was prepared by professionals and it is public, but it is not perfect, and some of the errors below are counsel's rather than the engine's.
Method
- Selection. For each docket, the first brief of the chosen side filed after certiorari was granted. Briefs in opposition, amicus briefs, reply briefs and briefs “in support” were skipped. Four replacement briefs were named before the first run. None was needed, so they became a holdout that was never inspected during development.
- Reading the PDF. pdf.js extracts the text of each page. Lines are rebuilt from glyph positions, superscript footnote numbers are dropped, and italics are read from the embedded font names.
- Pages. The printed TOA is found by its heading. Body page 1 is the first page printed “1”. The body ends where the appendix or addendum starts. Every page reference is the brief's own printed page number, not the PDF page.
- Answer key. The TOA parser joins wrapped lines until an entry ends in dot leaders and a page list, then keys each case by volume, reporter and first page (parallel cites all count). No brief had to be excluded for an unreadable TOA.
- Detection. eyecite, the Free Law Project's citation parser (JavaScript port), runs over the body with page offsets. Full citations, short forms, *Id.* and *supra* are resolved to their case. Main text and footnotes are laid out separately, so a citation cut by a page break or a footnote block stays whole.
- Scoring. A case counts as found if any of its TOA citations is detected. A page list is exact only if the set of pages matches counsel's exactly. Entries marked *passim* are left out of the page check, and ranges are expanded. Extras are detected case citations that match no TOA entry.
Results by brief
| Brief (PDF) | No. | Side | Filed | Cases in TOA | Found | Exact page lists | Extras |
|---|---|---|---|---|---|---|---|
| Greer v. United States | 19-8709 | Petitioner | 2021 | 70 | 70 | 58/66 (87.9%) | 8 |
| Bouarfa v. Mayorkas | 23-583 | Petitioner | 2024 | 62 | 62 | 55/62 (88.7%) | 2 |
| Timbs v. Indiana | 17-1091 | Respondent | 2018 | 103 | 103 | 84/100 (84.0%) | 6 |
| Lorenzo v. SEC | 17-1077 | Petitioner | 2018 | 53 | 53 | 41/47 (87.2%) | 12 |
| Bostock v. Clayton County | 17-1618 | Respondent | 2019 | 84 | 84 | 70/77 (90.9%) | 7 |
| DHS v. Regents of the University of California | 18-587 | Petitioner | 2019 | 28 | 28 | 20/26 (76.9%) | 8 |
| Espinoza v. Montana Department of Revenue | 18-1195 | Respondent | 2019 | 22 | 22 | 14/22 (63.6%) | 1 |
| Dobbs v. Jackson Women's Health Organization | 19-1392 | Petitioner | 2021 | 45 | 45 | 36/42 (85.7%) | 2 |
| New York State Rifle & Pistol Ass'n v. Bruen | 20-843 | Respondent | 2021 | 55 | 53 | 47/50 (94.0%) | 4 |
| 303 Creative LLC v. Elenis | 21-476 | Petitioner | 2022 | 73 | 73 | 60/66 (90.9%) | 0 |
| Loper Bright Enterprises v. Raimondo | 22-451 | Respondent | 2023 | 72 | 72 | 66/72 (91.7%) | 4 |
| Kahler v. Kansas | 18-6135 | Petitioner | 2019 | 60 | 60 | 53/58 (91.4%) | 2 |
| Cedar Point Nursery v. Hassid | 20-107 | Respondent | 2021 | 58 | 58 | 45/49 (91.8%) | 5 |
| Sackett v. EPA | 21-454 | Petitioner | 2022 | 42 | 42 | 34/39 (87.2%) | 3 |
| Kelly v. United States | 18-1059 | Respondent | 2019 | 24 | 24 | 18/23 (78.3%) | 5 |
| Kennedy v. Bremerton School District | 21-418 | Petitioner | 2022 | 31 | 31 | 24/27 (88.9%) | 6 |
| Great Lakes Insurance SE v. Raiders Retreat Realty Co. | 22-500 | Respondent | 2023 | 36 | 36 | 28/36 (77.8%) | 3 |
| Free Speech Coalition, Inc. v. Paxton | 23-1122 | Petitioner | 2024 | 52 | 52 | 46/52 (88.5%) | 4 |
| Kousisis v. United States | 23-909 | Respondent | 2024 | 46 | 45 | 39/45 (86.7%) | 9 |
| Chiles v. Salazar | 24-539 | Petitioner | 2025 | 66 | 66 | 60/66 (90.9%) | 0 |
| First Choice Women's Resource Centers, Inc. v. Platkin | 24-781 | Respondent | 2025 | 72 | 72 | 63/72 (87.5%) | 9 |
| Villarreal v. Texas | 24-557 | Petitioner | 2025 | 39 | 39 | 36/39 (92.3%) | 2 |
| Students for Fair Admissions v. President & Fellows of Harvard College | 20-1199 | Respondent | 2022 | 33 | 33 | 29/31 (93.5%) | 1 |
| Bost v. Illinois State Board of Elections | 24-568 | Petitioner | 2025 | 46 | 46 | 41/46 (89.1%) | 2 |
| Nestlé USA, Inc. v. Doe | 19-416 | Respondent | 2020 | 46 | 44 | 32/39 (82.1%) | 10 |
| All 25 briefs | 1,318 | 1,313 | 1,099/1,252 (87.8%) | 115 |
The weakest result, the Montana respondents' brief in Espinoza (63.6%), shows the main disagreement. Counsel's table lists Trinity Lutheran on 6 pages, but the engine credits 12, adding pages where the brief cites the case by short form, Id. or name. With only 22 cases in that table, a handful of entries like this moves the score a lot. The four holdout briefs (Barrett, Bondi v. VanDerStok, Fulton, United States v. Skrmetti) are listed in the downloads. On those, the engine found 188 of 190 cases, with 81.1% exact page lists.
What each change did
Each change was tested against the same 25 briefs, so the final score is optimistic by construction. That is the reason the unmodified baseline and the untouched holdout are both published. eyecite's built-in pin-cite reference extraction lowered accuracy and is not used.
| Configuration | Cases found | Exact page lists | Missing page refs | Extra page refs |
|---|---|---|---|---|
| Baseline: full citations, short forms, Id., supra | 1,308 | 85.7% | 309 | 91 |
| + eyecite's pin-cite reference extraction (not adopted) | 1,308 | 84.7% | 282 | 125 |
| + italic name-only references | 1,308 | 86.8% | 124 | 165 |
| + credit each citation to the page where it starts | 1,308 | 87.7% | 140 | 147 |
| + short-form repair | 1,308 | 88.2% | 119 | 147 |
| + footnotes laid out after the main text (final) | 1,313 | 87.8% | 122 | 149 |
Where it goes wrong
We classified two random samples by hand: 30 of the 115 extra case citations, and 20 of the case entries whose page list was not exact. Every sampled item is listed in the JSON.
| Failure mode | Real example | How often |
|---|---|---|
| Name-only references. Counsel indexes some mentions of a case by name and not others | Greer: the TOA lists Rehaif on pages 1, 2 and 4, but the brief names Rehaif on about ten pages. In the Government's Loper Bright brief, Kisor is indexed on 14 pages, most of them name-only. | 9 of 20 inexact entries. Adding name detection alone cut missing page refs from 309 to 124 but raised extra page refs from 91 to 165 |
| Citations a TOA does not list as a case | “Opinions below” citations (e.g. 45 F.4th 359 in Loper Bright), subsequent history (“cert. denied, 555 U.S. 1137”), briefs cited by Westlaw number, and one English statute (2 Edw. III, ch. 3) read as a case | 13 + 5 + 5 + 1 of 30 sampled extras |
| Real cases missing from counsel's TOA | Rose v. Clark, 478 U.S. 570, cited on page 15 of the Greer brief. Hudson v. United States, 522 U.S. 93, cited by reporter only in Indiana's Timbs brief | 4 of 30 sampled extras; the engine was right to flag these |
| Page breaks | A quoted parenthetical that runs onto the next page: Hellenic Lines v. Rhoditis in Indiana's Timbs brief, which counsel credits to pages 15 and 16 | 4 of 20 inexact entries |
| Short forms not credited | “Garcia, 469 U.S. at 547” in the Dobbs petitioners' brief; “Moore, 293 Ky. at 55” in Indiana's Timbs brief | 2 of 20 inexact entries. The repair step now catches short forms typed without “at” (“Loughrin, 573 U.S. 363”) |
| Errors in the answer key | Greer's TOA lists Rosales-Mireles only on page 15, where it does not appear; the brief cites it on pages 29, 40 and 41. The Bruen respondents' TOA prints “568 U.S. 3980” for Clapper and “561 U.S. 14” for Holder | 3 of the 5 cases not found are typos in the TOA or the brief; 4 of 20 inexact entries could not be explained from the extracted text |
| Citations eyecite cannot read | Fisher v. University of Texas, No. 11-345 (U.S.); Regina v. Kenrick, 5 Q.B. 49; LEXIS agency decisions; international tribunal judgments | 12 TOA entries, not counted in the 99.6% figure |
Statutes (experimental)
As a side check, we compared single-section U.S. Code entries in the printed TOAs (127 entries) with full “U.S.C.” citations found by a simple pattern. 112 were found (88.2%), and 80 of 106 had an exact page list (75.5%). Short forms such as “§ 922(g)” are not credited yet. The case figures are the benchmark. This one is not.
Limits
- Cases only. Statutes, rules, constitutional provisions and secondary sources are not scored, apart from the experimental U.S. Code check.
- U.S. Supreme Court merits briefs only: booklet format, professionally prepared tables. Circuit and state briefs, scanned PDFs and other layouts were not tested.
- The answer key is counsel's printed table, which has its own errors and inconsistent habits, name-only mentions above all.
- Pre-launch engine. These numbers describe this run on these PDFs, not a guarantee for any brief. BriefAuthority formats and indexes; it does not check whether a citation is good law.
Download the data and reproduce it
The full results are in toa-benchmark-2026-09.json and toa-benchmark-2026-09.csv. They include per-brief rows for all 29 briefs (25 plus the holdout), the docket number, brief side, filing date and supremecourt.gov PDF link, every metric above, the cases not found, scores for every configuration tried, and both hand-labelled samples. The engine is pdf.js 4.10.38 and eyecite (JavaScript port) 2.7.6, run in Node 25. Anyone can download the same PDFs from the docket pages and score a tool against the printed tables in the same way.
Frequently asked
How accurate is an automatic table of authorities?
In this benchmark on 25 Supreme Court merits briefs, the engine found 99.6% of the cases in counsel's printed tables, and 87.8% of those had exactly the same page list. The remaining differences are mostly judgement calls about name-only mentions, citations that run across a page break, and short forms. Review every entry before filing.
Why use the brief's own table as the answer key?
Because it is what a court actually received, it was prepared by professionals, and anyone can check it: every brief here is public on supremecourt.gov. The limitation is that counsel's tables contain errors too. We found wrong volume or page numbers and cases listed on pages where they do not appear.
Does the benchmark cover statutes and other authorities?
Only cases are scored. A separate, experimental check of U.S. Code citations found 88.2% of single-section U.S.C. entries, with 75.5% exact page lists. We do not treat that as a benchmark result.
What are the extra citations the engine finds?
Case citations that are in the brief but not in counsel's table. In a hand-checked sample of 30, most were citations a TOA leaves out by convention: the opinions below, subsequent history such as “cert. denied”, and briefs cited by Westlaw number. 4 were real cases that counsel's table missed.
Is 87.8% exact good enough to file without checking?
No. About one case in eight still needs a page added or removed. BriefAuthority is built around a review screen for that reason, and it does not validate citations. Spot-check the table against the final PDF before you file.
Can I rerun the benchmark?
Yes. The JSON and CSV list every brief with its docket number and supremecourt.gov PDF link, and they record the library versions, the configuration, and the hand-labelled samples. You can download the same PDFs and score any tool against the printed tables.
Sources
- Supreme Court of the United States — Docket search (brief PDFs)supremecourt.gov
- Supreme Court of the United States — Rules and Guidance (Rules 24 and 34.2)supremecourt.gov
- Free Law Project — eyecitegithub.com
- Mozilla — PDF.jsmozilla.github.io
- BriefAuthority TOA benchmark data (JSON)
BriefAuthority · free for small briefs
In development · early access