Engine Validation
4,190 Real Contracts.
100% Deterministic.
100% Replayable.
A finding you can't reproduce or explain isn't defensible in a legal workflow. Every production release is benchmarked against real, publicly filed contracts before it ships — not sampled, not synthetic — specifically to prove ours are.
This page is the evidence, not a claim about it.
Benchmark Overview
Reference run 20260802T164957Z
4,131 of 4,190 contracts (98.59%) reached a clean terminal state; the remaining 59 (1.41%) are a non-blocking, secondary parsing cross-check flag, detailed in Known Limitations — not a rule-engine, export, or replay defect.
Research
Built on published work, not ad hoc prompting
Every layer on this page traces back to a peer-reviewed architecture, not a prompt that happened to work. Two papers, one architecture, one benchmark, one product — in that order.
Why Benchmark?
Legal software has a different bar than most software
A finding that can't be reproduced, traced, or explained isn't useful in a legal workflow — it's a liability. TriageBench exists to make four specific properties measurable, not assumed.
Reproducible
The same contract in produces the same findings out, every time, on every run.
Auditable
Every finding cites the rule ID and exact clause text that produced it — nothing is opaque.
Explainable
The AI only explains an already-final finding in plain English — it never decides.
Deterministic
189 explicit rules decide. No statistical weighting, no learned parameters, no randomness.
What Is TriageBench?
The same ten-stage pipeline, run against every contract
TriageBench runs TriageCounsel's real deterministic pipeline in-process — independent of the web application and database, with no dependency on the LLM layer. Hover any stage for detail.
Real Contract Corpus
4,190 real SEC-filed exhibits — nothing synthetic
Employment
58.6% of corpus
Purchases
24.0% of corpus
Services
15.8% of corpus
Lease
1.5% of corpus
Benchmark Methodology
Every contract, the same ten checks, in order
Results
Reference run: full metrics
| Metric | Value |
|---|---|
| Contracts scheduled / accounted for | 4,190 / 4,190 |
| Contracts reaching clean terminal state | 4,131 (98.59%) |
| Parser success rate (no P0/P1 defects) | 100.0% |
| Verify / replay success rate | 100.0% |
| DOCX export success rate | 100.0% |
| Negotiation Package success rate | 100.0% |
| Critical blockers | 0 |
| Engine reliability score | 99 / 100 |
| Parser reliability score | 100 / 100 |
| Determinism score | 100 / 100 |
| Evidence-anchoring score | 100 / 100 |
| DOCX / export reliability score | 100 / 100 |
| Commercial readiness score | 97 / 100 |
| Go / No-Go verdict | GO |
Scores below 100 are stated, not smoothed over. Engine reliability sits at 99 because 59 contracts (1.41%) were flagged by a secondary, independent parsing cross-check — see Known Limitations. Commercial readiness (97) is a composite score, not a marketing number.
TriageBench Live
A separate suite validates the deployed application itself
TriageBench validates the deterministic engine, in-process. TriageBench Live validates the deployed product — exactly as a customer uses it, against the real running server. It never imports an engine or application module; that boundary is enforced automatically by an AST-based test, not just convention.
TriageBench Live's concurrency suite is what actually found a real production defect this release: a database-session lifecycle leak under sustained load. See Engineering Defects Found below for exactly how it was caught, fixed, and re-verified with zero leakage under both sequential and concurrent sustained traffic.
Engineering Defects Found
The benchmark's job is to find real defects — here's what it found
We are not showing this section to claim a spotless system. We're showing it because a benchmark that never catches anything isn't validating anything. These are the real defects TriageBench and TriageBench Live found in this release cycle, and exactly how each was fixed and re-verified.
DOCX export crash on XML-illegal characters
✓ Fixed & verifiedFound by: the full 4,190-contract TriageBench run — a legacy page-break artifact (a form-feed character) in one SEC filing's extracted text was illegal under the XML 1.0 character spec and crashed DOCX export.
Fixed by: implementing the exact XML 1.0 Char production,
verified character-by-character against lxml's own acceptance behavior, applied everywhere raw
contract text enters a generated document. Verified by: 15 new targeted
tests plus independent cross-validation against Mammoth, a third-party OOXML parser, then a full corpus re-run.
Database session-lifecycle leak under sustained load
✓ Fixed & verifiedFound by: TriageBench Live's concurrency suite — 59 route and exception-handler call sites obtained a database session outside FastAPI's dependency injection, so nothing guaranteed the session closed; under sustained traffic the connection pool exhausted permanently.
Fixed by: converting every call site to FastAPI's own
Depends(get_db), plus SQLite WAL mode with an explicit busy timeout.
Verified by: a dedicated proof suite exercising sequential,
at-capacity, and above-capacity concurrent bursts — zero leakage confirmed under both sequential and concurrent sustained load.
Production deployment packaging defect
✓ Fixed & verifiedFound by: a direct audit of the live deployment — the benchmark corpus had been committed to the repository without a bundle-exclusion rule, pushing the deployed function to 795 MB against a 500 MB platform ceiling. Every deployment since had been silently failing to build.
Fixed by: correcting the deployment exclusion rules. Verified by: confirming the resulting build completed, then independently checking the live production domain returns HTTP 200 with working registration and login flows.
Known Limitations
What this benchmark does — and does not — validate
This benchmark validates
This benchmark does not validate
Every report TriageCounsel generates states this directly — not only this page. A clean benchmark run means the engine did what it was built to do reliably. It does not mean the engine covers every legal question a document might raise.
Benchmark Metadata
Reproducible by design
Interactive Benchmark Explorer
Inspect results by contract category
| Contracts in category | 2,455 |
| Rules executed per contract | 189 |
| Mean redline coverage | 6.94% |
| Median analysis latency | 938 ms |
| Parser / verify / export success | 100% |
| Contracts in category | 1,007 |
| Rules executed per contract | 189 |
| Mean redline coverage | 6.77% |
| Median analysis latency | 3,247 ms |
| Parser / verify / export success | 100% |
| Contracts in category | 664 |
| Rules executed per contract | 189 |
| Mean redline coverage | 7.65% |
| Median analysis latency | 812 ms |
| Parser / verify / export success | 100% |
| Contracts in category | 64 |
| Rules executed per contract | 189 |
| Mean redline coverage | 11.49% |
| Median analysis latency | 2,487 ms |
| Parser / verify / export success | 100% |
Latency figures are median wall-clock engine time per contract from the reference run, not a promised SLA.
Timeline
Ruleset growth, release over release
Bar height is rule count, scaled to v7.2. Every release is re-benchmarked against the full corpus before shipping.
Downloads
Take the evidence with you
Benchmark Executive Report
The full reference-run report: every metric on this page, in the raw generated document.
Download .md →Technical Executive Brief
Architecture, severity model, and full benchmark results in one PDF.
Download PDF →Research Paper
The peer-reviewed ICCS 2026 paper behind the architecture, plus published benchmarks.
Read on /research →Methodology
The exact ten-stage pipeline, corpus, and scoring definitions used to produce this page.
Download .md →Trust should be measurable.
Don't take our word for it. Inspect the benchmark. Read the methodology. Run your own contract.
Start Free Review