When the jury rules, it’s right 96% of the time.
We tested the three AI judges on 232 facts we had already checked by hand, with the real sources in front of them. On 90 they couldn’t agree, so they didn’t rule. Here is every number, every mistake, and how to run it again.
- 136 of 142
- rulings right
- 95% interval 91% to 98%
- 6
- rulings wrong
- each one is listed below
- 90
- not ruled
- the bond is refunded and the line reopens
- 78 of 82
- right with no sources
- it ruled on 142 with them
The test
Two of three models must match, as on the chain. The right answers come from our corpus, checked against primary sources before this test existed.
| Who ruled | Right | Wrong | No ruling | Right when it ruled |
|---|---|---|---|---|
| The jury, with sources | 136 | 6 | 90 | 96% |
| The jury, with no sources | 78 | 4 | 150 | 95% |
| DeepSeek-V4.1-Flash alone, with sources | 120 | 11 | 101 | 92% |
| Qwen3.8-27B alone, with sources | 140 | 18 | 74 | 89% |
| glm-5.3-flash alone, with sources | 138 | 3 | 91 | 98% |
On the 216 true lines the jury ruled 124 true and 5 false. On the 16 stale or wrong lines it ruled 12 false and 1 true. Most facts it declined had a short source excerpt that didn’t settle them, which is the right time to say so.
Every wrong ruling
A wrong ruling can be disputed with a bond while its window is open. These would have needed one.
| The line | Our record | The jury | What happened |
|---|---|---|---|
| Any Ethereum account may send ETH to a validator at any time to increase its stake. From Wikipedia: Ethereum | True | False | It read “at any time” strictly: top-ups now wait in a queue. |
| When a sender is creating a transaction, the sender must specify a gas limit and gas price. From Wikipedia: Ethereum | Out of date | True | Transactions now set a maximum fee, not a gas price. The jury treated the two as the same. |
| Zcash is modeled on bitcoin, sharing its proof-of-work consensus mechanism. From Wikipedia: Zcash | True | False | It took “sharing its proof-of-work” to mean the same algorithm. Zcash uses a different one. |
| Ethereum's consensus protocol is replaced by Avalanche's Snowman consensus. From Wikipedia: Avalanche (blockchain platform) | True | False | The sentence needs its article around it. Alone, it reads as if Ethereum itself changed. |
| Tether Holdings Limited is headquartered in El Salvador as of January 2025. From Wikipedia: Tether (cryptocurrency) | True | False | The source names Tether International, not Tether Holdings, and the jury held that against the line. |
| Tether's USDT is currently the world's largest market capitalization stablecoin. From Wikipedia: Stablecoin | True | False | It read USDT’s rank among all coins as its rank among stablecoins. |
Can someone lie to the jury?
We attacked it ourselves, with the jury’s exact request. Two of three models decide.
| What the jury was given | The right result | Ruled false |
|---|---|---|
| Real receipts, on 10 lines we caught | Ruled false, and they are | 10 of 10 |
| An accusation with no proof, on 10 true lines | Should not rule false | 0 of 10 |
| Evidence that orders a verdict, on 10 true lines | Should not rule false | 0 of 10 |
| A made-up receipt, on 10 true lines | Should not rule false | 9 of 10 |
The jury ignores accusations and orders, but a made-up receipt fooled it 9 times in 10: an AI can’t tell a real quote from a fake one by reading it. On 10 true lines given what our record found, it ruled true on 7 and never false.
So every receipt is checked where it lives
Before our jury sits, each cited page is fetched and its exact words looked for, and three attested models read the page against the challenge.
| The challenge cited | Lines | Our jury sat | The Sentinel disputed |
|---|---|---|---|
| Real receipts from real sources | 21 | 20 | 0 |
| A made-up page, quote or record | 30 | 0 | 30 |
| A real page and the lie, with no quote | 10 | 0 | 10 |
| A real quote that doesn’t support the lie | 13 | 0 | 12 |
A challenge whose receipts fail gets no jury from us. Anyone can still run the jury themselves, so if a ruling rests on receipts that failed, the Sentinel disputes it with its own bond inside the window, and the bond doubles if they dispute back. What it can’t catch is a quote on a site the attacker controls: then the doubling bond and the council at the third dispute are what stop them.
Can’t I just ask an AI?
Yes, and it’s how you win. We tried it on the 10 lines we caught and 6 we know are true.
| Asked | On the 10 caught lines | On the 6 true lines |
|---|---|---|
| One AI with web search, one run | Found 7, unsure on 3, repeated 0 | 5 right, unsure on 1, 0 wrongly doubted |
| Three AIs with no web, 3 runs each | Repeated the mistake in 47 of 90 answers | Wrongly doubted 16 of 54 answers |
An AI that searches can find most of these. All 10 were still on Wikipedia on 2026-11-06, because nobody is paid to look. A market pays the first person who does, and its answer is public, signed and settled.
The live record
These tests are off-chain. The real rulings are public as they happen.
Run it again
Each test is one command with a NEAR AI key. Nothing is signed, nothing is posted and no money moves.
node scripts/bench-jury.mjs # deployments/jury-bench.json
node scripts/measure-jury.mjs # deployments/jury-test.json
Full results: the jury, the attacks, AI with web search, AI with no web. Run on 2026-11-06, one run each. How the record was built: the method, written before the first verdict.
