Can AI call real-world events better than the market?

Detailed scoreboard

We backtested on 44,771 resolved Polymarket binary markets, got an attractive positive result, then found its leak ourselves and retracted it publicly. We are now testing forward: every day we forecast markets that are still trading, anchor the sha256 of the forecast file to the Bitcoin blockchain with OpenTimestamps, and score it after resolution. No orders were ever sent, no wallet exists, and the cost is $0.

Score history

The scoreboard is recomputed after every daily run. A new row is added whenever any number changes; old rows are never edited. Format: settled count · ours / market (the models column is skill relative to the market).

Full history: scoreboard.jsonl

Iterations

The approach is developed in rounds. Each round is pre-registered before it is run, and its result is recorded here whether it helped or not. Method details are not published.

1. Forward shadow run (in progress)

Any backtest has to assume what was known at the time. That assumption is exactly where v1 went wrong. A shadow run does not need it: forecasts are locked before the outcome exists, and a third party timestamps them.

Settled forecasts in the decision band (market price 0.15–0.85): 256 of 350. The threshold follows the power estimate in Foresight Arena: a Brier-skill edge of 0.02 needs about 350 settled forecasts for 80% power. Until then this page states no verdict.

Current point estimates on the settled decision band (below the sample threshold, not a result): M1 BSS +0.012 · M2 BSS -0.002 · M5 BSS -0.001

Competitors (pre-registered, frozen)

market (the market price at snapshot time, i.e. the opponent) and frozen models labelled M1, M2, M5, M10 and M11. Model internals are not published; their outputs are, and are timestamped before resolution.

Snapshot ledger

2. High-confidence picks

We only call the markets we are most confident about, and report how often those calls are right. A high hit rate alone proves little, because picking heavy favourites is easy. So next to our hit rate we show the hit rate the market price itself implied for the same calls; the gap between the two is the part that counts.

Validation estimate, frozen before any pick was made: picking the most confident 70% of markets gave a hit rate of 99.8% / 99.2% / 98.4% (1 / 7 / 30 days) against a market-implied 99.5% / 98.4% / 97.6%. Rules: picks_rules.json, timestamped.

3. Paper account

Rules: $100 flat per order, one decision per market at its first appearance in a snapshot, held to resolution. Models trade only when |model − mid| ≥ 0.05. Fills use the best bid/ask recorded in the snapshot; markets with a missing side or a spread above 0.10 are skipped. Only top-of-book is modelled, so fills are optimistic. A verdict needs ≥ 200 settled orders per account and a 95% interval that excludes zero.

Rules file paper_rules.json · sha256 f1dd3d2d901f546dd522233fd8a69437c745ec0e90d4b17cc96247fd9975fd15 · pending · .ots

Order files (derived deterministically from snapshot + rules)

How to verify it yourself

# 1. download a file and its timestamp proof
curl -O https://renlab.ai/poly/evidence/snap-20261112T003002Z.json
curl -O https://renlab.ai/poly/evidence/snap-20261112T003002Z.json.ots
# 2. the hash must match the ledger above
shasum -a 256 snap-20261112T003002Z.json
# 3. check the hash is in a Bitcoin block (either way)
#    a) no install: drop the .json and the .ots onto https://opentimestamps.org
#    b) pip install opentimestamps-client && ots info snap-20261112T003002Z.json.ots
# 4. re-derive the paper orders from snapshot + rules and diff them
#    the rules are in paper_rules.json and picks_rules.json, both timestamped

Fresh proofs show “pending” for a few hours until a calendar commits them to a block. A file never changes after it is written; changing one byte breaks its hash. Last third-party archive copy: 2026-11-12 00:50 UTC

3b. Capital account

Accuracy is only half of it. This paper account decides which markets to fund and how much, after spread, fees and the time money is locked until resolution, with per-market, per-event, per-theme and total-exposure caps and pre-set pause rules. Its rules and every order file are committed by hash and Bitcoin timestamp now, and will be published in full when the test concludes, so anyone can check they were not changed afterwards.

Running test (anytime-valid e-values)

An e-value can be checked every day without inflating false alarms. A value of 20 or more is evidence at the 5% level; when all three pick buckets are read together the bar is 60. Our earlier plan to decide at 200 settled picks was dropped: at hit rates near 99% that sample has almost no power.

Commitments

Rules file capital_rules.json · sha256 5371146739df9dfbeef1eee167b06d332fa52c47b37eb150796f2289cf10b370 · pending · .ots

3c. Arm K: an independent forecaster that never sees prices

In each batch, 40 open markets are drawn by a fixed hash rule. The forecasters get only the question and its resolution rules, with no price, and research the question on their own. Their forecasts are hashed and timestamped first. Only after that do we record the market price, which is the benchmark. Forecasts are revealed when the test ends.

3d. Arm C: crypto price questions, priced from an outside market

Questions of the form “will the price be above X on date D” also have a price in a separate, much larger market. Arm C turns that price into a probability with a fixed rule and records it at the same moment as the Polymarket price. Each market is recorded once, at first sight. Files are committed by hash and timestamp and revealed when the test ends.

4. Backtest: a public retraction

Brier skill score vs market price, decision band (0.15 ≤ price ≤ 0.85); > 0 means more accurate

Realisable decision-band samples: 651 / 870 / 1,035. “sig.” = the 95% paired bootstrap interval (10,000 resamples) excludes zero.

The leak is visible in the v1 data

The two groups differ by under 3 points in price but 9–14 points in YES rate: whether a market resolved early carries the outcome, and the price cannot see it. Across all 44,761 markets, 39% resolved more than a day early. The literature calls this logical leakage (Paleka et al. 2025). Our four leakage checks, including a future-price canary, only catch feature leakage; they cannot catch this sample-selection leak.

5. Open method search

After the retraction we searched freely. Seven candidates were fitted on search-train and compared only on search-val; one per horizon (★) was scored once on a held-out test set. All candidates are shown, not just winners.

search-val, decision-band BSS

Selected method on test

Winners on validation shrink or flip sign on test, the usual signature of picking the maximum on a small validation set. This matches the literature: public information inside the market rarely beats Polymarket (Le 2026, Cardozo & Rivero-Wildemauwe 2026).

6. Information beyond the price

We also tested information that is not in the price. Some of it carries a real, persistent signal, but in our pre-registered validation it did not make the forecasts more accurate than the best price-only models, so it did not join the shadow run on that basis. We also tested four further ideas taken from other practitioners and from published research; none beat the frozen picks on validation. Details are not published.

7. Scope and limits

• Accuracy (Brier) is the headline, not profit. The paper account ignores depth, slippage and capital lock-up. Nothing here is investment advice.
• Backtest prices are daily samples; intraday information is lost.
• Markets in one event are strongly correlated; intervals resample whole events, but event id is only a proxy.
• The held-out test set has been viewed three times; after that, new methods are judged only by the shadow run.
• Why no third-party paper-trading site (AgentPit, Polysandbox, PaperPoly…): their records are held by the operator and cannot be audited by outsiders. Bitcoin-timestamped files give stronger evidence.
• The daily job runs on one machine; a day it is off has no snapshot. Gaps are visible in the ledger and are never back-filled.

Raw data

anchor_leak.json · settlement.json · ledger.jsonl · paper_rules.json · picks_rules.json · paper_account.json · paper_ledger.jsonl · archive_log.jsonl · eprocess.json