Audit and lessons (2026-10-25)

Twelve generations of trading research, January to October 2026. None reached a claim. In this platform: 6 in-sample screen-gate passes; none survived to a claim; both sealed tests KILL.

#	What it tried	Outcome
G1	Crypto reinforcement learning	Headline came from bear-market beta and same-day leakage.
G2	RL tournament on crypto	A single-seed winner; the multi-seed rerun showed no edge.
G3	Classical crypto sweep (trend, carry, reversion, volatility premium)	Nothing passed; a kill fired and the plan kept the idea anyway.
G4	High-frequency bitcoin analog forecaster	Too few trades per trial: the idea was never really tested.
G5	LLM-debate equity signals	Lost after costs: more reasoning over the same public data added turnover, not signal.
G6	Large-cap equal weight with optimiser search	The objective saw the test set; the result matched the index.
G7	Research wiki and committee-written plans	A pre-registered momentum test missed its null; the futures plan was never backtested.
G8	Following third-party paid alerts	Results are restricted under that source's terms.
G9	Pre-registered strategies on third-party paid alerts	Results are restricted under that source's terms.
G10	This platform: registry, sealed holdouts, stock screens, rates calendar, options, prediction markets	No claim; both sealed tests KILL.
G11	Following third-party paid alerts, on options	Results are restricted under that source's terms.
G12	Our own index-options research in simulation	Every test ended PARK or KILL; results restricted under the data vendors' terms.

Three independent audits

Three AI models audited the work in parallel on the same brief, none seeing the others; their claims were then re-checked against the files and the registry. They agree that:

the machinery is sound: hash-chained registry, sealed slices with opening budgets, specification before returns, double-cost stress, honest kills, including kills of our own screen passes;
there is no claim after 441 registry events (237 trials, 72 screens, 38 families);
futures trend plus carry on commercial data is the one evidence line worth starting, and it was chosen weeks earlier and never started;
a private round is not reachable on the current path: it needs a forward record, licensable data and a mechanism.

Where they disagreed (is the proof bar too strict?), the evidence sides with "no": the sealed tests, the only true out-of-sample tests, ended KILL. That points to weak candidates, not an over-strict bar.

What recurs

Every positive headline came from the evaluation, not the market: leakage, test-set selection, a single seed, market beta.
Do not build on someone else's signals. Their data and results are not ours to show, and the edge is not ours.
The cheap killing number came last. Gross edge against cost is now the first check on every screen.
Platform and site work outran evidence. Earlier sites were private showcases built before any result; this one is a record.
Kills fired but the plan did not change. Decisions are now recorded, and kills feed back into them.
The method is the durable asset: pre-registration, deflated Sharpe, overfitting probability, sealed data, honest kills.
Data rights were an afterthought. Nothing tested so far runs on data an investor could be shown, which is why this log shows verdicts and not numbers.
Slow, cheap, behavioural or fundamental ideas beat fast ones; intraday and short-dated options ideas kept none.
Uncommitted results get lost or misquoted. The hash chain closes that.

What follows from it

No new mining of academic data; forward logs already running continue.
Freeze the futures trend plus carry specification before any futures data is bought or read (see What's next).
Keep the forward logs running and fix machine uptime so they stop missing days.
Test whether the method itself is worth paying for: an evidence-audit service for other researchers.