METHOD AND AUDIT

How this is measured, and what the measurement cannot say.

Every figure on this page is read from the exported evidence package at build time. None of them is written into the page, so none can drift from the run that produced it.

Question

The research question

Given the set of 8-K filings accepted before a session's entry point, can they be ranked so that the ones followed by an unusually large benchmark-adjusted move appear near the top?

This is a ranking question, scored per session, not a return question. It asks which filings deserve attention first, and it is answerable with public data and checkable against what happened.

Method

Labels, windows and validation

Label: A benchmark-adjusted move of at least 2 standard deviations in the event window.
Event window: 2 sessions from the first eligible entry.
Estimation window: 120 sessions of market-model estimation, ending 20 sessions before the event.
Validation: purged_embargoed_walk_forward

Reaction convention: prior close (market-model event study convention); see reaction_capture.csv

Point in time

What the model is allowed to know

Entry happens at the first session that opens after the filing's acceptance timestamp. A filing accepted after the close is entered the next session, never the same one.
Trailing features end the session before entry. A trailing volatility window that includes the event day is a window that contains the answer.
The universe at each date is the universe as it was known then, not the set of issuers that still exist today.
Issuer history is restricted to filings whose outcomes had already resolved at the decision point.

Leakage audit

The leakage ladder

Each row removes one source of hindsight and re-runs the whole evaluation. The first row is a pipeline that looks strong and is not; the last is the one that produces every published number.

Each stage removes one source of hindsight, measured end to end.

Stage	Events	Average precision	ROC AUC	Daily top-five share	Lift vs random	Impossible entries
Naive pipeline	11,672	59.7%	0.841	31.2%	1.73	96%
+ purged, embargoed CV	9,727	59.0%	0.838	31.2%	1.71	96%
+ shifted trailing features	9,727	43.9%	0.782	27.4%	1.50	96%
+ point-in-time universe	9,727	43.9%	0.782	27.4%	1.50	96%
+ point-in-time entry	9,721	39.7%	0.764	19.8%	1.70	0%

Naive pipeline: Shuffled K-fold, today's index, trailing windows that include the event day, entry on the filing date.
+ purged, embargoed CV: Stop training on the future and on labels whose outcome windows reach into the test fold.
+ shifted trailing features: Trailing volatility and turnover must end the session before entry, not include it.
+ point-in-time universe: Screen on index membership as of the event date, restoring the issuers that were later dropped.
+ point-in-time entry: Enter at the first open after the acceptance time. This also changes the measured window; the invariant is that every impossible entry disappears.

Metrics

Ranking metrics

Headline metrics from the honest pipeline.

Metric	Value	What it measures
Daily precision@5	19.8%	The average share of each session's top five that turned out to be material.
Random precision@5	11.6%	The same measure with five filings drawn at random from the same session.
Average precision	39.7%	Area under the precision-recall curve over all filings, not per session.
ROC AUC	0.764	The chance a random material filing is ranked above a random immaterial one.
Base rate	12.7%	The share of all evaluated filings that were followed by a material move.

Precision@5 is an average share, not a hit rate. It is the average share of each day's top five that turned out to be material — not the probability of finding at least one material filing among them, and not a per-day success rate.

Intervals come from a cluster bootstrap over sessions; paired for baselines with 2,000 draws over 957 sessions. Average precision: 36.5% to 43.0% at 95%.

Baselines

Compared with rules that need no model

A model is only interesting against the cheap rules it is meant to beat. All of these are scored on the same sessions, the same way.

Daily top-five share for each selection rule.

Selection rule	Daily top-five share
The model's ranking	19.8%
Five at random	11.6%
Earliest arrival first	9.3%
Item 2.02 (results of operations) first	13.4%
Perfect foresight (upper bound)	28.3%

Capacity

What if a reader has time for more, or fewer, than five?

Precision at each reading capacity, from the exported capacity profile. Sessions with fewer than k filings are left out at that k.

Precision at k against a shuffled queue and the best possible order that day.

Read k	Sessions	Random	Model	Ceiling	Lift	Span captured
1	942	11.7%	31.6%	58.3%	2.71	43%
2	920	11.7%	26.5%	45.9%	2.26	43%
3	887	11.8%	23.5%	37.8%	1.99	45%
5	766	11.6%	19.8%	28.3%	1.70	49%
10	365	13.6%	18.7%	22.7%	1.38	56%
20	60	16.0%	18.5%	19.7%	1.16	68%

The ceiling is far below 100% because a day with one material filing cannot put five material filings in a top five, and some sessions have none.

Tradability

Could this be traded?

The share of each filing's abnormal move that is already in the opening print. This is a measurement of what was left to act on, not a trading rule.

For material filings accepted after the close, 45.7% of the reaction is already in the opening print.

Median share of the abnormal move realised by the next open.

Filings	Count	Already in the open
all filings	11,665	6.4%
not material	10,170	3.1%
material (>= threshold)	1,495	27.7%
material, accepted pre	27	25.5%
material, accepted open	381	10.4%
material, accepted post	1,086	45.7%
material, accepted closed	1	3.9%

Scored again with the label anchored at the open instead of the prior close, average precision is 15.5%. That is a harder question, not a second ranker.

Model comparison

Does the choice of model matter?

Every family is scored on the same filings. The difference table is the paired comparison against the shipped family, not a second experiment.

Out-of-sample scores by model family.

Family	Average precision	ROC AUC	Average precision, 95%
hist_gbdt	37.3%	0.746	34.3% – 40.4%
logistic	35.6%	0.736	32.7% – 38.6%
random_forest	39.7%	0.764	36.5% – 43.0%
stratified_dummy	12.9%	0.506	12.0% – 13.8%

Paired difference versus the reference family. A negative difference is behind the reference.

Family	Reference	Difference	95% on the difference
hist_gbdt	random_forest	-0.024	-0.038 – -0.009
logistic	random_forest	-0.041	-0.058 – -0.024
stratified_dummy	random_forest	-0.269	-0.298 – -0.240

An interval that crosses zero does not establish that one family is ahead. Nested selection, not the top row of this table, is what chooses the shipped family.

Calibration

Do the scores mean what they say?

Three calibrators scored on the same held-back folds. The shipped method is the one named in the export, not the one with the friendliest row.

The export ships identity.

Brier score, expected calibration error and ranking metrics by calibrator.

Method	Brier	Expected calibration error	Average precision	ROC AUC
identity	0.155	0.011	37.0%	0.693
isotonic	0.157	0.027	35.7%	0.675
sigmoid	0.158	0.019	35.9%	0.675

A lower Brier score and a lower expected calibration error are better. Ranking metrics are reported beside them so a calibrator cannot win by flattening the ranking.

Volatility

How turbulent is the next month?

A 20-session volatility band. It is not an input to the filing ranker, and it does not say which way the price moves.

Pinball loss and how often the stated bands contain the outcome.

Forecaster	Pinball loss	80% band holds	Band width	Calibrated
random_walk	0.036	79.1%	0.331	Yes
climatology	0.030	80.3%	0.270	Yes
har	0.028	77.4%	0.235	Yes
chronos	0.030	75.5%	0.237	No

The shipped forecaster is har. Its 80% band holds 79.6% of outcomes in the calm third and 75.3% in the turbulent third.

A forecaster ships only when both bands hold close to the coverage they claim. A wider band can look calibrated while saying less.

Ablations

What each group of features adds

Each row removes one group and re-scores. A group that changes nothing is reported as changing nothing.

Feature-group ablations, re-scored end to end.

group	features	pr_auc	roc_auc	brier
structured	51	0.37195895820068187	0.6960733089385638	0.154947204577749
deterministic_text	55	0.370106899238709	0.69309281896201	0.1553201571519176
transformer_text	57	0.3664431392150513	0.69263908703564	0.15546405113887993
all	61	0.36495395099528927	0.6897501972812635	0.155896896978032

Limitations

What this cannot tell you

The universe is a convenience sample of currently listed large caps. It is not survivorship-controlled, so it cannot speak to issuers that delisted.
No transaction cost, spread, slippage or capacity assumption is applied anywhere. A ranking result is not a strategy result.
The label is the magnitude of a move, not its direction. Nothing here says which way a price went or will go.
Dynamic events are published for AAPL, JPM and NVDA only. Other companies still have filings, filed results and price history.
Results are measured on US 8-K disclosures over the evaluated sessions and should not be assumed to carry to other filing types, markets or periods.

Provenance

Evidence exported: 2026-08-28T02:43:23.788742+00:00
Schema version: 1.1
events: 11,716 rows · sha256 fec33e5bdc9a…
prices: 1,982,600 rows · sha256 d52cb1f58823…
membership: 193 rows · sha256 f25717f8df47…
note: — rows · sha256 …

Glossary

Method index

Precision@k: Of the k items ranked highest, the share that turned out to be material. Averaged over sessions when it is called daily precision@k.
Average precision: The area under the precision-recall curve: one number summarising how well the ranking separates material from immaterial across every threshold.
ROC AUC: The probability that a randomly chosen material filing is ranked above a randomly chosen immaterial one. 0.5 is chance.
Base rate: How often the thing being predicted happens at all. Every precision figure has to be read against it.
Walk-forward validation: Train on the past, test on the future, repeatedly moving forward in time — rather than shuffling all dates together.
Purging and embargo: Dropping training examples whose outcome windows overlap the test period, so an answer cannot reach the model through an overlapping window.
Point in time: Using only what was knowable at the decision moment: the universe as it was, the prices as they had printed, and the filings already accepted.