Explore the interactive research site →
This repository is the clean release of the final recommended model only: a five-seed ensemble of histogram gradient-boosted trees, trained only on firm-derived SEC features and evaluated with expanding fiscal-year walk-forward holdouts.
Given information available by the end of fiscal year t-1, can we identify which U.S. public
companies will deliver stronger revenue growth in fiscal year t than comparable firms in the
same two-digit SIC industry?
The intended use is cross-sectional candidate selection. Overall binary classification is useful but moderate; the model is substantially stronger at ranking the most likely outperformers. The ensemble's rank score expresses how strongly a company stands out relative to the other firms scored in that year. It is not a calibrated probability.
| Component | Final specification |
|---|---|
| Unit of observation | Company–prediction-year |
| Feature timing | SEC information from fiscal year t-1 or earlier |
| Outcome | Revenue growth in t minus aligned leave-one-out SIC2 peer median |
| Predictors | 164 constructed firm_* fields; 163 numeric fields enter the model |
| Excluded predictors | FRED macro fields, industry dummies, identifiers, and target-derived fields |
| Learner | HistGradientBoostingClassifier |
| Hyperparameters | learning rate 0.05; 300 iterations; 31 leaves; L2 0.1; early stopping |
| Ensemble | Five random seeds; average within-year percentile ranks |
| Validation | Expanding walk-forward fiscal-year holdouts, 2016–2024 |
| Primary use | Top-decile and narrower annual candidate lists |
These are the independently rerun results produced by the code in this repository against the final
rebuilt dataset. The JSON version is in results/reference_metrics.json.
| Metric | Result | Meaning |
|---|---|---|
| Overall pooled accuracy | 68.22% | Correct binary calls across all held-out company-years at a 0.5 mean-probability threshold |
| Mean annual accuracy | 68.26% | Accuracy calculated separately by test year, then averaged so large years do not dominate |
| Pooled classification precision | 69.20% | Among firms called outperformers at mean probability ≥ 0.5, the share that actually outperformed |
| Mean annual rank-ensemble AUC | 74.95% | Mean discrimination after averaging within-year percentile ranks across five seeds |
| Mean annual top-10% precision | 89.61% | Mean success rate among the final ensemble's highest-ranked 10% in each year |
| Mean annual top-5% precision | 93.63% | Mean success rate among the highest-ranked 5% in each year |
| Mean annual top-1% precision | 98.00% | Mean success rate among the highest-ranked 1% in each year |
| Holdout observations | 25,221 | Company-years evaluated outside their training window in the locked reference run |
The 89.61% figure is not overall accuracy. It applies only to the roughly highest-scoring 10% of firms in each held-out year. This is why the recommended use is to identify likely outperformers, not to treat every company classification as equally reliable.
A five-year Dirichlet perturbation analysis tests whether averaging older company observations into
each feature improves the final model. The weights are ordered from the most recent fiscal-year
observation to four years older, so [1, 0, 0, 0, 0] is exactly the released current-only model.
The current-only vector ranked first in the 2021–2024 screen and remained stronger than the best
noncurrent challenger in the full five-seed 2016–2024 verification: 89.61% versus 88.55% mean
top-decile precision and 74.95% versus 74.10% mean AUC. Reproducible code, candidate-level results,
charts, and Wolfram Language visualization source are documented in
analysis/README.md.
A separate 3D PCA of annual feature-distribution fingerprints shows substantial displacement in fiscal feature years 2020–2022. Across the nine evaluated years, distance from the 2015–2019 centroid is associated with lower top-decile precision (Pearson r = -0.73), providing descriptive evidence that COVID-era covariate shift made the highest-confidence selections less reliable.
flowchart LR
A["SEC quarterly Financial Statement Data Set ZIPs"] --> B["Select one U.S. 10-K per CIK and fiscal year"]
B --> C["Extract 53 canonical annual accounting metrics"]
A --> D["Extract qtrs=1 revenue facts from 10-Q/10-K"]
D --> E["Build Q1-Q4 trajectory; derive Q4 only when Q1-Q3 reconcile to annual revenue"]
C --> F["Annual ratios, growth, changes, lags, and structural-zero flags"]
E --> F
G["Nine downloaded FRED series"] --> H["29 annual macro descriptors: dataset/audit only"]
F --> I["Company-year panel, features dated t-1"]
H --> I
I --> J["Target: growth in t minus aligned leave-one-out SIC2 peer median"]
J --> K["Final model selects only firm_* columns"]
K --> L["Five-seed walk-forward HGB rank ensemble"]
The annual pipeline establishes the firm/fiscal-year spine and most accounting characteristics.
The quarterly pipeline contributes revenue trajectory features to the matching annual row. A fourth
quarter is derived as annual revenue minus Q1, Q2, and Q3 only when exactly three interim quarters
are present and the residual is positive. Quarterly observations never move beyond the annual
feature period, so a prediction for year t uses quarters ending no later than fiscal year t-1.
For company i, prediction year t, and two-digit SIC industry g:
growth(i,t) = [revenue(i,t) - revenue(i,t-1)] / revenue(i,t-1)
target(i,t) = growth(i,t) - median{growth(j,t) : j != i, SIC2(j)=g,
|period_end(j)-period_end(i)| <= 92 days}
outperform(i,t) = 1[target(i,t) > 0]
Peer medians are leave-one-out, require at least 20 aligned peers, and use a ±92-day period-end window so June fiscal-year companies are not benchmarked against unrelated December filers.
Every test year is untouched until the model for that year is fitted. Imputation medians, winsorization cutoffs, and constant-column removal are learned from historical rows only.
| Test year | Training years |
|---|---|
| 2016 | 2011–2015 |
| 2017 | 2011–2016 |
| 2018 | 2011–2017 |
| … | … |
| 2024 | 2011–2023 |
For each test year, five models are fitted with different seeds. Their probability outputs are converted into within-year percentile ranks and averaged. The resulting rank-ensemble score is the final selection score. Raw mean probability remains available for binary decisions at 0.5.
src/public_mna/
sec_bulk.py SEC audit, 10-K selection, canonical fact extraction, annual panel
quarterly_features.py Quarterly revenue extraction and trajectory features
macro.py Optional FRED annualization for dataset construction and auditing
target_rebuild.py Forecast-safe firm features and aligned industry-relative target
final_model.py The only retained model and evaluator
cli.py Reproducible command-line interface
docs/
DATA.md Sources, accounting fields, macro fields, and calculated feature groups
MODEL_CARD.md Intended use, evaluation logic, limitations, and leakage controls
results/
reference_metrics.json Locked metrics from the verified rerun
reference_annual_metrics.csv Year-by-year holdout results
tests/ Timing, leakage, peer-alignment, preprocessing, and ranking tests
python -m venv .venv
# Windows: .venv\Scripts\activate
# macOS/Linux: source .venv/bin/activate
python -m pip install -e ".[dev]"
pytestPlace SEC Financial Statement Data Set quarterly ZIP files (2010q1.zip, …) in
data/raw/sec_fsds/. Place the nine FRED CSV exports listed in docs/DATA.md in
data/raw/fred/. Raw and processed data are deliberately ignored by Git.
revenue-model audit-sec
revenue-model select-annual --start-fy 2010 --end-fy 2025
revenue-model extract-facts
revenue-model build-panel
revenue-model extract-quarterly
revenue-model build-fred --start-year 2010 --end-year 2025
revenue-model build-dataset --start-year 2011 --end-year 2024
revenue-model evaluate --start-year 2016 --end-year 2024The evaluation command writes summary.json, annual metrics, walk-forward split metadata, the exact
feature list, and per-company holdout predictions under results/run/.
To run directly against the archived reference dataset without copying it into this repository:
revenue-model evaluate --dataset /path/to/independent_year_rebuilt_dataset.csvThe repository includes code, configuration, tests, documentation, and small aggregate results. It
does not include SEC bulk archives, FRED downloads, the 90+ MB processed company-year dataset, or
per-row predictions. This keeps the Git history reviewable and avoids redistributing source data.
The reference dataset fingerprint and software versions are recorded in
results/reference_metrics.json.