Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 8 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,9 @@ Core upstream artifacts:
2. `data/output/latest_ranking.csv`
3. `data/output/live_pool.json`
4. `data/output/live_pool_legacy.json`
5. `data/output/release_manifest.json`
6. `data/output/release_status_summary.json`
5. `data/output/artifact_manifest.json`
6. `data/output/release_manifest.json`
7. `data/output/release_status_summary.json`

## Upstream Boundary

Expand All @@ -28,7 +29,7 @@ Core upstream artifacts:

In practice, that means:

- upstream publishes and explains `latest_universe`, `latest_ranking`, `live_pool`, `release_manifest`, and release-status summaries
- upstream publishes and explains `latest_universe`, `latest_ranking`, `live_pool`, `artifact_manifest`, `release_manifest`, and release-status summaries
- downstream consumes the official live-pool contract plus publish metadata and emits only runtime/execution status
- research CSVs, shadow-track diagnostics, and monthly review outputs stay upstream and are not part of the minimum downstream execution contract

Expand All @@ -40,7 +41,7 @@ The repository is now intentionally split into two tracks:
- data source: `Binance Spot only`
- universe mode: `core_major`
- publish cadence: `monthly`
- default outputs: `latest_universe.json`, `latest_ranking.csv`, `live_pool.json`, `live_pool_legacy.json`
- default outputs: `latest_universe.json`, `latest_ranking.csv`, `live_pool.json`, `live_pool_legacy.json`, `artifact_manifest.json`
- `Experimental external-data track`
- used for research, comparison, and validation only
- not enabled by default
Expand Down Expand Up @@ -245,10 +246,10 @@ Validate the local production artifacts before publish or rollback:
.venv/bin/python scripts/validate_release_contract.py --mode core_major --expected-pool-size 5
```

Require a generated manifest as part of the check:
Require generated release and artifact manifests as part of the production check:

```bash
.venv/bin/python scripts/validate_release_contract.py --mode core_major --expected-pool-size 5 --require-manifest
.venv/bin/python scripts/validate_release_contract.py --mode core_major --expected-pool-size 5 --require-manifest --require-artifact-manifest
```

Operator workflow details, rollback steps, and research-vs-production boundaries are documented in `docs/operator_runbook.md`.
Expand Down Expand Up @@ -370,7 +371,7 @@ Downstream consumers should rely on these core fields in `data/output/live_pool.
- `symbol_map`
- `source_project`

Publish-time pointer fields such as `storage_prefix`, `current_prefix`, `live_pool_uri`, `live_pool_legacy_uri`, `latest_universe_uri`, and `latest_ranking_uri` are stable when present in the published Firestore payload, but they are release/distribution metadata rather than research features.
Publish-time pointer fields such as `storage_prefix`, `current_prefix`, `live_pool_uri`, `live_pool_legacy_uri`, `artifact_manifest_uri`, `latest_universe_uri`, and `latest_ranking_uri` are stable when present in the published Firestore payload, but they are release/distribution metadata rather than research features.

Optional additive research extensions:

Expand Down
25 changes: 24 additions & 1 deletion README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,10 @@
1. `data/output/latest_universe.json`
2. `data/output/latest_ranking.csv`
3. `data/output/live_pool.json`
4. `data/output/live_pool_legacy.json`
5. `data/output/artifact_manifest.json`
6. `data/output/release_manifest.json`
7. `data/output/release_status_summary.json`

## 当前状态

Expand All @@ -18,7 +22,7 @@
- 数据源:仅 `Binance Spot`
- universe mode:`core_major`
- 发布频率:`monthly`
- 默认输出:`latest_universe.json`、`latest_ranking.csv`、`live_pool.json`、`live_pool_legacy.json`
- 默认输出:`latest_universe.json`、`latest_ranking.csv`、`live_pool.json`、`live_pool_legacy.json`、`artifact_manifest.json`
- `Experimental external-data track`
- 仅用于研究、比较和验证
- 默认不启用
Expand Down Expand Up @@ -147,6 +151,20 @@ pip install -r requirements.txt
- 输出设置
- GCS / Firestore 发布设置

## 发布契约检查

发布或回滚前,先校验本地生产产物:

```bash
.venv/bin/python scripts/validate_release_contract.py --mode core_major --expected-pool-size 5
```

生产发布链应同时要求 release manifest 和 profile-aware artifact manifest:

```bash
.venv/bin/python scripts/validate_release_contract.py --mode core_major --expected-pool-size 5 --require-manifest --require-artifact-manifest
```

## 最小可运行流程

1. 下载历史数据
Expand Down Expand Up @@ -215,12 +233,15 @@ pip install -r requirements.txt
- `data/output/live_pool_legacy.json`
- Firestore summary document

`data/output/artifact_manifest.json` 是 profile-aware wrapper,负责声明 artifact contract version、主 artifact、相关文件路径和校验和;它不是 `live_pool.json` 的字段复制。

一些发布期辅助字段,例如:

- `storage_prefix`
- `current_prefix`
- `live_pool_uri`
- `live_pool_legacy_uri`
- `artifact_manifest_uri`
- `latest_universe_uri`
- `latest_ranking_uri`

Expand Down Expand Up @@ -290,6 +311,7 @@ make monthly-shadow-build
- official baseline
- `data/output/live_pool.json`
- `data/output/live_pool_legacy.json`
- `data/output/artifact_manifest.json`
- `data/output/release_manifest.json`
- shadow candidate tracks
- `data/output/shadow_candidate_tracks/track_summary.csv`
Expand Down Expand Up @@ -322,6 +344,7 @@ make monthly-build-telegram

- 只发送简短的 monthly build/publish health summary
- 使用已有的 `monthly_shadow_build_summary.json`、`live_pool.json`、`release_manifest.json`、`track_summary.csv`
- 生产发布链还会检查 `artifact_manifest.json`,但 Telegram 文本只展示摘要状态
- 如果 Telegram 凭证缺失,会跳过而不是报错中断
- 不改变 monthly build 行为,也不是 review 包生成器

Expand Down
43 changes: 42 additions & 1 deletion docs/integration_contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,41 @@ The upstream project publishes a monthly `core_major` live pool and exposes it t

## Canonical Downstream Files

### `artifact_manifest.json`

This profile-aware manifest is the canonical contract wrapper for downstream
runtimes. It lets the execution platform validate the artifact family without
binding to `CryptoLeaderRotation` internals.

Required stable fields:

- `manifest_type = strategy_artifact`
- `contract_version = crypto_leader_rotation.live_pool.v1`
- `strategy_profile = crypto_leader_rotation`
- `artifact_type = live_pool`
- `artifact_name = crypto_leader_rotation_live_pool`
- `as_of_date`
- `snapshot_as_of`
- `version`
- `mode`
- `symbol_count`
- `symbols`
- `source_project`
- `generated_at`
- `primary_artifact = live_pool`
- `artifacts`

The `artifacts` mapping includes relative file paths and SHA-256 checksums for:

- `latest_universe`
- `latest_ranking`
- `live_pool`
- `live_pool_legacy`

Downstream platforms should treat this manifest as the strategy artifact
contract and keep legacy `live_pool_legacy.json` parsing as a compatibility
path, not as the only contract shape.

### `live_pool_legacy.json`

This is the most convenient file for older downstream scripts that expect a direct symbol mapping.
Expand Down Expand Up @@ -123,9 +158,12 @@ Payload example:
"current_prefix": "gs://example-bucket/crypto-leader-rotation/current",
"live_pool_legacy_uri": "gs://example-bucket/crypto-leader-rotation/current/live_pool_legacy.json",
"live_pool_uri": "gs://example-bucket/crypto-leader-rotation/current/live_pool.json",
"artifact_manifest_uri": "gs://example-bucket/crypto-leader-rotation/current/artifact_manifest.json",
"latest_universe_uri": "gs://example-bucket/crypto-leader-rotation/current/latest_universe.json",
"latest_ranking_uri": "gs://example-bucket/crypto-leader-rotation/current/latest_ranking.csv",
"versioned_live_pool_legacy_uri": "gs://example-bucket/crypto-leader-rotation/releases/2026-03-13-core_major/live_pool_legacy.json",
"versioned_artifact_manifest_uri": "gs://example-bucket/crypto-leader-rotation/releases/2026-03-13-core_major/artifact_manifest.json",
"artifact_contract_version": "crypto_leader_rotation.live_pool.v1",
"generated_at": "2026-03-13T13:00:00+00:00",
"source_project": "crypto-leader-rotation"
}
Expand All @@ -136,8 +174,9 @@ The Firestore document intentionally excludes the full ranking CSV. Downstream r
Stable vs additive fields:

- stable core fields: `as_of_date`, `version`, `mode`, `pool_size`, `symbols`, `symbol_map`, `source_project`
- publish-only pointer fields: `storage_prefix`, `current_prefix`, `live_pool_uri`, `live_pool_legacy_uri`, `latest_universe_uri`, `latest_ranking_uri`, `versioned_live_pool_legacy_uri`
- publish-only pointer fields: `storage_prefix`, `current_prefix`, `live_pool_uri`, `live_pool_legacy_uri`, `artifact_manifest_uri`, `latest_universe_uri`, `latest_ranking_uri`, `versioned_live_pool_legacy_uri`, `versioned_artifact_manifest_uri`
- additive observability field: `generated_at`
- artifact wrapper field: `artifact_contract_version`

## GCS Path Layout

Expand All @@ -148,6 +187,7 @@ gs://<bucket>/crypto-leader-rotation/releases/<YYYY-MM-DD-mode>/latest_universe.
gs://<bucket>/crypto-leader-rotation/releases/<YYYY-MM-DD-mode>/latest_ranking.csv
gs://<bucket>/crypto-leader-rotation/releases/<YYYY-MM-DD-mode>/live_pool.json
gs://<bucket>/crypto-leader-rotation/releases/<YYYY-MM-DD-mode>/live_pool_legacy.json
gs://<bucket>/crypto-leader-rotation/releases/<YYYY-MM-DD-mode>/artifact_manifest.json
```

Current pointers:
Expand All @@ -157,6 +197,7 @@ gs://<bucket>/crypto-leader-rotation/current/latest_universe.json
gs://<bucket>/crypto-leader-rotation/current/latest_ranking.csv
gs://<bucket>/crypto-leader-rotation/current/live_pool.json
gs://<bucket>/crypto-leader-rotation/current/live_pool_legacy.json
gs://<bucket>/crypto-leader-rotation/current/artifact_manifest.json
```

## Local Shadow Release History
Expand Down
12 changes: 7 additions & 5 deletions docs/operator_runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ Primary production outputs:
- `data/output/latest_ranking.csv`
- `data/output/live_pool.json`
- `data/output/live_pool_legacy.json`
- `data/output/artifact_manifest.json`
- `data/output/release_manifest.json`
- `data/output/release_status_summary.json`
- `data/output/release_status_summary.md`
Expand Down Expand Up @@ -86,7 +87,7 @@ Boundary rules:
3. Run explicit contract validation:

```bash
.venv/bin/python scripts/validate_release_contract.py --mode core_major --expected-pool-size 5
.venv/bin/python scripts/validate_release_contract.py --mode core_major --expected-pool-size 5 --require-manifest --require-artifact-manifest
```

4. Run publish preflight without external writes:
Expand Down Expand Up @@ -125,7 +126,7 @@ Boundary rules:

- `requirements-lock.txt` is present and matches the intended release dependency set.
- Local artifacts are non-empty and pass `scripts/validate_release_contract.py`.
- `live_pool.json`, `live_pool_legacy.json`, and `release_manifest.json` agree on `as_of_date`, `version`, `mode`, `pool_size`, and `source_project`.
- `live_pool.json`, `live_pool_legacy.json`, and `release_manifest.json` agree on `as_of_date`, `version`, `mode`, `pool_size`, and `source_project`; `artifact_manifest.json` agrees on `as_of_date`, `version`, `mode`, `source_project`, and `symbol_count`.
- `GCP_PROJECT_ID`, `GCS_BUCKET`, `FIRESTORE_COLLECTION`, and `FIRESTORE_DOCUMENT` are set correctly for real publish.
- Historical backfills use `--allow-stale` explicitly; do not silently publish stale artifacts.

Expand All @@ -141,7 +142,7 @@ Symptoms:
Actions:

- Re-run `scripts/build_live_pool.py`
- Inspect `data/output/latest_universe.json`, `live_pool.json`, `live_pool_legacy.json`, and `latest_ranking.csv`
- Inspect `data/output/latest_universe.json`, `latest_ranking.csv`, `live_pool.json`, `live_pool_legacy.json`, and `artifact_manifest.json`
- Confirm `pool_size`, `symbols`, `symbol_map`, `version`, and `source_project` are present and aligned

### Stale artifacts
Expand Down Expand Up @@ -192,17 +193,18 @@ Use rollback only when the newest publish is clearly bad or malformed.
- GCS `crypto-leader-rotation/releases/<version>/`
- the last good `data/output/release_manifest.json`

2. Restore the four canonical artifacts from that version into `data/output/`:
2. Restore the five canonical artifacts from that version into `data/output/`:

- `latest_universe.json`
- `latest_ranking.csv`
- `live_pool.json`
- `live_pool_legacy.json`
- `artifact_manifest.json`

3. Validate the restored payload locally:

```bash
.venv/bin/python scripts/validate_release_contract.py --mode core_major --expected-pool-size 5
.venv/bin/python scripts/validate_release_contract.py --mode core_major --expected-pool-size 5 --require-artifact-manifest
```

4. Regenerate the manifest and verify publish preflight:
Expand Down
3 changes: 3 additions & 0 deletions docs/validation_status.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ The repository is now frozen around this production default:
- `latest_ranking.csv`
- `live_pool.json`
- `live_pool_legacy.json`
- `artifact_manifest.json`

This is the only path that should be treated as the formal production baseline.

Expand Down Expand Up @@ -125,6 +126,7 @@ Validated in-repo:

- `scripts/build_live_pool.py` produces the default `Production v1` live output
- `scripts/publish_release.py --dry-run` builds a correct production release manifest
- `scripts/validate_release_contract.py --require-artifact-manifest` validates the profile-aware artifact contract
- `scripts/write_release_heartbeat.py` writes a small logs-branch heartbeat file
- GitHub Actions workflow YAML parses correctly
- release versioning, GCS object keys, and Firestore payload layout are consistent
Expand All @@ -136,6 +138,7 @@ Validated artifacts:
- `data/output/latest_ranking.csv`
- `data/output/live_pool.json`
- `data/output/live_pool_legacy.json`
- `data/output/artifact_manifest.json`
- `data/output/release_manifest.json`
- `data/output/heartbeat/monthly/<version>.json`

Expand Down
4 changes: 3 additions & 1 deletion scripts/build_live_pool.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,13 +62,15 @@ def main() -> None:
),
expected_pool_size=int(config["export"]["live_pool_size"]),
max_age_days=args.contract_max_age_days,
require_artifact_manifest=True,
require_freshness=not bool(args.as_of_date or args.allow_stale),
)
logger.info(
"Release contract validated | version=%s | pool_size=%s | manifest_present=%s",
"Release contract validated | version=%s | pool_size=%s | release_manifest_present=%s | artifact_manifest_present=%s",
validation["version"],
validation["pool_size"],
validation["manifest_present"],
validation["artifact_manifest_present"],
)
logger.info("Export payload:\n%s", result["live_payload"])

Expand Down
7 changes: 7 additions & 0 deletions scripts/run_release_status_summary.py
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ def build_release_status_payload(
universe = load_json(root / "latest_universe.json")
live_pool = load_json(root / "live_pool.json")
manifest = load_optional_json(root / "release_manifest.json") or {}
artifact_manifest = load_optional_json(root / "artifact_manifest.json") or {}
ranking = pd.read_csv(root / "latest_ranking.csv")

validation = validate_release_outputs(
Expand All @@ -94,6 +95,7 @@ def build_release_status_payload(
expected_pool_size=live_pool.get("pool_size"),
max_age_days=max_age_days,
require_manifest=True,
require_artifact_manifest=True,
require_freshness=require_freshness,
)

Expand Down Expand Up @@ -134,6 +136,7 @@ def build_release_status_payload(
"latest_universe_symbol_count": len(list(universe.get("symbols", []))),
"latest_ranking_row_count": int(len(ranking)),
"latest_ranking_selected_count": int(selected_mask.sum()) if not selected_mask.empty else 0,
"artifact_contract_version": str(artifact_manifest.get("contract_version", "")).strip(),
"ranking_preview": ranking_preview_rows,
},
"publish_summary": {
Expand All @@ -147,6 +150,7 @@ def build_release_status_payload(
"validation": {
"ok": bool(validation.get("ok")),
"manifest_present": bool(validation.get("manifest_present")),
"artifact_manifest_present": bool(validation.get("artifact_manifest_present")),
"age_days": validation.get("age_days"),
"errors": list(validation.get("errors", [])),
"warnings": list(validation.get("warnings", [])),
Expand All @@ -156,6 +160,7 @@ def build_release_status_payload(
"latest_ranking": str(root / "latest_ranking.csv"),
"live_pool": str(root / "live_pool.json"),
"release_manifest": str(root / "release_manifest.json"),
"artifact_manifest": str(root / "artifact_manifest.json"),
},
}

Expand Down Expand Up @@ -190,6 +195,7 @@ def render_markdown(payload: dict[str, Any]) -> str:
- latest_universe symbol count: {artifact['latest_universe_symbol_count']}
- latest_ranking row count: {artifact['latest_ranking_row_count']}
- latest_ranking selected count: {artifact['latest_ranking_selected_count']}
- artifact contract version: {artifact['artifact_contract_version'] or 'n/a'}

### Ranking preview

Expand All @@ -207,6 +213,7 @@ def render_markdown(payload: dict[str, Any]) -> str:

- ok: {validation['ok']}
- manifest_present: {validation['manifest_present']}
- artifact_manifest_present: {validation['artifact_manifest_present']}
- age_days: {validation['age_days']}

### Errors
Expand Down
6 changes: 6 additions & 0 deletions scripts/validate_release_contract.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,11 @@ def parse_args() -> argparse.Namespace:
action="store_true",
help="Require release_manifest.json and validate it against the live pool contract.",
)
parser.add_argument(
"--require-artifact-manifest",
action="store_true",
help="Require artifact_manifest.json and validate the profile-aware artifact contract.",
)
parser.add_argument(
"--reference-date",
default=None,
Expand All @@ -55,6 +60,7 @@ def main() -> None:
reference_date=args.reference_date,
max_age_days=args.max_age_days,
require_manifest=args.require_manifest,
require_artifact_manifest=args.require_artifact_manifest,
require_freshness=not args.allow_stale,
)
print(json.dumps(validation, ensure_ascii=False, indent=2))
Expand Down
Loading