interpretable-rn-analyzer

interpretable-rn-analyzer is an interpretable MusicXML to RomanText analyzer. It was built as a research alternative to opaque end-to-end harmony models: every prediction is produced from explicit score features, shallow decision trees, count-based grammar tables, and a semi-Markov Viterbi decoder.

The current best checked-in decoder profile is configs/decoder/balanced_short_spans.json. In the latest local comparison it improved the interpretable system, but it still trails pretrained AugmentedNet; see docs/research_status.md. The repository is meant to be a clear starting point for further interpretable MIR and music-theory work, not a claim that this approach is already state of the art.

MusicXML
-> strip embedded harmony annotations
-> parse note, timing, beat, bass, and pitch-class evidence
-> build a candidate harmonic-boundary lattice
-> generate explicit local-key candidates
-> generate explicit Roman-numeral candidates
-> score boundaries, keys, and Roman numerals with shallow trees
-> add count-based grammar, key-sequence, duration, pitch, and bass terms
-> decode the globally best semi-Markov path
-> write WiR-style RomanText

The main analyzer uses no neural networks, embeddings, random forests, gradient boosting, XGBoost, LightGBM, or opaque ensembles.

Installation

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,parquet]"

music21, pandas, numpy, and scikit-learn are the core dependencies. The parquet extra installs pyarrow, which is recommended for the full training pipeline.

Repository Layout

src/interpretable_rn_analyzer/
  data/          WiR discovery, RomanText parsing, score slicing, augmentation
  features/      transparent boundary, key, RN, and segment features
  theory/        Roman numeral helpers, candidate generation, RomanText writer
  models/        shallow-tree training, grammar training, semi-Markov decoding
  evaluation/    reference-agreement metrics and Roman Umpire consistency
  baselines/     optional AugmentedNet inference wrapper

experiments/
  augmentednet_comparison/  reproducible train/predict/evaluate scripts

configs/
  decoder/       named decoder parameter profiles

docs/
  algorithm.md        detailed algorithm walk-through linked to code
  api_reference.md    public function and module reference
  research_status.md  current results, limitations, and improvement roadmap

examples/
  when_in_rome/  small MusicXML examples copied from When-in-Rome

tests/
  unit tests for parsing, features, decoding, augmentation, and metrics

Generated data, trained models, predictions, and evaluation outputs are written under data/ and outputs/ by default. They are intentionally ignored by Git.

Data Policy

Training and evaluation references must be human When-in-Rome analysis.txt files only. Generated analyses such as analysis_AugmentedNet_*.txt, analysis_automatic.rntxt, or this tool's analysis_predicted.txt outputs are not gold labels.

AugmentedNet support is only a comparison baseline. ChordGNN is intentionally not part of the active workflow because the original model is not available in a directly reproducible form.

Original files under external/When-in-Rome/ should not be modified. Scripts write sanitized score copies under data/sanitized_scores/ when an external system needs harmony-free MusicXML.

Quick Prediction

You need trained models under outputs/models/semimarkov/. After training, run one of the bundled examples:

interpretable-rn-analyzer \
  --score examples/when_in_rome/clara_schumann_liebst_du_um_schoenheit/score.mxl \
  --model-dir outputs/models/semimarkov \
  --decoder-params configs/decoder/balanced_short_spans.json \
  --out-analysis outputs/examples/clara_schumann_analysis_predicted.txt

Equivalent module form:

PYTHONPATH=src python -m interpretable_rn_analyzer.models.predict_semimarkov \
  --score examples/when_in_rome/clara_schumann_liebst_du_um_schoenheit/score.mxl \
  --model-dir outputs/models/semimarkov \
  --decoder-params configs/decoder/balanced_short_spans.json \
  --out-analysis outputs/examples/clara_schumann_analysis_predicted.txt

If --out-analysis is omitted, the RomanText is printed to stdout.

Full Interpretable Pipeline

Prepare When-in-Rome data:

experiments/augmentednet_comparison/00_prepare_data.sh

Train the interpretable system. Train-set transposition augmentation is enabled by default, and decoder tuning compares the practical smoke profiles unless you restrict it:

experiments/augmentednet_comparison/01_train_interpretable.sh

For the public best-known three-profile comparison:

DECODER_TUNE_MODE=smoke \
DECODER_TUNE_PROFILES=baseline_segmentation,recall_low_threshold,balanced_short_spans \
experiments/augmentednet_comparison/01_train_interpretable.sh

Predict the held-out test split:

experiments/augmentednet_comparison/02_predict_interpretable.sh

Evaluate:

experiments/augmentednet_comparison/05_evaluate_all.sh

Main generated outputs:

outputs/models/semimarkov/
outputs/predictions/semimarkov_test.csv
outputs/romantext/semimarkov_test/
outputs/evaluation/system_comparison/report.md
outputs/evaluation/system_comparison/system_metrics.csv

AugmentedNet Comparison

AugmentedNet is optional and used only as a pretrained baseline on the same held-out test split.

experiments/augmentednet_comparison/setup_augmentednet_env.sh
export AUGMENTEDNET_PYTHON="conda run -n augmentednet-cpu python"
experiments/augmentednet_comparison/03_predict_augmentednet_pretrained.sh
experiments/augmentednet_comparison/05_evaluate_all.sh

This is a same-test-split pretrained comparison, not a retraining of AugmentedNet on the local train split.

Useful CLIs

interpretable-rn-analyzer --help
interpretable-rn-predict --help
interpretable-rn-tune-semimarkov --help
interpretable-rn-augment-transpositions --help
interpretable-rn-compare --help
interpretable-rn-roman-umpire --help

Module equivalents:

PYTHONPATH=src python -m interpretable_rn_analyzer.data.find_wir_pairs --help
PYTHONPATH=src python -m interpretable_rn_analyzer.data.parse_romantext --help
PYTHONPATH=src python -m interpretable_rn_analyzer.data.score_slices --help
PYTHONPATH=src python -m interpretable_rn_analyzer.data.build_dataset --help
PYTHONPATH=src python -m interpretable_rn_analyzer.models.train_candidate_tree --help
PYTHONPATH=src python -m interpretable_rn_analyzer.models.train_key_tree --help
PYTHONPATH=src python -m interpretable_rn_analyzer.models.train_boundary_tree --help
PYTHONPATH=src python -m interpretable_rn_analyzer.models.harmonic_grammar --help
PYTHONPATH=src python -m interpretable_rn_analyzer.models.tune_semimarkov --help
PYTHONPATH=src python -m interpretable_rn_analyzer.models.predict_semimarkov --help
PYTHONPATH=src python -m interpretable_rn_analyzer.evaluation.compare_systems --help

Model Artifacts

Each trained shallow-tree directory contains:

model.joblib
feature_columns.json
tree_rules.txt
feature_importances.csv
validation_metrics.json
grid_search_results.csv
parameter_test_report.md

The count-based grammar is saved as outputs/models/semimarkov/harmonic_grammar.json. Decoder tuning writes decoder_params.json, decoder_tuning/all_trials.csv, and a markdown report under the model directory.

Documentation

Start with:

• docs/algorithm.md, for the full model explanation.
• docs/api_reference.md, for the function and module map.
• docs/research_status.md, for current quality and next steps.
• CONTRIBUTING.md, for development and data-policy rules.
• THIRD_PARTY_NOTICES.md, for example-score attribution.

License

Unless otherwise noted, this repository's source code and documentation are licensed under the Apache License 2.0.

The bundled files under examples/when_in_rome/ were copied from When-in-Rome and remain under that corpus' CC BY-SA 4.0 terms; see THIRD_PARTY_NOTICES.md.

Tests

PYTHONPATH=src python -m compileall -q src tests
PYTHONPATH=src pytest -q

The GitHub Actions workflow in .github/workflows/ci.yml runs the same checks.

Known Limitations

• The current interpretable system is substantially behind pretrained AugmentedNet on full-label reference agreement.
• Local-key prediction remains the largest bottleneck.
• Boundary recall improved with short-span profiles, but over/under-segmentation still changes by repertoire.
• Roman Umpire consistency metrics depend on music21 parsing and are useful as consistency checks, not as absolute truth.
• WiR references are human analyses and may contain legitimate analytical choices. Reports should say "human reference" or "reference agreement", not "ground truth".