Per-sample modality contribution with ViLT

This repository uses ViLT for per-sample modality-contribution analysis. It contains an inference-verification stage and a contribution-estimation stage.

Checkpoints

Benchmark	Checkpoint	Task
VQAv2	dandelin/vilt-b32-finetuned-vqa	visual question answering
NLVR2	dandelin/vilt-b32-finetuned-nlvr2	two-image binary reasoning
COCO retrieval	dandelin/vilt-b32-finetuned-coco	image-text matching

Stage 1: inference and internal-state proof

Stage 1 verifies preprocessing, prediction, hidden states, attention tensors, and deterministic inference.

uv sync
uv run python main.py --stage 1 --benchmark vqav2

Stage 2: per-sample modality contribution

Stage 2 estimates image, text, and interaction contribution for each sample using two complementary families of evidence:

1. Four-condition ablation with inclusion–exclusion

   • f(I,T): full sample
   • f(I,T0): original image and masked text
   • f(I0,T): black image and original text
   • f(I0,T0): black image and masked text

   The original predicted output remains the target in all four runs. The decomposition is:

   image       = f(I,T0) - f(I0,T0)
   text        = f(I0,T) - f(I0,T0)
   interaction = f(I,T) - f(I,T0) - f(I0,T) + f(I0,T0)
   

2. Gradient × activation

   • text-token importance
   • image-patch importance
   • modality gradient norms

Run a single benchmark demo:

uv run python main.py --stage 2 --benchmark vqav2
uv run python main.py --stage 2 --benchmark nlvr2
uv run python main.py --stage 2 --benchmark coco_retrieval

Data

src.data owns dataset loading and conversion. It streams validation data from VQAv2, NLVR2, and COCO Captions and converts every source row to the same CanonicalSample contract:

sample_id, benchmark, images, text, target, metadata

Use load_benchmark_samples(..., count=..., seed=...) for a deterministic benchmark sample. Dataset records are not hardcoded in the CLI or notebook.

Outputs

Stage 2 writes:

• a CSV containing one row per sample and scalar features suitable for pandas, statistics, and later forgetting prediction;
• a compressed NPZ file containing CLS embeddings and variable-length token/patch attribution arrays indexed by sample ID;
• optional figures generated by src.analysis.visualization.

Important columns include:

confidence, margin, entropy,
image_contribution, text_contribution, interaction_contribution,
image_share, text_share, interaction_share,
image_gradient_norm, text_gradient_norm,
image_gradient_x_input, text_gradient_x_input,
top_patch_index, top_token

Notebook

Open notebooks/modality_contribution.ipynb. The committed notebook is executed with 50 validation samples per benchmark (150 total) and walks through:

1. loading all three task-specific checkpoints;
2. making and validating predictions;
3. estimating per-sample contribution;
4. building one combined DataFrame;
5. saving CSV and NPZ artifacts;
6. plotting contribution distributions, image-vs-text scatter, benchmark means, correlations, token importance, and patch importance;
7. preparing CLS embeddings for PCA once multiple samples are available.