Order-level classifier: options for handling non-insect (out-of-distribution) crops

[mihow]

Summary

When the order-level classifier runs on detections from real trap imagery, it confidently assigns an insect order to crops that are not animals at all — smudges on the sheet, plant matter, debris, and similar artifacts. This is expected behaviour for a closed-set softmax model rather than a defect in the model or its training: the classifier has 16 classes and no "not an insect" option, so every crop it is handed must be assigned one of them, and neural networks are generally confident on inputs far outside their training distribution.

Opening this to collect options and decide whether it is worth addressing, and where.

Background — why this happens

The order classifier (convnext_tiny_in22k, 16 classes: 14 insect orders plus Araneae and Opiliones) is trained on GBIF occurrence images (the three Darwin Core Archive downloads listed in research/order_level_classifier/README.md). Those are photographs of identified insects: in focus, centred, and framed by a person who chose to photograph an insect. Detections from trap imagery include a much wider range of inputs, including things that are not organisms at all.

Two things worth noting so they are not mistaken for solutions:

• The moth / non-moth binary head does not cover this. The training loss is p·L_binary + (1−p)·L_order, where the binary term separates moths from non-moth insects. It does not distinguish an insect from a non-animal crop.
• Temperature scaling does not address it either. Following up on Explore cross-model confidence calibration for multi-model prediction selection #75, we fitted temperature for the order model (T = 1.403 on the GBIF val split, T = 1.165 on AMI-Traps; ECE improves in both cases — details in this comment). Temperature is a single global monotonic transform, so it rescales confident-and-correct and confident-but-meaningless predictions identically. Worth flagging too that our calibration evaluation used AMI-Traps crops filtered to those carrying a species-rank GBIF key — that is, confirmed insects — so it does not measure this out-of-distribution tail at all.

Also relevant: a new general-purpose insect detector is being trialled upstream, and it may produce fewer non-animal detections than the current one. That would reduce how often this arises, but probably not remove it, since any detector operating on trap imagery will pass through some non-animal crops.

Options

Listed roughly cheapest to most involved. Not mutually exclusive, and we do not have a preference yet.

1. Post-hoc out-of-distribution scoring on the existing checkpoint (no retraining). Max-logit, energy score, or Mahalanobis distance over penultimate features. Requires no change to the model or the category map, and can be evaluated immediately. This connects to the open-set recognition work in [P2] Train and evaluate open set recognition models for NENA macro species (GBIF) #25 / [P3] Evaluate open set recognition models for NENA macro species (Trap) #26 and the Mahalanobis tooling already in the repo, which were applied at species level for NENA rather than at order level.
2. Add a reject / background class. Fine-tune with negative crops (non-animal detections mined from trap data, which are plentiful). Likely the most effective option, but it needs labelled negatives and a training run, and it changes the checkpoint and category map, which is a downstream contract for consumers.
3. Filter upstream of the classifier. A dedicated blank/junk gate, or using the detector's own confidence, so obvious non-animal crops never reach the order model. Leaves the order classifier untouched at the cost of an extra stage.
4. Apply the open-set recognition approaches from [P2] Train and evaluate open set recognition models for NENA macro species (GBIF) #25 / [P3] Evaluate open set recognition models for NENA macro species (Trap) #26 at order level, rather than only at species level.
5. Handle it downstream only (threshold or flag in the consuming application). Included for completeness, but it looks weak on the evidence: neither the softmax score nor normalised entropy separates these crops, so there is not much signal to threshold on.

Suggested first step

Measure the size of the problem before choosing, since as far as we can tell it has not been quantified. Concretely: assemble an in-distribution set (labelled insect crops) and an out-of-distribution set (non-animal crops from real trap imagery), then compare the order model's score distributions across the two and test whether a post-hoc score (max-logit, energy, Mahalanobis) separates them better than softmax does. If one of them separates cleanly, option 1 becomes a cheap improvement with no retraining.

Happy to run that and post the results here. Comments and corrections very welcome — particularly if any of this has already been looked at and we have missed it.

Refs: #75 (confidence calibration, incl. the order-model temperature results), #60, #25, #26.

[Whatsonyourmind]

Following the #75 thread through to here — the "why this happens" framing is exactly right, and the instinct to measure it before touching the checkpoint is the correct call. A few things I'd add to make that measurement decisive rather than just descriptive, plus a view on how to order the options.

Make the "first step" a detection benchmark, not just a histogram comparison. Once you have the ID set (labeled insect crops) and the OOD set (non-animal trap crops), score each crop with several candidates and report AUROC + FPR@95%TPR for each — those two numbers turn "do the distributions separate" into a decision you can act on. Concretely, cheapest first:

• MSP (max softmax prob) — the baseline you already know fails; include it so the improvement is quantified against something.
• Max-logit and energy (−T·logsumexp(logits/T)). Energy is the one I'd bet on here: it keeps the absolute magnitude that softmax normalizes away, which is exactly the signal a confident-but-meaningless crop destroys. And it's essentially free — you can reuse the temperature you already fit (T = 1.403 GBIF-val / 1.165 AMI-Traps) directly in the expression, so it costs nothing beyond a forward pass you're already running.
• Mahalanobis over penultimate features (per-class Gaussians fit on GBIF-val, taken from the shared trunk before the binary/order split). More setup, but on far-OOD like smudges and plant matter it's usually the strongest separator — and since you note the Mahalanobis tooling from the [P2] Train and evaluate open set recognition models for NENA macro species (GBIF) #25/[P3] Evaluate open set recognition models for NENA macro species (Trap) #26 NENA work is already in the repo, this is largely re-pointing it at the order layer rather than new code.

If energy or Mahalanobis clears something like AUROC > 0.9 at a usable FPR95, option 1 is a genuine gate and you can ship it with no retraining.

Options 1 and 2 are a pipeline, not a fork. The cheapest durable path is to use option 1 to bootstrap option 2: run the OOD score over unlabeled trap detections, take the highest-scoring (most confidently non-animal) crops, and that's your negative-mining pool for the reject-class fine-tune — near-zero labeling cost because the score does the pre-filtering, and you're mining exactly the hard negatives that matter. So I'd sequence it as (1) now → quantify and ship as a gate, (2) later → reject class trained on the negatives mined by (1), with (3) the new upstream detector reducing volume into both. Option 5 is correctly weak: no in-distribution-calibrated softmax transform can recover a signal it never saw.

One honest caveat on where conformal fits, since I offered APS/RAPS in #75 and don't want to oversell it for this ticket: split-conformal set-size is the right tool for the in-distribution "which model is more certain / should I abstain" axis (coverage-calibrated, cross-model comparable), but it is not an OOD detector. The coverage guarantee only holds over the calibration distribution, so on genuine non-animal inputs there's no guarantee at all — set-size does tend to inflate off-manifold, but that's a side effect, not what CP is designed to certify. So I'd keep the two in separate lanes: a dedicated OOD score (energy / Mahalanobis) for the non-animal tail this issue is about, and APS/RAPS set-size behind it for the in-distribution-but-uncertain crops and the cross-model selection from #75. They compose cleanly — gate first, size behind.

Happy to write the evaluation harness for the first step (score distributions + AUROC/FPR95 for MSP / max-logit / energy / Mahalanobis, reusing the calibration setup from #75) if you can point me at a sample of the non-animal crops alongside the ID labeled set — that sample is the one input I don't have. I'll fold the APS/RAPS set-size in on the same pass so both axes come out of one run.