`match_features` runs on a threading backend — the GIL caps pair matching at ~2 effective cores regardless of `processes`

[haraldschilly]

Environment

• ODM 3.6.2, official docker image (opendronemap/odm@sha256:a96f56db…), joblib 1.5.3
• 16-core Intel (Panther Lake), 30 GB RAM, CPU-only
• 704 images (4K video frames), --feature-quality ultra, --min-num-features 20000,
  HAHOG + WORDS (BOW) matcher, no/partial GPS → 247,456 candidate pairs

Observed behavior

match_features logs Computing pair matching with 12 processes, but host CPU
utilization stays at ~2 cores. Measured throughput: 362 pairs/min
(≈ 11.4 h for this dataset's matching phase alone).

Cause

opensfm/context.py::parallel_map runs everything through joblib's
threading backend:

def parallel_map(func, args, num_proc, max_batch_size=1, backend="threading"):
    ...
    with parallel_backend(backend, n_jobs=num_proc):

The backend parameter already exists, but the call site in
opensfm/matching.py::match_images_with_pairs never passes it:

matches = context.parallel_map(match_unwrap_args, args, processes, jobs_per_process)

So processes: N is actually N threads serialized on the Python GIL. The
C++ parts (FLANN/OpenCV) release the GIL only partially; the Python glue
(feature loading/decompression, candidate filtering) does not scale.

Experiment: one-line change, measured results

Patch (env-switchable, default unchanged):

matches = context.parallel_map(
    match_unwrap_args, args, processes, jobs_per_process,
    backend=os.environ.get("ODM_MATCHING_BACKEND", "threading"))

Results on the dataset above (container memory limit 18 GiB):

backend	workers	pairs/min	outcome
threading	"12"	362	stable; ~2 effective cores
loky	12	—	workers OOM-SIGKILLed after ~4 min
loky	6	—	OOM after ~90 s
loky	4	570	ran at the memory limit (reclaim thrash, per-worker rate degraded), OOM after ~10 min
loky	2	508	stable at ~11 GiB total; 1.4× threading

Key numbers: one loky worker sustains ~256 pairs/min — i.e. 12 threads
together deliver about 1.4 worker-equivalents. Per worker, real processes
are ~2.5× faster than the threading pool's aggregate.

Why processes alone don't scale further

With fork/spawn workers nothing is shared: each worker loads its own copies of
feature .npz files and builds its own FLANN indexes, so memory scales
linearly with workers (~5 GiB/worker on ultra features here) and >2–3 workers
exceed a 30 GB machine.

Suggestions (in increasing order of impact)

1. Expose the backend at the match_features call site (config option or
   env var), documenting the memory trade-off. One-line change; on machines
   with enough RAM this alone is a big win.
2. Share the feature store between workers (e.g. memory-mapped arrays /
   joblib memmapping instead of per-worker npz loading).
3. Cache FLANN indexes per image (bounded LRU) instead of rebuilding per
   pair — helps both backends.

With (2)+(3), 12 real workers would fit comfortably in RAM; extrapolating the
per-worker rate, matching for this dataset would drop from ~11 h to roughly
1–1.5 h.