Speed up dot plots: integrate preprocessed data on frontend (SCP-5992)

[eweitz]

This dramatically speeds up dot plots for datasets where the preprocessing pipeline has run.

Previously, dot plots would load much more data than they needed from the backend. Dot plots only display two metrics: scaled mean expression and percent of cells expressing for a given annotation label. They don't need each cell's expression data, but they would load it.

Now, dot plots can use preprocessed data that only provides those two summary metrics. This makes dot plots much faster, and also (in principle) enables them to scale to many more genes than the current 50-gene cutoff.

The technical implementation adds a monkey patch to the underlying JavaScript of Morpheus. It also only enables this behind a feature flag, as at least two known issues need to be resolved before this is made generally available:

• Heatmaps need to request non-preprocessed data when that tab is clicked
• Genes (rows) and labels (columns) need to use proper ordering

Demo video:

Fast_dot_plots_frontend__SCP__2025-11-24.mov

[codecov]

Codecov Report

❌ Patch coverage is 65.95745% with 32 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.61%. Comparing base (299abd5) to head (57c82c0).
⚠️ Report is 9 commits behind head on development.

Files with missing lines	Patch %	Lines
app/javascript/lib/dot-plot-precompute-patch.js	71.64%	16 Missing and 3 partials ⚠️
...pp/javascript/components/visualization/DotPlot.jsx	47.05%	9 Missing ⚠️
app/javascript/lib/scp-api.jsx	0.00%	4 Missing ⚠️

Additional details and impacted files

@@               Coverage Diff               @@
##           development    #2324      +/-   ##
===============================================
- Coverage        72.62%   72.61%   -0.01%     
===============================================
  Files              333      335       +2     
  Lines            27745    27834      +89     
  Branches          2651     2565      -86     
===============================================
+ Hits             20151    20213      +62     
- Misses            7453     7477      +24     
- Partials           141      144       +3     

Files with missing lines	Coverage Δ
.../js/visualization/dot-plot-precompute.test-data.js	100.00% <100.00%> (ø)
app/javascript/lib/scp-api.jsx	44.20% <0.00%> (-0.22%)	⬇️
...pp/javascript/components/visualization/DotPlot.jsx	23.29% <47.05%> (+1.07%)	⬆️
app/javascript/lib/dot-plot-precompute-patch.js	71.64% <71.64%> (ø)

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[bistline]

I'm very glad to see this didn't look to terribly complicated in the end, though I'm sure it took quite a while to figure out due to the lack of documentation. Very nice work! The speedup locally is incredible.

[bistline]

This shouldn't ever be the case - all records are destroyed on study deletion, though clearly at some point something went sideways locally and you ended up with orphaned data. I did a quick check on staging & production and we don't have any of these (good), but leaving this check in place is harmless.