Auto streamtrees - #2080
Draft
trvrb wants to merge 31 commits into
Draft
Conversation
The previous interface was slightly cumbersome - I often found myself wanting to source both datasets & narratives from some directory X and having to type in the rather long `--datasetDir X --narrativeDir X`, and often I'd misspell / pluralise the argument names; using a simple positional argument `X` is much nicer. The ability to use multiple directories is also functionality I've long wanted. The previous CLI interface is preserved in its entirety for backwards compatibility and is mutually exclusive with the new positional-args style.
Stands up the default getDataset handler on an ephemeral port and makes real HTTP requests against the JSONs in test/fetched-jsons, covering both the main (v2) dataset and the tip-frequencies sidecar. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Convert all server-side code (cli/, auspice.js, index.js, src/version.js) and the test file from CommonJS require/module.exports syntax to ES module import/export syntax. This also addresses the downstream effects on webpack bundling for the frontend code in src/. package.json: Added "type": "module" to declare the package as ESM. This makes Node treat all .js files as ES modules by default. Replaced the "main" field with "exports" to use the modern package entry point specification. Added babel-plugin-transform-import-meta as a dev dependency for Jest compatibility. Server code (cli/, auspice.js, index.js, src/version.js): Converted all require() calls to import statements and all module.exports/exports assignments to named or default exports. For packages that use CommonJS internally (argparse, webpack config), createRequire(import.meta.url) is used to obtain a require function that can load CJS modules from an ESM context. __dirname shim: Since __dirname is not available in ES modules, files that reference it (cli/utils.js, cli/build.js, cli/develop.js, cli/view.js) now derive it via path.dirname(fileURLToPath(import.meta.url)). The test file also uses this pattern. Dynamic require(handlersPath) in cli/view.js: The user-provided custom handlers path was previously loaded with require(). This is now await import(handlersPath), which required making customRouteHandlers() and run() async functions. Config file renames: webpack.config.js and babel.config.js are renamed to .cjs since they use CommonJS syntax (module.exports) and would otherwise be treated as ESM due to the package-level "type": "module" setting. Babel transform for Jest: Jest runs tests through babel-jest which transforms ESM to CJS. However, babel leaves import.meta.url untouched, causing a runtime error when Jest executes the transformed code. Added babel-plugin-transform-import-meta (test env only) to convert import.meta.url to its CJS equivalent during testing. The test/server-fetch.test.js file itself was also converted to ESM syntax. webpack.config.cjs updates: Added resolve.fullySpecified: false (both top-level and as a per-module rule) to allow extensionless imports in the bundled frontend code -- without this, webpack enforces mandatory file extensions for all imports in "type": "module" packages. The JS/TS rule also sets type: "javascript/auto" so webpack parses source files in mixed mode, recognizing both require() and import syntax. module.hot to import.meta.webpackHot: In src/root.js and src/store.ts, the webpack HMR API was accessed via module.hot which is a CJS-only global. Replaced with import.meta.webpackHot, the ESM equivalent. The HMR accept callback in store.ts now uses dynamic import() instead of require() to reload reducers. Frontend require() calls for assets: Several React components used require() to load images (SVG, PNG, JPG). These are converted to static import statements at the top of each file, which webpack asset/resource loader handles identically. Affected files: navBar/content.js, splash/splash.js, framework/fine-print.js, framework/footer-descriptions.js, info/byline.js. Dynamic extension loading (src/util/extensions.ts): The extensions.ts file retains its synchronous require(`@extensions/...`) pattern. This works because webpack's javascript/auto parser mode (set via the module rule) handles require() at build time even in a "type": "module" package. An earlier approach using top-level await with dynamic import() caused async module semantics to cascade through the dependency graph (via util/globals.js), which broke component rendering by wrapping module exports in async module namespace objects. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Convert 8 cli files from .js to .ts, leveraging Node 24's native type stripping to run them directly without a build step. tsc is used for type-checking only (noEmit). Files converted: - cli/utils.ts - cli/server/getDataset.ts - cli/server/getDatasetHelpers.ts - cli/server/getAvailable.ts - cli/server/getNarrative.ts - cli/server/processPaths.ts - cli/view.ts - cli/develop.ts - cli/build.ts Supporting changes: - Add cli/tsconfig.json extending root with nodenext module resolution and allowImportingTsExtensions for Node-native .ts imports - Add allowImportingTsExtensions to root tsconfig.json (needed because cli .ts files are pulled in transitively via test imports) - Exclude cli/**/* from root tsconfig include/add to exclude (prevents checking cli files under the wrong moduleResolution) - Update package.json type-check script to run both tsconfigs - Update all import paths across 11 files (.js -> .ts extensions) - Update webpack.config.cjs require for cli/utils.ts - Add eslint-disable for explicit-function-return-type and consistent-type-assertions in converted files (incremental migration) - Apply chalk 2.x workaround (type assertion) for nodenext compat - Fix TypeScript strict errors: unknown in catch blocks, Object.entries casting for Set methods, undefined-in-object check Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The docs and code had fallen out of sync over the years.
* `parseNarrativeFile` was not only deprecated as the docs indicated
("This function is deprecated as of vXXX"), it no longer existed in the codebase.
* The docs for the getNarrative route incorrectly indicated the `type` query as optional
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Triggered by existing warnings: WARNING: You are currently running a version of TypeScript which is not officially supported by @typescript-eslint/typescript-estree.
Errors were new since version bump in parent commit
This is an old and now unused test that's been fully superseded by the Playwright smoke tests. It was the only test that used "chai" (and "mocha", although that wasn't listed as a dep). I only noticed this as the eslint version bump flagged up an error in the test file
This functionality never worked as well as hoped. Bigger picture, this represents a move away from pre-provisioned datasets / narratives and towards "bring your own data", which is the main use case after all!
(This was from an ancient method of deploying auspice)
to be compatible in a world without the main 'data/' directory. This made me realise our "smoke-tests" are basically a single test! Nonetheless it's worth keeping around the architecture for future improvements
This is a relic of earlier times where it was kept in-sync with the canonical nextstrain datasets. We maintain similar functionality in the fetch-test-data script but for testing purposes only. This is a key part of the bigger re-design which moves auspice to a visualisation program/library which you point at your own data.
It has previously been useful to run Auspice as a stand-alone Heroku (review-) app, but this has been superseded by standing up a testing nextstrain.org review app as the range of datasets/narratives available is much greater, or a auspice.us app for datasets not available via nextstrain.org. It's simpler (and cheaper!) to drop the Auspice app entirely
Slightly out of date!!!
[v3] [cli] Allow one or more positional path args
[v3] Various JS/CS CJS/ESM updates
[v3] Default data changes
Automatically partition a large tree into streamtrees by a tip-count threshold (AUTO_STREAM_TIP_THRESHOLD) as an alternative to author-defined branch labels, so any big tree can be aggregated for performance/legibility. Streams render "in place": each stream start is positioned at the midpoint of its member-tip display-order band during setDisplayOrder (not overridden afterwards), so the surrounding tree doesn't move when streams toggle, disjoint clades can't overlap, and the connecting tees reach each stream. This reworks the stream display-order layout, which previously collapsed streams into KDE-derived heights. Also adds a "Show stream labels" toggle (default off), since the synthetic auto-partition names are visual noise, and fixes a redux state-mutation error that prevented re-enabling streamtrees after toggling them off. On the 35k-tip spike-sm tree this collapses ~134k SVG elements to ~6k. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Colour the stream connector (joiner + backbone) by the stream's dominant colour category (most member tips) instead of a fixed gray, so thin streams show their colour rather than a gray line. - In streamtree mode render all branches, tees and stream connectors at a uniform thickness (branchStrokeWidth); per-node branch thickness is redundant/confusing when streams already encode count via their height. Normal (non-stream) mode is unchanged. - Tune defaults: branchStrokeWidth 1.5, auto-partition tip threshold 200. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Fix zoom/filter throwing in dev by exempting tree.streams from the redux immutability check (processStreams mutates it in place, like tree.nodes which is already exempt) — clicking a ripple/backbone/branch to zoom now works. - Branch thickness in streamtree mode now matches the stream backbone width for any branch with visible descendants (tipCount > 0), while fully-filtered branches (no visible descendants) stay thin. Fixes date-filtered trunk branches rendering thinner than the backbone they connect to, and keeps clade-filtered dead-ends thin. - End the stream backbone at the farthest-right visible tip (streamVisibleMax) rather than the full unfiltered extent, so date filtering retracts the right side; the left side still extends back to connect with the parent lineage. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
drawStreams used `.transition("500")`, where "500" is a d3 transition *name*, not
a duration — so stream ripples/connectors/labels animated at d3's default 250ms
while branches used the real 500ms transitionTime, leaving stream transitions
visibly out of sync (e.g. when toggling divergence <-> time). Thread the actual
transitionTime through drawStreams and use `.transition().duration(transitionTime)`
so streams and branches share one duration.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace the absolute AUTO_STREAM_TIP_THRESHOLD (which needed per-tree tuning) with AUTO_STREAM_TARGET_COUNT: autoPartitionStreams now greedily splits the largest clade (by fullTipCount) until it reaches ~N streams, so a single N gives consistent on-screen stream density regardless of tree size (verified ~consistent counts on 1.5k-tip ebola and 35k-tip spike-sm). Same output contract, so downstream stream processing/rendering is unchanged. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Auto-streamtrees previously computed a static partition (by fullTipCount) once at load/toggle/label-change, so zooming or filtering showed a subset of a fixed partition rather than ~N streams of whatever is on screen. Now the partition is re-derived on discrete on-screen-tip changes (zoom, clade/trait filter, date settle) from the visible tipCount, always splitting the visible tips into ~AUTO_STREAM_TARGET_COUNT streams — coarse when zoomed out, finer when zoomed/filtered in. Continuous events (date-slider drag, animation frames) keep the cheap in-place processStreams via the quickdraw gate, so drags/playback stay smooth. Streams render in place (xy-identical), so re-granularizing subdivides within existing bands without moving the tree. Also guards two stale-hovered-node crashes surfaced by re-partitioning: the clear pass now only resets stream membership (streamName/inStream) and keeps render data so live ripple hover/leave handlers don't deref deleted streamCategories; and the hover tooltip bails when the hovered node is no longer a stream in the current map (stale streamName key). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Performance (treeStreams.ts): speed up the per-re-partition KDE without changing output (verified sub-pixel-identical ripple paths on ebola/ncov). - Replace the O(n^2) `categoryInfo.nodes.includes(...)` per-node filter with a Set lookup, and bucket observedCategories' node indices in one pass rather than re-filtering the full list per category. - Hoist the per-stream `new Set(visibility)` to one `everythingVisible` compute. - Fix the skipPivots guard: it tested `Object.hasOwn(s, 'streamPivots')` on the StreamSummary, but pivots live on the start node, so the pivot block always re-ran; key it off `nodes[s.startNode]` so the in-place drag path skips it. - 3-sigma cutoff on the KDE inner loop: evaluate each tip's gaussian only over the pivot window within +/-3 sigma (evenly-spaced grid), turning it from O(tips x allPivots) into O(tips x ~30). Tail truncation is <0.3% of peak. - Add perf.js timers around processStreams / autoPartitionStreams. Transitions: animate stream updates in place so a coarse<->fine re-partition cross-dissolves instead of hard-swapping. - change.ts: give streams their own transition budget (streamTransitionTime), suppressed only by the rapid-render guard (keeps drags/playback snappy), not by skipTreeAnimation -- so streams animate even on trees too large to animate tip-by-tip. - renderers.ts: fade entering stream groups in (exiting groups already fade out). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
On a re-partition, a splitting stream now animates the coarse parent's stacked area breaking apart into its finer child streams (a true area-morph) instead of cross-fading. Each entering child ripple starts as its display-order slice of the parent's already-rendered pixel band and tweens to its final shape; because children bands tile the parent band (tip displayOrder is partition-invariant), the slices refill the parent's silhouette at t=0. Mechanics: snapshot the old streams + rendered shapes in reactD3Interface/ change.ts before they're overwritten, thread through change() to drawStreams, which classifies each new stream against the old partition (split / merge / 1:1 / none) and injects a d3 attrTween at the ripple enter for splits. Streams get their own transition budget so this animates even on large trees. Merges (zoom-out / coarsening) and unclassifiable cases fall back to the existing cross-fade; NaN-guarded and a no-op when no snapshot is present. New module streamMorph.ts: snapshot, old<->new correspondence, split t=0 carve, pivot-value resampling, point interpolation. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The auto-partition target stream count was a hardcoded constant (AUTO_STREAM_TARGET_COUNT = 100). Make it a runtime controls field (streamTreeTargetCount) driven by a new sidebar toggle modeled on the animation SLOW/MEDIUM/FAST buttons, showing FINE / MEDIUM / COARSE (150 / 100 / 50). Changing it ships a fresh streams map, so the coarse<->fine change flows through the existing split/merge area-morph. - treeStreams.ts: AUTO_STREAM_TARGET_COUNTS presets (constant kept as the MEDIUM default); autoPartitionStreams already took the count as a param. - controls reducer: streamTreeTargetCount field + default + reducer case. - new CHANGE_STREAM_TREE_TARGET_COUNT action + changeStreamTreeTargetCount thunk (re-partitions over the current view; auto-only); tree reducer stores the fresh streams map via a fallthrough case. - thread controls.streamTreeTargetCount into all three autoPartitionStreams call sites (initial load, branch-label switch, zoom/filter re-partition), so every re-partition honors the chosen granularity. - choose-stream-trees.tsx: three SidebarButtons, shown only in auto mode. - URL sync: ?streamTarget= written on change (default omitted) and validated/restored on load, mirroring ?streamLabel. - en sidebar i18n: Fine / Coarse / Streamtree granularity. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Completes the streamtree area-morph. On a coarsening re-partition (zoom-out, FINE->COARSE, any merge), the fine child streams now visibly converge into the coarse parent rather than cross-fading. Since one SVG path per (stream, category) can't depict K children in the entering parent, we morph the exiting child paths (separate DOM elements) toward their slice of the new parent, while the new parent fades in at its final shape via the existing enter path. This is the reverse of the split carve, so it reuses that geometry: the split carve is extracted into a shared carveSlice helper, and merge calls it with the roles swapped (new parent envelope/band as the target, each child's old shape as the start). The snapshot now records the ripple key so merge targets match the exiting DOM ripples; computeEnvelope/computeBand are extracted for reuse. prepareStreamMorph fills in the merge branch (mergeExit + mergingOldNames), and drawStreams' streamGroup exit morphs the merging children (attrTween to target, then fade + remove); areaGenerator is hoisted so the exit callback can use it. Where the new parent is invisible (e.g. the coarsened outside on a zoom-in) the merge falls back to the existing cross-fade. Split behavior is unchanged; NaN-guarded and a no-op when no snapshot / transitionTime 0. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a sidebar toggle (under "Show stream labels") that switches the streamtree layout between the current render-in-place layout (xy-concordant with the rect tree; the default, off) and the original "v3" layout (on), where each stream gets its own KDE-sized display-order band and the tree re-flows. This makes the two strategies easy to compare live. The v3 algorithm is restored verbatim from git (commit 82dfcda, i.e. the parent of the render-in-place rework): StreamInfo / traverseConnectedStreams / setDisplayOrdersForStream, and the global weightToDisplayOrderScaleFactor path (previously dead) becomes live again in v3 mode. setDisplayOrderRecursively now threads a `false | "inPlace" | StreamInfo` mode, and setRippleDisplayOrders branches between the global scale (v3) and the per-stream band-fit scale (in-place); the ripple-centering loop is shared. New controls field streamTreeUpdateLayout (default false) + toggle action, mirroring showStreamTreeLabels through the controls -> prop -> phylotree.params plumbing. Toggling routes through the existing `updateLayout` change flag (NOT streamDefinitionChange), so it re-runs setDisplayOrder + mapToScreen on the smooth modifySVG path; the partition is unchanged, so ripples morph via the keyed update transition and branches/tips glide. Session-only (no URL sync), mirroring the stream-labels toggle. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
jameshadfield
force-pushed
the
v3
branch
2 times, most recently
from
July 13, 2026 02:16
fb4fe78 to
4d9e17a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is currently an experimental prototype branch. I'm only doing a draft PR to get a Heroku review app up.