Docs: recipe for reshaping columns to/from JSON arrays ("STAN format") - #2216
Merged
Conversation
Adds a "Columns as JSON arrays" section to shapes-of-data.md covering the reshape-to-column-arrays and reshape-back techniques from issue 392 (e.g. for feeding data to Stan), using arrayify()/emit1 to produce real JSON arrays rather than a dedicated file format. Closes #392.
johnkerl
marked this pull request as ready for review
July 16, 2026 23:33
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a "Columns as JSON arrays" section to
shapes-of-data.md, documenting the mkstan/unstan recipe from #392 as a doc-only solution rather than a new--istan/--ostanfile format.Why doc-only, not a new format: #392 is a 2020 issue with a single external commenter and no activity since 2022. The "STAN format" isn't a distinct format — it's a JSON object whose values are arrays, one per column (useful for feeding the Stan modeling tool), and the reshape is already a few lines of DSL. Adding
--istan/--ostanflags would mean maintaining a format flag indefinitely for a shape convention that only matters to one downstream tool, with unresolved questions (per the issue thread itself) about how it should generalize to nested/ragged data. Thecolumns-to-arrays/arrays-to-columnsverb idea floated later in the issue is a separable, more general feature and isn't addressed here.What's in this recipe
Placed directly after the existing "Transposing very wide data" section, which uses the same underlying out-of-stream-variable technique (
@x[k][NR] = v), so the two sections build on each other.arrayify()+emit1to produce real JSON arrays, e.g.{"shape": [...], "rate": [...]}.One correction versus the original issue thread: the 2020 comment's
emit @output_recordsnippet doesn't produce the object-of-arrays shape on current Miller (6.20.2-dev) — plainemiton a 2-level map splits into separate records and drops the field-name keys. I tested against a live build and fixed it toemit1 arrayify(@output_record), which does produce the intended output. Both directions are verified round-tripping correctly againstexample.csv/ the newdata/stan-example.jsonfixture.Testing
docs/src/shapes-of-data.mdis regenerated output (./genmds shapes-of-data.md.in, run against a localmake build) — the embedded command output is live, not hand-typed, so it'll be caught by doc-build CI if it ever drifts from actualmlrbehavior.Known gaps
columns-to-arrays/arrays-to-columnsverbs floated later in Prep JSON data for stan #392 — left as a separate decision if reshaping demand shows up independently of the Stan use case.