Skip to content

Docs: recipe for reshaping columns to/from JSON arrays ("STAN format") - #2216

Merged
johnkerl merged 1 commit into
mainfrom
johnkerl/stan-docs
Jul 17, 2026
Merged

Docs: recipe for reshaping columns to/from JSON arrays ("STAN format")#2216
johnkerl merged 1 commit into
mainfrom
johnkerl/stan-docs

Conversation

@johnkerl

@johnkerl johnkerl commented Jul 16, 2026

Copy link
Copy Markdown
Owner

Summary

Adds a "Columns as JSON arrays" section to shapes-of-data.md, documenting the mkstan/unstan recipe from #392 as a doc-only solution rather than a new --istan/--ostan file format.

Why doc-only, not a new format: #392 is a 2020 issue with a single external commenter and no activity since 2022. The "STAN format" isn't a distinct format — it's a JSON object whose values are arrays, one per column (useful for feeding the Stan modeling tool), and the reshape is already a few lines of DSL. Adding --istan/--ostan flags would mean maintaining a format flag indefinitely for a shape convention that only matters to one downstream tool, with unresolved questions (per the issue thread itself) about how it should generalize to nested/ragged data. The columns-to-arrays/arrays-to-columns verb idea floated later in the issue is a separable, more general feature and isn't addressed here.

What's in this recipe

Placed directly after the existing "Transposing very wide data" section, which uses the same underlying out-of-stream-variable technique (@x[k][NR] = v), so the two sections build on each other.

  • Forward direction (columns → JSON arrays): uses arrayify() + emit1 to produce real JSON arrays, e.g. {"shape": [...], "rate": [...]}.
  • Reverse direction (JSON arrays → one record per row): finds the longest array, then re-keys by row index.

One correction versus the original issue thread: the 2020 comment's emit @output_record snippet doesn't produce the object-of-arrays shape on current Miller (6.20.2-dev) — plain emit on a 2-level map splits into separate records and drops the field-name keys. I tested against a live build and fixed it to emit1 arrayify(@output_record), which does produce the intended output. Both directions are verified round-tripping correctly against example.csv / the new data/stan-example.json fixture.

Testing

docs/src/shapes-of-data.md is regenerated output (./genmds shapes-of-data.md.in, run against a local make build) — the embedded command output is live, not hand-typed, so it'll be caught by doc-build CI if it ever drifts from actual mlr behavior.

Known gaps

  • Doesn't implement the columns-to-arrays/arrays-to-columns verbs floated later in Prep JSON data for stan #392 — left as a separate decision if reshaping demand shows up independently of the Stan use case.

Adds a "Columns as JSON arrays" section to shapes-of-data.md covering
the reshape-to-column-arrays and reshape-back techniques from issue
392 (e.g. for feeding data to Stan), using arrayify()/emit1 to
produce real JSON arrays rather than a dedicated file format.

Closes #392.
@johnkerl johnkerl changed the title Docs: recipe for reshaping columns to/from JSON arrays (closes #392) Docs: recipe for reshaping columns to/from JSON arrays ("STAN format") Jul 16, 2026
@johnkerl
johnkerl marked this pull request as ready for review July 16, 2026 23:33
@johnkerl
johnkerl merged commit dbc2454 into main Jul 17, 2026
8 checks passed
@johnkerl
johnkerl deleted the johnkerl/stan-docs branch July 17, 2026 15:07
@johnkerl johnkerl mentioned this pull request Jul 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant