perf: improve field indexing in JSON StructArrayDecoder (1.7x speed up) #9086

Weijun-H · 2026-01-02T13:31:04Z

Which issue does this PR close?

Closes #NNN.

Rationale for this change

Optimize JSON struct decoding on wide objects by reducing per-row allocations and repeated field lookups.

What changes are included in this PR?

Reuse a flat child-position buffer in StructArrayDecoder and add an optional field-name index for object mode.
Skip building the field-name index for list mode; add overflow/allocation checks.

decode_wide_object_i64_json
                        time:   [11.828 ms 11.865 ms 11.905 ms]
                        change: [−67.828% −67.378% −67.008%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe

decode_wide_object_i64_serialize
                        time:   [7.6923 ms 7.7402 ms 7.7906 ms]
                        change: [−75.652% −75.483% −75.331%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

Are these changes tested?

Yes

Are there any user-facing changes?

No

scovich

Not sure I understand the indexing code well enough to say whether that part is correct, but the idea of using an optional index for field name lookups makes a lot of sense to me.

scovich · 2026-01-05T21:33:03Z

arrow-json/src/reader/struct_array.rs

    }
 }
+
+fn build_field_index(fields: &Fields) -> Option<HashMap<String, usize>> {


qq: Do lifetimes coincide so that we could return Option<HashMap<&str, usize>> instead?

Yes, the lifetimes do coincide. we can use HashMap<&'a str, usize> by taking fields: &'a Fields as a parameter, which avoids the self-referential struct problem. However, this would require threading the lifetime parameter <'a> through the entire decoder system across many files. Since the lookup performance is identical, I don’t think it’s worth the added complexity.

maybe it would be a good follow on PR

alamb

Thanks @Weijun-H and @scovich

alamb · 2026-01-06T22:30:41Z

arrow-json/benches/reader.rs

+use std::fmt::Write;
+use std::sync::Arc;
+
+fn build_schema(field_count: usize) -> Arc<Schema> {


can you please add some comments here with an example of what this code does / what patterns of input it creates?

Also, it would help me to reproduce your results if you could make a separate PR with the benchmarks (so I can compare main to the PR)

separate benchmark here

#9107

alamb · 2026-01-06T22:31:31Z

arrow-json/src/reader/struct_array.rs

    }
 }
+
+fn build_field_index(fields: &Fields) -> Option<HashMap<String, usize>> {


maybe it would be a good follow on PR

alamb · 2026-01-10T12:38:20Z

run benchmark json-reader

alamb-ghbot · 2026-01-10T12:38:44Z

🤖 Hi @alamb, thanks for the request (#9086 (comment)).

scrape_comments.py only supports whitelisted benchmarks.

Standard: (none)
Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels

Please choose one or more of these with run benchmark <name> or run benchmark <name1> <name2>...

…exing in StructArrayDecoder

…ecoders and field index creation

Weijun-H · 2026-01-10T12:55:49Z

run benchmark json-reader

alamb-ghbot · 2026-01-10T12:55:52Z

🤖 Hi @Weijun-H, thanks for the request (#9086 (comment)). scrape_comments.py only responds to whitelisted users. Allowed users: Dandandan, Omega359, adriangb, alamb, comphead, geoffreyclaude, klion26, rluvaton, xudong963, zhuqi-lucas.

alamb-ghbot · 2026-01-10T17:30:18Z

🤖 ./gh_compare_arrow.sh gh_compare_arrow.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing optimize-json-scan (06ded8b) to b2aeab1 diff
BENCH_NAME=json-reader
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench json-reader
BENCH_FILTER=
BENCH_BRANCH_NAME=optimize-json-scan
Results will be posted here when complete

alamb-ghbot · 2026-01-10T17:46:26Z

🤖: Benchmark completed

Details

group                                        main                                   optimize-json-scan
-----                                        ----                                   ------------------
decode_binary_hex_json                       1.05     93.1±0.89ms        ? ?/sec    1.00     88.5±1.03ms        ? ?/sec
decode_binary_view_hex_json                  1.05     94.2±0.61ms        ? ?/sec    1.00     89.6±1.39ms        ? ?/sec
decode_fixed_binary_hex_json                 1.05     92.9±1.20ms        ? ?/sec    1.00     88.3±1.40ms        ? ?/sec
decode_wide_object_i64_json                  1.38  1468.8±33.63ms        ? ?/sec    1.00  1065.8±27.55ms        ? ?/sec
decode_wide_object_i64_serialize             1.46  1268.0±13.45ms        ? ?/sec    1.00   866.5±14.04ms        ? ?/sec
decode_wide_projection_full_json/131072      1.64       3.0±0.03s    57.4 MB/sec    1.00  1845.3±18.20ms    94.3 MB/sec
decode_wide_projection_narrow_json/131072    1.00   780.7±12.09ms   222.9 MB/sec    1.01   791.4±10.94ms   219.9 MB/sec

alamb

Thanks @Weijun-H -- I think this PR is a nice improvement. I have some suggestions on how to make it faster and improve the comments, but overall very nice 👍

alamb · 2026-01-10T19:43:47Z

arrow-json/src/reader/struct_array.rs

    is_nullable: bool,
    struct_mode: StructMode,
+    field_name_to_index: Option<HashMap<String, usize>>,
+    child_pos: Vec<u32>,


Could you add a comment that explains what child_pos is? It isn't clear here (the idea of caching rather than recreating it looks good though)

Specifically I think it is important to document what is stored at each index (e.g. each index the tape position of at field_idx * row_count + row)

renamed and commented in df9e710

alamb · 2026-01-10T19:48:17Z

arrow-json/src/reader/struct_array.rs

+                    ))
+                })?;
+        }
+        self.child_pos.resize(total_len, 0);


This seems like it would set some elements to zero twice -- I think you can get the same result without the extra setting via

self.child_pos.clear(); self.child_pos.resize(total_len, 0);

Also, I think resize calls reserve internally (it internally calls extend_with which calls reserve), so there is no need to also call child_pos.reserve above

(also the rest of this crate just calls reserve so I think using try_reserve just here seems unecessary)

addressed in. df9e710

alamb · 2026-01-10T19:50:41Z

arrow-json/src/reader/struct_array.rs

+                                    fields.len()
+                                )));
+                            }
+                            child_pos[entry_idx * row_count + row] = cur_idx;


👍 this is a nice way to avoid allocations

…ecoder for improved clarity and performance

github-actions bot added the arrow Changes to the arrow crate label Jan 2, 2026

Weijun-H marked this pull request as ready for review January 2, 2026 13:57

Weijun-H changed the title ~~perf: improve field indexing in StructArrayDecoder~~ perf: improve field indexing in StructArrayDecoder (1.5x speed up) Jan 2, 2026

Weijun-H changed the title ~~perf: improve field indexing in StructArrayDecoder (1.5x speed up)~~ perf: improve field indexing in StructArrayDecoder (2x speed up) Jan 2, 2026

Weijun-H changed the title ~~perf: improve field indexing in StructArrayDecoder (2x speed up)~~ perf: improve field indexing in StructArrayDecoder (1.7x speed up) Jan 2, 2026

scovich reviewed Jan 5, 2026

View reviewed changes

alamb reviewed Jan 6, 2026

View reviewed changes

alamb changed the title ~~perf: improve field indexing in StructArrayDecoder (1.7x speed up)~~ perf: improve field indexing in JSON StructArrayDecoder (1.7x speed up) Jan 7, 2026

alamb added the performance label Jan 10, 2026

apache deleted a comment from alamb-ghbot Jan 10, 2026

Weijun-H added 4 commits January 10, 2026 14:54

feat: add benchmark for JSON reader performance and improve field ind…

726d1bf

…exing in StructArrayDecoder

refactor: streamline StructArrayDecoder initialization by combining d…

5109576

…ecoders and field index creation

chore

e593630

chore

06ded8b

Weijun-H force-pushed the optimize-json-scan branch from 7e3077e to 06ded8b Compare January 10, 2026 12:55

alamb approved these changes Jan 10, 2026

View reviewed changes

refactor: replace child_pos with field_tape_positions in StructArrayD…

df9e710

…ecoder for improved clarity and performance

alamb mentioned this pull request Jan 11, 2026

Andrew Lamb Weekly-ish Open Source plan - 2026-01-05 apache/datafusion#19652

Open

40 tasks

perf: improve field indexing in JSON StructArrayDecoder (1.7x speed up) #9086

Are you sure you want to change the base?

perf: improve field indexing in JSON StructArrayDecoder (1.7x speed up) #9086

Uh oh!

Conversation

Weijun-H commented Jan 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

scovich left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Jan 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb-ghbot commented Jan 10, 2026

Uh oh!

Weijun-H commented Jan 10, 2026

Uh oh!

alamb-ghbot commented Jan 10, 2026

Uh oh!

alamb-ghbot commented Jan 10, 2026

Uh oh!

alamb-ghbot commented Jan 10, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Weijun-H commented Jan 2, 2026 •

edited

Loading

alamb commented Jan 10, 2026 •

edited

Loading