doc: add example of RowFilter usage #9115
Open
+84
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sonhmai
force-pushed
the
doc/row-filter-usage-9096
branch
from
37be4e1 to
f286dfd
Compare
sonhmai
changed the title
sonhmai
force-pushed
the
doc/row-filter-usage-9096
branch
from
f286dfd to
bc8e06f
Compare
sonhmai
changed the title
Contributor
Author
|
@alamb would you mind reviewing this? Thanks! |
Jefffrey
reviewed
Jan 10, 2026
| use parquet::errors::Result; | ||
| use std::fs::File; | ||
|
|
||
| // RowFilter / with_row_filter usage. For background and more |
Contributor
There was a problem hiding this comment.
Are we better off removing this and keeping only the doctest to reduce duplication?
Contributor
There was a problem hiding this comment.
Yes, I agree -- I think the doc examples are easier to find so I recommend removing this example file
Actually, looking at the existing examples I think many of them are redundant / would be easier to find if we moved them into the documentation:
https://github.com/apache/arrow-rs/tree/main/parquet/examples
| /// more efficient skipping over data pages. See [`ArrowReaderOptions::with_page_index`]. | ||
| /// | ||
| /// For a running example see `parquet/examples/read_with_row_filter.rs`. | ||
| /// See <https://arrow.apache.org/blog/2025/12/11/parquet-late-materialization-deep-dive/> |
Contributor
There was a problem hiding this comment.
/// See the [blog post on late materialization] for a more technical explanation.
///
/// ...
///
/// [blog post on late materialization]: https://arrow.apache.org/blog/2025/12/11/parquet-late-materialization-deep-dive/
Slightly nice formatting this way
alamb
approved these changes
Jan 11, 2026
| use parquet::errors::Result; | ||
| use std::fs::File; | ||
|
|
||
| // RowFilter / with_row_filter usage. For background and more |
Contributor
There was a problem hiding this comment.
Yes, I agree -- I think the doc examples are easier to find so I recommend removing this example file
Actually, looking at the existing examples I think many of them are redundant / would be easier to find if we moved them into the documentation:
https://github.com/apache/arrow-rs/tree/main/parquet/examples
| /// more efficient skipping over data pages. See [`ArrowReaderOptions::with_page_index`]. | ||
| /// | ||
| /// For a running example see `parquet/examples/read_with_row_filter.rs`. | ||
| /// See <https://arrow.apache.org/blog/2025/12/11/parquet-late-materialization-deep-dive/> |
| /// let builder = ParquetRecordBatchReaderBuilder::try_new(file)?; | ||
| /// let schema_desc = builder.metadata().file_metadata().schema_descr_ptr(); | ||
| /// | ||
| /// // Create predicate: column id > 4. This col has index 0. |
Contributor
There was a problem hiding this comment.
Suggested change
| /// // Create predicate: column id > 4. This col has index 0. | |
| /// // Create predicate that evaluates `id > 4`. The `id` column has index 0. |
| /// // Create predicate: column id > 4. This col has index 0. | ||
| /// let projection = ProjectionMask::leaves(&schema_desc, [0]); | ||
| /// let predicate = ArrowPredicateFn::new(projection, |batch| { | ||
| /// let id_col = batch.column(0); |
Contributor
There was a problem hiding this comment.
As a minor suggestion, I think it would make a nicer example if you picked a different column from the file other than 0 so that it is clear the batch passed to the predicate only contains the selected projection column
For example, perhaps you could use the int_col (column index 4)
> select * from './parquet-testing/data/alltypes_plain.parquet';
+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+
| id | bool_col | tinyint_col | smallint_col | int_col | bigint_col | float_col | double_col | date_string_col | string_col | timestamp_col |
+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+
| 4 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30332f30312f3039 | 30 | 2009-03-01T00:00:00 |
| 5 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30332f30312f3039 | 31 | 2009-03-01T00:01:00 |
| 6 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30342f30312f3039 | 30 | 2009-04-01T00:00:00 |
| 7 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30342f30312f3039 | 31 | 2009-04-01T00:01:00 |
| 2 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30322f30312f3039 | 30 | 2009-02-01T00:00:00 |
| 3 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30322f30312f3039 | 31 | 2009-02-01T00:01:00 |
| 0 | true | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 30312f30312f3039 | 30 | 2009-01-01T00:00:00 |
| 1 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30312f30312f3039 | 31 | 2009-01-01T00:01:00 |
+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+
8 row(s) fetched.
Elapsed 0.039 seconds.
> describe './parquet-testing/data/alltypes_plain.parquet';
+-----------------+---------------+-------------+
| column_name | data_type | is_nullable |
+-----------------+---------------+-------------+
| id | Int32 | YES |
| bool_col | Boolean | YES |
| tinyint_col | Int32 | YES |
| smallint_col | Int32 | YES |
| int_col | Int32 | YES |
| bigint_col | Int64 | YES |
| float_col | Float32 | YES |
| double_col | Float64 | YES |
| date_string_col | BinaryView | YES |
| string_col | BinaryView | YES |
| timestamp_col | Timestamp(ns) | YES |
+-----------------+---------------+-------------+
11 row(s) fetched.
Elapsed 0.005 seconds.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
The RowFilter API does exist and can evaluate predicates during evaluation, but it has no examples.
What changes are included in this PR?
ParquetRecordBatchReaderBuilder::with_row_filter.parquet/examples/read_with_row_filter.rsAre these changes tested?
Yes
Are there any user-facing changes?
Yes, doc only. No API changes.