[Varint] Implement ShreddingState::AllNull variant #8093

codephage2020 · 2025-08-08T07:42:05Z

Which issue does this PR close?

Closes [Variant] Implement ShreddingState::AllNull variant #8088 .

What changes are included in this PR?

handles the "all null" case

Are these changes tested?

Yes.

Are there any user-facing changes?

no.

…t handling Signed-off-by: codephage2020 <[email protected]>

Signed-off-by: codephage2020 <[email protected]>

alamb · 2025-08-11T20:50:27Z

parquet-variant-compute/src/variant_array.rs

+    #[test]
+    fn all_null_variant_array_construction() {
+        let metadata = BinaryViewArray::from(vec![b"test" as &[u8]; 3]);
+        let nulls = NullBuffer::from(vec![false, false, false]); // all null


Are we sure that this case (where there is a value field present, but it is all null) should be treated as though there was no value field?

I haven't double checked the spec (probably the Arrow one) but this feels like it may be out of compliance

As far as I know, the spec's requirement that "both value and typed_value are null" meaning "the value is missing."
We can see from the spec's example:
refer to here.

Oh, I got it. I will add a test for the case.

I added a comprehensive test that demonstrates when a value field exists in the schema but contains all null values, it correctly remains in the Unshredded state rather than AllNull.

From what I understand, the treatment of NULL/NULL depends on context:

For a top-level variant value, it's interpreted as Variant::Null

For a shredded variant object field, it's interpreted as missing (SQL NULL)

So I guess there are two ways to get SQL NULL -- null mask on the struct(value, typed_value), or if both value and typed_value are themselves NULL?

Ah -- the spec requires that the struct ("group") containing value and typed_value columns must be non-nullable:

The group for each named field must use repetition level required. A field's value and typed_value are set to null (missing) to indicate that the field does not exist in the variant. To encode a field that is present with a [variant/JSON, not SQL] null value, the value must contain a Variant null: basic type 0 (primitive) and physical type 0 (null).

So effectively, the NULL/NULL combo becomes the null mask for that nested field. Which is why a top-level NULL/NULL combo is incorrect -- the top-level field already has its own nullability.

alamb

Thank you @codephage2020 -- this looks great -- I had one corner case question

…t handling Signed-off-by: codephage2020 <[email protected]>

Signed-off-by: codephage2020 <[email protected]>

alamb

Thank you @codephage2020 -- I think this looks good to me

FYI @scovich in case you would also like to review

scovich

Lots of messiness in the shredding spec here... but I think the current code is probably ok-ish, because we only support top-level variant so far?

Main question is whether we should forbid the AllNull case at top level, since it's technically valid only for nested fields?

scovich · 2025-08-12T21:35:29Z

parquet-variant-compute/src/variant_array.rs

@@ -187,6 +187,7 @@ impl VariantArray {
                    typed_value_to_variant(typed_value, index)
                }
            }
+            ShreddingState::AllNull { .. } => Variant::Null,


This is tricky... see #8122 (comment)

For a top-level variant, null/null is illegal (tho returning Variant::Null is arguably a correct way to compensate)

For a shredded variant field, null/null means SQL NULL, and returning Variant::Null is arguably incorrect (causes SQL IS NULL operator to return FALSE). But we don't even have a way to return SQL NULL here (it would probably correspond to Option::None?)

scovich · 2025-08-12T21:56:11Z

parquet-variant-compute/src/variant_array.rs

+        // Should succeed and create an AllNull variant when neither value nor typed_value are present
+        let variant_array = VariantArray::try_new(Arc::new(array)).unwrap();


By a strict reading of the spec, this should actually fail, because this ShreddingState does not represent a shredded object field, but rather represents a top-level variant value:

value typed_value Meaning

null null The value is missing; only valid for shredded object fields

But maybe that's a validation VariantArray::try_new should perform, not ShreddingState::try_new?

Also, it would quickly become annoying if variant_get has to replace a missing or all-null value column with an all-Variant::Null column just to comply with the spec? Maybe that's why there's this additional tidbit?

If a Variant is missing in a context where a value is required, readers must return a Variant null (00): basic type 0 (primitive) and physical type 0 (null). For example, if a Variant is required (like measurement above) and both value and typed_value are null, the returned value must be 00 (Variant null).

scovich · 2025-08-12T22:04:59Z

parquet-variant-compute/src/variant_array.rs

+    #[test]
+    fn all_null_variant_array_construction() {
+        let metadata = BinaryViewArray::from(vec![b"test" as &[u8]; 3]);
+        let nulls = NullBuffer::from(vec![false, false, false]); // all null


From what I understand, the treatment of NULL/NULL depends on context:

For a top-level variant value, it's interpreted as Variant::Null

For a shredded variant object field, it's interpreted as missing (SQL NULL)

So I guess there are two ways to get SQL NULL -- null mask on the struct(value, typed_value), or if both value and typed_value are themselves NULL?

scovich · 2025-08-12T23:00:47Z

parquet-variant-compute/src/variant_array.rs

+    #[test]
+    fn all_null_variant_array_construction() {
+        let metadata = BinaryViewArray::from(vec![b"test" as &[u8]; 3]);
+        let nulls = NullBuffer::from(vec![false, false, false]); // all null


Ah -- the spec requires that the struct ("group") containing value and typed_value columns must be non-nullable:

The group for each named field must use repetition level required. A field's value and typed_value are set to null (missing) to indicate that the field does not exist in the variant. To encode a field that is present with a [variant/JSON, not SQL] null value, the value must contain a Variant null: basic type 0 (primitive) and physical type 0 (null).

So effectively, the NULL/NULL combo becomes the null mask for that nested field. Which is why a top-level NULL/NULL combo is incorrect -- the top-level field already has its own nullability.

parquet-variant-compute/src/variant_get/output/primitive.rs

Co-authored-by: Ryan Johnson <[email protected]> Co-authored-by: Ryan Johnson <[email protected]>

Signed-off-by: codephage2020 <[email protected]>

codephage2020 · 2025-08-13T07:48:54Z

Thanks for the thorough review! I’ll go through all the comments carefully and respond later today.

codephage2020 added 2 commits August 8, 2025 15:38

Implement support for AllNull state in VariantArray and related outpu…

1ceeae9

…t handling Signed-off-by: codephage2020 <[email protected]>

fmt

bd43fbf

Signed-off-by: codephage2020 <[email protected]>

codephage2020 marked this pull request as ready for review August 9, 2025 03:10

alamb reviewed Aug 11, 2025

View reviewed changes

codephage2020 added 2 commits August 12, 2025 16:15

Implement support for AllNull state in VariantArray and related outpu…

1120b07

…t handling Signed-off-by: codephage2020 <[email protected]>

fmt

b5d4c5d

Signed-off-by: codephage2020 <[email protected]>

codephage2020 force-pushed the feat/shreddingstate-allnull-variant branch from bd43fbf to b5d4c5d Compare August 12, 2025 08:15

codephage2020 added 3 commits August 12, 2025 22:34

Add test for unshredded state when value field is present but all null

b237cb1

Signed-off-by: codephage2020 <[email protected]>

Add test for unshredded state when value field is present but all null

6705cb5

Delete duplicates

d266031

Signed-off-by: codephage2020 <[email protected]>

alamb approved these changes Aug 12, 2025

View reviewed changes

scovich reviewed Aug 13, 2025

View reviewed changes

Update parquet-variant-compute/src/variant_get/output/primitive.rs

4b52a03

Co-authored-by: Ryan Johnson <[email protected]> Co-authored-by: Ryan Johnson <[email protected]>

github-actions bot added the parquet-variant parquet-variant* crates label Aug 13, 2025

fmt and clippy

b16c95b

Signed-off-by: codephage2020 <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Varint] Implement ShreddingState::AllNull variant #8093

[Varint] Implement ShreddingState::AllNull variant #8093

Uh oh!

codephage2020 commented Aug 8, 2025 •

edited

Loading

Uh oh!

alamb Aug 11, 2025

Uh oh!

codephage2020 Aug 12, 2025

Uh oh!

codephage2020 Aug 12, 2025

Uh oh!

codephage2020 Aug 12, 2025

Uh oh!

scovich Aug 12, 2025

Uh oh!

scovich Aug 12, 2025

Uh oh!

alamb left a comment

Uh oh!

alamb left a comment

Uh oh!

scovich left a comment

Uh oh!

scovich Aug 12, 2025

Uh oh!

scovich Aug 12, 2025

Uh oh!

scovich Aug 12, 2025

Uh oh!

scovich Aug 12, 2025

Uh oh!

Uh oh!

codephage2020 commented Aug 13, 2025

Uh oh!

Uh oh!

		// Should succeed and create an AllNull variant when neither value nor typed_value are present
		let variant_array = VariantArray::try_new(Arc::new(array)).unwrap();

[Varint] Implement ShreddingState::AllNull variant #8093

Are you sure you want to change the base?

[Varint] Implement ShreddingState::AllNull variant #8093

Uh oh!

Conversation

codephage2020 commented Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

scovich left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codephage2020 commented Aug 13, 2025

Uh oh!

Uh oh!

codephage2020 commented Aug 8, 2025 •

edited

Loading