Support simplify not for physical expr #18970

xudong963 · 2025-11-27T13:37:33Z

Part of the #18868

datafusion/physical-expr/src/simplifier/not.rs

datafusion/physical-expr/src/simplifier/mod.rs

datafusion/physical-expr/src/simplifier/not.rs

xudong963 · 2025-12-01T11:31:16Z

Thanks for the review @martin-g

alamb

Thank you @xudong963 and @martin-g

I think prior to releasing this code we should make the following changes:

Avoid explicit recursion in not simplification (and instead use the tree node API)
Avoid the potential infinite loop

It would be nice to make them as part of this PR, but I also think it would be ok to do it as a follow on

Other things I think would be good to do but not strictly necessary:

Move the tests to the public interface (PhysicalExprSimplifier)
Avoid duplications of Operator::negate

alamb · 2025-12-01T17:14:20Z

datafusion/physical-expr/src/simplifier/mod.rs

        );
-        Ok(unwrapped)
+        // Combine transformation results
+        let final_transformed = transformed || unwrapped.transformed;


I think you can use transform_data here instead: https://docs.rs/datafusion/latest/datafusion/common/tree_node/struct.Transformed.html#method.transform_data

So something like

// Apply NOT expression simplification first let rewritten = simplify_not_expr(&node, self.schema)?.transform_data(|node| { unwrap_cast::unwrap_cast_in_comparison(node, self.schema) })?;

That handles combining the transformed flag for you

neat, done in be077db.

Need to catch up on the tree node APIs, haha

alamb · 2025-12-01T17:14:37Z

datafusion/physical-expr/src/simplifier/mod.rs

+
        // Apply unwrap cast optimization
        #[cfg(test)]
        let original_type = node.data_type(self.schema).unwrap();


It would be nice to move the original type check before applying the not and the verification after the not

Somethg like

impl<'a> TreeNodeRewriter for PhysicalExprSimplifier<'a> { type Node = Arc<dyn PhysicalExpr>; fn f_up(&mut self, node: Self::Node) -> Result<Transformed<Self::Node>> { #[cfg(test)] let original_type = node.data_type(self.schema).unwrap(); // Apply NOT expression simplification first let rewritten = simplify_not_expr(&node, self.schema)?.transform_data(|node| { unwrap_cast::unwrap_cast_in_comparison(node, self.schema) })?; #[cfg(test)] assert_eq!( rewritten.data.data_type(self.schema).unwrap(), original_type, "Simplified expression should have the same data type as the original" ); Ok(rewritten) } }

done in be077db

alamb · 2025-12-01T17:15:57Z

datafusion/physical-expr/src/simplifier/not.rs

+        ])
+    }
+
+    #[test]


Since this function is not really run in isolation (it is always run as part of PhysicalExprSimplifier) I think it would be better if these tests were in datafusion/physical-expr/src/simplifier/mod.rs rather than here.

I don't think this change is required

alamb · 2025-12-01T17:16:30Z

datafusion/physical-expr/src/simplifier/not.rs

+}
+
+/// Returns the negated version of a comparison operator, if possible
+fn negate_operator(op: &Operator) -> Option<Operator> {


This looks the same as Operator::negate: https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.Operator.html#method.negate

I recommend using that code, or if there is a reason not to use Operator::negate add a comment explaining why it is not used

alamb · 2025-12-01T17:24:26Z

datafusion/physical-expr/src/simplifier/not.rs

+                    Arc::new(NotExpr::new(Arc::clone(binary_expr.right())));
+
+                // Recursively simplify the NOT expressions
+                let simplified_left = simplify_not_expr(&not_left, schema)?;


I think since simplify_not_expr is used in PhysicalExprSimplifier which is doing a walk up the plan (using f_up) there is no reason to also recurse explicitly in this rule.

You should be able to apply the rewrite rules only NOT(A OR B) --> NOT A AND NOT B rather than also changing the exprs

alamb · 2025-12-01T17:27:07Z

datafusion/physical-expr/src/simplifier/not.rs

+    let mut current_expr = Arc::clone(expr);
+    let mut overall_transformed = false;
+
+    loop {


I am also somewhat concerned about this loop -- it seems like it may be trying to handle a non complete recursion and it could potentially lead to infinite recursion if a rewrite flip/flopped

If the loop is needed I suggest:

Put it in the higher level PhysicalExprSimplifier

add a counter that breaks after some number of iterations (e.g. 5 or something)

I put it into PhysicalExprSimplifier with a counter to limit the loop

alamb · 2025-12-01T17:34:38Z

datafusion/physical-expr/src/simplifier/not.rs

+
+        assert!(result.transformed);
+        // Should be simplified back to the original b > 5
+        assert_eq!(result.data.to_string(), inner_expr.to_string());


What is the reason for using to_string()?

I tried comparing the two directly and it seems to work

assert_eq!(&result.data, &inner_expr);

yeah, I didn't notice the expr is comparable

alamb · 2025-12-01T17:35:29Z

datafusion/physical-expr/src/simplifier/not.rs

+        assert!(result.transformed);
+        // Should be simplified back to the original b > 5
+        assert_eq!(result.data.to_string(), inner_expr.to_string());


It might also make these queries easier to read if you made a helper like

fn assert_transformed(expr, expected_expr);

and

fn assert_not_transformed(expr, expected_expr);

alamb · 2025-12-01T17:37:08Z

datafusion/physical-expr/src/simplifier/not.rs

+        if let Some(literal) = result.data.as_any().downcast_ref::<Literal>() {
+            assert_eq!(literal.value(), &ScalarValue::Boolean(Some(false)));
+        } else {
+            panic!("Expected literal result");
+        }


You could potentially make this a function to make the tests easier to read - something like

let literal = as_literal(&result);

And handle the panic internally in as_literal

That way the mechics of the test wouldn't obscure the test logic so much

I think you could do something similar with BinaryExpr

alamb · 2025-12-01T17:37:40Z

datafusion/physical-expr/src/simplifier/not.rs

+        let schema = test_schema();
+
+        // Create a deeply nested NOT expression: NOT(NOT(NOT(...NOT(b > 5)...)))
+        // This tests that we don't get stack overflow with many nested NOTs


…arison

xudong963 · 2025-12-02T08:53:33Z

Thank you @xudong963 and @martin-g

I think prior to releasing this code we should make the following changes:

Avoid explicit recursion in not simplification (and instead use the tree node API)

Avoid the potential infinite loop

It would be nice to make them as part of this PR, but I also think it would be ok to do it as a follow on

Other things I think would be good to do but not strictly necessary:

Move the tests to the public interface (PhysicalExprSimplifier)

Avoid duplications of Operator::negate

Thanks @alamb. I addressed all suggestions in the latest commit 27c4ef1

Support simplify not for physical expr

c01c335

github-actions bot added physical-expr Changes to the physical-expr crates execution Related to the execution crate labels Nov 27, 2025

xudong963 mentioned this pull request Nov 27, 2025

Row group limit pruning #18868

Open

martin-g reviewed Nov 27, 2025

View reviewed changes

datafusion/physical-expr/src/simplifier/not.rs Outdated Show resolved Hide resolved

alamb mentioned this pull request Nov 30, 2025

Andrew Lamb Weekly-ish Open Source plan - 2025-11-24 #18888

Closed

37 tasks

xudong963 added 3 commits December 1, 2025 09:57

not(not a and not b)

a0116ce

support not list

0d075ba

replace recursive and add test

ceb85ff

github-actions bot removed the execution Related to the execution crate label Dec 1, 2025

xudong963 requested a review from zhuqi-lucas December 1, 2025 07:31

martin-g approved these changes Dec 1, 2025

View reviewed changes

alamb mentioned this pull request Dec 1, 2025

Andrew Lamb Weekly-ish Open Source plan - 2025-12-01 #19016

Open

31 tasks

alamb approved these changes Dec 1, 2025

View reviewed changes

Use transform_data to connect simplify_not_expr & unwrap_cast_in_comp…

be077db

…arison

xudong963 force-pushed the simplify_not branch from d749da7 to be077db Compare December 2, 2025 07:40

github-actions bot added the execution Related to the execution crate label Dec 2, 2025

resolve all comments

27c4ef1

xudong963 force-pushed the simplify_not branch from e9731f3 to 27c4ef1 Compare December 2, 2025 08:53

xudong963 requested a review from alamb December 2, 2025 08:54

+                      ])
+                  }
+                  #[test]

Support simplify not for physical expr #18970

Are you sure you want to change the base?

Support simplify not for physical expr #18970

Conversation

xudong963 commented Nov 27, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xudong963 commented Dec 1, 2025

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xudong963 commented Dec 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants