Physical distinctness expressions incorrectly report nullable output

[sesteves]

Describe the bug

The physical BinaryExpr::nullable implementation reports IS DISTINCT FROM and IS NOT DISTINCT FROM as nullable when either operand is nullable:

fn nullable(&self, input_schema: &Schema) -> Result<bool> {
    Ok(self.left.nullable(input_schema)? || self.right.nullable(input_schema)?)
}

SQL distinctness operators always produce a non-null Boolean, including when one or both operands are null. The logical expressions correctly report non-nullable output, but physical planning lowers Boolean truth predicates such as b IS TRUE through IsNotDistinctFrom. A nullable input therefore creates a logical/physical schema mismatch.

This can fail deterministically during aggregate physical planning when common-subexpression elimination extracts a repeated nullable truth predicate:

Physical input schema should be the same as the one converted from logical input schema.
Differences: field nullability at index 0 [__common_expr_1]:
(physical) true vs (logical) false

I confirmed the generic nullable implementation is still present on DataFusion main at commit 2dd1a14232b8021cb8979b0438367f1fde7c97e7. The behavior is also present in DataFusion 53.1.0 and 54.0.0.

To Reproduce

The following query repeats a truth predicate over a nullable Boolean, causing aggregate CSE to extract it:

SELECT
  SUM(CASE WHEN b IS TRUE THEN 1 ELSE 0 END),
  COUNT(CASE WHEN b IS TRUE THEN 1 END),
  SUM(CASE WHEN b IS FALSE THEN 1 ELSE 0 END),
  SUM(CASE WHEN b IS NOT TRUE THEN 1 ELSE 0 END),
  SUM(CASE WHEN b IS NOT FALSE THEN 1 ELSE 0 END)
FROM (
  VALUES
    (TRUE),
    (FALSE),
    (CAST(NULL AS BOOLEAN))
) AS t(b);

The expected result is:

1, 1, 1, 2, 2

A lower-level regression can construct physical BinaryExpr values using nullable operands and verify that both Operator::IsDistinctFrom and Operator::IsNotDistinctFrom report nullable() == false.

Expected behavior

Physical distinctness expressions should always report non-nullable output:

match self.op {
    Operator::IsDistinctFrom | Operator::IsNotDistinctFrom => Ok(false),
    _ => Ok(self.left.nullable(input_schema)? || self.right.nullable(input_schema)?),
}

The repeated aggregate query should plan and execute while aggregate logical/physical schema validation remains enabled.

Regression coverage should include:

• Nullable operands for both physical distinctness operators.
• Repeated nullable IS TRUE conditions that trigger aggregate CSE.
• IS TRUE, IS FALSE, IS NOT TRUE, and IS NOT FALSE semantics for TRUE, FALSE, and NULL inputs.

Additional context

This is related to the broad class of logical/physical schema nullability mismatches discussed in #17801, but that issue concerned CASE/COALESCE inference and does not fix the physical BinaryExpr::nullable path.

It is also distinct from #23692, which concerns SQL parser precedence for multiple IS NOT DISTINCT FROM predicates in join conditions.

A downstream workaround can rewrite truth predicates to non-nullable CASE expressions before physical planning, but the underlying physical distinctness nullability should be fixed in DataFusion so downstream engines do not need optimizer compatibility rules.

[blinding-pixels]

take