Can I use an EquivalenceGroup to tell datafusion that a date partition column is derived from a datetime column?

[samuelcolvin]

I have a table that looks something like:

create table eggs (
    created_at timestamp,
    size int,
    day: date  -- this column is injected from hive partitioning
);

Data for the table comes from hive partitioned files, something like:

/date=2024-05-05/1.parquet
/date=2024-05-06/2.parquet
/date=2024-05-06/3.parquet

Now the point is that day was calculated as the date part of created_at. So If a user runs

select * from eggs where created_at>'2024-05-06T00:00:00'

Then datafusion should only need to read /date=2024-05-06/3.parquet.

---

My question is, can I use an EquivalenceGroup defined by something like

    let created_at_to_date = ScalarFunction {
        fun: functions::BuiltinScalarFunction::Cast,
        args: vec![col("timestamp_x")],
        return_type: Arc::new(DataType::Date32),
    };

    let equivalence_group = vec![col("day"), created_at_to_date];
    let eq_group = EquivalenceGroup::new(equivalence_group);

? Is there a better / more canonical way to do this?

(Given I've got this far, perhaps I should just try it and see, but I've written the question now, so may as well post it.)

[samuelcolvin]

I tried this and it seems EquivalenceGroup doesn't work, datafusion assumes any EquivalenceGroup applies to columns that are in the projection, I'm looking for a way to communicate equivalence between columns that are including in the queries, and ones that are not.

[mustafasrepo]

Just to confirm, does select * from eggs where day>'2024-05-06' works as you expect? If this works, we should have a support for your use case theoretically. However, I think in the analysis we might have overfitted on Column equalities. If you can post a reproducer, we can examine the root cause why information added is not communicated.