Apache Iceberg Rust version
None
Describe the bug
If we write a data file and then evolve our table's schema to add a complex type field like list, map, struct, proceeding to read the data file back will error out.
To Reproduce
- Create a table and write a data file (without column xs).
- Evolve the schema to add xs list (or a map/struct column).
- Read the table.
- Reading the older file errors with unexpected target column type List(...).
Expected behavior
A column present in the table schema but absent from a data file is filled with a typed NULL array of the correct (possibly nested) Arrow type via new_null_array, for nested types just as for primitives. The read should succeed with the added column all-NULL for older files.
Willingness to contribute
None
Apache Iceberg Rust version
None
Describe the bug
If we write a data file and then evolve our table's schema to add a complex type field like list, map, struct, proceeding to read the data file back will error out.
To Reproduce
Expected behavior
A column present in the table schema but absent from a data file is filled with a typed NULL array of the correct (possibly nested) Arrow type via new_null_array, for nested types just as for primitives. The read should succeed with the added column all-NULL for older files.
Willingness to contribute
None