Skip to content

Data files persisted without complex columns in the table schema hit "unexpected target column" on read #2618

@jordepic

Description

@jordepic

Apache Iceberg Rust version

None

Describe the bug

If we write a data file and then evolve our table's schema to add a complex type field like list, map, struct, proceeding to read the data file back will error out.

To Reproduce

  1. Create a table and write a data file (without column xs).
  2. Evolve the schema to add xs list (or a map/struct column).
  3. Read the table.
  4. Reading the older file errors with unexpected target column type List(...).

Expected behavior

A column present in the table schema but absent from a data file is filled with a typed NULL array of the correct (possibly nested) Arrow type via new_null_array, for nested types just as for primitives. The read should succeed with the added column all-NULL for older files.

Willingness to contribute

None

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions