`rewrite_table_path` does not update `manifest_length` in rewritten manifest lists, causing Trino to fail with "Incorrect file size"

[leeyam24]

Apache Iceberg version

1.11.0 (latest release)

Query engine

Trino

Please describe the bug 🐞

When using rewrite_table_path to migrate an Iceberg table between clusters (e.g., different HDFS prefixes), the rewritten manifest list files contain stale manifest_length values. Specifically, RewriteTablePathUtil.rewriteManifestList() copies each ManifestFile entry and only updates field 0 (the path), leaving field 1 (the length) unchanged from the original:

// core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java
for (ManifestFile file : manifestFiles) {
    ManifestFile newFile = file.copy();
    ((StructLike) newFile).set(0, newPath(newFile.path(), sourcePrefix, targetPrefix));
    writer.add(newFile);  // length (field 1) is NOT updated
    ...
}

The rewritten manifest .avro files have new path strings embedded inside them (data file paths with the new prefix). If the target prefix is longer than the source prefix, the rewritten manifest files are physically larger on disk than what manifest_length records in the manifest list. The Iceberg spec requires manifest_length (field 501) to reflect the actual byte length of the manifest file.

Trino's Iceberg connector enforces this field strictly and fails with:

Incorrect file size for file <manifest.avro> (end of stream not reached)

Spark does not validate manifest_length and succeeds.

To Reproduce

1. Have an Iceberg table on cluster A with prefix hdfs://cluster-a/warehouse/.
2. Run rewrite_table_path with source_prefix = hdfs://cluster-a/warehouse/ and target_prefix = hdfs://cluster-b/longer-warehouse-path/ (target prefix is longer).
3. distcp the staged files to cluster B and register the table.
4. Query the table (any snapshot) with Trino → fails with Incorrect file size for file (end of stream not reached).
5. Query the same table with Spark → succeeds.

Additional context: rewrite_manifests does not fix historical snapshots

Running rewrite_manifests after the migration does not resolve the issue for historical snapshots. BaseRewriteManifests.apply() only operates on base.currentSnapshot(), so it creates a new snapshot with correct manifest sizes, but all prior snapshots still point to manifest lists with stale manifest_length values. Querying any old snapshot via Trino will continue to fail.

Environment

• Apache Iceberg version: 1.11.0
• Query engine: Trino (fails), Spark (succeeds)
• Source/target filesystem: HDFS → HDFS (different clusters)

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time