[Spark] Iceberg V3 row-lineage metadata columns fail with ValidationException during aggregate push-down

[abjain123]

Apache Iceberg version

1.11.0 (latest release)

Query engine

Spark

Please describe the bug 🐞

When querying Iceberg V3 tables in Spark, row-lineage metadata columns cannot be used in SQL aggregates when aggregate push-down is enabled. The queries fail at query-planning time with a ValidationException. The same queries succeed when aggregate push-down is disabled via spark.sql.iceberg.aggregate-push-down.enabled=false.

Steps to Reproduce:

1. Initialize a PySpark session with the Iceberg catalog and extensions enabled.
2. Create an Iceberg table with 'format-version'='3'.
3. Insert mock data.
4. Execute an aggregate function on _row_id

CAT = "ds2_standalone"
DB = "iceberg_v3_qe"
NS = f"{CAT}.{DB}"
TBL_SHORT = "v3_tc014_v3"
TBL = f"{NS}.{TBL_SHORT}"
OPTS = "USING ICEBERG TBLPROPERTIES ('format-version'='3', 'write.format.default'='parquet')"
spark.sql(f"CREATE DATABASE IF NOT EXISTS {NS}")
spark.sql(f"DROP TABLE IF EXISTS {TBL}")
spark.sql(f"CREATE TABLE {TBL} (id INT, name STRING) {OPTS}")
spark.sql(f"INSERT INTO {TBL} VALUES (1, 'Alice'), (2, 'Bob'), (3, 'Carol')")
spark.sql(f"SELECT _row_id from {TBL}").show() # Works
spark.sql(f"SELECT MAX(_row_id) FROM {TBL}").show() # Fails here

Metrics:
Spark Version: 4.1.1
Iceberg Table Format: V3

Error Trace:
org.apache.iceberg.exceptions.ValidationException: Cannot find field '_row_id' in struct: struct<1: id: optional int, 2: name: optional string>

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time