DESCRIBE TABLE truncates complex STRUCT/ARRAY type strings in Databricks

[EhabEasee]

Summary

DatabricksEngineAdapter.columns() (inherited from BaseEngineAdapter) uses DESCRIBE TABLE to fetch column types. Databricks silently truncates the type string for deeply-nested STRUCT/ARRAY columns, replacing trailing fields with ... N more fields>>. This causes create_external_models to write incomplete type definitions into external_models.yaml, silently dropping fields with no warning.

Root cause

This is intentional Databricks behaviour. From the Databricks docs for DESCRIBE TABLE:

The non-JSON output of DESCRIBE TABLE can change, and new columns might be appended over time. Avoid relying on the output meeting a specific size or schema. For programmatic consumption, use DESCRIBE TABLE ... AS JSON.

Reproducible example

import itertools
from sqlmesh import Context

ctx = Context(paths=["<path-to-your-sqlmesh-project>"])
adapter = ctx.engine_adapter

adapter.execute("""
    CREATE TEMPORARY TABLE sqlmesh_describe_repro (
        id INT,
        nested ARRAY<STRUCT<
            field_one: STRING, field_two: STRING, field_three: STRING,
            field_four: STRING, field_five: STRING, field_six: STRING,
            field_seven: STRING, field_eight: STRING, field_nine: STRING,
            field_ten: STRING,
            inner_items: ARRAY<STRUCT<
                item_id: STRING,
                item_location: STRUCT<location_id: STRING, location_name: STRING>,
                item_warehouse: STRUCT<warehouse_id: STRING, warehouse_name: STRING>,
                item_quantity: DOUBLE,
                item_uom: STRING,
                item_expiration_date: TIMESTAMP,
                item_description: STRING
            >>,
            field_eleven: STRING, field_twelve: STRING, field_thirteen: STRING,
            field_fourteen: STRING, field_fifteen: STRING, field_sixteen: STRING,
            field_seventeen: STRING, field_eighteen: STRING,
            field_nineteen: STRING, field_twenty: STRING,
            field_twenty_one: STRING, field_twenty_two: STRING,
            field_twenty_three: STRING, field_twenty_four: STRING,
            field_twenty_five: DOUBLE, field_twenty_six: DOUBLE,
            field_twenty_seven: INT, field_twenty_eight: INT,
            field_twenty_nine: TIMESTAMP
        >>
    )
""")

cols = adapter.columns("sqlmesh_describe_repro")
print(cols["nested"].sql(dialect="databricks"))
# ARRAY<STRUCT<..., field_twenty_four: STRING, ... 5 more fields>>

[EhabEasee]

This has already been fixed in #5783. I was running an older version