[Bug]: Bug Report: TypeError in Serverless Execution, pyarrow.lib.ChunkedArray

[bhattarai842]

Prior Issue Search

• I have searched existing issues and confirmed this is not a duplicate.

Bug Description

The SAT execution fails specifically when run_on_serverless = true. The process terminates with a TypeError indicating a failure to convert a PyArrow ChunkedArray to a standard Array. This error does not occur when running the same process on a Classic Databricks Cluster.

TypeError: Cannot convert pyarrow.lib.ChunkedArray to pyarrow.lib.Array

Steps to Reproduce

Execute any SAT pipeline.

Expected Behavior

No response

Cloud Provider

AWS

Environment Details

No response

Additional Context

No response

[manish05485]

Yes I also faced same issue. The reason for me is Databricks serverless 5 now has python version 3.12 , it was working earlier on serverless version 4 which has python 3.11. The notebook which is failing is security_analysis_initializer

[shreelshah12]

We are aware of this issue with the latest serverless version, and while we work on a fix, we recommend running the workflows/notebooks in a serverless environment that does not use Python 3.12.