Google Cloud SDK Connection Reinitializes with Each Query After Recent Update

[MrShops]

Question:
Has anyone experienced performance issues when running queries on BigQuery using the Google Cloud SDK after updating the dbt Power User extension? Specifically, I've noticed that since version 34.0, each query seems to reinitialize the connection, causing a significant delay.

Details:
Previously, in version 33.2, the session was persistent, and queries executed quickly. However, starting with version 34.0, it seems like a new connection is established for each query, leading to delays of 40-60 seconds before the query returns results. Our team has been utilising 33.2 until this point but after upgrading to dbt 1.8.x we were forced to switch leading to it now being a large blocker.

What I’ve observed:
It appears that the PythonBridge instance is recreated for each query execution in the newer versions. Is this the intended behaviour, or is there a way to maintain a persistent session like in the previous versions?

Plea for help
@mdesmet I'm aware you were the original author of this change seen here: #906

Is there anything I'm missing on how I can circumvent this?

[anandgupta42]

@MrShops is this still an issue?

[MrShops]

@anandgupta42 unfortunately yes, it’s frustrating enough to the point that I would be happy to put personal funds to see a fix put in place.

[anandgupta42]

@MrShops, removing the fix will significantly impact other adapters. Additionally, we have not received any complaints about BQ thus far, so we are unsure if the issue is related to your setup. Can you be available for a Zoom call to help us debug further? If so, please send me an email at anand [at] altimate [dot] ai , and we can continue from there.

[MrShops]

Thank you or for the quick response. I'll reach out over email :)