Allow `DatabricksSQL()` to connect to a cluster, not just a warehouse

[margaret-colville]

My organisation primarily uses clusters for connecting to Databricks, rather than warehouses.

The simple "DBI example" snippet from the README seems very close to what I need for my use case:

• Connect to Databricks
• Execute some arbitrary SQL to return as a table
• Run further queries/filters on it from there, return the results instantly or later when required

However, as far as I can tell, the DatabricksSQL() function used in the example only supports warehouses for connection, not clusters.

Could the function be updated (or could there be a second one) to have the option to connect via clusters instead?

It's possible my confusion is user error; I did try supplying a cluster ID to the warehouse_id parameter, but if it supports that as a solution, I may have done so incorrectly.

[zacdav-db]

Hey @margaret-colville currently DatabricksSQL only supports a warehouse, there are two reasons for this:

1. Warehouses are significantly more price/performant for query execution
2. The backend is the statement execution API, this only supports warehouses

Due to (2) its not possible to support what you are asking for unfortunately.

The alternative would be to use {odbc} (docs) - which will let you use a cluster.