Skip to content

Commit 117e6f1

Browse files
mday-iomday-io
authored
Merge branch 'main' into feat/invalidate-cleanup-snapshots
2 parents 4ded31b + 13787d7 commit 117e6f1

31 files changed

Lines changed: 2171 additions & 186 deletions

‎docs/concepts/models/overview.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -184,7 +184,7 @@ This table lists each engine's support for `TABLE` and `VIEW` object comments:
184184
| DuckDB <=0.9 | N | N |
185185
| DuckDB >=0.10 | Y | Y |
186186
| MySQL | Y | Y |
187-
| MSSQL | N | N |
187+
| MSSQL | Y | Y |
188188
| Postgres | Y | Y |
189189
| GCP Postgres | Y | Y |
190190
| Redshift | Y | N |

‎docs/concepts/overview.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,4 +68,4 @@ SQLMesh automatically runs audits when you apply a `plan` to an environment, or
6868
## Infrastructure and orchestration
6969
Every company's data infrastructure is different. SQLMesh is flexible with regard to which engines and orchestration frameworks you use &mdash; its only requirement is access to the target SQL/analytics engine.
7070

71-
SQLMesh keeps track of model versions and processed data intervals using your existing infrastructure. SQLMesh it automatically creates a `sqlmesh` schema in your data warehouse for its internal metadata.
71+
SQLMesh keeps track of model versions and processed data intervals using your existing infrastructure. It automatically creates a `sqlmesh` schema in your data warehouse for its internal metadata.

‎docs/guides/configuration.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -170,6 +170,16 @@ The examples specify a Snowflake connection whose password is stored in an envir
170170
account: <account>
171171
```
172172

173+
!!! tip "Base64-encoded secrets"
174+
175+
If a secret is distributed base64-encoded in a single environment variable (for example a BigQuery service-account key), pipe the variable through the built-in `b64decode` filter to decode it to text inline:
176+
177+
```yaml
178+
keyfile_json: {{ env_var('BIGQUERY_KEY_B64') | b64decode }}
179+
```
180+
181+
A matching `b64encode` filter is also available. Both return UTF-8 text, so they are intended for string/JSON secrets rather than arbitrary binary data.
182+
173183
=== "Python"
174184

175185
Python accesses environment variables via the `os` library's `environ` dictionary.

‎docs/integrations/engines/azuresql.md‎

Lines changed: 35 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -14,23 +14,41 @@ pip install "sqlmesh[azuresql]"
1414
```
1515
pip install "sqlmesh[azuresql-odbc]"
1616
```
17+
Set `driver: "pyodbc"` in your connection options.
18+
19+
20+
#### Python Driver (Official Microsoft driver for Azure SQL):
21+
See [`mssql-python`](https://pypi.org/project/mssql-python/) for more information.
22+
23+
```
24+
pip install "sqlmesh[azuresql-mssql-python]"
25+
```
26+
27+
Set `driver: "mssql-python"` in your connection options. This driver supports
28+
[Entra ID auth](https://github.com/microsoft/mssql-python/wiki/Microsoft-Entra-ID-support),
29+
for detailed connection options see [this link](https://github.com/microsoft/mssql-python/wiki/Connection-to-SQL-Database).
30+
31+
!!! note
32+
The `mssql-python` driver [requires](https://pypi.org/project/mssql-python/) `python >= 3.10`.
33+
1734

1835
### Connection options
1936

20-
| Option | Description | Type | Required |
21-
| ----------------- | ---------------------------------------------------------------- | :----------: | :------: |
22-
| `type` | Engine type name - must be `azuresql` | string | Y |
23-
| `host` | The hostname of the Azure SQL server | string | Y |
24-
| `user` | The username / client ID to use for authentication with the Azure SQL server | string | N |
25-
| `password` | The password / client secret to use for authentication with the Azure SQL server | string | N |
26-
| `port` | The port number of the Azure SQL server | int | N |
27-
| `database` | The target database | string | N |
28-
| `charset` | The character set used for the connection | string | N |
29-
| `timeout` | The query timeout in seconds. Default: no timeout | int | N |
30-
| `login_timeout` | The timeout for connection and login in seconds. Default: 60 | int | N |
31-
| `appname` | The application name to use for the connection | string | N |
32-
| `conn_properties` | The list of connection properties | list[string] | N |
33-
| `autocommit` | Is autocommit mode enabled. Default: false | bool | N |
34-
| `driver` | The driver to use for the connection. Default: pymssql | string | N |
35-
| `driver_name` | The driver name to use for the connection. E.g., *ODBC Driver 18 for SQL Server* | string | N |
36-
| `odbc_properties` | The dict of ODBC connection properties. E.g., authentication: ActiveDirectoryServicePrincipal. See more [here](https://learn.microsoft.com/en-us/sql/connect/odbc/dsn-connection-string-attribute?view=sql-server-ver16). | dict | N |
37+
| Option | Description | Type | Required |
38+
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------: | :------: |
39+
| `type` | Engine type name - must be `azuresql` | string | Y |
40+
| `host` | The hostname of the Azure SQL server | string | Y |
41+
| `user` | The username / client ID to use for authentication with the Azure SQL server | string | N |
42+
| `password` | The password / client secret to use for authentication with the Azure SQL server | string | N |
43+
| `port` | The port number of the Azure SQL server | int | N |
44+
| `database` | The target database | string | N |
45+
| `charset` | The character set used for the connection | string | N |
46+
| `timeout` | The query timeout in seconds. Default: no timeout | int | N |
47+
| `login_timeout` | The timeout for connection and login in seconds. Default: 60 | int | N |
48+
| `login_attempts` | The number of reconnection attempts before failing. Default: 1 <br><br>*This option only applies to the `mssql-python` driver. | int | N |
49+
| `appname` | The application name to use for the connection | string | N |
50+
| `conn_properties` | The list of connection properties | list[string] | N |
51+
| `autocommit` | Is autocommit mode enabled. Default: false | bool | N |
52+
| `driver` | The driver to use for the connection. Default: pymssql | string | N |
53+
| `driver_name` | The driver name to use for the connection (e.g., *ODBC Driver 18 for SQL Server*). | string | N |
54+
| `odbc_properties` | The dict of ODBC connection properties (e.g., *authentication: ActiveDirectoryServicePrincipal*). See more [here](https://learn.microsoft.com/en-us/sql/connect/odbc/dsn-connection-string-attribute?view=sql-server-ver16).<br><br>*For the `mssql-python` driver, please see [this link](https://github.com/microsoft/mssql-python/wiki/Connection-to-SQL-Database). | dict | N |

‎docs/integrations/engines/databricks.md‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -271,6 +271,28 @@ The only relevant SQLMesh configuration parameter is the optional `catalog` para
271271
| `disable_databricks_connect` | When running locally, disable the use of Databricks Connect for all model operations (so use SQL Connector for all models) | bool | N |
272272
| `disable_spark_session` | Do not use SparkSession if it is available (like when running in a notebook). | bool | N |
273273

274+
### Query tags
275+
276+
Databricks SQL Connector supports per-query tags through the `query_tags` model session property. Specify tags as a `MAP(...)` of string keys to string or `NULL` values:
277+
278+
```sql
279+
MODEL (
280+
name sqlmesh_example.tagged_model,
281+
dialect databricks,
282+
session_properties (
283+
query_tags = MAP(
284+
'team', 'data-eng',
285+
'app', 'sqlmesh',
286+
'feature', NULL
287+
)
288+
)
289+
);
290+
291+
SELECT 1 AS id;
292+
```
293+
294+
Query tags are only applied when SQLMesh executes SQL through the Databricks SQL Connector. They are not applied when SQLMesh routes execution through Databricks Connect, a Databricks notebook SparkSession, or the Spark engine adapter.
295+
274296
## Model table properties to support altering tables
275297

276298
If you are making a change to the structure of a table that is [forward only](../../guides/incremental_time.md#forward-only-models), then you may need to add the following to your model's `physical_properties`:
@@ -288,3 +310,25 @@ MODEL (
288310

289311
If you attempt to alter without having this property set, you will get an error similar to `databricks.sql.exc.ServerOperationError: [DELTA_UNSUPPORTED_DROP_COLUMN] DROP COLUMN is not supported for your Delta table.`.
290312
[Databricks Documentation for more details](https://docs.databricks.com/en/delta/column-mapping.html#requirements).
313+
314+
## Liquid Clustering
315+
316+
SQLMesh supports the liquid clustering keywords AUTO and NONE
317+
318+
```sql
319+
MODEL (
320+
name sqlmesh_example.new_model,
321+
...
322+
clustered_by AUTO
323+
)
324+
```
325+
326+
To cluster by a column called `auto` or `none`, use parentheses and backticks
327+
328+
```sql
329+
MODEL (
330+
name sqlmesh_example.new_model,
331+
...
332+
clustered_by (`auto`)
333+
)
334+
```

0 commit comments

Comments
 (0)