Skip to content

Commit e9fee38

Browse files
iawanishiawanish
authored
Merge branch 'main' into fix/sqlglot-30-9-compatibility-clean
2 parents 3d993c9 + 9a25aa1 commit e9fee38

57 files changed

Lines changed: 3256 additions & 244 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Makefile‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -49,11 +49,13 @@ install-dev-dbt-%:
4949
$(MAKE) install-dev; \
5050
if [ "$$version" = "1.6.0" ]; then \
5151
echo "Applying overrides for dbt 1.6.0"; \
52-
$(PIP) install 'pydantic>=2.0.0' 'google-cloud-bigquery==3.30.0' 'databricks-sdk==0.28.0' --reinstall; \
52+
$(PIP) install 'pydantic>=2.0.0' 'google-cloud-bigquery==3.30.0' 'databricks-sdk==0.28.0' \
53+
'pyOpenSSL>=24.0.0' --reinstall; \
5354
fi; \
5455
if [ "$$version" = "1.7.0" ]; then \
5556
echo "Applying overrides for dbt 1.7.0"; \
56-
$(PIP) install 'databricks-sdk==0.28.0' --reinstall; \
57+
$(PIP) install 'databricks-sdk==0.28.0' \
58+
'pyOpenSSL>=24.0.0' --reinstall; \
5759
fi; \
5860
if [ "$$version" = "1.5.0" ]; then \
5961
echo "Applying overrides for dbt 1.5.0"; \

‎docs/concepts/models/overview.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -184,7 +184,7 @@ This table lists each engine's support for `TABLE` and `VIEW` object comments:
184184
| DuckDB <=0.9 | N | N |
185185
| DuckDB >=0.10 | Y | Y |
186186
| MySQL | Y | Y |
187-
| MSSQL | N | N |
187+
| MSSQL | Y | Y |
188188
| Postgres | Y | Y |
189189
| GCP Postgres | Y | Y |
190190
| Redshift | Y | N |

‎docs/concepts/models/python_models.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -369,6 +369,33 @@ def entrypoint(
369369
)
370370
```
371371

372+
Blueprint variables can also be used as **column names and column types** in the `columns` dictionary. For example, if each blueprint produces a model with a different set of column names and types, both can be parameterized using the same `@{variable}` syntax:
373+
374+
```python linenums="1"
375+
import pandas as pd
376+
from sqlmesh import ExecutionContext, model
377+
378+
@model(
379+
"@{customer}.metrics",
380+
kind="FULL",
381+
blueprints=[
382+
{"customer": "customer1", "primary_metric": "revenue", "primary_type": "int", "secondary_metric": "cost", "secondary_type": "double"},
383+
{"customer": "customer2", "primary_metric": "sales", "primary_type": "text", "secondary_metric": "profit", "secondary_type": "double"},
384+
],
385+
columns={
386+
"@{primary_metric}": "@{primary_type}",
387+
"@{secondary_metric}": "@{secondary_type}",
388+
},
389+
)
390+
def entrypoint(context: ExecutionContext, **kwargs) -> pd.DataFrame:
391+
return pd.DataFrame({
392+
context.blueprint_var("primary_metric"): [1],
393+
context.blueprint_var("secondary_metric"): [1.5],
394+
})
395+
```
396+
397+
Global variables (defined in the project config) can also be used as column names and types in the same way.
398+
372399
Note the use of curly brace syntax `@{customer}` in the model name above. It is used to ensure SQLMesh can combine the macro variable into the model name identifier correctly - learn more [here](../../concepts/macros/sqlmesh_macros.md#embedding-variables-in-strings).
373400

374401
Blueprint variable mappings can also be constructed dynamically, e.g., by using a macro: `blueprints="@gen_blueprints()"`. This is useful in cases where the `blueprints` list needs to be sourced from external sources, such as CSV files.

‎docs/concepts/overview.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,4 +68,4 @@ SQLMesh automatically runs audits when you apply a `plan` to an environment, or
6868
## Infrastructure and orchestration
6969
Every company's data infrastructure is different. SQLMesh is flexible with regard to which engines and orchestration frameworks you use &mdash; its only requirement is access to the target SQL/analytics engine.
7070

71-
SQLMesh keeps track of model versions and processed data intervals using your existing infrastructure. SQLMesh it automatically creates a `sqlmesh` schema in your data warehouse for its internal metadata.
71+
SQLMesh keeps track of model versions and processed data intervals using your existing infrastructure. It automatically creates a `sqlmesh` schema in your data warehouse for its internal metadata.

‎docs/guides/configuration.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -170,6 +170,16 @@ The examples specify a Snowflake connection whose password is stored in an envir
170170
account: <account>
171171
```
172172

173+
!!! tip "Base64-encoded secrets"
174+
175+
If a secret is distributed base64-encoded in a single environment variable (for example a BigQuery service-account key), pipe the variable through the built-in `b64decode` filter to decode it to text inline:
176+
177+
```yaml
178+
keyfile_json: {{ env_var('BIGQUERY_KEY_B64') | b64decode }}
179+
```
180+
181+
A matching `b64encode` filter is also available. Both return UTF-8 text, so they are intended for string/JSON secrets rather than arbitrary binary data.
182+
173183
=== "Python"
174184

175185
Python accesses environment variables via the `os` library's `environ` dictionary.

‎docs/integrations/dlt.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -28,12 +28,12 @@ This will create the configuration file and directories, which are found in all
2828

2929
SQLMesh will also automatically generate models to ingest data from the pipeline incrementally. Incremental loading is ideal for large datasets where recomputing entire tables is resource-intensive. In this case utilizing the [`INCREMENTAL_BY_TIME_RANGE` model kind](../concepts/models/model_kinds.md#incremental_by_time_range). However, these model definitions can be customized to meet your specific project needs.
3030

31-
#### Specify the path to the pipelines directory
31+
#### Specify the path to the pipelines working directory
3232

33-
The default location for dlt pipelines is `~/.dlt/pipelines/<pipeline_name>`. If your pipelines are in a [different directory](https://dlthub.com/docs/general-usage/pipeline#separate-working-environments-with-pipelines_dir), use the `--dlt-path` argument to specify the path explicitly:
33+
The default location for dlt pipeline working state is `~/.dlt/pipelines/<pipeline_name>`. If dlt stores your pipeline state in a [different pipelines working directory](https://dlthub.com/docs/general-usage/pipeline#separate-working-environments-with-pipelines_dir), use the `--dlt-path` argument to specify that directory explicitly. This should be the directory where dlt stores pipeline state, not the directory containing your pipeline scripts:
3434

3535
```bash
36-
sqlmesh init -t dlt --dlt-pipeline <pipeline-name> --dlt-path <pipelines-directory> dialect
36+
sqlmesh init -t dlt --dlt-pipeline <pipeline-name> --dlt-path <pipelines-working-directory> dialect
3737
```
3838

3939
### Generating models on demand
@@ -58,10 +58,10 @@ sqlmesh dlt_refresh <pipeline-name> --force
5858
sqlmesh dlt_refresh <pipeline-name> --table <dlt-table>
5959
```
6060

61-
- **Provide the explicit path to the pipelines directory** (using `--dlt-path`):
61+
- **Provide the explicit path to the pipelines working directory** (using `--dlt-path`):
6262

6363
```bash
64-
sqlmesh dlt_refresh <pipeline-name> --dlt-path <pipelines-directory>
64+
sqlmesh dlt_refresh <pipeline-name> --dlt-path <pipelines-working-directory>
6565
```
6666

6767
#### Configuration

‎docs/integrations/engines/azuresql.md‎

Lines changed: 35 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -14,23 +14,41 @@ pip install "sqlmesh[azuresql]"
1414
```
1515
pip install "sqlmesh[azuresql-odbc]"
1616
```
17+
Set `driver: "pyodbc"` in your connection options.
18+
19+
20+
#### Python Driver (Official Microsoft driver for Azure SQL):
21+
See [`mssql-python`](https://pypi.org/project/mssql-python/) for more information.
22+
23+
```
24+
pip install "sqlmesh[azuresql-mssql-python]"
25+
```
26+
27+
Set `driver: "mssql-python"` in your connection options. This driver supports
28+
[Entra ID auth](https://github.com/microsoft/mssql-python/wiki/Microsoft-Entra-ID-support),
29+
for detailed connection options see [this link](https://github.com/microsoft/mssql-python/wiki/Connection-to-SQL-Database).
30+
31+
!!! note
32+
The `mssql-python` driver [requires](https://pypi.org/project/mssql-python/) `python >= 3.10`.
33+
1734

1835
### Connection options
1936

20-
| Option | Description | Type | Required |
21-
| ----------------- | ---------------------------------------------------------------- | :----------: | :------: |
22-
| `type` | Engine type name - must be `azuresql` | string | Y |
23-
| `host` | The hostname of the Azure SQL server | string | Y |
24-
| `user` | The username / client ID to use for authentication with the Azure SQL server | string | N |
25-
| `password` | The password / client secret to use for authentication with the Azure SQL server | string | N |
26-
| `port` | The port number of the Azure SQL server | int | N |
27-
| `database` | The target database | string | N |
28-
| `charset` | The character set used for the connection | string | N |
29-
| `timeout` | The query timeout in seconds. Default: no timeout | int | N |
30-
| `login_timeout` | The timeout for connection and login in seconds. Default: 60 | int | N |
31-
| `appname` | The application name to use for the connection | string | N |
32-
| `conn_properties` | The list of connection properties | list[string] | N |
33-
| `autocommit` | Is autocommit mode enabled. Default: false | bool | N |
34-
| `driver` | The driver to use for the connection. Default: pymssql | string | N |
35-
| `driver_name` | The driver name to use for the connection. E.g., *ODBC Driver 18 for SQL Server* | string | N |
36-
| `odbc_properties` | The dict of ODBC connection properties. E.g., authentication: ActiveDirectoryServicePrincipal. See more [here](https://learn.microsoft.com/en-us/sql/connect/odbc/dsn-connection-string-attribute?view=sql-server-ver16). | dict | N |
37+
| Option | Description | Type | Required |
38+
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------: | :------: |
39+
| `type` | Engine type name - must be `azuresql` | string | Y |
40+
| `host` | The hostname of the Azure SQL server | string | Y |
41+
| `user` | The username / client ID to use for authentication with the Azure SQL server | string | N |
42+
| `password` | The password / client secret to use for authentication with the Azure SQL server | string | N |
43+
| `port` | The port number of the Azure SQL server | int | N |
44+
| `database` | The target database | string | N |
45+
| `charset` | The character set used for the connection | string | N |
46+
| `timeout` | The query timeout in seconds. Default: no timeout | int | N |
47+
| `login_timeout` | The timeout for connection and login in seconds. Default: 60 | int | N |
48+
| `login_attempts` | The number of reconnection attempts before failing. Default: 1 <br><br>*This option only applies to the `mssql-python` driver. | int | N |
49+
| `appname` | The application name to use for the connection | string | N |
50+
| `conn_properties` | The list of connection properties | list[string] | N |
51+
| `autocommit` | Is autocommit mode enabled. Default: false | bool | N |
52+
| `driver` | The driver to use for the connection. Default: pymssql | string | N |
53+
| `driver_name` | The driver name to use for the connection (e.g., *ODBC Driver 18 for SQL Server*). | string | N |
54+
| `odbc_properties` | The dict of ODBC connection properties (e.g., *authentication: ActiveDirectoryServicePrincipal*). See more [here](https://learn.microsoft.com/en-us/sql/connect/odbc/dsn-connection-string-attribute?view=sql-server-ver16).<br><br>*For the `mssql-python` driver, please see [this link](https://github.com/microsoft/mssql-python/wiki/Connection-to-SQL-Database). | dict | N |

0 commit comments

Comments
 (0)