Skip to content

Commit 238d52e

Browse files
committed
feat: accuracy-focused v1.1.0 (canonical types, tiered extraction, mapping)
Re-architect schema extraction and comparison so a red check always means a real problem and a green check is genuinely safe to deploy. - Add canonical type system (core/types.py): dialect-aware normalization of warehouse SQL types and Python hints into a neutral CanonicalType vocabulary; the validator compares canonical types, killing the bulk of false "type mismatch" warnings (dbt varchar == Pydantic str). - Tiered dbt extractor: catalog.json (real types, high confidence) -> sqlglot (proper SQL parse, trusted column names, SELECT * detection) -> regex (low confidence, never hard-fails). Manifest type enrichment. - Confidence-aware validation: incomplete sources (e.g. SELECT *) downgrade missing-column from critical to warning; low-confidence sources skip type warnings. Never falsely block a build. - Explicit mapping config (mapping.tables / mapping.columns) for when name heuristics aren't enough; normalized name matching (userId == user_id). - Clean base extractor (remove Python-specific type mapping; add _make_column). - Harden GitHub rate-limit handling against non-mapping headers. - Bump to 1.1.0; add sqlglot dependency; rewrite README + CHANGELOG.
1 parent fbcec5d commit 238d52e

14 files changed

Lines changed: 1462 additions & 546 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 51 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,55 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
77

88
## [Unreleased]
99

10+
## [1.1.0] - 2026-06-30
11+
12+
This release is focused on **accuracy** — making a red check always mean a real
13+
problem and a green check genuinely safe, so the tool can be trusted to gate a
14+
deploy.
15+
16+
### Added
17+
- **Canonical type system** (`core/types.py`): every extractor now normalizes
18+
its native types (warehouse SQL types, Python hints) into a shared, neutral
19+
vocabulary (`CanonicalType`). The validator compares canonical types instead
20+
of raw strings, eliminating the bulk of false "type mismatch" warnings
21+
(e.g. dbt `varchar` vs Pydantic `str` are now correctly equal).
22+
- Dialect-aware normalization: Snowflake `NUMBER(38,0)`→bigint, BigQuery
23+
`INT64`/`FLOAT64`, Redshift `SUPER`, Postgres `jsonb`, and more.
24+
- **Tiered dbt extraction** with graceful degradation:
25+
1. `catalog.json` — real warehouse types (high confidence).
26+
2. `sqlglot` — a proper SQL parser. Handles CTEs, `||`, window functions, and
27+
quoted identifiers that the old regex parser mangled. Detects `SELECT *`
28+
and flags the schema as incomplete.
29+
3. regex — last-resort best effort (low confidence, never hard-fails).
30+
- **Confidence-aware validation**: when source columns can't be fully resolved
31+
(e.g. `SELECT *`), a missing column is reported as a **warning, not a
32+
build-blocking critical**. Type warnings are suppressed for low-confidence
33+
(regex-tier) sources. This is the core false-positive guard.
34+
- **Explicit mapping config** (`mapping:` in `.retl-validator.yml`) for when
35+
name heuristics aren't enough — map a target table/column to a differently
36+
named source model/column:
37+
```yaml
38+
mapping:
39+
tables:
40+
user_analytics: user_analytics_summary
41+
columns:
42+
user_analytics:
43+
userId: user_id
44+
```
45+
- **Name normalization**: tables/columns now match across snake_case, camelCase
46+
and casing differences (`userId` == `user_id` == `USER_ID`).
47+
48+
### Changed
49+
- `Schema` now carries `confidence` and `is_complete` (via `metadata`).
50+
- `BaseExtractor` no longer contains Python-specific type mapping; type
51+
normalization lives in the canonical type system. Added `_make_column` helper.
52+
- Added `sqlglot` as a dependency (imported optionally; falls back to regex if
53+
absent).
54+
55+
### Fixed
56+
- Hardened GitHub API rate-limit handling against non-dict response headers
57+
(previously could raise when headers weren't a mapping).
58+
1059
## [1.0.5] - 2025-01-24
1160

1261
### Fixed
@@ -66,6 +115,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
66115
- Limited type inference from SQL
67116
- No support for complex nested types
68117

69-
[Unreleased]: https://github.com/OGsiji/data-contract-validator/compare/v1.0.5...HEAD
118+
[Unreleased]: https://github.com/OGsiji/data-contract-validator/compare/v1.1.0...HEAD
119+
[1.1.0]: https://github.com/OGsiji/data-contract-validator/releases/tag/v1.1.0
70120
[1.0.5]: https://github.com/OGsiji/data-contract-validator/releases/tag/v1.0.5
71121
[1.0.0]: https://github.com/OGsiji/data-contract-validator/releases/tag/v1.0.0

0 commit comments

Comments
 (0)