Commit 238d52e
committed
feat: accuracy-focused v1.1.0 (canonical types, tiered extraction, mapping)
Re-architect schema extraction and comparison so a red check always means a
real problem and a green check is genuinely safe to deploy.
- Add canonical type system (core/types.py): dialect-aware normalization of
warehouse SQL types and Python hints into a neutral CanonicalType vocabulary;
the validator compares canonical types, killing the bulk of false
"type mismatch" warnings (dbt varchar == Pydantic str).
- Tiered dbt extractor: catalog.json (real types, high confidence) ->
sqlglot (proper SQL parse, trusted column names, SELECT * detection) ->
regex (low confidence, never hard-fails). Manifest type enrichment.
- Confidence-aware validation: incomplete sources (e.g. SELECT *) downgrade
missing-column from critical to warning; low-confidence sources skip type
warnings. Never falsely block a build.
- Explicit mapping config (mapping.tables / mapping.columns) for when name
heuristics aren't enough; normalized name matching (userId == user_id).
- Clean base extractor (remove Python-specific type mapping; add _make_column).
- Harden GitHub rate-limit handling against non-mapping headers.
- Bump to 1.1.0; add sqlglot dependency; rewrite README + CHANGELOG.1 parent fbcec5d commit 238d52e
14 files changed
Lines changed: 1462 additions & 546 deletions
File tree
- data_contract_validator
- core
- extractors
- tests
- test_core
- test_extractors
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
10 | 59 | | |
11 | 60 | | |
12 | 61 | | |
| |||
66 | 115 | | |
67 | 116 | | |
68 | 117 | | |
69 | | - | |
| 118 | + | |
| 119 | + | |
70 | 120 | | |
71 | 121 | | |
0 commit comments