Some of the code snippets for SQL streaming in declarative pipelines in https://github.com/databricks-solutions/ai-dev-kit/blob/main/databricks-skills/databricks-spark-declarative-pipelines/references/sql/3-streaming-patterns.md are not correct
- Deduplication Pattern snippet
Bronze: Ingest all (may contain duplicates)
CREATE OR REFRESH STREAMING TABLE bronze_events AS
SELECT *, current_timestamp() AS _ingested_at
FROM STREAM read_files(...);
-- Silver: Deduplicate by event_id
CREATE OR REFRESH STREAMING TABLE silver_events_dedup AS
SELECT
event_id, user_id, event_type, event_timestamp, _ingested_at
FROM (
SELECT
*,
ROW_NUMBER() OVER (PARTITION BY event_id ORDER BY event_timestamp) AS rn
FROM STREAM bronze_events
)
WHERE rn = 1;
These types of window functions do not work with streaming tables.
- Time window deduplication snippet:
CREATE OR REFRESH STREAMING TABLE silver_events_dedup AS
SELECT
event_id, user_id, event_type, event_timestamp,
MIN(_ingested_at) AS first_seen_at
FROM STREAM bronze_events
GROUP BY
event_id, user_id, event_type, event_timestamp,
window(event_timestamp, '1 hour')
HAVING COUNT(*) >= 1;
The having clause should be just be "=1" instead of ">=1"
Some of the code snippets for SQL streaming in declarative pipelines in https://github.com/databricks-solutions/ai-dev-kit/blob/main/databricks-skills/databricks-spark-declarative-pipelines/references/sql/3-streaming-patterns.md are not correct
These types of window functions do not work with streaming tables.
The having clause should be just be "=1" instead of ">=1"