Skip to content

Issues with SQL streaming patterns code snippets #583

Description

@cyril-brk

Some of the code snippets for SQL streaming in declarative pipelines in https://github.com/databricks-solutions/ai-dev-kit/blob/main/databricks-skills/databricks-spark-declarative-pipelines/references/sql/3-streaming-patterns.md are not correct

  1. Deduplication Pattern snippet
Bronze: Ingest all (may contain duplicates)
CREATE OR REFRESH STREAMING TABLE bronze_events AS
SELECT *, current_timestamp() AS _ingested_at
FROM STREAM read_files(...);

-- Silver: Deduplicate by event_id
CREATE OR REFRESH STREAMING TABLE silver_events_dedup AS
SELECT
  event_id, user_id, event_type, event_timestamp, _ingested_at
FROM (
  SELECT
    *,
    ROW_NUMBER() OVER (PARTITION BY event_id ORDER BY event_timestamp) AS rn
  FROM STREAM bronze_events
)
WHERE rn = 1;

These types of window functions do not work with streaming tables.

  1. Time window deduplication snippet:
CREATE OR REFRESH STREAMING TABLE silver_events_dedup AS
SELECT
  event_id, user_id, event_type, event_timestamp,
  MIN(_ingested_at) AS first_seen_at
FROM STREAM bronze_events
GROUP BY
  event_id, user_id, event_type, event_timestamp,
  window(event_timestamp, '1 hour')
HAVING COUNT(*) >= 1;

The having clause should be just be "=1" instead of ">=1"

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions