Refactor + resolve & persist docs errors

CHANGELOG.md

@@ -1,3 +1,26 @@
+# dbt_jira v1.7.0
+
+[PR #175](https://github.com/fivetran/dbt_jira/pull/175) includes the following updates:
+
+## Schema/Data Change (--full-refresh required after upgrading)
+**3 total changes • 3 possible breaking changes**
+
+| Data Model(s) | Change type | Old | New | Notes |
+| ---------- | ----------- | -------- | -------- | ----- |
+| `jira__daily_issue_field_history` <br> `jira__daily_sprint_issue_history` <br> `jira__issue_enhanced` | Removed default columns (**Breaking Change**) | `story_points` and `story_point_estimate` included by default | Columns only included when added to `var('issue_field_history_columns')` | These fields are Scrum-specific custom fields not universally used. Add the relevant fields to the `issue_field_history_columns` variable to retain them. |
+| `jira__sprint_enhanced` | Removed default columns (**Breaking Change**) | `story_points_committed`, `story_points_end`, `story_points_completed`, `story_point_estimate_committed`, `story_point_estimate_end`, and `story_point_estimate_completed` included by default | Columns only included when `story points` and/or `story point estimate` are added to `var('issue_field_history_columns')` | These metrics depend on `story_points`/`story_point_estimate` being tracked in field history. Add the relevant fields to the `issue_field_history_columns` variable to retain them. |
+| `jira__daily_sprint_issue_history` <br> `jira__sprint_enhanced` | Column type change | `story_points` and `story_point_estimate` cast as `float` | Cast as `numeric` | Improves precision for fractional story point values. |
+
+## Bug Fix
+- Updated `stg_jira__sprint` to filter out soft-deleted sprint records using the `_fivetran_deleted` flag.
+
+## Under the Hood
+- Adds a new `convert_string_to_numeric` macro with warehouse-specific dispatch implementations to safely cast story point string values to numeric, handling edge cases such as comma-formatted numbers.
+
+## Documentation
+- Updates model descriptions to remove static documentation for `story_points`, `story_point_estimate`, and `team` columns. Removes story point-related column documentation from `jira__sprint_enhanced` as these columns are dynamic and only present when added to `var('issue_field_history_columns')`.
+  - To retain story point data, add `story points` and/or `story point estimate` to the `issue_field_history_columns` variable in your `dbt_project.yml` ([See the README for details](https://github.com/fivetran/dbt_jira?tab=readme-ov-file#define-daily-issue-field-history-columns)). Quickstart users can add these fields in the Issue Field History Columns setting.
+
 # dbt_jira v1.6.0
 
 [PR #170](https://github.com/fivetran/dbt_jira/pull/170) includes the following update:

README.md

@@ -66,7 +66,7 @@ Include the following jira package version in your `packages.yml` file:
 ```yaml
 packages:
   - package: fivetran/jira
-    version: [">=1.6.0", "<1.7.0"]
+    version: [">=1.7.0", "<1.8.0"]
 ```
 
 > All required sources and staging models are now bundled into this transformation package. Do not include `fivetran/jira_source` in your `packages.yml` since this package has been deprecated.

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'jira'
-version: '1.6.0'
+version: '1.7.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<3.0.0"]
 

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'jira_integration_tests'
-version: '1.6.0'
+version: '1.7.0'
 config-version: 2
 profile: 'integration_tests'
 

integration_tests/seeds/sprint.csv

@@ -1,4 +1,4 @@
-id,_fivetran_synced,board_id,complete_date,end_date,name,start_date
-4,2020-11-22 12:20:59.513,5,,2020-11-26 21:25:12,TNGP Sprint 1,2020-11-12 21:25:17.778
-1,2020-11-22 12:20:57.012,0,2020-11-10 20:52:38.262,2020-11-17 10:29:44.653,Sample Sprint 2,2020-11-03 10:09:44.653
-3,2020-11-22 12:20:57.298,7,,2020-11-26 21:55:00,Sample Sprint 2,2020-11-12 21:55:42.220
+id,_fivetran_synced,_fivetran_deleted,board_id,complete_date,end_date,name,start_date
+4,2020-11-22 12:20:59.513,false,5,,2020-11-26 21:25:12,TNGP Sprint 1,2020-11-12 21:25:17.778
+1,2020-11-22 12:20:57.012,false,0,2020-11-10 20:52:38.262,2020-11-17 10:29:44.653,Sample Sprint 2,2020-11-03 10:09:44.653
+3,2020-11-22 12:20:57.298,false,7,,2020-11-26 21:55:00,Sample Sprint 2,2020-11-12 21:55:42.220

integration_tests/tests/consistency/consistency_daily_sprint_issue_history.sql

@@ -10,9 +10,7 @@ with prod as (
         issue_id,
         sprint_id,
         sprint_name,
-        days_left_in_sprint,
-        story_point_estimate,
-        story_points
+        days_left_in_sprint
     from {{ target.schema }}_jira_prod.jira__daily_sprint_issue_history
 ),
 
@@ -22,9 +20,7 @@ dev as (
         issue_id,
         sprint_id,
         sprint_name,
-        days_left_in_sprint,
-        story_point_estimate,
-        story_points
+        days_left_in_sprint
     from {{ target.schema }}_jira_dev.jira__daily_sprint_issue_history
 ),
 

macros/convert_string_to_numeric.sql

@@ -0,0 +1,27 @@
+{%- macro convert_string_to_numeric(column) -%}
+{{ return(adapter.dispatch('convert_string_to_numeric', 'jira')(column)) }}
+{%- endmacro -%}
+
+{%- macro default__convert_string_to_numeric(column) -%}
+cast(replace({{ column }}, ',', '') as {{ dbt.type_numeric() }})
+{%- endmacro -%}
+
+{%- macro bigquery__convert_string_to_numeric(column) -%}
+cast(regexp_extract(replace({{ column }}, ',', ''), r'-?\d+(?:\.\d+)?') as {{ dbt.type_numeric() }})
+{%- endmacro -%}
+
+{%- macro snowflake__convert_string_to_numeric(column) -%}
+cast(regexp_substr(replace({{ column }}, ',', ''), '-?[0-9]+(\\.[0-9]+)?') as {{ dbt.type_numeric() }})
+{%- endmacro -%}
+
+{%- macro postgres__convert_string_to_numeric(column) -%}
+cast(substring(replace({{ column }}, ',', '') from '(-?[0-9]+(\.[0-9]+)?)') as {{ dbt.type_numeric() }})
+{%- endmacro -%}
+
+{%- macro redshift__convert_string_to_numeric(column) -%}
+cast(regexp_substr(replace({{ column }}, ',', ''), '(-?[0-9]+(\.[0-9]+)?)') as {{ dbt.type_numeric() }})
+{%- endmacro -%}
+
+{%- macro spark__convert_string_to_numeric(column) -%}
+cast(regexp_extract(replace({{ column }}, ',', ''), '(-?\\d+(\\.\\d+)?)', 0) as {{ dbt.type_numeric() }})
+{%- endmacro -%}

macros/split_sprint_ids.sql

@@ -1,10 +1,10 @@
-{% macro split_sprint_ids(using_teams) %}
+{% macro split_sprint_ids(using_teams, include_story_points=false, include_story_point_estimate=false) %}
 
-{{ adapter.dispatch('split_sprint_ids', 'jira') (using_teams) }}
+{{ adapter.dispatch('split_sprint_ids', 'jira') (using_teams, include_story_points, include_story_point_estimate) }}
 
 {% endmacro %}
 
-{% macro default__split_sprint_ids(using_teams) %}
+{% macro default__split_sprint_ids(using_teams, include_story_points=false, include_story_point_estimate=false) %}
 {# bigquery  #}
     select
         daily_issue_field_history.issue_id,
@@ -13,81 +13,101 @@
         daily_issue_field_history.date_week,
         daily_issue_field_history.status,
         {{ "daily_issue_field_history.team," if using_teams }}
-        cast(daily_issue_field_history.story_points as {{ dbt.type_float() }}) as story_points,
-        cast(daily_issue_field_history.story_point_estimate as {{ dbt.type_float() }}) as story_point_estimate,
+        {% if include_story_points %}
+        {{ jira.convert_string_to_numeric('daily_issue_field_history.story_points') }} as story_points,
+        {% endif %}
+        {% if include_story_point_estimate %}
+        {{ jira.convert_string_to_numeric('daily_issue_field_history.story_point_estimate') }} as story_point_estimate,
+        {% endif %}
         sprints as sprint_id
 
     from daily_issue_field_history
-    cross join 
+    cross join
         unnest(cast(split(sprint, ", ") as array<string>)) as sprints
 
 {% endmacro %}
 
-{% macro snowflake__split_sprint_ids(using_teams) %}
+{% macro snowflake__split_sprint_ids(using_teams, include_story_points=false, include_story_point_estimate=false) %}
     select
         daily_issue_field_history.issue_id,
         daily_issue_field_history.source_relation,
         daily_issue_field_history.date_day,
         daily_issue_field_history.date_week,
         daily_issue_field_history.status,
         {{ "daily_issue_field_history.team," if using_teams }}
-        cast(daily_issue_field_history.story_points as {{ dbt.type_float() }}) as story_points,
-        cast(daily_issue_field_history.story_point_estimate as {{ dbt.type_float() }}) as story_point_estimate,
+        {% if include_story_points %}
+        {{ jira.convert_string_to_numeric('daily_issue_field_history.story_points') }} as story_points,
+        {% endif %}
+        {% if include_story_point_estimate %}
+        {{ jira.convert_string_to_numeric('daily_issue_field_history.story_point_estimate') }} as story_point_estimate,
+        {% endif %}
         sprints.value as sprint_id
-    
+
     from daily_issue_field_history
-    cross join 
+    cross join
         table(flatten(STRTOK_TO_ARRAY(sprint, ', '))) as sprints
 
 {% endmacro %}
 
-{% macro redshift__split_sprint_ids(using_teams) %}
+{% macro redshift__split_sprint_ids(using_teams, include_story_points=false, include_story_point_estimate=false) %}
 	select
         unnest_sprint_id_array.issue_id,
         unnest_sprint_id_array.source_relation,
         unnest_sprint_id_array.date_day,
         unnest_sprint_id_array.date_week,
         unnest_sprint_id_array.status,
         {{ "unnest_sprint_id_array.team," if using_teams }}
-        unnest_sprint_id_array.story_points,
-        unnest_sprint_id_array.story_point_estimate,
+        {% if include_story_points %}
+        {{ jira.convert_string_to_numeric('unnest_sprint_id_array.story_points') }} as story_points,
+        {% endif %}
+        {% if include_story_point_estimate %}
+        {{ jira.convert_string_to_numeric('unnest_sprint_id_array.story_point_estimate') }} as story_point_estimate,
+        {% endif %}
         cast(sprint_id as {{ dbt.type_string() }}) as sprint_id
     from (
-        select 
+        select
             daily_issue_field_history.issue_id,
             daily_issue_field_history.source_relation,
             daily_issue_field_history.date_day,
             daily_issue_field_history.date_week,
             daily_issue_field_history.status,
             {{ "daily_issue_field_history.team," if using_teams }}
-            cast(daily_issue_field_history.story_points as {{ dbt.type_float() }}) as story_points,
-            cast(daily_issue_field_history.story_point_estimate as {{ dbt.type_float() }}) as story_point_estimate,
+            {% if include_story_points %}
+            daily_issue_field_history.story_points,
+            {% endif %}
+            {% if include_story_point_estimate %}
+            daily_issue_field_history.story_point_estimate,
+            {% endif %}
             split_to_array(sprint, ', ') as super_sprint_ids
 
         from daily_issue_field_history
     ) as unnest_sprint_id_array, unnest_sprint_id_array.super_sprint_ids as sprint_id
 
 {% endmacro %}
 
-{% macro postgres__split_sprint_ids(using_teams) %}
+{% macro postgres__split_sprint_ids(using_teams, include_story_points=false, include_story_point_estimate=false) %}
     select
         daily_issue_field_history.issue_id,
         daily_issue_field_history.source_relation,
         daily_issue_field_history.date_day,
         daily_issue_field_history.date_week,
         daily_issue_field_history.status,
         {{ "daily_issue_field_history.team," if using_teams }}
-        cast(daily_issue_field_history.story_points as {{ dbt.type_float() }}) as story_points,
-        cast(daily_issue_field_history.story_point_estimate as {{ dbt.type_float() }}) as story_point_estimate,
+        {% if include_story_points %}
+        {{ jira.convert_string_to_numeric('daily_issue_field_history.story_points') }} as story_points,
+        {% endif %}
+        {% if include_story_point_estimate %}
+        {{ jira.convert_string_to_numeric('daily_issue_field_history.story_point_estimate') }} as story_point_estimate,
+        {% endif %}
         sprints as sprint_id
 
     from daily_issue_field_history
-    cross join 
+    cross join
         unnest(string_to_array(sprint, ', ')) as sprints
 
 {% endmacro %}
 
-{% macro spark__split_sprint_ids(using_teams) %}
+{% macro spark__split_sprint_ids(using_teams, include_story_points=false, include_story_point_estimate=false) %}
 {# databricks and spark #}
     select
         daily_issue_field_history.issue_id,
@@ -96,10 +116,14 @@
         daily_issue_field_history.date_week,
         daily_issue_field_history.status,
         {{ "daily_issue_field_history.team," if using_teams }}
-        cast(daily_issue_field_history.story_points as {{ dbt.type_float() }}) as story_points,
-        cast(daily_issue_field_history.story_point_estimate as {{ dbt.type_float() }}) as story_point_estimate,
+        {% if include_story_points %}
+        {{ jira.convert_string_to_numeric('daily_issue_field_history.story_points') }} as story_points,
+        {% endif %}
+        {% if include_story_point_estimate %}
+        {{ jira.convert_string_to_numeric('daily_issue_field_history.story_point_estimate') }} as story_point_estimate,
+        {% endif %}
         sprints as sprint_id
     from daily_issue_field_history
     lateral view explode(split(sprint, ', ')) sprints_view as sprints
 
-{% endmacro %}
+{% endmacro %}

macros/staging/get_sprint_columns.sql

@@ -1,6 +1,7 @@
 {% macro get_sprint_columns() %}
 
 {% set columns = [
+    {"name": "_fivetran_deleted", "datatype": "boolean"},
     {"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
     {"name": "board_id", "datatype": dbt.type_int()},
     {"name": "complete_date", "datatype": dbt.type_timestamp()},

models/intermediate/field_history/int_jira__pivot_daily_field_history.sql

@@ -169,7 +169,7 @@ limit_to_relevant_fields as (
     from get_valid_dates
 
     where lower(field_id) = 'status'
-        or lower(field_name) in ('sprint', 'sprint_name', 'story points', 'story point estimate'
+        or lower(field_name) in ('sprint', 'sprint_name'
         {%- for col in issue_field_history_columns -%}
             ,'{{ (col|lower) }}'
         {%- endfor -%} )
@@ -217,7 +217,7 @@ int_jira__daily_field_history as (
 
 pivot_out as (
 
-    -- pivot out default columns (status and sprint) and others specified in the var(issue_field_history_columns)
+    -- pivot out default columns (status, sprint, sprint_name) and others specified in the var(issue_field_history_columns)
     -- only days on which a field value was actively changed will have a non-null value. the nulls will need to
     -- be backfilled in the final jira__daily_issue_field_history model
     select
@@ -227,12 +227,10 @@ pivot_out as (
         valid_starting_at_week,
         max(case when lower(field_id) = 'status' then field_value end) as status,
         max(case when lower(field_name) = 'sprint' then field_value end) as sprint,
-        max(case when lower(field_name) = 'sprint_name' then field_value end) as sprint_name,
-        max(case when lower(field_name) = 'story points' then field_value end) as story_points,
-        max(case when lower(field_name) = 'story point estimate' then field_value end) as story_point_estimate
+        max(case when lower(field_name) = 'sprint_name' then field_value end) as sprint_name
 
         {% for col in issue_field_history_columns -%}
-        {% if col|lower not in ['sprint', 'sprint_name', 'story points', 'story point estimate'] %}
+        {% if col|lower not in ['sprint', 'sprint_name'] %}
             , max(case when lower(field_name) = '{{ col|lower }}' then field_value end) as {{ dbt_utils.slugify(col) | replace(' ', '_') | lower }}
         {% endif %}
         {% endfor -%}

models/intermediate/field_history/int_jira__timestamp_field_history_scd.sql

@@ -1,7 +1,7 @@
 {%- set custom_columns = [] -%}
 {%- for col in var('issue_field_history_columns', []) -%}
     {%- set clean_col = dbt_utils.slugify(col) | replace(' ', '_') | lower -%}
-    {%- if clean_col not in ['sprint', 'sprint_name', 'story_points', 'story_point_estimate'] -%}
+    {%- if clean_col not in ['sprint', 'sprint_name'] -%}
         {%- do custom_columns.append(clean_col) -%}
     {%- endif -%}
 {%- endfor -%} 
@@ -28,16 +28,6 @@ with change_data as (
         , sprint_name
         , sum( case when sprint_name is null then 0 else 1 end) over ( partition by issue_id {{ jira.partition_by_source_relation() }}
             order by updated_at rows unbounded preceding) as sprint_name_field_partition
-        {% if 'story points' in var('issue_field_history_columns', []) | map('lower') | list %}
-        , story_points
-        , sum( case when story_points is null then 0 else 1 end) over ( partition by issue_id {{ jira.partition_by_source_relation() }}
-            order by updated_at rows unbounded preceding) as story_points_field_partition
-        {% endif %}
-        {% if 'story point estimate' in var('issue_field_history_columns', []) | map('lower') | list %}
-        , story_point_estimate
-        , sum( case when story_point_estimate is null then 0 else 1 end) over ( partition by issue_id {{ jira.partition_by_source_relation() }}
-            order by updated_at rows unbounded preceding) as story_point_estimate_field_partition
-        {% endif %}
 
         {% for col in custom_columns %}
         , {{ col }}
@@ -65,16 +55,6 @@ with change_data as (
         , first_value( sprint_name ) over (
             partition by issue_id, sprint_name_field_partition {{ jira.partition_by_source_relation() }}
             order by updated_at asc rows between unbounded preceding and current row) as sprint_name
-        {% if 'story points' in var('issue_field_history_columns', []) | map('lower') | list %}
-        , first_value( story_points ) over (
-            partition by issue_id, story_points_field_partition {{ jira.partition_by_source_relation() }}
-            order by updated_at asc rows between unbounded preceding and current row) as story_points
-        {% endif %}
-        {% if 'story point estimate' in var('issue_field_history_columns', []) | map('lower') | list %}
-        , first_value( story_point_estimate ) over (
-            partition by issue_id, story_point_estimate_field_partition {{ jira.partition_by_source_relation() }}
-            order by updated_at asc rows between unbounded preceding and current row) as story_point_estimate
-        {% endif %}
 
         {% for col in custom_columns %}
         -- grab the value that started this batch/partition