shopify__discounts table skipped casuses downstream models to be skipped

[drago-ssh]

Hi,

After talking to Mriganka he suggested I raise a blank issue here to see if we can solve it.

Problem
I have updated our shopify fivetrann dbt package from 0.13 to 0.19.
When doing the dbt build I notice that I get some skipped models. I would like to make sure that everything builds as before.

The logs don't quite say why it was skipped so I'm left wondering.

I also notice some tests failing, specifically these 2:

For the not null test I get this error:
12:32:07 Began running node test.shopify_source.not_null_stg_shopify__media_image_image_id.a62b445b8b
12:32:07 559 of 1059 START test not_null_stg_shopify__media_image_image_id .............. [RUN]
12:32:07 Began compiling node test.shopify_source.not_null_stg_shopify__media_image_image_id.a62b445b8b
12:32:07 Writing injected SQL for node "test.shopify_source.not_null_stg_shopify__media_image_image_id.a62b445b8b"
12:32:07 Began executing node test.shopify_source.not_null_stg_shopify__media_image_image_id.a62b445b8b
12:32:07 Writing runtime sql for node "test.shopify_source.not_null_stg_shopify__media_image_image_id.a62b445b8b"
12:32:07 On test.shopify_source.not_null_stg_shopify__media_image_image_id.a62b445b8b: /* {"app": "dbt", "dbt_version": "2025.6.25+dd77c38", "profile_name": "user", "target_name": "default", "node_id": "test.shopify_source.not_null_stg_shopify__media_image_image_id.a62b445b8b"} */

select
  count(*) as failures,
  count(*) != 0 as should_warn,
  count(*) != 0 as should_error
from (

select image_id
from rokoko-dw.dbt_dc_stg_shopify.stg_shopify__media_image
where image_id is null

) dbt_internal_test

12:32:07 Opening a new connection, currently in state closed
12:32:08 BigQuery adapter: https://console.cloud.google.com/bigquery?project=rokoko-dw&j=bq:EU:15ca1972-1097-4996-bb94-8c1cd2f2ed14&page=queryresults
12:32:08 559 of 1059 FAIL 275 not_null_stg_shopify__media_image_image_id ................ [�[31mFAIL 275�[0m in 1.51s]
12:32:08 Finished running node test.shopify_source.not_null_stg_shopify__media_image_image_id.a62b445b8b
12:37:15 �[31mFailure in test not_null_stg_shopify__media_image_image_id (models/stg_shopify.yml)�[0m
12:37:15 Got 275 results, configured to fail if != 0
12:37:15 compiled code at target/compiled/shopify_source/models/stg_shopify.yml/not_null_stg_shopify__media_image_image_id.sql

For the unique combination test I get:

12:33:15 Began running node test.shopify.dbt_utils_unique_combination_of_columns_int_shopify__discount_code_enriched_code__source_relation.43df67b091
12:33:15 810 of 1059 START test dbt_utils_unique_combination_of_columns_int_shopify__discount_code_enriched_code__source_relation [RUN]
12:33:15 Began compiling node test.shopify.dbt_utils_unique_combination_of_columns_int_shopify__discount_code_enriched_code__source_relation.43df67b091
12:33:15 Writing injected SQL for node "test.shopify.dbt_utils_unique_combination_of_columns_int_shopify__discount_code_enriched_code__source_relation.43df67b091"
12:33:15 Began executing node test.shopify.dbt_utils_unique_combination_of_columns_int_shopify__discount_code_enriched_code__source_relation.43df67b091
12:33:15 Writing runtime sql for node "test.shopify.dbt_utils_unique_combination_of_columns_int_shopify__discount_code_enriched_code__source_relation.43df67b091"
12:33:15 On test.shopify.dbt_utils_unique_combination_of_columns_int_shopify__discount_code_enriched_code__source_relation.43df67b091: /* {"app": "dbt", "dbt_version": "2025.6.25+dd77c38", "profile_name": "user", "target_name": "default", "node_id": "test.shopify.dbt_utils_unique_combination_of_columns_int_shopify__discount_code_enriched_code__source_relation.43df67b091"} */

select
  count(*) as failures,
  count(*) != 0 as should_warn,
  count(*) != 0 as should_error
from (

with __dbt__cte__int_shopify__discount_code_enriched as (
with discount_redeem_codes as (

select *
from `rokoko-dw`.`dbt_dc_stg_shopify`.`stg_shopify__discount_redeem_code`

),

discount_applications as (

select *
from `rokoko-dw`.`dbt_dc_stg_shopify`.`stg_shopify__discount_application`

),

unified_discount_codes as (

select
    discount_code_id,
    'basic' as discount_type,
    applies_once_per_customer,
    usage_count,
    codes_count,
    codes_precision,
    combines_with_order_discounts,
    combines_with_product_discounts,
    combines_with_shipping_discounts,
    created_at,
    customer_selection_all_customers,
    ends_at, 
    starts_at,
    status,
    title,
    total_sales_amount,
    total_sales_currency_code,
    updated_at,
    usage_limit,
    source_relation
from `rokoko-dw`.`dbt_dc_stg_shopify`.`stg_shopify__discount_code_basic`

union all

select
    discount_code_id,
    'bxgy' as discount_type, 
    applies_once_per_customer,
    usage_count,
    codes_count,
    codes_precision,
    combines_with_order_discounts,
    combines_with_product_discounts,
    combines_with_shipping_discounts,
    created_at,
    customer_selection_all_customers,
    ends_at,
    starts_at,
    status,
    title,
    total_sales_amount,
    total_sales_currency_code,
    updated_at,
    usage_limit,
    source_relation
from `rokoko-dw`.`dbt_dc_stg_shopify`.`stg_shopify__discount_code_bxgy`

union all

select
    discount_code_id,
    'free_shipping' as discount_type, 
    applies_once_per_customer,
    usage_count,
    codes_count,
    codes_precision,
    combines_with_order_discounts,
    combines_with_product_discounts,
    combines_with_shipping_discounts,
    created_at,
    customer_selection_all_customers,
    ends_at, 
    starts_at,
    status,
    title,
    total_sales_amount,
    total_sales_currency_code,
    updated_at,
    usage_limit,
    source_relation
from `rokoko-dw`.`dbt_dc_stg_shopify`.`stg_shopify__discount_code_free_shipping`

),

discounts_with_codes as (

select
    discount_redeem_codes.code,
    unified_discount_codes.*
from unified_discount_codes 
left join discount_redeem_codes 
    on unified_discount_codes.discount_code_id = discount_redeem_codes.discount_id
    and unified_discount_codes.source_relation = discount_redeem_codes.source_relation

),

discounts_with_applications as (

select
    discounts_with_codes.*,
    discount_applications.allocation_method,
    discount_applications.description,
    discount_applications.target_selection,
    discount_applications.target_type,
    discount_applications.type as application_type,
    discount_applications.value,
    discount_applications.value_type
from discounts_with_codes
left join discount_applications 
    on discounts_with_codes.code = discount_applications.code
    and discounts_with_codes.source_relation = discount_applications.source_relation

)

select *
from discounts_with_applications
), validation_errors as (

select
    code, source_relation
from __dbt__cte__int_shopify__discount_code_enriched
group by code, source_relation
having count(*) > 1

)

select *
from validation_errors

) dbt_internal_test

12:33:15 Opening a new connection, currently in state closed
12:33:17 BigQuery adapter: https://console.cloud.google.com/bigquery?project=rokoko-dw&j=bq:EU:bc79f7c1-c751-4d05-889c-c26999c9b3bd&page=queryresults
12:33:17 810 of 1059 FAIL 4 dbt_utils_unique_combination_of_columns_int_shopify__discount_code_enriched_code__source_relation [�[31mFAIL 4�[0m in 2.16s]
12:33:17 Finished running node test.shopify.dbt_utils_unique_combination_of_columns_int_shopify__discount_code_enriched_code__source_relation.43df67b091
12:37:15 �[31mFailure in test dbt_utils_unique_combination_of_columns_int_shopify__discount_code_enriched_code__source_relation (models/intermediate/intermediate.yml)�[0m
12:37:15 Got 4 results, configured to fail if != 0
12:37:15 compiled code at target/compiled/shopify/models/intermediate/intermediate.yml/dbt_utils_unique_combination_o_5368aaf9111a76f0f2253a79922522a8.sql

[fivetran-joemarkiewicz]

@drago-ssh thanks for opening this issue and sorry to see you're running into build failures after upgrading. It looks like you have three questions:

1. Some models are being skipped and you're not sure why
2. A not_null test is failing for the stg_shopify__media_image model
3. A unique combination of columns test is failing for the int_shopify__discount_code_enriched model

A few observations and follow up questions:

Some models are being skipped and you're not sure why

• It's not entirely clear from the screenshot. Would you be able to clarify which models are being skipped? Also, do you see any mention of upstream model failing? The only reason I would believe you would see skipped models is if an upstream model has failed. If we can identify which models have failed we should be able to address that issue and ensure the models aren't skipped. Additionally, I would be curious if you see the same types of errors if you execute a dbt run instead of a dbt build. The tests won't run, but I'm curious if the failing tests are somehow the cause of the skipped models.
• Potential Solution: Unsure, but will find out more upon response to the above questions. Additionally, it may be that addressing the below issues will resolve the skipped models issue.

A not_null test is failing for the stg_shopify__media_image model

• These not_null tests are defined here and are meant to fail if either the media_id or image_id are null. When looking at your staging model or raw data do you see these fields are null? If so, I would wonder if that's expected. My initial thought is that maybe the image_id test should be removed, but will dive deeper into this after hearing if your data resembles this behavior in certain cases.
• Potential Solution: Remove the not_null test on the image_id field within the stg_shopify__media_image model.

A unique combination of columns test is failing for the int_shopify__discount_code_enriched model

• This test is defined here and is checking to ensure the code and source_relation columns are unique. When you inspect the results of your shopify__discounts end model do you see duplicates of the code field? I notice that in the v0.13.0 (and still in v0.19.0) release we have this test defined to warn at the end model instead of fail as it's best practice to have unique discount codes. However, it's not required. If you're seeing a warning in that end model and the duplicate codes are expected then it probably will be best for us to address this in the package by updating the intermediate model to either remove or adjust the severity of the code uniqueness test.
• Potential Solution: Update the int_shopify__discount_code_enriched model test to either remove or adjust the severity of the code uniqueness test.

[drago-ssh]

It's not entirely clear from the screenshot. Would you be able to clarify which models are being skipped? Also, do you see any mention of upstream model failing? The only reason I would believe you would see skipped models is if an upstream model has failed. If we can identify which models have failed we should be able to address that issue and ensure the models aren't skipped. Additionally, I would be curious if you see the same types of errors if you execute a dbt run instead of a dbt build. The tests won't run, but I'm curious if the failing tests are somehow the cause of the skipped models.

Potential Solution: Unsure, but will find out more upon response to the above questions. Additionally, it may be that addressing the below issues will resolve the skipped models issue.

The skipped models are:

shopify__discounts (dbt package)
stg_shopify_discounts (self made)
mart_ecom_shopify_discounts (self made, basically select * from stg_shopify_discounts)

If I do dbt run then it works

Another thing I notice is that stg_shopify_discounts has a only null's for created_at which is why it fails since I'm using that field to join with the exchange rates on a given day. Why would shopify__discounts have only nulls in that field?

select 
    * except (
             source_relation
            , avg_order_discount_amount
            , total_order_discount_amount
            , total_abandoned_checkout_discount_amount
            , total_order_line_items_price
            , total_order_shipping_cost
            , total_abandoned_checkout_shipping_price
            , total_order_refund_amount
            )
    , case
        when source_relation = 'fivetran_shopify_usa'
        then 'usa'
        when source_relation = 'fivetran_shopify_jp'
        then 'jp'
        when source_relation = 'fivetran_shopify_row'
        then 'row'
        when source_relation = 'fivetran_shopify_eu'
        then 'eu'
    end as shopify_store
    , avg_order_discount_amount / coalesce(exchange_rate, 1) as avg_order_discount_amount
    , total_order_discount_amount / coalesce(exchange_rate, 1) as total_order_discount_amount
    , total_abandoned_checkout_discount_amount / coalesce(exchange_rate, 1) as total_abandoned_checkout_discount_amount
    , total_order_line_items_price / coalesce(exchange_rate, 1) as total_order_line_items_price
    , total_order_shipping_cost / coalesce(exchange_rate, 1) as total_order_shipping_cost
    , total_abandoned_checkout_shipping_price / coalesce(exchange_rate, 1) as total_abandoned_checkout_shipping_price
    , total_order_refund_amount / coalesce(exchange_rate, 1) as total_order_refund_amount
from {{ ref('shopify__discounts') }} as discounts

These not_null tests are defined here and are meant to fail if either the media_id or image_id are null. When looking at your staging model or raw data do you see these fields are null? If so, I would wonder if that's expected. My initial thought is that maybe the image_id test should be removed, but will dive deeper into this after hearing if your data resembles this behavior in certain cases.

Potential Solution: Remove the not_null test on the image_id field within the stg_shopify__media_image model.

Actually do not need any media specific data so I would believe this test/model can just be disregarded/removed.

This test is defined here and is checking to ensure the code and source_relation columns are unique. When you inspect the results of your shopify__discounts end model do you see duplicates of the code field? I notice that in the v0.13.0 (and still in v0.19.0) release we have this test defined to warn at the end model instead of fail as it's best practice to have unique discount codes. However, it's not required. If you're seeing a warning in that end model and the duplicate codes are expected then it probably will be best for us to address this in the package by updating the intermediate model to either remove or adjust the severity of the code uniqueness test.

Potential Solution: Update the int_shopify__discount_code_enriched model test to either remove or adjust the severity of the code uniqueness test.

Seems like shopify__discount has a bunch of nulls.

select * from {{ ref('int_shopify__discount_code_enriched') }} will also generate a lot of nulls.

I wonder if this could be because we have multiple fivetran_shopify connectors?

[fivetran-joemarkiewicz]

If I do dbt run then it works

• This confirms to me that the failing dbt tests are the reason these models are being skipped. We should be able to address this by making the above proposed changes/removals of the tests.

Seems like shopify__discount has a bunch of nulls.

• Thanks for sharing. That is interesting. I wonder if this could be because you don't have all the newer DISCOUNT_* tables from the connector. From the April 2025 release notes you can see the list of newer DISCOUNT_* tables which take the place of the previous DISCOUNT_CODE table. Would you be able to confirm that you have these (or at least a few of them) syncing and that data is available in them.

[drago-ssh]

Would you be able to confirm that you have these (or at least a few of them) syncing and that data is available in them.

Before updating to 0.19 - I decided to select * the tables from the schema tab in the connector settings on fivetran. So I am asking fivetran to give everything what can be fetched.

Based on the link you shared with the release notes, I went ahead and checked wether these tables land in our data lake. Below you can see which ones end up there and which ones are not there. For all the ✅ ones - I also made sure that the tables are populated with data.

DISCOUNT_AUTOMATIC_APP ❌
DISCOUNT_AUTOMATIC_BASIC ✅ (but only in 1 shop out of 4, meaning 1 fivetran shopify connection out of 4)
DISCOUNT_AUTOMATIC_BXGY ✅ (but only in 1 shop out of 4, meaning 1 fivetran shopify connection out of 4)
DISCOUNT_AUTOMATIC_FREE_SHIPPING ❌
DISCOUNT_CODE_APP ❌
DISCOUNT_CODE_BASIC ✅
DISCOUNT_CODE_BXGY ❌
DISCOUNT_CODE_FREE_SHIPPING ✅
DISCOUNT_REDEEM_CODE ✅
DISCOUNT_SHAREABLE_URL ✅
DISCOUNT_DESTINATION_SELECTION_COUNTRY ❌
DISCOUNT_CUSTOMER_SEGMENT_SELECTION ❌
DISCOUNT_CUSTOMER_SELECTION ✅
DISCOUNT_CUSTOMER_GETS_COLLECTION ✅ (but only in 1 shop out of 4, meaning 1 fivetran shopify connections out of 4)
DISCOUNT_CUSTOMER_GETS_PRODUCT ✅
DISCOUNT_CUSTOMER_GETS_PRODUCT_VARIANT ✅ (but only in 3 shop out of 4, meaning 3 fivetran shopify connections out of 4)
DISCOUNT_CUSTOMER_BUYS_COLLECTION ❌
DISCOUNT_CUSTOMER_BUYS_PRODUCT ✅ (but only in 1 shop out of 4, meaning 1 shopify fivetran connections out of 4)
MEDIA ✅
MEDIA_IMAGE ✅
PRODUCT_MEDIA ✅
EXTERNAL_VIDEO ❌
VIDEO ❌
MODEL_3_D ❌
MEDIA_SOURCE ❌
MEDIA_WANRING ❌
MEDIA_FILE_ERROR ❌
MEDIA_ERROR ❌
PRODUCT_VARIANT_MEDIA ❌

[drago-ssh]

In regards to disabling the 2 tests that fail. It does not seem like I can disable them in dbt_project.yml as the shopify dbt package overrides everything I would disable there?

11:34:37 561 of 1059 START test not_null_stg_shopify__media_image_image_id .............. [RUN]
11:34:38 561 of 1059 FAIL 275 not_null_stg_shopify__media_image_image_id ................ [�[31mFAIL 275�[0m in 1.30s]
11:39:50 �[31mFailure in test not_null_stg_shopify__media_image_image_id (models/stg_shopify.yml)�[0m
11:39:50 Got 275 results, configured to fail if != 0
11:39:50 compiled code at target/compiled/shopify_source/models/stg_shopify.yml/not_null_stg_shopify__media_image_image_id.sql

(models/stg_shopify.yml) is something that the package creates as it does not exist in the models folder? How would you go about telling the package which tests to disregard?

[fivetran-joemarkiewicz]

@drago-ssh I'm still looking into why you would be seeing so many nulls with the tables you have in your destination for the discounts end model.

Regarding the disabling of the tests. You can disable them in your root dbt_project.yml (not the package's) and it won't be overwritten. You would use the following in your dbt_project.yml

tests:
  shopify_source:
    not_null_stg_shopify__media_image_image_id:
      +enabled: false
  shopify:
    intermediate:
      dbt_utils_unique_combination_of_columns_int_shopify__discount_code_enriched_code__source_relation:
        +enabled: false

We will plan to deploy a release in the coming sprint to remove these entirely from the dbt package.

[drago-ssh]

Thank you!

So the only way I manged to make it build is if I disregard these 3 tests in my dbt_project.yml.

tests:
  shopify_source:
    not_null_stg_shopify__media_image_image_id:
      +enabled: false
  shopify:
    unique_shopify__discounts_discounts_unique_key:
      +enabled: false
    intermediate:
      dbt_utils_unique_combination_of_columns_int_shopify__discount_code_enriched_code__source_relation:
        +enabled: false

But of course this is problematic because it causes downstream problems since shopify__discounts is packed with nulls

Is it because we are using the wrong source tables? Can't see many of the ones mentioned in this post

heres my src file

models
/
01_staging
/
shopify
/
src_shopify.yml

version: 2

sources:
  # Shared table configuration for all Shopify stores
  - name: fivetran_shopify_eu
    schema: fivetran_shopify_eu
    database: "{{ var('shopify_database', target.database) }}"
    loader: Fivetran
    config:
      enabled: true
    freshness:
      warn_after: {count: 24, period: hour}
    loaded_at_field: _fivetran_synced
    tables: &shopify_tables
      - name: discount_code_bxgy
      - name: media_image
      - name: discount_redeem_code
      - name: abandoned_applied_discount
      - name: abandoned_checkout
      - name: abandoned_checkout_discount_code
      - name: abandoned_checkout_line
      - name: abandoned_checkout_shipping_line
      - name: abandoned_checkout_tax_line
      - name: abandoned_checkout_url_tag
      - name: collection
      - name: collection_product
      - name: customer
        columns: 
          - name: id
            data_tests:
              - unique
              - not_null
      - name: customer_address
      - name: customer_segment
      - name: customer_tag
      - name: discount_allocation
      - name: discount_application
      - name: discount_code
      - name: discount_code_basic
      - name: discount_code_free_shipping
      - name: draft_order
      - name: fulfillment
      - name: inventory_item
      - name: inventory_level
      - name: inventory_quantity
      - name: location
      - name: metafield
      - name: media
      - name: order
        columns: 
          - name: id
            data_tests:
              - unique
              - not_null
      - name: order_adjustment
      - name: order_discount_code
      - name: order_fulfillment
        identifier: fulfillment    # Actual table in DB

      - name: fulfillment_event
      - name: fulfillment_order_line
      - name: order_line
      - name: order_line_refund
      - name: order_note_attribute
      - name: order_shipping_line
      - name: order_shipping_tax_line
      - name: order_tag
      - name: order_url_tag
      - name: price_rule
      - name: product
      - name: product_image
      - name: product_media
      - name: product_tag
      - name: product_variant
      - name: order_refund
        identifier: refund

      - name: order_risk
      - name: refund
      - name: shop
      - name: tax_line
      - name: tender_transaction
      - name: transaction
      - name: shopify_inventory_levels_wms

  # Other stores using the same table configuration
  - name: fivetran_shopify_jp
    schema: fivetran_shopify_jp
    database: "{{ var('shopify_database', target.database) }}"
    loader: Fivetran
    freshness:
      warn_after: {count: 10, period: day}
    loaded_at_field: _fivetran_synced
    config:
      enabled: true
    tables: *shopify_tables

  - name: fivetran_shopify_row
    schema: fivetran_shopify_row
    database: "{{ var('shopify_database', target.database) }}"
    loader: Fivetran
    freshness:
      warn_after: {count: 24, period: hour}
    loaded_at_field: _fivetran_synced
    config:
      enabled: true
    tables: *shopify_tables

  - name: fivetran_shopify_usa
    schema: fivetran_shopify_usa
    database: "{{ var('shopify_database', target.database) }}"
    loader: Fivetran
    freshness:
      warn_after: {count: 24, period: hour}
    loaded_at_field: _fivetran_synced
    config:
      enabled: true
    tables: *shopify_tables

[drago-ssh]

Would appreciate any updates on this as our data has been stale for more than a week now :(
Happy to jump on a call if needed.

in case you might want to take a look at the project yml

# Name your project! Project names should contain only lowercase characters
# and underscores. A good package name should reflect your organization's
# name or the intended use of these models
name: 'rokoko'
version: '1.0.0'
config-version: 2

# This setting configures which "profile" dbt uses for this project.
profile: 'default'

# These configurations specify where dbt should look for different types of files.
# The `source-paths` config, for example, states that models in this project can be
# found in the "models/" directory. You probably won't need to change these!
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]

target-path: "target"  # directory which will store compiled SQL files
clean-targets:         # directories to be removed by `dbt clean`
  - "target"
  - "dbt_packages"


# Configuring models
# Full documentation: https://docs.getdbt.com/docs/configuring-models

# In this example config, we tell dbt to build all models in the example/ directory
# as tables. These settings can be overridden in the individual model files
# using the `{{ config(...) }}` macro.
models:
  rokoko:
    # Applies to all files under models/marts
    03_marts:
      +materialized: table
      +tags: ['mart']
      product:
        +tags: ['daily']
      sales:
        sales_forecast:
          +tags: ['daily']
    01_staging:
      google_ads:
        +tags: ['daily']
      hibob:
        +tags: ['daily']
      mixpanel:
        +materialized: table
        +tags: ['daily']
  facebook_ads:
    +tags: ['daily']
  facebook_ads_source:
    +tags: ['daily']
  instagram_business:
    +tags: ['daily']
  instagram_business_source:
    +tags: ['daily']
  zendesk_source:
    +tags: ['daily']
  zendesk: 
    +tags: ['daily']
  ga4:
    +tags: ['daily']
    staging:
      recommended_events:
        stg_ga4__event_purchase:
          +enabled: true
        stg_ga4__event_add_to_cart:
          +enabled: true
        stg_ga4__event_begin_checkout:
          +enabled: true
        stg_ga4__event_add_payment_info:
          +enabled: true
          
vars:

### Fivetran Shopify Package
  shopify_union_schemas: ['fivetran_shopify_eu','fivetran_shopify_jp','fivetran_shopify_row','fivetran_shopify_usa'] # use this if the data is in different schemas/datasets of the same database/project
  shopify_using_fulfillment_event: true # false by default
  shopify_timezone: "Europe/Copenhagen" # Replace with your timezone
  shopify__using_refund : true
  shopify_refund_identifier: order_refund
  shopify_source:
    has_defined_sources: true
  shopify:
    has_defined_sources: true
### Google Ads Data Transfer package
  t_campaignStats: '`gcp_google_ads.ads_CampaignStats_5372755939`'
  t_campaign: '`gcp_google_ads.ads_Campaign_5372755939`'
  t_ads: '`gcp_google_ads.ads_Ad_5372755939`'
  t_adStats: '`gcp_google_ads.ads_AdBasicStats_5372755939`'
  t_campaignConv: '`gcp_google_ads.ads_CampaignConversionStats_5372755939`'
  t_adGroupStats: '`gcp_google_ads.ads_AdGroupStats_5372755939`'
  t_adGroup: '`gcp_google_ads.ads_AdGroup_5372755939`'
  t_adGroupConv: '`gcp_google_ads.ads_AdGroupConversionStats_5372755939`'
  convName1: 'PURCHASE'
  convName2: 'SUBMIT_LEAD_FORM'
  convName3: 'BOOK_APPOINTMENT'
### Fivetran Facebook Ads Package
  #facebook_ads_database: your_destination_name
  facebook_ads_schema: fivetran_facebook_ads
### Fivetran Facebook Pages Package
  facebook_pages_schema: fivetran_facebook
### Fivetran Stripe Package
  stripe_schema: fivetran_stripe
### Fivetran Hubspot Package
  # hubspot_schema: fivetran_hubspot
  # hubspot_email_event_enabled: false
### Fivetran Zendesk Package
  zendesk_schema: fivetran_zendesk
  using_user_tags: False
  using_organization_tags: False
### GA4 Package
  ga4:
    source_project: "rokoko-dw"
    property_ids: ['313227498']
    start_date: "20230412" # Earliest date to load
    frequency: "daily" # daily|streaming|daily+streaming. See 'Export Frequency' below.
    conversion_events: 'purchase'
    static_incremental_days: 4
    derived_session_properties:
      - event_parameter: "engagement_time_msec"
        session_property_name: "engagement_duration"
        value_type: "float_value"
### Fivetran Linkedin Package
  linkedin_ads_schema: fivetran_linkedin_ads
### Fivetran Instagram Business Package
  instagram_business_schema: fivetran_instagram
## Fivetran Linkedin Pages Package
  linkedin_pages_schema: fivetran_linkedin
### Social Media Reporting
  social_media_rollup__twitter_enabled: False
  social_media_rollup__facebook_enabled: True
  social_media_rollup__linkedin_enabled: True
  social_media_rollup__instagram_enabled: True

[fivetran-joemarkiewicz]

@drago-ssh apologies for the delay. After looking into the issue I believe this can be resolved by ensuring all the columns are enabled for these discount tables at the connection level. While the table may be enabled, unless all necessary columns are selected, the source table may be nearly empty. Would you be able to enable the required fields (you can either enable all fields or reference the docs to enable required fields) then re-run the transformation and see if that sorts out the stale data issue.

Regarding the test fixes, we will be developing these updates in the current sprint. I'll keep you updated here regarding the progress of those updates.

[drago-ssh]

@fivetran-joemarkiewicz thanks a lot!
They are now enabled but I cannot see any data being populated in the data lake.

1. Re-authorized all shopify connectors
2. Selected all the columns and resynced the specific table (did not populate)
3. Made a full resync (did not populate)

Why could that be?

[fivetran-joemarkiewicz]

@drago-ssh that's very interesting that you aren't seeing the full data persist after selecting all the columns. I would recommend following up in your Fivetran support ticket with this information as they will be better suited to help from the connection side.

[drago-ssh]

@fivetran-joemarkiewicz it seems like it appeared 1 day later now. After succesfully building shopofy__discounts should I expect the amount of discount_code_id's to be the same?

select distinct discount_code_id from `dbt_dc_shopify.shopify__discounts`
where created_at <= '2025-06-29 10:14:26 UTC'

=10.3k
Also the value seems to be null in a lot of instances as opposed to the old version based on an older dbt package + price_rule tables.

select distinct discount_code_id from `dbt_prod_shopify.shopify__discounts`
where created_at <= '2025-06-29 10:14:26 UTC'

=49.2k

Based on these sources

      - name: abandoned_applied_discount
        identifier: abandoned_applied_discount
      - name: abandoned_checkout
        identifier: abandoned_checkout
      - name: abandoned_checkout_discount_code
        identifier: abandoned_checkout_discount_code
      - name: abandoned_checkout_line
        identifier: abandoned_checkout_line
      - name: abandoned_checkout_shipping_line
        identifier: abandoned_checkout_shipping_line
      - name: abandoned_checkout_tax_line
        identifier: abandoned_checkout_tax_line
      - name: abandoned_checkout_url_tag
        identifier: abandoned_checkout_url_tag
      - name: collection
        identifier: collection
      - name: collection_product
        identifier: collection_product
      - name: customer
        identifier: customer
        columns: 
          - name: id
            data_tests:
             - unique
             - not_null 
      - name: customer_address
        identifier: customer_address
      - name: customer_segment
        identifier: customer_segment
      - name: customer_tag
        identifier: customer_tag
      - name: discount_allocation
        identifier: discount_allocation
      - name: discount_application
        identifier: discount_application
      - name: discount_code
        identifier: discount_code
      - name: draft_order
        identifier: draft_order
      - name: fulfillment
        identifier: fulfillment
      - name: inventory_item
        identifier: inventory_item
      - name: inventory_level
        identifier: inventory_level
      - name: location
        identifier: location
      - name: metafield
        identifier: metafield
      - name: order
        identifier: order
        columns: 
          - name: id
            data_tests:
             - unique
             - not_null        
      - name: order_adjustment
        identifier: order_adjustment
      - name: order_discount_code
        identifier: order_discount_code
      - name: order_fulfillment
        identifier: fulfillment
      - name: fulfillment_event
        identifier: fulfillment_event
      - name: fulfillment_order_line
        identifier: fulfillment_order_line
      - name: order_line
        identifier: order_line
      - name: order_line_refund
        identifier: order_line_refund
      - name: order_shipping_line
        identifier: order_shipping_line
      - name: order_shipping_tax_line
        identifier: order_shipping_tax_line
      - name: order_url_tag
        identifier: order_url_tag
      - name: order_tag
        identifier: order_tag
      - name: price_rule
        identifier: price_rule
      - name: product
        identifier: product
      - name: product_image
        identifier: product_image
      - name: product_tag
        identifier: product_tag
      - name: product_variant
        identifier: product_variant
      - name: order_note_attribute
        identifier: order_note_attribute        
      - name: order_refund
        identifier: refund
      - name: order_risk
        identifier: order_risk
      - name: refund
        identifier: refund
      - name: shop
        identifier: shop
      - name: tax_line
        identifier: tax_line
      - name: tender_transaction
        identifier: tender_transaction
      - name: transaction
        identifier: transaction




      - name: discount_code_basic
        identifier: discount_code_basic
      - name: discount_code_free_shipping
        identifier: discount_code_free_shipping
      - name: discount_redeem_code
        identifier: discount_redeem_code
      - name: discount_shareable_url
        identifier: discount_shareable_url
      - name: discount_customer_selection
        identifier: discount_customer_selection
      - name: discount_customer_gets_product
        identifier: discount_customer_gets_product
      - name: media
        identifier: media
      - name: product_media
        identifier: product_media       
      - name: media_image
        identifier: media_image  
      - name: inventory_quantity
        identifier: inventory_quantity  
      - name: discount_code_bxgy
        identifier: discount_code_bxgy

Bottom line is that I cannot recreate the shopify__discounts model even if I have data in the imported tables. I get too many nulls/duplicates.