Updating to UC and Streaming Tables

[shabbirk-db]

Changes made:

• Fixed _resources to create and write to Volumes instead of dbfs.
• Commented out autoloader in the notebook
• Created streaming_table ingestion layer directly in dbt for the three streams, reading from Volumes.
• This will create full lineage in UC from Volumes all the way to Gold layer.

To do:

• Update documentation to reflect the changes
• Re-spec cluster to be UC native when created.
• Potentially generate the data as part of the dbdemos installation so we can skip the data-ingestion notebook entirely and make a more streamlined experience (e.g. dbt + ML only)
• Update the ML model to read from the new tables and make it UC friendly if needed