Versatile Data Kit 0.2

[antoniivanov]

Summary

Major features include:

• Improvements in control-service security to be compliant with best Kubernetes practices (run jobs as unprivileged/non-root);
• Plugins can now hook into the ingestion process before, during or after the ingestion execution; these hooks can also be chained;
• Added support for Kimball templates (SCD1, SCD2, Snapshot Accumulating Fact Table) for the vdk-impala plugin.

Package versions

See installation instructions here.
The versions of VDK components released under VDK 0.2 are:

Main components

vdk-heartbeat==0.5.476585195
vdk-core==0.1.476585195
vdk-control-cli==1.2.476585195
pipelines-control-service==1.4.476585195

Plugins

vdk-trino==0.2.476585195
vdk-test-utils==0.2.476585195
vdk-kerberos-auth==0.2.476585195
vdk-ingest-http==0.2.476585195
vdk-impala==0.2.476585195
quickstart-vdk==0.2.476585195

What's Changed

• vdk-wiki: Created Life Expectancy Scenario by @alod83 in Created Life Expectancy Scenario #616
• vdk-wiki: life-expectancy minor fixes to work end-to-end by @tozka in vdk examples: life-expectancy minor fixes to work end-to-end #700
• vdk-kerberos-auth: Fix keytab file in job directory by @doks5 in [vdk-kerberos-auth] Fix keytab file in job directory #721
• vdk-plugins: Update ingestion interfaces used in plugins by @doks5 in [vdk-plugins] Update ingestion interfaces used in plugins #689
• vdk-test-utils: Add test pre-ingest, ingest, post-ingest plugins by @doks5 in [vdk-plugins][vdk-test-utils] Add test pre-ingest, ingest, post-ingest plugins #679
• control-service: Allow job builder run as non-root by @doks5 in control-service: Allow job builder run as non-root #625
• control-service: add counter to track data job watching task executions by @tpalashki in control-service: add counter to track data job watching task executions #692
• control-service: add job-base-image folder and CI step/job by @tozka in control-service: add job-base-image folder and CI step/job #711
• control-service: add security context to Data Job template by @mivanov1988 in control-service: add security context to Data Job template #713
• control-service: add timeouts to shedlock's database operations by @tpalashki in control-service: add timeouts to shedlock's database operations #693
• control-service: enable logging on update cron job failure by @mivanov1988 in control-service: enable logging on update cron job failure #674
• control-service: fix role permissions for pod/logs by @tozka in control-service: fix role permissions for pod/logs #699
• control-service: graphQL job executions filter by teamName by @mrMoZ1 in control-service: graphQL job executions filter by teamName #702
• control-service: remove unnecessary directory of legacy builder by @tozka in control-service: remove unnecessary directory of legacy builder #712
• control-service: remove unused code by @tozka in control-service: remove unused code #705
• control-service: revert swagger path changes by @ivakoleva in control-service: revert swagger path changes #704
• control-service: run data job as non-root user by @tozka in control-service: run data job as non-root user #710
• control-service: vdk sdk docker repository secret by @mivanov1988 in control-service: vdk sdk docker repository secret #694
• control-service: add amazon-ecr-credential-helper to the job builder by @tpalashki in job-builder: add amazon-ecr-credential-helper to the job builder #723
• control-service: Swagger UI path changes docs update by @ivakoleva in pipelines-control-service: Swagger UI path changes docs update #715
• control-service: publish Swagger UI to /data-jobs path by @ivakoleva in pipelines-control-service: publish Swagger UI to /data-jobs path #677
• control-service: publish Swagger UI to /data-jobs path by @ivakoleva in pipelines-control-service: publish Swagger UI to /data-jobs path #714
• control-service: redirect Swagger webjars resources by @ivakoleva in pipelines-control-service: redirect Swagger webjars resources #697
• control-service: release a new version by @ivakoleva in pipelines-control-service: release a new version #681
• quickstart-vdk: Run vdk-heartbeat before release by @YanaZhivkova in quickstart-vdk: Run vdk-heartbeat before release #665
• vdk-control-cli: Expand missing resource error message by @gageorgiev in vdk-control-cli: Expand missing resource error message #706
• vdk-control-cli: Make list command print all jobs on empty team param by @gageorgiev in vdk-control-cli: Make list command print all jobs on empty team param  #709
• vdk-control-cli: Parse contacts with both comma "," as delimiter as well by @tozka in vdk-control-cli: Parse contacts with both comma "," as delimiter as well #719
• vdk-control-cli: clarify api token documentation by @tozka in vdk-control-cli: clarify api token documentation #722
• vdk-core: add flag to enable synchronous/blockng ingestion by @tozka in vdk-core: add flag to enable synchronous/blockng ingestion #698
• vdk-core: adjust defined type for configuration values by @tozka in vdk-core: adjust defined type for configuration values #684
• vdk-core: fix error message by @tozka in vdk-core: fix error message #686
• vdk-core: Add ingestion functional tests by @doks5 in [vdk-core] Add ingestion functional tests #691
• vdk-core: Implementation of new ingestion interfaces by @doks5 in [vdk-core] Implementation of new ingestion interfaces #690
• vdk-core: Introduce post-ingest-sequence env var by @doks5 in [vdk-core] Introduce post-ingest-sequence env var #682
• vdk-heartbeat: Successful data job run test mode by @ivakoleva in vdk-heartbeat: Successful data job run test mode #718
• vdk-heartbeat: Fix successful run status check by @YanaZhivkova in [vdk-heartbeat] Fix successful run status check #724
• vdk-impala: Impala docker image upgrade by @ivakoleva in vdk-impala: Impala docker image upgrade #675
• vdk-impala: impala templates by @mrMoZ1 in vdk-impala: impala templates #671
• vdk-impala: template .sql missing from python distro by @ivakoleva in vdk-impala: template .sql missing from python distro #695
• vdk-impala: Improve error handling to handle view errors by @doks5 in [plugins][vdk-impala] Improve error handling to handle view errors #717
• vdk-ingest-http: additional request parameters support by @ivakoleva in vdk-ingest-http: additional request parameters support #701
• vdk-ingest-http: configurable allow JSON float NaN capability by @ivakoleva in vdk-ingest-http: configurable allow JSON float NaN capability #725
• vdk-trino: trino ingest to handle type casting and missing values by @tozka in vdk-trino: trino ingest to handle type casting and missing values #685
• versatile-data-kit: Establish release process by @gageorgiev in versatile-data-kit: Establish release process #673

New Contributors

• @alod83 made their first contribution in Created Life Expectancy Scenario #616

Full Changelog: 0.1...0.2

---

This discussion was created from the release Versatile Data Kit 0.2.