experimental: add databricks-lakeflow-connect skill - #1
Closed
jralfonsog wants to merge 1 commit into
Closed
Conversation
Initial scope-first commit for a draft PR. GA-first deep coverage, PuPr listed but deferred to follow-up commits. This commit includes: - SKILL.md (routing + 3-tier catalog: GA / PuPr / Beta-PrPr + workflow + key concepts + common issues) - references/4-ingestion-decision-tree.md (LFC vs Auto Loader vs Lakehouse Federation vs Delta Sharing vs Zerobus + cost considerations + escape hatches) - agents/openai.yaml + assets/ via scripts/skills.py generate - manifest.json updated To follow in subsequent commits: - references/1-saas-connectors.md (Salesforce, Workday Reports, ServiceNow, GA4, HubSpot, Confluence — all GA) - references/2-database-connectors.md (SQL Server cloud + on-prem + gateway pattern intro — GA) - references/5-troubleshooting-and-monitoring.md (GA-focused) Public Preview connectors (NetSuite, Dynamics 365, PG/MySQL CDC, query-based databases, Foreign Catalog query-based, SFTP) are production-supported and listed in SKILL.md; deep coverage will be added incrementally as PuPr connectors stabilize. SharePoint/Google Drive (Beta currently, GA Jun 1 target) and other Beta/PrPr connectors are not first-class in this skill. references/3-file-and-streaming-connectors.md will be created when SFTP + SharePoint/Drive get deep coverage (post-v1). Signed-off-by: Jose Alfonso <jralfonsog@gmail.com>
Owner
Author
|
Superseded by upstream PR databricks#103 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
New experimental skill
databricks-lakeflow-connectfor managed ingestion pipelines (Salesforce, Workday Reports, ServiceNow, Google Analytics 4, HubSpot, Confluence, SQL Server cloud + on-prem, plus Public Preview database / file connectors).Marked as draft for scope-first review. GA-first deep coverage; PuPr connectors are listed in
SKILL.mdas production-supported but their deep coverage will land in follow-up commits. Nodatabricks-pipelinesoverlap — Lakeflow Connect pipelines reuse the pipelines API surface viaingestion_definition, and this skill cross-links toskills/databricks-pipelines/from the decision tree and Related Skills.Changes
experimental/databricks-lakeflow-connect/SKILL.md(~200 lines) — routing + 3-tier catalog (GA / PuPr / Beta-PrPr) + workflow + key concepts + common issues.experimental/databricks-lakeflow-connect/references/4-ingestion-decision-tree.md(~130 lines) — Lakeflow Connect vs Auto Loader vs Lakehouse Federation vs Delta Sharing vs Zerobus + cost considerations + escape hatches. Cross-links to the Auto Loader work in databricks-solutions/ai-dev-kit#539.experimental/databricks-lakeflow-connect/agents/openai.yaml+assets/databricks.{svg,png}— auto-generated viascripts/skills.py generate.manifest.json— updated byscripts/skills.py generateto register the new skill.SharePoint / Google Drive (Beta as of May 2026; GA target Jun 1) are not first-class in v1 — they appear in the Beta/PrPr note in
SKILL.md.databricks-zerobus-ingestis pointed to from the catalog and decision tree (push-vs-pull dichotomy), not re-covered.To follow
references/1-saas-connectors.md— 6 GA SaaS connectors deep coveragereferences/2-database-connectors.md— SQL Server cloud + on-prem + gateway patternreferences/5-troubleshooting-and-monitoring.md— GA-focused issues + event log queriesreferences/3-file-and-streaming-connectors.md— created when SFTP + SharePoint/Drive get deep coverageCross-repo
#ai-dev-kit-teamSlack on 2026-05-27; maintainers signed off on Databricks Agent Skillsexperimental/as the destination.Test plan
python3 scripts/skills.py generateclean.python3 scripts/skills.py validatepasses (Everything is up to date.).skills/databricks-pipelines/,skills/databricks-dabs/,skills/databricks-jobs/,experimental/databricks-zerobus-ingest/,experimental/databricks-unity-catalog/).stf audit(L3 doc quality) — overall 8.2/10 (PASS all dimensions). Per-dimension scores below.databricks-field-eng/skillforge/SKILL.mdafterreferences/1-saas-connectors.mdlands (L1 link validity will be clean once the deferred references ship).ground_truth.yamlbefore promoting out of draft.stf auditper-dimension (L3)This pull request was AI-assisted by Isaac.