Running the databricks-mlflow-evaluation skill end to end during a user study surfaced four gaps in the skill guidance. The first one broke the user's Python environment. All four are self-contained edits to the skill markdown (GOTCHAS.md, CRITICAL-interfaces.md, patterns-datasets.md).
1 (highest impact): inspecting a dataset's schema installs databricks-agents and can break the environment
The skill teaches mlflow.genai.datasets.get_dataset() / .to_df() to inspect a persistent dataset. On a Databricks backend these import databricks-agents, which depends on databricks-connect (pinned numpy<2). When a coding agent hits the import error and runs pip install databricks-agents, the install downgrades numpy in an environment that already has numpy 2.x and breaks every downstream import (AttributeError: module 'numpy' has no attribute 'long').
Suggested fix: add a GOTCHA plus an inline note saying to inspect a dataset's columns with w.tables.get() or DESCRIBE TABLE over a SQL warehouse (needs only databricks-sdk, already installed), and never pip install databricks-agents at inspection time.
2: the databricks-agents dependency is not stated up front
The install section in CRITICAL-interfaces.md does not name databricks-agents or its numpy<2 pin, so the conflict above is only discovered by hitting it.
3: mlflow.openai.autolog() is the default even for LangGraph agents
openai.autolog() captures only the raw LLM call and drops LangGraph node spans. LangChain and LangGraph should use mlflow.langchain.autolog(), which needs only langchain-core. An autolog selector table plus a GOTCHA would prevent the wrong choice.
4: attaching to an experiment by numeric ID is not shown
The setup example omits the mlflow.set_experiment(experiment_id="<id>") form, and the numeric-name failure mode is not documented.
The full proposed diff is on the closed #589 if it is useful as a starting point. I can also send the change through the internal source repo instead.
Running the
databricks-mlflow-evaluationskill end to end during a user study surfaced four gaps in the skill guidance. The first one broke the user's Python environment. All four are self-contained edits to the skill markdown (GOTCHAS.md,CRITICAL-interfaces.md,patterns-datasets.md).1 (highest impact): inspecting a dataset's schema installs
databricks-agentsand can break the environmentThe skill teaches
mlflow.genai.datasets.get_dataset()/.to_df()to inspect a persistent dataset. On a Databricks backend these importdatabricks-agents, which depends ondatabricks-connect(pinnednumpy<2). When a coding agent hits the import error and runspip install databricks-agents, the install downgrades numpy in an environment that already has numpy 2.x and breaks every downstream import (AttributeError: module 'numpy' has no attribute 'long').Suggested fix: add a GOTCHA plus an inline note saying to inspect a dataset's columns with
w.tables.get()orDESCRIBE TABLEover a SQL warehouse (needs onlydatabricks-sdk, already installed), and neverpip install databricks-agentsat inspection time.2: the
databricks-agentsdependency is not stated up frontThe install section in
CRITICAL-interfaces.mddoes not namedatabricks-agentsor itsnumpy<2pin, so the conflict above is only discovered by hitting it.3:
mlflow.openai.autolog()is the default even for LangGraph agentsopenai.autolog()captures only the raw LLM call and drops LangGraph node spans. LangChain and LangGraph should usemlflow.langchain.autolog(), which needs onlylangchain-core. An autolog selector table plus a GOTCHA would prevent the wrong choice.4: attaching to an experiment by numeric ID is not shown
The setup example omits the
mlflow.set_experiment(experiment_id="<id>")form, and the numeric-name failure mode is not documented.The full proposed diff is on the closed #589 if it is useful as a starting point. I can also send the change through the internal source repo instead.