A hands-on Braintrust workshop built around Super Stonks, a small LangGraph stock-chat agent instrumented with the Braintrust SDK. The agent has an unknown issue baked in — and the workshop is the story of finding and fixing it.
Super Stonks looks like a helpful stock assistant. But somewhere in its behavior there's a recurring failure that users keep running into — one that isn't obvious until you look at the traces. You won't be told what it is up front; you'll uncover it the way you would in the real world.
Over the session you'll run the full Braintrust flywheel to discover and remediate that issue:
- Observe production traces in a shared, pre-seeded project.
- Find a recurring failure with Patterns, which analyzes the traces automatically.
- Curate the affected traces into a dataset in your own project.
- Score them with a scorer and run a baseline experiment.
- Fix the agent once you understand what's going wrong.
- A/B test the fixed agent against the baseline to prove the improvement.
- Deploy an online score so the same regression can't sneak back in.
By the end you'll have run the loop end-to-end — from a surfaced production failure to a verified fix guarded by online scoring.
Follow docs/PARTICIPANT.md. It's the step-by-step
guide you run top to bottom during the workshop: install, configure, run the
agent, investigate the failure, build the dataset, score, fix it, and validate.
Before the session, the install steps also live on the welcome screen —
docs/WELCOME.md.
Using Codex? You can run it on the Braintrust API key in your .env (via the
Braintrust AI gateway) — no OpenAI account needed. See docs/CODEX.md.
Use these resources to set the workshop up again and run it:
| Resource | Purpose |
|---|---|
docs/SEEDING_MILESTONES.md |
Build plan for provisioning the shared, pre-seeded super-stonks project (Topics + Patterns + seeded traces) and the trace/scorer design. |
docs/WELCOME.md |
Attendee install/welcome screen. |
AGENTS.md |
Single source of truth for coding-agent guidance, conventions, and the shared-seed vs. own-project split. |
- Provision the shared
super-stonksproject perdocs/SEEDING_MILESTONES.md:BRAINTRUST_DEFAULT_PROJECT=super-stonks make prepareenables Topics + Patterns and seeds the traffic (including the planted share-class tickers). Runmake top-upwith the same override the day before so the Patterns stay active. - Distribute a
BRAINTRUST_API_KEYfor thedataoutpost-workshoporg. No OpenAI key is handed out — OpenAI calls go through the Braintrust AI gateway using the OpenAI key configured in the org (make sure one is set under the org's AI providers). - Walk participants through
docs/PARTICIPANT.md— it's the run-of-show too.
A LangGraph agent using OpenAI gpt-4o-mini, wrapped with braintrust.wrap_openai;
market data comes from yfinance.
src/super_stonks/
app.py # Streamlit UI → `make agent`
agent/ # LangGraph agent: agent.py, tools.py, prompts.py, config.py, ...
evals/ # scorers.py + qa_eval.py
make setup # uv sync + create .env
make agent # launch the Streamlit app
make help # all targetsSee AGENTS.md for conventions, the shared-seed vs. own-project
split, and the coding-agent playbook.