diff --git a/Instructions/Consolidated/D-observe-evaluate-and-secure-agents.md b/Instructions/Consolidated/D-observe-evaluate-and-secure-agents.md index ed4a06963..7b50349e9 100644 --- a/Instructions/Consolidated/D-observe-evaluate-and-secure-agents.md +++ b/Instructions/Consolidated/D-observe-evaluate-and-secure-agents.md @@ -96,8 +96,8 @@ it runs". The **Optional** task then goes after safety. By completing the **Core** tasks of this exercise, you'll be able to: -- **Trace an agent** with OpenTelemetry, export the traces to Azure Monitor, and read them - in the Foundry portal — including custom spans you add around your own code. +- **Trace an agent** with OpenTelemetry, export the traces to Azure Monitor, and read them — + Foundry's own server-side trace and your custom spans are both there, in different views. - **Evaluate answer quality** against ground truth using built-in evaluators (groundedness, relevance, similarity) and a JSONL dataset. @@ -141,11 +141,10 @@ how it stands up to attack. - **Core only (~1h):** do Tasks 1–2. - **Everything (~1h 35m):** add **Task 3**, the red team scan. -> **One agent, three questions**: Task 1 traces an agent you create in code. Tasks 2 and 3 -> both point at the **grounded knowledge agent** from -> [Lab B](B-integrate-agents-with-enterprise-knowledge-and-m365.md). If you haven't done Lab B, -> one command creates an equivalent agent so this lab stands alone — see -> [Getting started](D0-getting-started.md). +> **One agent, three questions**: all three tasks point at the same **grounded knowledge +> agent** — Task 1 traces it, Tasks 2 and 3 measure it. If you haven't done +> [Lab B](B-integrate-agents-with-enterprise-knowledge-and-m365.md), one command creates an +> equivalent agent so this lab stands alone — see [Getting started](D0-getting-started.md). ## Measure, don't guess @@ -166,8 +165,8 @@ production. Across this lab you: -- **Instrumented an agent** with OpenTelemetry, exported traces to Application Insights, and - read them — including your own custom spans — in the Foundry portal. +- **Instrumented an agent** with OpenTelemetry and exported traces to Application Insights — + reading Foundry's automatic server-side trace and your own custom spans, side by side. - **Evaluated a grounded agent** against a ground-truth dataset with built-in groundedness, relevance and similarity evaluators, and got a score you can compare across changes. - (Optionally) **Red teamed the agent** with adversarial attack strategies and your own seed @@ -182,7 +181,7 @@ If you're finished, delete the resources you created to avoid unnecessary Azure 1. In the [Azure portal](https://portal.azure.com), navigate to the resource group that contains your Foundry resource. 1. On the toolbar, select **Delete resource group**, enter the resource group name, and confirm. -> The code you run in Task 1 deletes the agent version it creates. The agent Tasks 2 and 3 -> measure is removed when you delete the resource group. If you provisioned with `azd`, run -> `azd down` instead — but note that Application Insights, if you created it from the Foundry -> portal, is a separate resource and is deleted with the resource group rather than by `azd`. +> All three tasks measure the same `caldova-knowledge-agent`, so deleting the resource group +> removes it along with everything else. If you provisioned with `azd`, run `azd down` +> instead — but note that Application Insights, if you created it from the Foundry portal, is +> a separate resource and is deleted with the resource group rather than by `azd`. diff --git a/Instructions/Consolidated/D0-getting-started.md b/Instructions/Consolidated/D0-getting-started.md index e75c45b76..153ddb711 100644 --- a/Instructions/Consolidated/D0-getting-started.md +++ b/Instructions/Consolidated/D0-getting-started.md @@ -72,11 +72,16 @@ portal (the default), or provision them with one command using the Azure Develop ### Option B — Provision with azd (optional, one command) If you'd rather not click through the portal, the lab ships an optional `azd` template that -creates the Foundry resource, a project, and a model deployment for you. +creates the Foundry resource, a project, and a model deployment for you. This runs from inside +the repo, so clone it first if you haven't already: + +``` +git clone https://github.com/MicrosoftLearning/mslearn-ai-agents.git +``` 1. Install the [Azure Developer CLI](https://learn.microsoft.com/azure/developer/azure-developer-cli/install-azd). -1. From the `Labfiles/D-observe-evaluate-and-secure-agents` folder, run: +1. From the `Labfiles/D-observe-evaluate-and-secure-agents` folder in the repo you just cloned, run: ``` azd auth login @@ -90,29 +95,6 @@ creates the Foundry resource, a project, and a model deployment for you. > connect that in the portal using the steps below. When you're done with the lab, run > `azd down` to delete everything it created. -## Connect Application Insights (needed for Task 1) - -Foundry stores traces in an **Application Insights** resource connected to your project. Connect -one now — it takes a minute, and once it's connected Foundry starts recording server-side traces -for your agents without any code at all. - -1. In the [Foundry portal](https://ai.azure.com), open your project. - -1. In the left navigation, select **Agents**, then select **Traces** at the top. - -1. Select **Connect**, then either pick an existing Application Insights resource or select - **Create new** and complete the wizard. - - > If you don't see the **Connect** button, select **Manage** in the upper right, then - > **Project details** > **Connected resources** > **Add connection** > **Application Insights**. - -1. To *read* the traces you'll need the **Log Analytics Reader** role on that Application - Insights resource. If you created it yourself, you already have it. - -> **Why this matters**: your Foundry project can only hand your code a connection string if -> something is connected. Task 1 asks the project for that string, so this step has to happen -> first. - ## Get the starter code 1. In VS Code, open the Command Palette (**Ctrl+Shift+P**), run **Git: Clone**, and enter: @@ -121,6 +103,8 @@ for your agents without any code at all. https://github.com/MicrosoftLearning/mslearn-ai-agents.git ``` + > If you already cloned the repo for the `azd` option above, skip this and just open it. + 1. Open the cloned repo, then **File > Open Folder** and select `mslearn-ai-agents/Labfiles/D-observe-evaluate-and-secure-agents/Python`. This single folder holds the starter code for **every** task in this lab — you use one virtual environment and one `.env` throughout. 1. Right-click **requirements.txt** and choose **Open in Integrated Terminal**. Then create a virtual environment and install packages: @@ -138,9 +122,9 @@ for your agents without any code at all. > **Tip**: In the Foundry Toolkit VS Code extension, right-click your project deployment and select **Copy Project Endpoint** to get the endpoint URL. -## Get an agent to measure (needed for Tasks 2 and 3) +## Get an agent to measure (needed for every task) -Tasks 2 and 3 measure a **grounded** agent — one that answers from the Caldova +Every task uses a **grounded** agent — one that answers from the Caldova knowledge base rather than from the model's own memory. You have two ways to get one: - **You did [Lab B](B-integrate-agents-with-enterprise-knowledge-and-m365.md)**: set `AGENT_NAME` @@ -160,7 +144,28 @@ knowledge base rather than from the model's own memory. You have two ways to get This uploads the documents in `Python/knowledge/`, grounds an agent named `caldova-knowledge-agent` on them with File Search, and writes `AGENT_NAME` into your `.env`. -> Task 1 doesn't need this agent — it creates and deletes its own. +## Connect Application Insights (needed for Task 1) + +Foundry stores traces in an **Application Insights** resource connected to your project. Connect +one now — it takes a minute, and once it's connected Foundry starts recording server-side traces +for your agents without any code at all. + +1. In the [Foundry portal](https://ai.azure.com), open your project. + +1. In the left navigation, select **Agents**, then select **Traces** at the top. + +1. Select **Connect**, then either pick an existing Application Insights resource or select + **Create new** and complete the wizard. + + > If you don't see the **Connect** button, select **Manage** in the upper right, then + > **Project details** > **Connected resources** > **Add connection** > **Application Insights**. + +1. To *read* the traces you'll need the **Log Analytics Reader** role on that Application + Insights resource. If you created it yourself, you already have it. + +> **Why this matters**: your Foundry project can only hand your code a connection string if +> something is connected. Do this before starting Task 1, which asks the project for that +> string. ## Check you're ready for a task diff --git a/Instructions/Consolidated/D1-trace-your-agent.md b/Instructions/Consolidated/D1-trace-your-agent.md index 573cf3cca..5d4cb42c6 100644 --- a/Instructions/Consolidated/D1-trace-your-agent.md +++ b/Instructions/Consolidated/D1-trace-your-agent.md @@ -21,10 +21,12 @@ lab: *Part of the **Observe, evaluate, and secure your agents** lab. New here? Start with [Getting started](D0-getting-started.md).* > **Set up (start here):** This task needs a Foundry project, an **Application Insights -> resource connected to it**, and the starter code. If you haven't already, complete -> [Getting started](D0-getting-started.md) to create your project, connect Application -> Insights, clone the code, and set `PROJECT_ENDPOINT` and `MODEL_DEPLOYMENT_NAME` in -> `Python/.env`. Then, from the `Python` folder you opened in VS Code, verify you're ready: +> resource connected to it**, a grounded agent to trace, and the starter code. If you +> haven't already, complete [Getting started](D0-getting-started.md) to create your +> project, clone the code, set `PROJECT_ENDPOINT` and `AGENT_NAME` in `Python/.env` (point +> it at your [Lab B](B-integrate-agents-with-enterprise-knowledge-and-m365.md) agent, or +> create one with `python ../setup/bootstrap_agent.py`), and connect Application Insights. +> Then, from the `Python` folder you opened in VS Code, verify you're ready: ``` python ../setup/check_env.py --task 1 @@ -76,8 +78,10 @@ them at a different backend tomorrow without rewriting your instrumentation. > **Server-side traces come free.** Now that Application Insights is connected to your > project, Foundry already records traces for agents it hosts — no code required. What you -> add here is **client-side** instrumentation: spans around *your* code, so you can see your -> logic and the agent's work in one timeline. +> add here is **client-side** instrumentation: spans around *your* code. Both land in the same +> Application Insights resource, but Foundry's **Agents > Traces** page only renders its own +> server-side view (`Invoke Agent` / `Execute tool` / `Chat`) — to see your own spans and +> attributes alongside it, you'll look directly at Application Insights. Open the `Python` folder and activate the virtual environment from [Getting started](D0-getting-started.md) (`.\labenv\Scripts\Activate.ps1`), then continue below. @@ -93,7 +97,6 @@ Open **traced_agent.py** and add code at each commented placeholder. # Add references from azure.identity import DefaultAzureCredential from azure.ai.projects import AIProjectClient - from azure.ai.projects.models import PromptAgentDefinition from azure.monitor.opentelemetry import configure_azure_monitor from opentelemetry import trace ``` @@ -146,23 +149,18 @@ Open **traced_agent.py** and add code at each commented placeholder. tracer = trace.get_tracer(__name__) ``` -1. **Create the agent staff are talking to**: +1. **Look up the agent** — its `id` (not just its name) is needed to correlate traces with + this specific agent in the Foundry portal: ```python - # Create the agent staff are talking to - agent = project_client.agents.create_version( - agent_name=AGENT_NAME, - definition=PromptAgentDefinition( - model=model_deployment, - instructions=INSTRUCTIONS, - ), - ) - print(f"Agent created (name: {agent.name}, version: {agent.version})") + # Look up the agent so its id can be included in agent_reference + agent = project_client.agents.get(agent_name=agent_name) ``` 1. **Ask each question inside its own span** — this is the part that pays off. An outer span represents the review; each question gets a child span, tagged with attributes you choose - so you can tell them apart in the portal: + so you can tell them apart in the portal. This reuses `caldova-knowledge-agent` rather + than standing up a separate agent just for this task: ```python # Ask each question inside its own span @@ -176,21 +174,13 @@ Open **traced_agent.py** and add code at each commented placeholder. response = openai_client.responses.create( conversation=conversation.id, input=question, - extra_body={"agent_reference": {"name": agent.name, "type": "agent_reference"}}, + extra_body={"agent_reference": {"name": agent.name, "id": agent.id, "type": "agent_reference"}}, ) question_span.set_attribute("caldova.answer_length", len(response.output_text)) print(f"\nQ{number}: {question}") print(f"A{number}: {response.output_text}") ``` -1. **Clean up the agent version** so you don't leave test agents behind: - - ```python - # Clean up resources by deleting the agent version - project_client.agents.delete_version(agent_name=agent.name, agent_version=agent.version) - print("\nAgent deleted") - ``` - 1. Save the file (**Ctrl+S**). ### Run and test @@ -205,15 +195,11 @@ Open **traced_agent.py** and add code at each commented placeholder. python traced_agent.py ``` -1. You should see the three answers print, then the agent delete itself: +1. You should see the three answers print: ``` - Agent created (name: caldova-planning-assistant, version: 1) - Q1: How long does review take for a capacity request with a complete brief? A1: ... - - Agent deleted ``` > If you get an error about the connection string, Application Insights isn't connected to @@ -221,16 +207,29 @@ Open **traced_agent.py** and add code at each commented placeholder. ### Read the traces +**Two views, two purposes.** Foundry's **Agents > Traces** page shows the automatic +**server-side** trace of the agent's own turn (`Invoke Agent` > `Execute tool` / `Chat`) — it +confirms tracing is working, but it doesn't surface the **client-side** spans your script just +added. To see `morning-planning-review`, `planner-question`, and their custom attributes, look +at the Application Insights resource itself: + 1. In the [Foundry portal](https://ai.azure.com), open your project, select **Agents**, then - **Traces**. + **caldova-knowledge-agent**, then **Traces**, to confirm the run arrived (telemetry takes a + minute or two — wait and refresh if it isn't there yet). Selecting a trace here shows + Foundry's own `Invoke Agent` / `Execute tool` / `Chat` view, not your custom spans. + +1. Open the Application Insights resource in the [Azure portal](https://portal.azure.com): + open the resource group you created for this project, and select the Application Insights + resource in it. -1. Find the most recent trace and select it. Telemetry takes a minute or two to arrive — if - it isn't there, wait and refresh. +1. In Application Insights, expand **Investigate** in the left navigation, select **Search**, + and search for `morning-planning-review`. -1. Step through the spans. You should see your `morning-planning-review` span at the top with - three `planner-question` children, and inside each one the model call the SDK emitted. +1. Select a matching result to open its **end-to-end transaction details**. This is the raw + span tree: your `morning-planning-review` span at the top, three `planner-question` + children, and inside each one the model call the SDK emitted. -1. Select a `planner-question` span and look at its attributes. Your `caldova.question_number` +1. Select a `planner-question` span and look at its properties. Your `caldova.question_number` and `caldova.answer_length` are there alongside the standard GenAI attributes. 1. Compare the durations of the three questions. That's the planning lead's complaint, diff --git a/Instructions/Consolidated/D2-evaluate-answer-quality.md b/Instructions/Consolidated/D2-evaluate-answer-quality.md index 0254a2cc9..48d95cf89 100644 --- a/Instructions/Consolidated/D2-evaluate-answer-quality.md +++ b/Instructions/Consolidated/D2-evaluate-answer-quality.md @@ -33,8 +33,8 @@ python ../setup/check_env.py --task 2 ``` > **Continuing from a previous task?** If you just finished another task in the same -> `Python` folder, your project, virtual environment, and `.env` are already set — but Task 1 -> didn't need `AGENT_NAME`, so check that it's set before you start. +> `Python` folder, your project, virtual environment, and `.env` are already set — go +> straight to **Look at the dataset first** below. --- @@ -149,7 +149,7 @@ Open **evaluate_agent.py** and add code at each commented placeholder. 1. **Run the evaluation** — `evaluate()` reads the dataset, calls the target once per row, and passes each evaluator exactly the columns it needs. `column_mapping` is how you say which - column is which: `${data.x}` comes from the file, `${outputs.x}` comes back from the target: + column is which: `${data.x}` comes from the file, `${target.x}` comes back from the target: ```python # Run the evaluation @@ -166,20 +166,20 @@ Open **evaluate_agent.py** and add code at each commented placeholder. "column_mapping": { "query": "${data.query}", "context": "${data.context}", - "response": "${outputs.response}", + "response": "${target.response}", } }, "relevance": { "column_mapping": { "query": "${data.query}", - "response": "${outputs.response}", + "response": "${target.response}", } }, "similarity": { "column_mapping": { "query": "${data.query}", "ground_truth": "${data.ground_truth}", - "response": "${outputs.response}", + "response": "${target.response}", } }, }, diff --git a/Instructions/Consolidated/D3-red-team-your-agent.md b/Instructions/Consolidated/D3-red-team-your-agent.md index 7583c49b1..04a8c6fec 100644 --- a/Instructions/Consolidated/D3-red-team-your-agent.md +++ b/Instructions/Consolidated/D3-red-team-your-agent.md @@ -112,6 +112,8 @@ Open **red_team_agent.py** and add code at each commented placeholder. credential = DefaultAzureCredential() project_client = AIProjectClient(endpoint=project_endpoint, credential=credential) openai_client = project_client.get_openai_client() + # Look up the agent so its id can be included in agent_reference + agent = project_client.agents.get(agent_name=agent_name) ``` 1. **Build the callback that sends one attack prompt to your agent** — the red team calls this @@ -125,7 +127,7 @@ Open **red_team_agent.py** and add code at each commented placeholder. try: response = openai_client.responses.create( input=query, - extra_body={"agent_reference": {"name": agent_name, "type": "agent_reference"}}, + extra_body={"agent_reference": {"name": agent.name, "id": agent.id, "type": "agent_reference"}}, ) return response.output_text except Exception as error: # a blocked prompt is a result, not a crash @@ -173,7 +175,7 @@ Open **red_team_agent.py** and add code at each commented placeholder. AttackStrategy.Flip, AttackStrategy.Compose([AttackStrategy.Base64, AttackStrategy.ROT13]), ], - output_path=str(OUTPUT), + output_path=str(OUTPUT_DIR), ) ``` @@ -219,9 +221,10 @@ Open **red_team_agent.py** and add code at each commented placeholder. ] ``` -1. Open **redteam_scan.json** and look at `redteaming_data`. Every attack-response pair is - there, with the prompt that was sent, what your agent said, and whether the attack was - judged successful. Read a few — seeing the disguised prompts is most of the value. +1. Open **redteam_output/evaluation_result.json** and look at `redteaming_data`. Every + attack-response pair is there, with the prompt that was sent, what your agent said, and + whether the attack was judged successful. Read a few — seeing the disguised prompts is + most of the value. 1. Compare `baseline_asr` with the per-complexity numbers in `attack_technique_summary`. If a strategy scores higher than the baseline, that transformation is getting past something the diff --git a/Labfiles/C-build-multi-agent-solutions-with-agent-framework/Python/requirements.txt b/Labfiles/C-build-multi-agent-solutions-with-agent-framework/Python/requirements.txt index 3f6d95032..277297428 100644 --- a/Labfiles/C-build-multi-agent-solutions-with-agent-framework/Python/requirements.txt +++ b/Labfiles/C-build-multi-agent-solutions-with-agent-framework/Python/requirements.txt @@ -1,6 +1,10 @@ python-dotenv azure-identity==1.25.3 -agent-framework==1.12.1 +# Installs only core + foundry + openai (this lab doesn't need the other integrations +# the full agent-framework meta-package bundles). Pinned to a version tested together +# with a compatible agent-framework-core/agent-framework-openai resolution. +agent-framework-foundry==1.10.3 +agent-framework-orchestrations==1.0.1 httpx uvicorn starlette diff --git a/Labfiles/D-observe-evaluate-and-secure-agents/Python/agent_target.py b/Labfiles/D-observe-evaluate-and-secure-agents/Python/agent_target.py index c5b9e2044..4916dff30 100644 --- a/Labfiles/D-observe-evaluate-and-secure-agents/Python/agent_target.py +++ b/Labfiles/D-observe-evaluate-and-secure-agents/Python/agent_target.py @@ -8,7 +8,7 @@ evaluate() calls this once per line of caldova_eval.jsonl, passing the columns of that line as keyword arguments, and merges the returned dictionary back into -the row as `outputs.*`. So returning {"response": ...} makes `${outputs.response}` +the row as `target.*`. So returning {"response": ...} makes `${target.response}` available to every evaluator. """ @@ -37,15 +37,16 @@ def __init__(self): ) self.openai_client = self.project_client.get_openai_client() - # Fail early with a useful message rather than once per dataset row. - self.project_client.agents.get(agent_name=self.agent_name) + # Fail early with a useful message rather than once per dataset row, and keep the + # id so agent_reference can correlate traces with this agent. + self.agent = self.project_client.agents.get(agent_name=self.agent_name) def __call__(self, *, query: str, **kwargs) -> dict: """Answer one question. Extra dataset columns arrive in kwargs and are ignored.""" response = self.openai_client.responses.create( input=query, extra_body={ - "agent_reference": {"name": self.agent_name, "type": "agent_reference"} + "agent_reference": {"name": self.agent.name, "id": self.agent.id, "type": "agent_reference"} }, ) return {"response": response.output_text} diff --git a/Labfiles/D-observe-evaluate-and-secure-agents/Python/red_team_agent.py b/Labfiles/D-observe-evaluate-and-secure-agents/Python/red_team_agent.py index a917c2438..47aa15d29 100644 --- a/Labfiles/D-observe-evaluate-and-secure-agents/Python/red_team_agent.py +++ b/Labfiles/D-observe-evaluate-and-secure-agents/Python/red_team_agent.py @@ -15,7 +15,9 @@ agent_name = os.getenv("AGENT_NAME", "caldova-knowledge-agent") SEED_PROMPTS = Path("data/attack_objectives.json") -OUTPUT = Path("redteam_scan.json") +# scan() treats output_path as a directory and writes evaluation_result.json inside it +OUTPUT_DIR = Path("redteam_output") +OUTPUT = OUTPUT_DIR / "evaluation_result.json" # Connect to the project diff --git a/Labfiles/D-observe-evaluate-and-secure-agents/Python/traced_agent.py b/Labfiles/D-observe-evaluate-and-secure-agents/Python/traced_agent.py index 98fce6539..c087b1e82 100644 --- a/Labfiles/D-observe-evaluate-and-secure-agents/Python/traced_agent.py +++ b/Labfiles/D-observe-evaluate-and-secure-agents/Python/traced_agent.py @@ -7,7 +7,7 @@ # Load environment variables from .env file load_dotenv() project_endpoint = os.getenv("PROJECT_ENDPOINT") -model_deployment = os.getenv("MODEL_DEPLOYMENT_NAME") +agent_name = os.getenv("AGENT_NAME", "caldova-knowledge-agent") # Turn on GenAI tracing @@ -19,13 +19,6 @@ "Which contract manufacturers could fast-track us inside a three-month window?", ] -AGENT_NAME = "caldova-planning-assistant" -INSTRUCTIONS = ( - "You are the Caldova planning assistant. You answer questions from planning " - "and materials teams about capacity, contract manufacturers, and suppliers. " - "Keep answers short enough to read between meetings." -) - # Connect to the project # Read the Application Insights connection string and start exporting traces @@ -34,11 +27,8 @@ # Get a tracer for this script - # Create the agent staff are talking to + # Look up the agent so its id can be included in agent_reference # Ask each question inside its own span - - # Clean up resources by deleting the agent version - diff --git a/Labfiles/D-observe-evaluate-and-secure-agents/Solution/Python/agent_target.py b/Labfiles/D-observe-evaluate-and-secure-agents/Solution/Python/agent_target.py index c5b9e2044..4916dff30 100644 --- a/Labfiles/D-observe-evaluate-and-secure-agents/Solution/Python/agent_target.py +++ b/Labfiles/D-observe-evaluate-and-secure-agents/Solution/Python/agent_target.py @@ -8,7 +8,7 @@ evaluate() calls this once per line of caldova_eval.jsonl, passing the columns of that line as keyword arguments, and merges the returned dictionary back into -the row as `outputs.*`. So returning {"response": ...} makes `${outputs.response}` +the row as `target.*`. So returning {"response": ...} makes `${target.response}` available to every evaluator. """ @@ -37,15 +37,16 @@ def __init__(self): ) self.openai_client = self.project_client.get_openai_client() - # Fail early with a useful message rather than once per dataset row. - self.project_client.agents.get(agent_name=self.agent_name) + # Fail early with a useful message rather than once per dataset row, and keep the + # id so agent_reference can correlate traces with this agent. + self.agent = self.project_client.agents.get(agent_name=self.agent_name) def __call__(self, *, query: str, **kwargs) -> dict: """Answer one question. Extra dataset columns arrive in kwargs and are ignored.""" response = self.openai_client.responses.create( input=query, extra_body={ - "agent_reference": {"name": self.agent_name, "type": "agent_reference"} + "agent_reference": {"name": self.agent.name, "id": self.agent.id, "type": "agent_reference"} }, ) return {"response": response.output_text} diff --git a/Labfiles/D-observe-evaluate-and-secure-agents/Solution/Python/evaluate_agent.py b/Labfiles/D-observe-evaluate-and-secure-agents/Solution/Python/evaluate_agent.py index d07c688f0..347ce4a4f 100644 --- a/Labfiles/D-observe-evaluate-and-secure-agents/Solution/Python/evaluate_agent.py +++ b/Labfiles/D-observe-evaluate-and-secure-agents/Solution/Python/evaluate_agent.py @@ -62,20 +62,20 @@ def evaluator_endpoint(): "column_mapping": { "query": "${data.query}", "context": "${data.context}", - "response": "${outputs.response}", + "response": "${target.response}", } }, "relevance": { "column_mapping": { "query": "${data.query}", - "response": "${outputs.response}", + "response": "${target.response}", } }, "similarity": { "column_mapping": { "query": "${data.query}", "ground_truth": "${data.ground_truth}", - "response": "${outputs.response}", + "response": "${target.response}", } }, }, diff --git a/Labfiles/D-observe-evaluate-and-secure-agents/Solution/Python/red_team_agent.py b/Labfiles/D-observe-evaluate-and-secure-agents/Solution/Python/red_team_agent.py index 390f0c93e..4f211afcf 100644 --- a/Labfiles/D-observe-evaluate-and-secure-agents/Solution/Python/red_team_agent.py +++ b/Labfiles/D-observe-evaluate-and-secure-agents/Solution/Python/red_team_agent.py @@ -17,11 +17,15 @@ agent_name = os.getenv("AGENT_NAME", "caldova-knowledge-agent") SEED_PROMPTS = Path("data/attack_objectives.json") -OUTPUT = Path("redteam_scan.json") +# scan() treats output_path as a directory and writes evaluation_result.json inside it +OUTPUT_DIR = Path("redteam_output") +OUTPUT = OUTPUT_DIR / "evaluation_result.json" credential = DefaultAzureCredential() project_client = AIProjectClient(endpoint=project_endpoint, credential=credential) openai_client = project_client.get_openai_client() +# Look up the agent so its id can be included in agent_reference +agent = project_client.agents.get(agent_name=agent_name) # Build the callback that sends one attack prompt to your agent @@ -30,7 +34,7 @@ def caldova_agent(query: str) -> str: try: response = openai_client.responses.create( input=query, - extra_body={"agent_reference": {"name": agent_name, "type": "agent_reference"}}, + extra_body={"agent_reference": {"name": agent.name, "id": agent.id, "type": "agent_reference"}}, ) return response.output_text except Exception as error: # a blocked prompt is a result, not a crash @@ -75,7 +79,7 @@ async def main(): AttackStrategy.Flip, AttackStrategy.Compose([AttackStrategy.Base64, AttackStrategy.ROT13]), ], - output_path=str(OUTPUT), + output_path=str(OUTPUT_DIR), ) # Read the scorecard back and show the headline numbers diff --git a/Labfiles/D-observe-evaluate-and-secure-agents/Solution/Python/traced_agent.py b/Labfiles/D-observe-evaluate-and-secure-agents/Solution/Python/traced_agent.py index d831586e2..6e915ec7c 100644 --- a/Labfiles/D-observe-evaluate-and-secure-agents/Solution/Python/traced_agent.py +++ b/Labfiles/D-observe-evaluate-and-secure-agents/Solution/Python/traced_agent.py @@ -4,14 +4,13 @@ # Add references from azure.identity import DefaultAzureCredential from azure.ai.projects import AIProjectClient -from azure.ai.projects.models import PromptAgentDefinition from azure.monitor.opentelemetry import configure_azure_monitor from opentelemetry import trace # Load environment variables from .env file load_dotenv() project_endpoint = os.getenv("PROJECT_ENDPOINT") -model_deployment = os.getenv("MODEL_DEPLOYMENT_NAME") +agent_name = os.getenv("AGENT_NAME", "caldova-knowledge-agent") # Turn on GenAI tracing os.environ.setdefault("AZURE_EXPERIMENTAL_ENABLE_GENAI_TRACING", "true") @@ -24,13 +23,6 @@ "Which contract manufacturers could fast-track us inside a three-month window?", ] -AGENT_NAME = "caldova-planning-assistant" -INSTRUCTIONS = ( - "You are the Caldova planning assistant. You answer questions from planning " - "and materials teams about capacity, contract manufacturers, and suppliers. " - "Keep answers short enough to read between meetings." -) - # Connect to the project with ( DefaultAzureCredential() as credential, @@ -51,15 +43,8 @@ # Get a tracer for this script tracer = trace.get_tracer(__name__) - # Create the agent staff are talking to - agent = project_client.agents.create_version( - agent_name=AGENT_NAME, - definition=PromptAgentDefinition( - model=model_deployment, - instructions=INSTRUCTIONS, - ), - ) - print(f"Agent created (name: {agent.name}, version: {agent.version})") + # Look up the agent so its id can be included in agent_reference + agent = project_client.agents.get(agent_name=agent_name) # Ask each question inside its own span with tracer.start_as_current_span("morning-planning-review") as shift_span: @@ -72,12 +57,8 @@ response = openai_client.responses.create( conversation=conversation.id, input=question, - extra_body={"agent_reference": {"name": agent.name, "type": "agent_reference"}}, + extra_body={"agent_reference": {"name": agent.name, "id": agent.id, "type": "agent_reference"}}, ) question_span.set_attribute("caldova.answer_length", len(response.output_text)) print(f"\nQ{number}: {question}") print(f"A{number}: {response.output_text}") - - # Clean up resources by deleting the agent version - project_client.agents.delete_version(agent_name=agent.name, agent_version=agent.version) - print("\nAgent deleted") diff --git a/Labfiles/D-observe-evaluate-and-secure-agents/setup/check_env.py b/Labfiles/D-observe-evaluate-and-secure-agents/setup/check_env.py index 9af694962..1ba5f8250 100644 --- a/Labfiles/D-observe-evaluate-and-secure-agents/setup/check_env.py +++ b/Labfiles/D-observe-evaluate-and-secure-agents/setup/check_env.py @@ -19,7 +19,7 @@ Tasks and what they need: - Task 1 (core, code) PROJECT_ENDPOINT, MODEL_DEPLOYMENT_NAME + Task 1 (core, code) PROJECT_ENDPOINT, AGENT_NAME Task 2 (core, code) PROJECT_ENDPOINT, MODEL_DEPLOYMENT_NAME, AGENT_NAME Task 3 (optional, code) PROJECT_ENDPOINT, AGENT_NAME @@ -312,7 +312,7 @@ def _read_quoted(text, quote, escape_aware): # Which .env keys each task needs to run on its own. TASK_REQUIREMENTS = { - 1: ["PROJECT_ENDPOINT", "MODEL_DEPLOYMENT_NAME"], + 1: ["PROJECT_ENDPOINT", "AGENT_NAME"], 2: ["PROJECT_ENDPOINT", "MODEL_DEPLOYMENT_NAME", "AGENT_NAME"], 3: ["PROJECT_ENDPOINT", "AGENT_NAME"], } @@ -356,7 +356,7 @@ def looks_like_placeholder(value): "(for example, gpt-4o). You can see it in the Foundry portal under your project." ), "AGENT_NAME": ( - "Tasks 2 and 3 evaluate a grounded agent. Either reuse the knowledge agent " + "Every task uses a grounded agent. Either reuse the knowledge agent " "from Lab B and set AGENT_NAME to its name, or create one here by running, " "from the Python folder: python ../setup/bootstrap_agent.py" ),