Workflow CreateHandoffBuilderWith

[jefhughe]

I searching and found Composable workflows? #2258 but its not quite the same issue I think I'm having.

I am using the CreateHandoffBuilderWith and if I pass a single utterance it works pretty consistently. When I ask for a muliti-part question like: Can you get me the matter 'X' and then get me all the open tasks for the matter it's a 50\50 shot it it works.

I am also streaming the respones back.

MAIN ASK

1. Looking for some insight on what I am missing?
2. If I make any addtional updates or instructions the triage_agent is stops working or handing off.

STEPS TAKEN:

• More prompt tuning for each agent on aspects of complete your required:

Once I added this to the Instructions for the Matter agent it worked better. I was assuming that the triage_agent would be more of the orchestrator to know that I have these things to do and once and agent is done I can move on to the next action but this never occours.

Prompt Tuning

- **CRITICAL HANDOFF RULES:**
  - If the user requests TASKS related to a matter, after providing matter details, you MUST hand off to the Task specialty agent.
  - If the user requests NOTES related to a matter, after providing matter details, you MUST hand off to the Note specialty agent.
  - If the user requests CONTACTS related to a matter, after providing matter details, you MUST hand off to the Contact specialty agent.
  - Complete your matter-related work FIRST, then initiate the appropriate handoff.
  - Use clear language like "Now handing off to the Task agent to retrieve tasks for this matter" to trigger handoffs.
- If a request falls outside your scope (for example, questions about company contacts and employees), respond with: `"Sorry I don't have that capability yet."`
- Never attempt actions you cannot perform.

      ChatClientAgent triageAgent = new(client,
      """ 
      1) You determine which agent to use based on the user's request.  
      2) Create an execution plan then handoff to the first action based on the deterimed action needed.
      3) ALWAYS handoff to another agent.
      """,
            "triage_agent",
            "Routes messages to the appropriate specialist agent");           

// Create workflow - triage always hands off to worker agents
var workflow = AgentWorkflowBuilder.CreateHandoffBuilderWith(triageAgent)
      .WithHandoffs(triageAgent, [ activityAgent, taskAgent, matterAgent, noteAgent, contactAgent ])
      .WithHandoffs([activityAgent, taskAgent, matterAgent, noteAgent, contactAgent ], triageAgent)
      .WithHandoff(matterAgent, noteAgent)
      .WithHandoff(matterAgent, taskAgent)
      .Build();


await foreach(var chunk in WorkflowExecutor.RunWorkflowAsyncStreamWithContext(workflow, messages, req.ThreadId, workflowContextCache, distributedCheckpointManager, workflowContextConfig, ct))
    {
        await http.Response.WriteAsync(chunk, ct);
        await http.Response.Body.FlushAsync(ct);
    }

Example of Agent Creation

all of the agents passed in to the CreateHandoffBuilderWith are constructed in the same manner but have different configurations and personas assigned such as noteAgent, taskAgent....

    public AIAgent CreateAgent(IChatClient client, string openAIDeploymentName,
    AgentAISettings agentConfig,
    UserTool userTool,
    MatterTool matterTool)
    {
        if (!String.IsNullOrWhiteSpace(agentConfig.InstructionPrompt))
        {
            _persona = agentConfig.InstructionPrompt;  
        }

        var tools = new List<AITool>
        { 
            AIFunctionFactory.Create(userTool.GetCurrentUser), 
            AIFunctionFactory.Create(matterTool.SearchMatter)
        };

        var taskAgent = new ChatClientAgent(client, new ChatClientAgentOptions
        {
            Name = "Matter",
            Description = "Matter specialty agent for searching matters and obtaining data around the matter",
            Instructions = _persona,
            ChatOptions = new ChatOptions
            {
                Tools = tools,
                Temperature = agentConfig.Temperature,
                TopP = agentConfig.TopP,
                MaxOutputTokens = agentConfig.MaxOutputTokens,
                ToolMode = agentConfig.AgentToolMode,
                ResponseFormat = agentConfig.OutputFormat
            }
        });
        return taskAgent;
    }

[lokitoth]

Do you have an example trace? It generally helps in identifying the failure mode and what can be done to get it to stay on track.

[jefhughe]

the text # Next Question (5) (This Fails to Handoff) is where it started to have issues. This is all test dummy data just for clarity

This covers 5 turns from the chat to the agent workflow.

Process Flow

This shows the multi-turn failing on the 5th turn of a conversation where the handoff gets lost.

Initial Ask (1)

{
  "text": "get me all my open matters",
  "threadId": ""
}

Response (1)

HTTP 200 OK
Content-Type: text/plain; charset=utf-8

[meta threadToken=69ace5b7754948638d5ea0efb1262ebe]
[SUPER STEP STARTED] : 0]
[Executing: HandoffStart]
[Completed: HandoffStart]
[Executing: HandoffStart]
[Completed: HandoffStart]

[AUTO-UPDATE] ThreadId set to: 69ace5b7754948638d5ea0efb1262ebe
[Step Completed]
[SUPER STEP STARTED] : 1]
[Executing: triage_agent_c44ab6325ab04171a9be02e224ae0650]

To fulfill your request for all your open matters in XXXX, the XXXX Matter specialty agent is the right choice.

Execution plan:

1. Query XXXX for all matters with a status of "open."
2. Retrieve and display details about these open matters.

Now proceeding with the handoff to the XXXX
 Matter agent.
[Completed: triage_agent_c44ab6325ab04171a9be02e224ae0650]
[Step Completed]
[SUPER STEP STARTED] : 2]
[Executing: XXXXMatter_75b71d928e9243e488300dc53f3eb7eb]

Here are all your open matters:

 {
  "matters": [
    {
      "id": 1821078837,
      "etag": "\"6c8b753568ff9f8b6a68f6d0d2c36334\"",
      "number": 63
    },
    {
      "id": 1821724947,
      "etag": "\"6c955113686d67166a68f6d0d2c36334\"",
      "number": 64 
      }
  ]
} 

Would you like to view tasks, notes, or contacts associated with any of
 these matters?
[Completed: XXXXMatter_75b71d928e9243e488300dc53f3eb7eb]

[Step Completed]
[SUPER STEP STARTED] : 3]
[Executing: HandoffEnd]
[Workflow Complete]
[Completed: HandoffEnd]

[Step Completed]

Next Question (2)

threadId is the cache thread for the chat context

{
  "text": "get me all the tasks for 00063-MatterName",
  "threadId": "69ace5b7754948638d5ea0efb1262ebe"
}

Response (2)

HTTP 200 OK
Content-Type: text/plain; charset=utf-8

[SUPER STEP STARTED] : 0]
[Executing: HandoffStart]
[Completed: HandoffStart]
[Executing: HandoffStart]
[Completed: HandoffStart]

[Step Completed]
[SUPER STEP STARTED] : 1]
[Executing: triage_agent_6d5db1de77f44be6ad409f2f7b5c9add]

To retrieve all tasks related to matter 00063-MatterName, the XXXX Task specialty agent is best suited for this.

Execution plan:

1. Identify the matter ID for 00063-MatterName, which is 1821078837.
2. Query XXXX for all tasks associated with this matter.
3. Return a list of tasks including details like description, due date, and status.

Now proceeding with the handoff to the XXXX Task agent.
[Completed: triage_agent_6d5db1de77f44be6ad409f2f7b5c9add]

[Step Completed]
[SUPER STEP STARTED] : 2]
[Executing: XXXXTask_c11dcb9849684d4aa31138991bf7df09]

Here are all tasks associated with Matter 00063-MatterName:

{
  "tasks": [
    {
      "id": 1423351810,
      "etag": "\"54d6a002691e59aa6a68f6d0d2c36334\"",
      "name": "Draft contact document for research filings",
      "status": "complete" 
      },
      "assigner": {
        "id": 359367402,
        "name": "First Last"
      },
      "matter": {
        "id": 1821078837,
        "display_number": "00063-MatterName",
        "description": "Representation of ExxonMobil in a business litigation matter involving allegations of patent infringement related to horizontal drilling technologies.",
        "client": {
          "id": 2404767087,
          "name": "MICHAEL MatterName",
          "type": "Person"
        }
      } 
  ]
}

Would you like me to filter these tasks by status (e.g., only pending tasks)?
[Completed: XXXXTask_c11dcb9849684d4aa31138991bf7df09]

[Step Completed]
[SUPER STEP STARTED] : 3]
[Executing: HandoffEnd]
[Workflow Complete]
[Completed: HandoffEnd]

[Step Completed]

Next Question (3)

{
  "text": "bill 1 hour of work for matter 00063-MatterName for Paper shredding and document filings",
  "threadId": "69ace5b7754948638d5ea0efb1262ebe"
}

Response (3)

HTTP 200 OK
Content-Type: text/plain; charset=utf-8

[SUPER STEP STARTED] : 0]
[Executing: HandoffStart]
[Completed: HandoffStart]
[Executing: HandoffStart]
[Completed: HandoffStart]

[Step Completed]
[SUPER STEP STARTED] : 1]

[Executing: triage_agent_53b12e1d44854dd895bc368cf1901cbf]

To bill time for 1 hour of work related to matter 00063‑MatterName with the description “Paper shredding and document filings”, the request should go to the XXXX activity logging agent.

Execution plan:

1. Identify the matter ID (1821078837).
2. Create a new time entry in XXXX for 1 hour under that matter.
3. Include the description “Paper shredding and document filings” and link it to the appropriate user/account for proper billing.

Now proceeding with the handoff to the XXXX activity logging agent.
[Completed: triage_agent_53b12e1d44854dd895bc368cf1901cbf]

[Step Completed]
[SUPER STEP STARTED] : 2]
[Executing: XXXXActivity_b11a38c864c24daabe84099a193618c6]

It looks like the hourly rate for billing this time entry isn’t specified.  
Could you please confirm the rate to use for this matter 
(e.g., $250/hour )? Once I have that, I’ll finalize the entry.
[Completed: XXXXActivity_b11a38c864c24daabe84099a193618c6]
 

[Step Completed]
[SUPER STEP STARTED] : 3]
[Executing: HandoffEnd]
[Workflow Complete]
[Completed: HandoffEnd]
[Step Completed]

You can see from the highlighted text above I get a response back from the Activity agent which is valid since there is missing data. works as expected.

Response to Assistant (4)

{
  "text": "use 324 as the bill rate",
  "threadId": "69ace5b7754948638d5ea0efb1262ebe"
}

Response (4)

HTTP 200 OK
Content-Type: text/plain; charset=utf-8

[SUPER STEP STARTED] : 0]
[Executing: HandoffStart]
[Completed: HandoffStart]
[Executing: HandoffStart]
[Completed: HandoffStart]

[Step Completed]
[SUPER STEP STARTED] : 1]
[Executing: triage_agent_ed001f73ca7949e08f4c85b8bacc8822]

Got it — we'll bill **1 hour at  $324/hour** for matter **00063‑MatterName** with the description *“Paper shredding and document filings.”*

Executionplan:

1. Create a time entry in XXXX for 1 hour of billable work.
2. Link it to matter ID 1821078837 (00063‑MatterName).
3. Apply a $324/hour rate and ensure description accuracy for billing records.
4. Save the entry under your user account for reporting/invoicing.

Now handing off to the XXXX activity logging agent to record the billable time.
[Completed: triage_agent_ed001f73ca7949e08f4c85b8bacc8822]
 
[Step Completed]
[SUPER STEP STARTED] : 2]
[Executing: XXXXActivity_ad0ebcbc78f04b8bb7df520ebbdfa1d1]

Your time entry for 1 hour at $324/hour on matter 00063‑MatterName with the note “Paper shredding and document filings” has been successfully logged
and marked as billable.

Would you like me to prepare this entry for invoicing or leave it as a recorded activity for now?
[Completed: XXXXActivity_ad0ebcbc78f04b8bb7df520ebbdfa1d1]

[Step Completed]
[SUPER STEP STARTED] : 3]
[Executing: HandoffEnd]
[Workflow Complete]
[Completed: HandoffEnd]

[Step Completed]

Next Question (5) (This Fails to Handoff)

{
  "text": "what are the notes for this matter",
  "threadId": "69ace5b7754948638d5ea0efb1262ebe"
}

Response (5)

This is where it begins to fail on the workflow.

HTTP 200 OK
Content-Type: text/plain; charset=utf-8

[SUPER STEP STARTED] : 0]
[Executing: HandoffStart]
[Completed: HandoffStart]
[Executing: HandoffStart]
[Completed: HandoffStart]
[Step Completed]
[SUPER STEP STARTED] : 1]
[Executing: triage_agent_80d3b319a65846b3900ffa6255e71eee]

To get all notes associated with the matter 00063‑MatterName, the XXXX Note specialty agent should handle this.

Execution plan:

1. Identify the matter ID (1821078837) for 00063‑MatterName.
2. Query XXXX for all notes linked to that matter.
3. Display the note text, creation date, and author.

Now handing off to the XXXX Note agent to retrieve the notes.
[Completed: triage_agent_80d3b319a65846b3900ffa6255e71eee]

[Step Completed]
[SUPER STEP STARTED] : 2]
[Executing: HandoffEnd]
[Workflow Complete]
[Completed: HandoffEnd]

[Step Completed]

Issue

Never hands off to the Note Agent.

Observations:

• Usually works for the first few turns then the workflow handoffs stop working.
• this is regardless of the order of operations. It could be asking for notes first or tasks or add an activity.
• the permutations all show the same result.