Skip to content

Fix OpenAI Agents SDK guardrail span classification #1096

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
devin-ai-integration wants to merge 7 commits into from
Closed

Fix OpenAI Agents SDK guardrail span classification #1096

devin-ai-integration wants to merge 7 commits into from

Conversation

@devin-ai-integration
Copy link
Contributor

@devin-ai-integration devin-ai-integration bot commented Jun 20, 2025 •
edited
Loading

Fix OpenAI Agents SDK guardrail span classification

Problem

The OpenAI Agents SDK integration was incorrectly classifying guardrail events as generic events or agent spans instead of properly mapping them to GUARDRAIL span types. This prevented proper observability and tracing of guardrail events in the AgentOps platform.

Root Cause

Two functions in the OpenAI Agents SDK instrumentation were missing handlers for GuardrailSpanData:

  1. get_span_kind() in exporter.py - didn't handle GuardrailSpanData, causing it to fall through to default SpanKind.INTERNAL
  2. get_span_attributes() in attributes/common.py - didn't have a case for GuardrailSpanData, so these spans didn't get proper AgentOps span kind attributes

Solution

Added proper GuardrailSpanData handling following the established patterns in the codebase:

Changes Made

  1. Updated get_span_kind() function in exporter.py:

    • Added GuardrailSpanData to the HandoffSpanData case for SpanKind.INTERNAL

  2. Added GUARDRAIL_SPAN_ATTRIBUTES mapping in attributes/common.py:

    • Consistent attribute extraction pattern for guardrail spans

  3. Added get_guardrail_span_attributes() function in attributes/common.py:

    • Extracts attributes from GuardrailSpanData objects
    • Sets proper AGENTOPS_SPAN_KIND to GUARDRAIL value
    • Handles guardrail-specific attributes like name and triggered

  4. Updated get_span_attributes() dispatcher in attributes/common.py:

    • Added GuardrailSpanData case to route to the new handler function

Testing

✅ Verified with OpenAI Agents SDK guardrail example:

  • Ran examples/openai_agents/agent_guardrails.py
  • Successfully executed with "Math homework guardrail tripped" message
  • AgentOps session links generated, confirming instrumentation is active

✅ Lint checks: Pre-commit hooks passed with automatic formatting applied

Impact

  • Guardrail events from OpenAI Agents SDK are now properly classified as GUARDRAIL spans
  • Maintains existing functionality for other span types (Agent, Function, Generation, etc.)
  • Follows established codebase patterns for consistency
  • Enables proper observability and tracing of guardrail events in AgentOps

Link to Devin run

https://app.devin.ai/sessions/0764439e61a6472fa04739fd1db8fb08

Requested by: Alex ([email protected])

Fixes #1105

@devin-ai-integration @areibman
Fix OpenAI Agents SDK guardrail span classification
3713a89 
- Add GuardrailSpanData handling to get_span_kind() function
- Add get_guardrail_span_attributes() function with proper GUARDRAIL span kind
- Add GUARDRAIL_SPAN_ATTRIBUTES mapping for consistent attribute extraction
- Add GuardrailSpanData case to get_span_attributes() dispatcher

Fixes issue where GuardrailSpanData spans were falling through to generic
classification instead of being properly mapped to GUARDRAIL span type.

Co-Authored-By: Alex <[email protected]>
@codecov
Copy link

codecov bot commented Jun 20, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 85.71429% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
.../instrumentation/agentic/openai_agents/exporter.py 0.00% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

@areibman areibman requested a review from fenilfaldu June 20, 2025 06:41
@devin-ai-integration
Copy link
Contributor Author

devin-ai-integration bot commented Jun 20, 2025

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@areibman
Copy link
Contributor

areibman commented Jun 20, 2025

Vibe coded but looks good? Can you test @fenilfaldu ?

devin-ai-integration bot and others added 2 commits June 20, 2025 06:50
@devin-ai-integration @areibman
Add unit tests for guardrail span handling
3a5268f 
- Add test_guardrail_span_attributes() to test main functionality
- Add test_guardrail_span_attributes_without_optional_fields() for edge cases
- Add GuardrailSpanData test case to test_span_attributes_dispatcher()
- Import get_guardrail_span_attributes in test imports

Addresses codecov/patch failure by adding comprehensive test coverage
for the new guardrail span handling code.

Co-Authored-By: Alex <[email protected]>
@devin-ai-integration @areibman
Fix ruff formatting in test file
436afe7 
Apply automatic formatting changes from ruff-format to resolve
pre-commit check failure. Adds blank line after docstring.

Co-Authored-By: Alex <[email protected]>
@devin-ai-integration
Copy link
Contributor Author

devin-ai-integration bot commented Jun 20, 2025

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@fenilfaldu
Copy link
Contributor

fenilfaldu commented Jun 20, 2025

I think guardrail instrumentation needs rework !

Current behavior:

  • Each guardrail creates 2 spans: {guardrail_name} and {guardrail_name}.guardrail
  • This duplication occurs for both input and output guardrails
  • It's unclear which span represents the actual guardrail execution

Screenshot 2025-06-21 at 12 59 10 AM

@areibman
Copy link
Contributor

areibman commented Jun 20, 2025

I think guardrail instrumentation needs rework !

Current behavior:

  • Each guardrail creates 2 spans: {guardrail_name} and {guardrail_name}.guardrail
  • This duplication occurs for both input and output guardrails
  • It's unclear which span represents the actual guardrail execution

Screenshot 2025-06-21 at 12 59 10 AM

Actually I think this isn't a total guardrails rework more so than how we do instrumentation for OpenAI Agents SDK

devin-ai-integration bot and others added 2 commits June 20, 2025 19:42
@devin-ai-integration @areibman
Fix duplicate guardrail span creation
310b61e 
- Remove AgentOps @guardrail decorator from OpenAI Agents SDK examples
- OpenAI Agents SDK's @input_guardrail provides instrumentation automatically
- Add documentation explaining proper usage pattern
- Fixes issue where each guardrail created 2 spans: {name} and {name}.guardrail

Co-Authored-By: Alex <[email protected]>
@devin-ai-integration @areibman
Merge remote documentation updates with guardrail span duplication fix
37d1fb8 
Co-Authored-By: Alex <[email protected]>
@fenilfaldu
Copy link
Contributor

fenilfaldu commented Jun 20, 2025

Devin removed the wrong span - we were previously capturing usage metrics correctly in guardrail spans.

Before:
Screenshot 2025-06-21 at 1 35 56 AM

After:
Screenshot 2025-06-21 at 1 36 48 AM

@devin-ai-integration devin-ai-integration bot mentioned this pull request Jun 23, 2025
Open
5 tasks
@devin-ai-integration
Copy link
Contributor Author

devin-ai-integration bot commented Jun 29, 2025

Closing due to inactivity for more than 7 days. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fenilfaldu fenilfaldu Awaiting requested review from fenilfaldu

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

OpenAI Agents SDK Guardrail Span Classification Issues

3 participants

@areibman @fenilfaldu