Skip to content

docs(chat_settings): document that agent responses are untrusted text - #290

Open
northdpole wants to merge 1 commit into
polyai:mainfrom
northdpole:sec-304-document-response-rendering-contract
Open

docs(chat_settings): document that agent responses are untrusted text#290
northdpole wants to merge 1 commit into
polyai:mainfrom
northdpole:sec-304-document-response-rendering-contract

Conversation

@northdpole

Copy link
Copy Markdown
Contributor

Docs only, no behaviour change. Refs SEC-304.

Why

Nothing in the customer-facing docs told integrators how to treat the agent's reply. NCC found a client deployment where the model could be talked into emitting <script>alert(0123)</script> in a response body — the payload was smuggled across several turns as innocuous profile values, then the agent was asked to reassemble them (NCC-E033177-3UJ).

The platform is right not to strip markup: correct encoding depends on the render context and the platform cannot know it. That only works if integrators know it is their job.

The renderer that contained the finding was the customer's contact-centre widget, not ours — so the people who most needed this documented were exactly the ones reading these docs rather than our code.

What

Adds a "Rendering agent responses" section to the chat settings reference: a warning that responses are untrusted text, a table of what to encode where, and a note that embedding the PolyAI widget gives you DOMPurify sanitisation for free while consuming the Chat API directly does not.

It also calls out that safety filters are not an output encoder. They screen for harmful content, and an integrator who has configured them could reasonably assume they were covered. That conflation seemed like the most likely way someone ends up exposed.

Companion PR adds the internal equivalent to the webchat_handler Chat API README in poly_core: PolyAI-LDN/poly_core#45384

🤖 Generated with Claude Code

@northdpole
northdpole requested review from a team as code owners August 27, 2026 09:50
@northdpole

Copy link
Copy Markdown
Contributor Author

@AaronForinton — could I get your eyes on this when you have a moment? Docs-only, one new section in the chat settings reference.

Raising from a fork so I can't add you as a reviewer directly, hence the ping.

Context: it came out of an NCC pentest finding on a client deployment. The finding was that the agent can be talked into emitting HTML in a response, and the thing that contained it was the customer's own contact-centre widget rather than anything of ours — so the audience who needed this written down were the people reading these docs, not our code. Companion PR does the internal equivalent in poly_core (#45384).

Nothing in the customer-facing docs told integrators how to treat the agent's
reply. NCC found a client deployment where the model could be talked into
emitting <script>alert(0123)</script> in a response body — the payload was
smuggled across several turns as innocuous profile values, then the agent was
asked to reassemble them (NCC-E033177-3UJ).

The platform is right not to strip markup: the correct encoding depends on the
render context and the platform cannot know it. But that only works if the
people consuming responses know it is their job, and the docs never said so.

Adds a section to the chat settings reference covering what to encode where,
and calls out that safety filters screen content and are not an output encoder
— that conflation is the likely reason an integrator would assume they were
covered.

Refs SEC-304.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@northdpole
northdpole force-pushed the sec-304-document-response-rendering-contract branch from b33f071 to 639c16b Compare August 27, 2026 09:54

@Ruari-Phipps Ruari-Phipps left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if this is the right place for these docs. This section is about how to format the chat settings, not necessarily how to build a chat agent

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants