bug: ai-lakera-guard streaming action=alert ignores fail_open=false (fail-closed) on Lakera errors

[janiussyafiq]

Current Behavior

With ai-lakera-guard configured for response scanning (direction=output or both) on streaming traffic with action=alert, the plugin discards the deny result of the response scan. When the Lakera Guard API errors, times out, or is unreachable while fail_open=false, the streamed response is allowed through to the client instead of being blocked.

In the response body filter, moderate() correctly returns a deny code/body on a Lakera error when fail_open=false. The non-streaming path and the streaming block path both honor that return. But the streaming action=alert branch calls moderate_response(ctx, conf, text) and throws away its return value, then returns nothing — so the fail-closed decision is silently lost.

This contradicts the plugin's own documented contract. The action schema description states it affects flagged verdicts only — "Lakera API errors/timeouts stay governed by fail_open even in alert mode" — and fail_open defaults to false (fail-closed).

Location (PR #13606), apisix/plugins/ai-lakera-guard.lua:

• moderate() error branch: core.log.error(... "fail_open=false, blocking ...") ; return conf.deny_code, deny_message(...)
• streaming action=alert branch: calls moderate_response(ctx, conf, text) and ignores the returned (code, body).

Expected Behavior

In streaming action=alert, a Lakera error/timeout with fail_open=false should fail closed (block/replace the response), exactly as the non-streaming and streaming block paths do. action=alert should only suppress flagged verdicts; it must not change error handling.

Caveat for the fix: alert mode streams chunks to the client in real time, so by end-of-stream the body has already been delivered and cannot be retracted. Failing closed in alert mode therefore requires buffering the stream when fail_open=false, or explicitly rejecting/rewriting this config combination and documenting it. Streaming-alert error/timeout tests should be added.

Error Logs

On a Lakera error with fail_open=false, the plugin logs:

ai-lakera-guard: <err>; fail_open=false, blocking response

…yet the streamed response is still delivered in full. The log claims the response was blocked while the client receives the content — a silent fail-open (no crash).

Steps to Reproduce

1. Run APISIX with PR feat(ai-lakera-guard): scan LLM responses (direction output/both, non-streaming + streaming) #13606 applied.
2. Create a route with ai-proxy (or ai-proxy-multi) to an OpenAI-compatible upstream, plus ai-lakera-guard configured with direction=output, action=alert, fail_open=false.
3. Force the Lakera Guard endpoint to fail (point endpoint at an unreachable host, block egress, or force a timeout).
4. Send a streaming chat completion request ("stream": true).
5. Observe: the streamed assistant response is delivered in full even though the Lakera scan failed. Expected: a fail-closed (blocked) response.

Environment

• APISIX version: master with PR feat(ai-lakera-guard): scan LLM responses (direction output/both, non-streaming + streaming) #13606 (ai-lakera-guard response/output scanning).
• This is a code-path defect independent of OS / OpenResty / etcd versions; reproducible on any environment running that branch.

[janiussyafiq]

Reproduced on master (PR #13606, merged at 3ca5ddf3) with Test::Nginx

Setup: ai-proxy (openai-compatible) + ai-lakera-guard with direction=output, action=alert, fail_open=false. A mock Lakera endpoint returns HTTP 500 (error) when it scans the response. Same route config exercised twice: one non-streaming request, one streaming request.

Result:

• Non-streaming alert + Lakera error → correctly fails closed: client receives the deny body Response blocked by Lakera Guard.
• Streaming alert + Lakera error → fails open (the bug): the client receives the full assistant stream, even though the plugin logs the fail-closed decision:

ai-lakera-guard.lua:133: moderate_response(): ai-lakera-guard: Lakera Guard returned status 500; fail_open=false, blocking response

Leaked streamed body the client actually received (tokens delivered despite the "blocking response" log):

data: {... "delta":{"content":"trigger "} ...}
data: {... "delta":{"content":"lakera-error here"} ...}
data: {... "finish_reason":"stop","usage":{...} ...}
data: [DONE]

So moderate() returns the deny (code, body) on the Lakera error (and logs "blocking response"), but the streaming action=alert branch discards that return and the content streams through. Confirms the root cause and the fail-closed contract violation; the non-streaming path proves the same config blocks correctly.

Repro test (t/plugin/repro-p1-alert-failclosed.t) + 2 fixtures

t/fixtures/openai/chat-streaming-lakera-error.sse — assistant text contains lakera-error so the mock Lakera returns 500:

data: {"id":"chatcmpl-err123","object":"chat.completion.chunk","created":1700000000,"model":"gpt-4o-2024-05-13","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-err123","object":"chat.completion.chunk","created":1700000000,"model":"gpt-4o-2024-05-13","choices":[{"index":0,"delta":{"content":"trigger "},"finish_reason":null}]}

data: {"id":"chatcmpl-err123","object":"chat.completion.chunk","created":1700000000,"model":"gpt-4o-2024-05-13","choices":[{"index":0,"delta":{"content":"lakera-error here"},"finish_reason":null}]}

data: {"id":"chatcmpl-err123","object":"chat.completion.chunk","created":1700000000,"model":"gpt-4o-2024-05-13","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":10,"completion_tokens":8,"total_tokens":18}}

data: [DONE]

t/fixtures/openai/chat-lakera-error.json (non-streaming counterpart): a normal completion whose message.content is "trigger lakera-error here".

use t::APISIX 'no_plan';
log_level("info");
repeat_each(1);
no_long_string();
no_root_location();

add_block_preprocessor(sub {
    my ($block) = @_;
    if (!defined $block->request) { $block->set_value("request", "GET /t"); }
    my $http_config = $block->http_config // <<_EOC_;
        server {
            listen 6724;
            default_type 'application/json';
            location /v2/guard {
                content_by_lua_block {
                    local core = require("apisix.core")
                    local fixture_loader = require("lib.fixture_loader")
                    ngx.req.read_body()
                    local body = ngx.req.get_body_data() or ""
                    if core.string.find(body, "lakera-error") then
                        ngx.status = 500
                        ngx.say([[{"error":"simulated lakera error"}]])
                        return
                    end
                    ngx.status = 200
                    ngx.print(fixture_loader.load("lakera/scan-clean.json"))
                }
            }
        }
_EOC_
    $block->set_value("http_config", $http_config);
});
run_tests();

__DATA__

=== TEST 1: BASELINE (non-streaming) alert + fail_open=false + Lakera error => fails CLOSED
--- config
    location /t {
        content_by_lua_block {
            local t = require("lib.test_admin").test
            local code, body = t('/apisix/admin/routes/1', ngx.HTTP_PUT, [[{
                "uri": "/anything",
                "plugins": {
                  "ai-proxy": { "provider": "openai-compatible", "auth": { "header": { "Authorization": "Bearer token" } }, "options": { "model": "gpt-4" }, "override": { "endpoint": "http://127.0.0.1:1980/v1/chat/completions" }, "ssl_verify": false },
                  "ai-lakera-guard": { "api_key": "test-key", "lakera_endpoint": "http://127.0.0.1:6724/v2/guard", "direction": "output", "action": "alert", "fail_open": false }
                }
            }]])
            if code >= 300 then ngx.status = code end
            ngx.say(body)
        }
    }
--- response_body
passed

=== TEST 2: non-streaming alert + Lakera error is BLOCKED (deny body, content not leaked)
--- request
POST /anything
{ "messages": [ { "role": "user", "content": "say hi" } ] }
--- more_headers
X-AI-Fixture: openai/chat-lakera-error.json
--- response_body_like eval
qr/Response blocked by Lakera Guard/
--- error_log
fail_open=false, blocking response

=== TEST 3: SAME route, streaming request (direction=output, alert, fail_open=false)
--- config
    location /t {
        content_by_lua_block {
            local t = require("lib.test_admin").test
            local code, body = t('/apisix/admin/routes/1', ngx.HTTP_PUT, [[{
                "uri": "/anything",
                "plugins": {
                  "ai-proxy": { "provider": "openai-compatible", "auth": { "header": { "Authorization": "Bearer token" } }, "options": { "model": "gpt-4" }, "override": { "endpoint": "http://127.0.0.1:1980/v1/chat/completions" }, "ssl_verify": false },
                  "ai-lakera-guard": { "api_key": "test-key", "lakera_endpoint": "http://127.0.0.1:6724/v2/guard", "direction": "output", "action": "alert", "fail_open": false }
                }
            }]])
            if code >= 300 then ngx.status = code end
            ngx.say(body)
        }
    }
--- response_body
passed

=== TEST 4: BUG - streaming alert + Lakera error + fail_open=false LEAKS the response
--- request
POST /anything
{ "messages": [ { "role": "user", "content": "say hi" } ], "stream": true }
--- more_headers
X-AI-Fixture: openai/chat-streaming-lakera-error.sse
--- error_code: 200
--- response_body_like eval
qr/lakera-error here/
--- error_log
fail_open=false, blocking response

All four tests pass (the streaming case asserts the current, buggy behavior).