kong error ‘upstream prematurely closed connection while reading upstream’

[zyiqian]

Hi everyone..

I've got a weird problem here.

My KONG setup

kong:3.1
kong-ingress-controller:2.8

Current Behavior

Our two services are accessed across k8s clusters, routed through the kong gateway. Now we have captured the following error information in the production environment.

[error] 1126#0: *33991427 upstream prematurely closed connection while reading response header from upstream, client: 10.116.228.235, server: kong, request: "POST /nlp/filter/v1 HTTP/1.1", upstream: "http://10.116.186.204:8080/nlp/filter/v1", host: "xxxxx.com"

Client error:

{
  "message":"An invalid response was received from the upstream server"
}

I captured the TCP packet when the connection was closed.
Gateway ip: 10.116.212.39
Server ip: 10.116.186.204

I don't quite understand why the server has already responded to fin, and then kong sent another post?

service config:

{
  "retries": 5,
  "path": "/",
  "port": 80,
  "client_certificate": null,
  "updated_at": 1743594216,
  "connect_timeout": 60000,
  "protocol": "http",
  "read_timeout": 60000,
  "ca_certificates": null,
  "tags": null,
  "write_timeout": 60000,
  "created_at": 1743594216,
  "host": "service.prod.80.svc",
  "tls_verify": null,
  "id": "270fa588-34a6-5a49-a770-d45a8de5ff91",
  "enabled": true,
  "tls_verify_depth": null,
  "name": "prod.service-ingress.service.prod.80.svc"
}

I have tried the following configuration changes, but to no avail.

KONG_UPSTREAM_KEEPALIVE_IDLE_TIMEOUT = 0
KONG_UPSTREAM_KEEPALIVE_POOL_SIZE = 500

This phenomenon is sporadic and may only occur once or twice a day.

[zyiqian]

Adding these two configuration parameters "--timeout", "300", "--keep-alive", "10" on the server solved the problem.