grpc: the client connection is closing" msg="returning partial response"

[craigbryson]

Hey, we are getting a bunch of these errors intermittently:

level=warn ts=2022-10-19T06:08:03.527308807Z caller=proxy.go:531 component=proxy request="min_time:1665554280000 max_time:1666159680000 matchers:<name:\"device\" value:\"xxxx\" > matchers:<name:\"system\" value:\"xxx\" > matchers:<name:\"__name__\" value:\"events_per_system\" > max_resolution_window:12000 aggregates:COUNTER step:60000 range:600000 " err="receive series from Addr: 10.2.0.21:10901 LabelSets: {cluster=\"cluster\", prometheus=\"monitoring/monitoring-kube-prometheus-prometheus\", prometheus_replica=\"prometheus-monitoring-kube-prometheus-prometheus-0\"} Mint: 1642768639552 Maxt: 1666152000000: rpc error: code = Canceled desc = grpc: the client connection is closing" msg="returning partial response"

Where 10.2.0.21 is the service address for storegateway, but the storegateway log shows no errors - could somebody shed some light on the cause of this error?

[matej-g]

Hey @craigbryson, from that log it simply seems the request is being cancelled, might be a timeout or other issue why store is not responding. Can you take a look at your store metrics, if you see something suspicious?

[craigbryson]

Sure - nothing sticks out to me, except the duration can be a little high: