High tail latencies when doing `textDocument/inlayHint` requests

[crtschin]

Your environment

Which OS do you use?
Fedora
Which version of GHC do you use and how did you install it?
9.10.3 via nixpkgs
How is your project built (alternative: link to the project)?
This repository, via cabal using the nix shell.

Which LSP client (editor/plugin) do you use?
Helix
Which version of HLS do you use and how did you install it?
HEAD (7d02159).
Have you configured HLS in any way (especially: a hie.yaml file)?
No

Steps to reproduce

Using lsp-recorder, recorded a session where I took HEAD (or atleast close to it IIRC), and I added a reader pool in front of HieDb interactions (was playing around with this idea). Replaying the resulting trace gives the following results for inlay hints. You can find the snapshot and event traces here: hls-trace.tar.gz, which can be replayed locally with lsp-recorder replay.

> lsp-recorder replay --server-command "<insert HLS path here> --lsp" --trace hls-trace.jsonl --report hls-trace.report --speedup-factor 6 --timeout 60

Notably textDocument/inlayHint is the second highest sent request type in the above session, completion is the first with 500 occurences.

Expected behaviour

I did not expect such long latency tails, inlay hints happen quite often. From what I understand moving the editor around in Helix will already cause them. Though I might have to continue investigating this a bit more, as I only recorded the one HLS session.

Actual behaviour

"textDocument/inlayHint": {
  "count": 469,
  "p50_ms": 126.304996,
  "p95_ms": 6388.946637,
  "p99_ms": 9595.631002999999
},

Debug information