Significant Performance Drop Off Between v15.2 and v16.0

[rtkwlfhbousfield]

We attempted to update our graphite exporters from v15.2 to v16.0and saw a roughly 100% increase in the amount of compute required.

Container configs:

      containers:
        - name: graphite-exporter
          args:
            - --graphite.mapping-config=/etc/graphite-exporter/graphite-exporter.yaml
            - --web.listen-address=<environment dependent>
            - --web.telemetry-path=/metrics
            - --graphite.listen-address=2003
            - --graphite.cache-size=0
            - --graphite.sample-expiry=2m
            - --log.level=info
            - --graphite.mapping-strict-match

We are pushing more data than usual through these instances, roughly 10000 samples per thread. Were running 32 replicas per host.

Is there any known performance issues with the v16.0 or newer. At our scale it roughly doubled the computer requirements of our pipeline.

[matthiasr]

Hmm, interesting and concerning. Are you able to get CPU profiles from both versions?

The main thing that happened in 0.16 was the logging library switch, and it's entirely possible that we are now doing extra work in the hot path even if the log level ends up making this not produce any output.