-
Notifications
You must be signed in to change notification settings - Fork 54
Description
Given the rollup:
<default>
<function>avg</function>
<retention>
<age>0</age>
<precision>60</precision>
</retention>
<retention>
<age>864000</age>
<precision>900</precision>
</retention>
<retention>
<age>1728000</age>
<precision>1800</precision>
</retention>
<retention>
<age>3456000</age>
<precision>3600</precision>
</retention>
<retention>
<age>10368000</age>
<precision>21600</precision>
</retention>
<retention>
<age>34560000</age>
<precision>43200</precision>
</retention>
<retention>
<age>63072000</age>
<precision>86400</precision>
</retention>
<retention>
<age>94608000</age>
<precision>604800</precision>
</retention>
</default>
Prometheus queries like rate(net_bytes_sent{host="$host", interface="$net"}[$__interval])
Expr: rate(net_bytes_recv{host="X", interface="backup"}[3h]) * 8
Step: 3h0m0s
Range: last 6 months.
do not return any data if the step interval is above 1h. I think it is clear that it happens, because the data points far back have too much distance and it can no longer derive correctly. The no data behavior is described in the prometheus documentation to happen if there are less than 3 points in a window.
I debugged the prometheus code last night. The storage implementation correctly returns the data points and passed them to prometheus, but prometheus drops them all. This happens deep inside prometheus roughly at https://github.com/prometheus/prometheus/blob/release-2.54/promql/engine.go#L1618. Honestly, I was not able to understand exactly what happens there. I simply assume at this point that stock prometheus is purely meant for series with always the same resolution.
Untagged metrics have their own rollup pattern which gives them a higher resolution. Handling the tagged series over carbonapi works perfectly fine.
Maybe a pattern that matches all tagged metrics and gives them a higher resolution for a longer period might work better.
I think this is the only option there is atm to optimize this on the prometheus side?