Skip to content

Prometheus - Rate/Irate Function not returning any Data for low resolution data #306

@Hedius

Description

@Hedius

Given the rollup:

        <default>
            <function>avg</function>
            <retention>
                <age>0</age>
                <precision>60</precision>
            </retention>
            <retention>
                <age>864000</age>
                <precision>900</precision>
            </retention>
            <retention>
                <age>1728000</age>
                <precision>1800</precision>
            </retention>
            <retention>
                <age>3456000</age>
                <precision>3600</precision>
            </retention>
            <retention>
                <age>10368000</age>
                <precision>21600</precision>
            </retention>
            <retention>
                <age>34560000</age>
                <precision>43200</precision>
            </retention>
            <retention>
                <age>63072000</age>
                <precision>86400</precision>
            </retention>
            <retention>
                <age>94608000</age>
                <precision>604800</precision>
            </retention>
        </default>

Prometheus queries like rate(net_bytes_sent{host="$host", interface="$net"}[$__interval])

Expr: rate(net_bytes_recv{host="X", interface="backup"}[3h]) * 8
Step: 3h0m0s

Range: last 6 months.

do not return any data if the step interval is above 1h. I think it is clear that it happens, because the data points far back have too much distance and it can no longer derive correctly. The no data behavior is described in the prometheus documentation to happen if there are less than 3 points in a window.

I debugged the prometheus code last night. The storage implementation correctly returns the data points and passed them to prometheus, but prometheus drops them all. This happens deep inside prometheus roughly at https://github.com/prometheus/prometheus/blob/release-2.54/promql/engine.go#L1618. Honestly, I was not able to understand exactly what happens there. I simply assume at this point that stock prometheus is purely meant for series with always the same resolution.

Untagged metrics have their own rollup pattern which gives them a higher resolution. Handling the tagged series over carbonapi works perfectly fine.

Maybe a pattern that matches all tagged metrics and gives them a higher resolution for a longer period might work better.
I think this is the only option there is atm to optimize this on the prometheus side?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions