PromQL Alerts: Google Quotas

[mrsillydog]

This PR updates Google Quota alerts to use PromQL instead of the deprecated MQL.

All alerts in this PR are pretty much best effort - not sure what's going on here (maybe these are just templates?) but none of the MQL alerts could even be imported into Google Cloud as-is due to the metrics being template variables. So, I just converted everything best-effort, then mimicked other alert JSON changes. If you replace all the templated variables with actual meaningful content, everything should work fine.

[mrsillydog]

This had some pretty complicated timezone logic that didn't translate well to PromQL, so it has changed the most of any of these alerts.

[lyanco]

Daily quotas always roll over at UTC-8 midnight. So any MQL which references midnight West Coast time can be translated to hour()-8 = 0

It's still confusing to write because PromQL doesn't do "count from a fixed point in time until now" very well, but at least its feasible and time zones aren't needed.

[mrsillydog]

I'm not sure I'm following - this query basically wants to do "aggregate every 24 hours and then reset at midnight West Coast". I've done some exploration around the hour()-8, but it doesn't seem like it gives us that, at least not correctly for multiple consecutive days. Can you write a pretty basic example of what you're thinking of?

[lyanco]

Yeah - it's pretty hard to do - but I think I've figured out the basics -

Something like this will generate a time series that is 1 if and only if the timestamp's day in utc-8 is equal to the current day (probably want to sub out the up metric for the quota metric or some other GCP metric that everyone is guaranteed to have such as networking.googleapis.com/vm_flow/rtt):

clamp_max(day_of_year(max(timestamp(up @ end()))) == day_of_year(max(timestamp(up)) - (8*3600)), 1)

so I think then you can take the raw quota metric, filter it to just points where the above is 1, then run a sum_over_time[24h] on the result of that filtered raw quota metric

This should output a single point that gives you the sum since the day changed utc-8

[mrsillydog]

Couple screenshots:

Full query (single-line JSON-ified version is not particularly readable):

Just the clamp_max bit, as a proof of concept:

[lyanco]

I'm gonna need to pull in some of the cloud alerting folks to review this one. It's complicated.

[lyanco]

Note to self - the following is a better sub for clamp_max that doesn't incur 2 Monarch queries:

clamp_max(
day_of_year(max_over_time(vector(time())[1d:] @ end()) - (8*3600))
==
day_of_year(vector(time()) - (8*3600))
,1)