[Umbrella issue] How we monitor k8s-infra ?

[ameukam]

We initially had this conversation in #401.

Also kubernetes/test-infra#23317 (comment):

FYI @ameukam we don't have this feature enabled in kubernetes.io at the moment but will want to take a look at it soon

Some questions from thockin:

Cluster monitoring
a) What should we use?
GKE Workload metrics : https://cloud.google.com/stackdriver/docs/solutions/gke/managing-metrics#workload-metrics
Managed service for Prometheus : https://cloud.google.com/stackdriver/docs/managed-prometheus
b) How do we set it up with git-ops?
- #1376
- #1624
c) What exactly are we concerned about (signals)?
d) How are alerts delivered to a group of people?
e) How do we manage that group?
f) Do we need an on-call rotation?

App monitoring
a) Same tool as cluster monitoring?
b) What is the minimum expectation for an app to be deployed into community space
c) How do we manage groups of alerts for each app (ggroups?)
d) How do we manage on-call for each app?

GCP quotas monitoring
How do we monitoring them ?

More questions can be added.

/milestone v1.23
/are infra

[ameukam]

For this milestone, I would like to focus on how to flew out methods and practices about how we should do monitoring for k8s-infra.

/area infra

[spiffxp]

/priority important-longterm

[jimdaga]

If Prometheus is the tool picked I'm happy to jump in and help, I have a decent amount of experience.