ComputeDomain daemonset pods might fail because of not working around node taints

[jgehrcke]

In an environment where nodes have taints set a user might set tolerations on their workloads to counter those taints.

But then scheduling our ComputeDomain pods onto those very nodes might fail because they don't have comparable tolerations set.

Since ComputeDomain pods are tied to a user's workload the user knows best (and they do set tolerations on their things), so we have to follow this intent. We seemingly can just use a catch-all toleration.

[jgehrcke]

Taints and tolerations is already a beta feature and supports wildcard tolerations. Please see https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/#concepts.

(kubernetes/kubernetes#58981 (comment))

But I didn't see the word wildcard in there. Hm.

Reading on.

Ah!

If the key is empty, then the operator must be Exists, which matches all keys and values. Note that the effect still needs to be matched at the same time.

(from https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/#concepts)

So, should we set a wildcard toleration like this for all three effects? (noexecute, noschedule, prefernoschedule)

[jgehrcke]

$ kubectl explain pod.spec.tolerations

  key	<string>
    Key is the taint key that the toleration applies to. Empty means match all
    taint keys. If the key is empty, operator must be Exists; this combination
    means to match all values and all keys.

So maybe we want something like this:

tolerations:
- operator: "Exists"
  effect: "NoSchedule"
- operator: "Exists"
  effect: "NoExecute"
- operator: "Exists"
  effect: "PreferNoSchedule"

[jgehrcke]

Probably taken care of via #306.