Skip to content

Replace the per-machine probability-override kill-switch with a proper per-machine probability mechanism #268

Description

@robertodauria

Background

#267 added a locate-side kill-switch to suppress an individual machine registering with probability: 1. It was needed because autojoined nodes get their probability from per-org Datastore metadata (not per-machine), and disabling the whole org wasn't acceptable. It matches a machine name against heartbeat.ProbabilityOverrides, configured through the -probability-override flag and the PROBABILITY_OVERRIDE env var, and overrides the pick probability in filterSites.

This unblocked the immediate problem but is an explicit stopgap. This issue tracks removing it once a proper mechanism exists.

Why it's not a good long-term solution

  • Wrong source of truth / static. Config lives in locate's deploy config, not where machine metadata lives; changing or clearing an override needs a redeploy.
  • v2/v3 key footgun. The match key is host.Name.String(), which includes the service prefix for v3 but strips it for v2. We already mis-set this in sandbox (ndt-mlab2-chs0t… vs mlab2-chs0t…) and it silently did nothing.
  • Per-site, not per-machine. Picking is at the site level, so it dampens the whole site — fine for single-machine autojoin sites, wrong otherwise.
  • Name-bound & special-cased. If the node re-joins under a new generated name the override silently stops; it also deliberately overrides alwaysPick, which future readers won't expect.

Acceptance criteria

  • A per-machine probability/weight mechanism exists at the source of truth, controllable without redeploying locate and with a single unambiguous machine identifier.
  • Active overrides migrated to it.
  • Remove from locate: the -probability-override flag, PROBABILITY_OVERRIDE and _PROBABILITY_OVERRIDE, both app.yaml templates and cloudbuild.yaml, and heartbeat.ProbabilityOverrides with its filterSites handling.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions