[P2] Jobs are Queued - pet instances

[pytorch-alerts]

macos-m1-stable jobs are queueing, please investigate. Max queue time: 63 mins, Max queue size: 85 runners. Please visit http://hud.pytorch.org/metrics to see which runners are queueing.

Alert Details

• Occurred At: Nov 20, 5:15pm PST
• State: FIRING
• Team: pytorch-dev-infra
• Priority: P2
• Description: Alerts when any of the regular runner types is queuing for a long time or when many of them are queuing.
• Reason: [runner=macos-m1-stable] max_queue_size=85, max_queue_time_mins=63, queue_size_threshold=1, queue_time_threshold=0, threshold_breached=1
• Runbook: https://hud.pytorch.org/metrics
• View Alert: https://pytorchci.grafana.net/alerting/grafana/aez5q4um9pd6of/view?orgId=1
• Silence Alert: https://pytorchci.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Daez5q4um9pd6of&matcher=runner%3Dmacos-m1-stable&matcher=type%3Dalerting-infra&orgId=1
• Source: grafana
• Fingerprint: d3cc2810a9dda1d74867c3d2ac4d4a7b55d1072a084c8e9ba01e85849d025a36

[pytorch-alerts]

Alert Update

• State: FIRING
• Occurred At: Nov 27, 2:20pm PST
• Reason: [runner=macos-m1-stable] max_queue_size=15, max_queue_time_mins=122, queue_size_threshold=0, queue_time_threshold=1, threshold_breached=1
  The alert condition is still active.
  Fingerprint: d3cc2810a9dda1d74867c3d2ac4d4a7b55d1072a084c8e9ba01e85849d025a36