Feature: fallback pool (prefer on prem, fallback to another cloud)

[brandonkal]

It would be nice to give pools priority. So that you could either:

1. Only use a provider if the primary provider is in an error state
2. Only use a second cloud provider if on-prem pool is at capacity

[gabriel-samfira]

Hi @brandonkal !

You already can! 😁 Well, at least for the case where one pool is at capacity.

You can switch your entity pool balancer type from roundrobin to pack by running:

garm-cli repo update --pool-balancer-type pack YOUR_REPO_ID

Then you can set priorities to your pools:

garm-cli pool update YOUR_POOL_ID --priority 200
garm-cli pool update YOUR_BACKUP_POOL_ID --priority 100

Pools are sorted by priority in descending order. So if the priority is set to pack, GARM will try to fill up the highest priority pool before moving on to the next one.

And on that note, I did a terrible job at documenting things. I just noticed I have just one mention of priorities in the docs, but absolutely no example of how to use the feature 😓.

Give it a shot and let me know if it works.

[gabriel-samfira]

to view pool priorities when listing them, you can run:

garm-cli pool ls --all --long

[brandonkal]

Thank you this is good to know. My hypervisor host is without internet today so I have to set up an AWS fallback (which has been on my list but now is a must). Ultimately I want to:

• run arm jobs only on AWS
• run amd64 jobs on premises (with some queuing even)
• fallback amd64 jobs to AWS if the hypervisor is down or there are 10+ amd64 jobs waiting in the queue.

I suppose for now I can set my primary pool to have 0 capacity.

[gabriel-samfira]

You can use the priority option and just swap priorities when you notice an outage. Set a priority of 0 for the pool that has an outage and anything higher than 0 for the other pool(s). You might need to force remove erred runners.

You can define as many pools as you like. You can also have multiple pools with the same labels that can handle the same jobs. I recommend you use a single, unique label for pools (eg: linux-arm64-large). Define multiple pools with the same label but different priorities, and you're good to go.

If you only define a single pool for arm64 on aws, then that is the only place where arm64 will spin up.

[gabriel-samfira]

You can also enable/disable pools. So if you notice that a provider is down, you can disable pools for that provider.

[gabriel-samfira]

I migrated the issue into a discussion. We can continue here. If we find there is an issue we can act on, we can create an issue from the discussion.