Container build pool?

[prologic]

LHello,

I’ve noticed that disk I/O usage during container builds can significantly degrade the performance of the host machine, sometimes to the point of affecting the availability of running applications. This issue is particularly pronounced when the underlying hardware does not use NVMe storage.

Is there currently support in OpenEun for configuring container builds to run on separate machines from the primary workloads?

Thank you.

[akclace]

The container build currently runs on the same node as where the API processing happens. Yes, I have seen the same performance issue when multiple builds are done concurrently.

I will add a max_concurrent_builds options, to limit the perf impact. Default would be 1. If single build causes performance issues, running on a bigger machine would be the solution. The longer term plan is to add a way to run on Kubernetes, the container build would not happen on the api worker in that case.

[prologic]

Have you considers a cluster architecture here too where you can run a "build" node that registers itself to the openrun cluster?

That way I could run a couple of build nodes on some mac minis.

[akclace]

Pushed v0.14.6 with the change. Upgrade by running curl -sSL https://openrun.dev/install.sh | sh. Two new config options available in openrun.toml

[system]
max_concurrent_builds = 1 # max number of concurrent container builds
max_build_wait_secs = 120 # max wait time for a build lock

[prologic]

This is great! Thank you for doing this! 🙏 And so quickly. I would leave this discussion open though as in my experience, even 1 build data intensive job on a machine running production workloads can still kill performance. Hard learned lesson 🤣 It's almost unavoidable, unless you go with NVME disks or U.2, but even then I/O can still have an impact on latency, it'll just be less(er) with faster disks with faster IOPS.

[akclace]

even 1 build data intensive job on a machine running production workloads can still kill performance.

Yes, the option added is mainly for the single node scenario. For multi-node deployment (with k8s or non-k8s), I will look at other options to avoid the build related slowdown on the api handler