Support various K8S scheduling options for multi-tenancy, prevent deadlocks.

[ssuchter]

One of the things we'll want (definitely not in the alpha) to have a good answer to is how multiple tenants on a single Kubernetes cluster will run Spark apps at the same time. Since Big Data apps (including Spark) frequently use a lot of resources (e.g. lots of executor pods) scheduling can become complicated, with the possibility of deadlocks. There was a request that I share this doc that was written for explaining some of the history of the YARN approach to several challenges. We definitely might not choose to solve them the same way, but understanding the issues is important.

Please see the Google Doc here, and comment!

http://tiny.pepperdata.com/yarn-scheduling-choices

I imagine this might turn into some kind of uber-issue in the future...

[ash211]

Thanks for writing up that doc Sean -- it does a good job of describing the challenges with multitenancy. Specifically I think the tension between BG3 and BG4 is something we've a fair amount of effort tuning YARN for.

A lot of Spark support for multi-tenancy is going to come down to making use of the Kubernetes features that are exposed. This ties in to the dynamic allocation umbrella feature for Spark-on-K8s that will come in in later phases too.

I think from K8s we need:

• means to define relative priority between jobs (pods). Maybe driven by the account running the jobs, or labels they have, or namespaces they're in. This is required to know which containers should have priority.
• preemption -- in the case where a low priority job is running and has all the resources, and a high priority job starts up and needs resources, k8s will need to forcefully take resources from the low-pri job. It's not enough to just wait for the low-pri job to yield resources. This preemption functionality is something we'll need from k8s too.