feat: custom resource quotas

[oliverbaehler]

Describe the feature

We would like to request a feature regarding quota's of ObjectBucketClaim. Unfortunately, these objectBucketClaims can't be added in the native ResourceQuota object. The quota of a ObjectbucketClaim can only be set inside the ObjectBucketClaim, as following:

apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: test
  namespace: test
  finalizers:
    - objectbucket.io/finalizer
  labels:
    bucket-provisioner: openshift-storage.ceph.rook.io-bucket
spec:
  additionalConfig:
    maxSize: 2G
  generateBucketName: test
  objectBucketName: customer-test-test
  storageClassName: customer

We would like to manage the maxSize on tenant level. And of course the maxSize should be required on a ObjectBucketClaim, but maybe we can add that rule with kyverno. But we would like to manage the amount of storage available for a tenant on tenant level for objectBucketClaims. Is that something that could be added to Capsule?

Implementation

We are going to add a new namespaced resource called CustomQuota - an initial draft is here:

---
kind: CustomQuota
metadata:
  name: custom-quota
spec:
  source:
    version: v1
    kind: ObjectBucketClaim
    Path: .spec.additionalConfig.maxSize
  selectors (Optional):
    - CELExpressions?
  limit: 10Gi
status:
  used: 5Gi
  available 1Gi

With the source we identify fields where we expect compute quantities (2Gi eg.). Then there^s a limit for that. Essentially we need to create an admission which blocks the request, if the requested resource would allocate over limit.

Here we should probably make a dynamic admission webhook, to interrupt traffic as little as possible or tag resources via ssa that they are sent to the calculating admission webhook.

For Quantity Count we can use standard ResourceQuotas.