OAM Spec v0.3.0 Working Draft [Public Review and Comment]

[resouer]

OAM Spec v0.3.0 Working Draft

Hi all, this is the issue tracker to finalize the design of OAM spec v0.3.0. It reflects the high level ideas of the existing working draft in master branch with considerations on backward compatibility in existing releases.

As a working draft, the below design is open for revise, but we should try best to keep the changes minimal.

1. ComponentDefinition

Purpose: refactor Component in v0.2.1 and ComponentSchematic of v1alpha1 to combine all the schematic primitives in one object, w/o any instance values.

Target user: component provider, software distributor

apiVersion: core.oam.dev/v1beta1
kind: ComponentDefinition
metadata:
  name: webservice
  namespace: foo # declare this component is in namespace foo
spec:
  workload: # workload type descriptor
    type: containerizedworkloads.core.oam.dev # ref to a WorkloadDefinition via name
  schematic:
    kube: # this section is same with v0.2.1 component spec (i.e. raw k8s resource template)
      template:
        apiVersion: core.oam.dev/v1alpha2
        kind: ContainerizedWorkload
        spec:
        ...
          containers:
          - name: my-cool-workload
            image: example/very-cool-workload:0.1.2@sha256:verytrustworthyhash
            resources:
              cpu:
                required: 1.0
              memory:
                required: 100MB
            cmd:
            - "bash lscpu"
      parameters: 
      - name: image
        required: true
        fieldPaths: 
        - "spec.containers[0].image"
      - name: cmd
        required: true
        fieldPaths:
        - "spec.containers[0].cmd"
    cue:
      # cue template and parameters 
    helm:
      # a helm chart
status:
  parameters: # a generated json schema of parameters, equivalent to spec.parameters but friendly to k8s impl

The implementation will auto generate the jsonschema of module parameters and patch it to the status of ComponentDefinition.

2. WorkloadDefinition

Purpose: equivalent to WorkloadDefinition in v0.2.1, a missing concept in v1alpha1

Target user: this is an optional system level object, the main use case is for custom workload type discovering.

apiVersion: core.oam.dev/v1beta1
kind: WorkloadDefinition
metadata:
  name: containerizedworkloads.core.oam.dev
spec:
  definitionRef: # crd of this workload
    apiVersion: core.oam.dev/v1alpha2
    kind: ContainerizedWorkload
  childResources:
    - apiVersion: apps/v1
      kind: Deployment

3. Application

Purpose: equivalent to ApplicationConfiguration in v0.2.1 and v1alpha1

Target user: app dev, app ops, end users

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: application-sample
spec:
  components:
    - name: myweb
      type: foo/webservice # ref to ComponentDefinition, optionally, in namespace foo
      properties: # parameter values of the component
        image: "busybox"
        cmd:
        - sleep
        - "1000"
      traits: # assume traits are in oam system namespace for now
        - type: scaler
          properties:
            replicas: 10
        - type: sidecar
          properties:
            name: "sidecar-test"
            image: "nginx"

4. TraitDefinition

Purpose: equivalent to TraitDefinition in v0.2.1 and Trait of v1alpha1

Target user: infra ops, platform builders, traits are platform provided capabilities, end users should not modify traits - they only use them

apiVersion: core.oam.dev/v1alpha2
kind: TraitDefinition
metadata:
  name: scaler
spec:
  appliesToWorkloads:
    - containerizedworkloads.core.oam.dev # arbitrary workload definition
    # support k8s label selector in the future
  definitionRef:
    apiVersion: xxx
    kind: xxx
  schematic:
    # seems we don't need `kube` template for trait, definitionRef is enough
    cue:
      # cue template and parameters
    helm:
      # a helm chart
    ... # could have more module types in the future
status:
  parameters: # a optional generated json schema of parameters, equivalent to spec.parameters but friendly to k8s impl

5. ApplicationScope

No change with spec v0.2.1

Open Questions

1. Behavior of traits for revisioned components.

Traits are attached to component instances, and those instances are revisioned. Should we replica the traits then (e.g. ingress vs. {ingress-v1, ingress-v2})?

In previous code, as a workaround, we tagged TraitDefinition as revisionEnabled to control this behavior, though recently we fallback to assume ALL traits are replicated but have to leave rollout to a seperate field (since it is across-revisions).

[wonderflow]

package: # the encapsulation of this component. alternative name: module, componentSpec, encapsulation, templateModule, schematic

I vote for package.

[ryanzhang-oss]

  package: # the encapsulation of this component. alternative name: module, componentSpec, encapsulation, templateModule, schematic

I vote for "schematic"

[hongchaodeng]

Vote for schematic

[wonderflow]

Fine, let's use schematic

[kminder]

Below are points that I think are important to address with the current proposal.

1. Seems like this should be at least 0.2->0.3 vs a "patch" 0.2.1->0.2.2.
2. Needs to retain or replace scopes in 0.2.2 (0.3) resources.
3. Needs to support complex traits. eg https://github.com/oam-dev/catalog/blob/master/applications/istio/bookinfo/appconfig-1-bookinfo.yaml, https://github.com/oam-dev/catalog/tree/master/traits/ingresstrait
4. Needs to support Component isolation. Different users having access to different Components.
5. Needs to have an extensible template "plugin" model that does not require spec change.
6. Needs to normalize parameter handling between template types, including parameter schema.
7. Needs to normalize use of setting vs parameter.

[kminder]

More detailed questions and comments.

1. ComponentDefinition

• Can we (at least optionally) share a "contract" between ComponentDefinitions?
• At a minimum ComponentDefinitions will be hard to write and we expect users/component providers to do so.
• Seems spec:workload:definition: doesn't reference a WorkloadDefinition, right? Does the naming need alignment?
• Need more detail on how characteristics are handled especially if they will result in labels. How are these "namespaced" to prevent conflict?
• Requiring consumers to find the right location for parameter schema is non-optimal. Can this always be declared by template author?
• Rename childResource to childResources

2. WorkloadDefinition

• This has no value as currently proposed unless the "contract" question above is addressed.

3. Application

• Components are not provided by the platform but rather a component provider. Won't a namespace be required for isolation/security?
• Components and traits should use the same term for settings/properties
• Can settings really be either scalar or array? That needs to be in param schema somehow (especially for kube template type).

4. TraitDefinition

• The appliesToWorkload doesn't seem flexible enough to be useful. It should use something more like k8s label matching. It need or/in/notin at least.
• The appliesToWorkload need better details about how workload definition and childResources are translated to a characteristic if they are.
• Can appliesToWorkload use any label or just those in a ComponentDefinition's characteristics? If so what labels?
• Doesn't appliesToWorkload need to operate against the ComponentDefinition vs the workload instance for validation?

5. ApplicationScope

• Scope support must be included in 0.2.2 in some form for functional compatibility.

[kminder]

WRT

4. Needs to support Component isolation. Different users having access to different Components.

What I really mean is that a ComponentDefinition needs to be namespace qualified. Just adding a namespace appears awkward though as it isn't clear if it applies to the name or the type.

apiVersion: core.oam.dev/v1alpha2
kind: Application
...
spec:
  components:
    - name: my-comp-inst
      namespace: my-comp-def-ns
      type: my-comp-def

Honestly I've always felt Kubernetes needed standardized single value qualified reference formats (e.g. namespace/name and group/version.kind@namespace/name or similar).

spec:
  components:
    - name: my-comp-inst
      type: my-comp-def-ns/my-comp-def

[dhiguero]

Hi all, these are our comments on the spec proposal.

Component Definition

• Only one resource apiVersion/Kind is associated with a component. What if it generates multiple ones?
• Is it possible to instantiate a component multiple times? From the point of view of the Application it seems that we have a type and name, but at the same time the component has a status. Is the status expected to be populated with information about the generic component (e.g., template can be parsed), or will it be containing information about the instances. From the point of view of having a clear picture of what is happening on an application, having some type of ComponentInstance information is desirable. For example, where will be the information stored in case a particular instance of a component cannot be deployed because of quota, image not being available or any other common error in Kubernetes. Right now, some of this information is not available on the OAM layer.
• Parameters can be confusing as they can either be specified in two different places (spec.schematic.kube.parameters, or spec.parameters). It will seem more natural to consolidate those on a single place (spec.parameters).
• Is status.parameters generated? If so, how will the runtime be able to determine which are the parameters for a random component? Is there an expectation that the component will provide a schema for that?
• If we use an alternative method to specify a component such as Helm, it is not clear what is the association/impact with the generated resources. What is the expected apiVersion/Kind in that use case? Is the runtime expected to track/associate all of the generated resources? will it fail in case of a mismatch with the workload definition?
• From the point of view of maintainability of the runtime, adding other mechanism such as Helm, ARM, Cue, etc. adds a risk as once the PR is approve, it is likely that the runtime must be updated, maintained and tested with each new version. If we take for example the Kubernetes project, the current approach is to move provider specific code outside of the main project to improve maintainability. If those are implemented as Workloads, not as component we can gain extensibility without adding extra pressure to the runtime developers.
• Are we expecting that components are limited to "software components"? Previously we thought that the modeling implied that an application was composed by a set of software elements. With the addition of the ARM templates, it seems that we open the possibility for a component to be an infrastructure element (e.g., storage). If that is the case, the separation of responsibilities between the different user personas becomes confusing. If the proposal aims to support non-software component, a new InfrastructureComponent entity could be discussed.

Workload definition

• We think that having a workload definition is desirable as it provides information to the users (and/or runtimes). One alternative to the current proposal could be to move part of the extensibility by means of templates or support for other specs to the workload definition, and not in the component itself.

Application

• We should consolidate naming among all entities in the system (e.g., use "parameters" always) so it is easier for the users.
• It is still difficult to reason about the resulting application instance. For example, what is the expected number of replicas per component? can the application be checked to see if all components are running as expected? A solution to this could be to have a clear status information for each component instance.

Trait definition

• Having a set of general characteristics may be a good idea (e.g., replicable) but at the same time it seems that the trait needs to actually know the underlying type to work. For example, were is the number of replicas for a given K8s entity (could be a CRD). As such, it may happen that a trait is applicable to "replicable" entities, but an entity that is "replicable" is not supported by a version of the trait. One alternative is to use annotations on the components so "replicable" traits can check the workload.oam.dev/replicable-number-replicas-path annotation to identify which is the fields of the k8s entity against a patch is going to be generated.
• Parameters as in components appear in two places, either generated by the system, or predefined. Why is it not possible to agree that independently of how the user specifies what a trait is (the same with component) there is a need to specify parameters even if they could be inferred. Inferring parameters also means that traits may need some kind of interface/language to define this.
• We need to consider the implications of installing a Trait. Since those are cluster-wide resources and not namespaced, restrictions may apply in real life deployments about who is able to install a cluster-wide resource.
• appliesToWorkloads should have the same format (array, list, etc) as in the workload.

Application Scope

• We think it covers the need to having an entity that is able to group components where a feature is applied (scope, metric, etc.) so it would make sense to maintain it.

Other aspects

Versioning

• Given the amount of changes that are expected for this new version, we believe that calling it 0.3.0 will be more accurate.

Extra information on user facing entities

• Adding a "description" field to parameters will help the consuming user to identify the impact of changing that.

Component Security Context

We have submitted a PR for the OAM runtime controller to include security context information on components limiting the information to aspects of the container, not to the underlying orchestration system. We think that this information should also be part of the new OAM proposal at least for the spec.schematic.kube aspect.

OAM Compliance

With the growth of OAM in the industry we think that the spec should have a clear definition of which are the minimum requirements for an OAM runtime. Before this proposal, our understanding was that the differentiation from one provider to another would be the different workload types and traits that were present, not parts of the spec.

Trait consolidation

Traits are currently implemented as Kubernetes operators that modify the generated resource according to their internal logic. However, this opens up the door for consolidation failure scenarios where two traits may want to modify the same characteristic. For example, image two trait trying to set different number of replicas.

Maybe it is too much of a change for the current proposal/scope, but it will be nice to consider this issue in the future. If traits are applied in order by some mechanism, we feel the system becomes much more robust, even if that requires traits not being operators, but implementing a common gRPC interface to return the altered entity.

Configuration and secrets

This issue is not referenced by the spec, and may not be clear for a new user how this is expected to be supported. One option is to add this type of information as a new entity or include it on the Application one, while other could be to use a specific Trait. Either way, the spec should contain information about how to proceed when deploying an application that requires a config map. While many containerized application rely on environment variables to set options, others just use a plain config file on known locations.

[resouer]

@kminder @dhiguero Just updated the spec description based on your inputs, they are valuable!

We will leave some features as Non-Goal for this release, for example:

1. Make .schematic fully extensible. There's no intension to expand this until there're significant requirements to open this as an extension point and conformance design will be needed then. We will focus on the schematic types listed above which are all driven by real user requirements.
2. Advanced trait appliesTo behavior. This is a highly demanded feature (i.e. applies to characteristics instead of workload def'n name), but we decide won't couple it in this release and tackle it separately in issue Define workload types, replicable, and podspecable and make appliesTo work properly #392. A similar feature is ConflictsWith on traits.