[Change Proposal] Attribute-based auto-installation of integrations

[hegerchr]

Context

The auto-installation of integrations is based on the specification of a dataset. If the dataset exists and data is present, the matching integration will be installed. Integrations specify this in the discovery section of the manifest.ymlfile. To provide an example, the OpenTelemetry Collector Internal Telemetry integration defines:

discovery:
  datasets:
    - name: collectortelemetry.otel

OpenTelemetry data ingested by the SDKs, by default, is routed to data streams

• logs-generic.otel-default
• metrics-generic.otel-default
• traces-generic.otel-default

and the data streams to the generic.otel dataset.

New OpenTelemetry content-only integrations (also referred to as content packs) are often language and technology specific. The following scenarios are examples to make it more tangible.

Scenario 1

OpenTelemetry data from a Java application is ingested through the managed OTLP endpoint. The generic.otel dataset is available and data is present. Matching based on the dataset generic.otel installs the Android OpenTelemetry Assets and the RUM OpenTelemetry Assets automatically. The user is never ingesting OpenTelemetry Android or RUM data, the automatically installed dashboards have no data to show.

Scenario 2

Prometheus data from a Node.js application is ingested through the Prometheus remote write endpoint. The generic.prometheus dataset is available and data is present. A technology-specific Java SpringBoot integration is automatically installed because the data stream is available and data is present. The dashboard will not find any data.

Change

It should be possible to define attributes and attribute values to match in order to express the auto-installation condition in more detail. Something like:

discovery:
  datasets:
    - name: generic.otel
      attributes:
        resource.attributes.telemetry.sdk.language: java
        resource.attributes.telemetry.sdk.name: android
        scope.name: io.opentelemetry.okhttp-3.0

@juliaElastic mentioned this change impacts the package spec and Kibana.

Open

Data can be routed to custom datasets (PRW, OTel). IIUC the auto-installation mechanism would not install an integration, because of the custom name of the dataset. To mitigate this issue, should it be possible to omit the dataset definition and only provide attributes and attribute values to match? An integration could be installed if the attributes and values match independent of the dataset.

[cmacknz]

Agree that the current dataset based discovery doesn't fit well with OpenTelemetry data streams. One reason for the current dataset only check was that checking for the existence of a data stream is cheaper than querying into a data stream. That said, I think this proposal makes sense and I don't know how else we'd make this work for the generic OpenTelemetry data streams.

[nimarezainia]

right now this is what we have:

I think @hegerchr 's approach there is sound. I want to find a generic way that in a content pack user can define the datastream to look at and the specific field: value that would trigger the installation. that would extend this solution.

@hegerchr I suppose the list of attributes you have are an AND ? all those fields and values should match before this content is installed. correct?

[nimarezainia]

@bturquet for your attention. @mrodm curious to know if you think this is possible or if you see issues with this approach.

[hegerchr]

@hegerchr I suppose the list of attributes you have are an AND ? all those fields and values should match before this content is installed. correct?

@nimarezainia yes they are AND. I currently can't think of a case where OR is needed.

[mrodm]

@bturquet for your attention. @mrodm curious to know if you think this is possible or if you see issues with this approach.

From the package-spec side, this looks feasible. My main concern is whether Kibana would actually need this exposed as a package-registry discovery query, or whether Fleet instead fetches packages via a generic content-type search and evaluates discovery client-side, never adding the discovery= in EPR queries.

What exists today in package-spec and package-registry:

• Dataset-based discovery (at least one of the listed datasets must be present):

 curl -s "https://epr.elastic.co/search?discovery=datasets:nginx.access.otel"	

discovery:
  datasets:
    - name: nginx.access.otel
    - name: nginx.error.otel

• Field-based discovery (all listed fields must be present, presence-only — no value matching):

 curl -s "https://epr.elastic.co/search?discovery=fields:process.pid,host.os.name"

discovery:
  fields:
    - name: process.pid
    - name: host.os.name

• Combining both in one query ANDs the two conditions — all fields present and at least one dataset present:

 curl -s "https://epr.elastic.co/search?discovery=fields:process.pid,host.os.name&discovery=datasets:generic.otel"

• Note: value matching on discovery.fields was attempted before (Add support for an optional value in discovery fields #917) and reverted in favor of explicit discovery.datasets (Revert "Add support for an optional value in discovery fields" #923, [Change Proposal] Add a new discovery parameter based on datasets #919)

cc @juliaElastic can you confirm whether the current Kibana auto-install task uses discovery= queries at all today or it reads directly the information from search queries without that query parameter?

If it is not used by Kibana (as it seems to be the case now), we could skip adding specific query parameters for now in the Package Registry. I'm not sure about how the syntax of those queries would look if those attributes with their values are added to the EPR queries along with the dataset. Or, should we try to do it for completeness even if Kibana does not require it now? WDYT?

Given the scenarios described in the description, I have some doubts.
To decide which content packages should be installed, do we need to match on field values, or is it enough that each integration has at least one field name that only it uses (so presence alone is enough to distinguish them).

Following the example in the description:

• If Java/Android/RUM OTel emit differently-named fields (not just different values of the same field), presence-only discovery.fields already solves this with no spec change.

discovery:
  fields:
    - name: field1.specific.name
    - name: field2.specific.name

• If they share the same field/attribute keys and only differ by value, we need value matching.
  • In that case: should value be added to the existing flat discovery.fields (global, reopening the design reverted in Add support for an optional value in discovery fields #917), or should attributes be scoped per dataset, so a package with multiple datasets can assign different expected values per dataset? Something like:

discovery:
  datasets:
    - name: generic.otel
      attributes:
        resource.attributes.telemetry.sdk.language: java
        scope.name: io.opentelemetry.okhttp-3.0
    - name: golang.otel
      attributes:
        resource.attributes.telemetry.sdk.language: go

[juliaElastic]

Confirming the Kibana side: the auto-install task does not use discovery= EPR queries at all. It calls the registry /search with only type=content, pulls all content packages, and evaluates discovery.datasets client-side.

On whether EPR needs the query params: since Kibana already reads the whole discovery object out of the search response and evaluates it itself, we don't need new discovery= query syntax in EPR for this to work it's enough that the new attributes are returned in the /search response payload. I'd defer adding discovery= attribute query syntax to EPR unless another consumer needs it.

About using document queries to match on attribute values, we actually have a blocker. kibana_system (the internal user that runs the auto install task) doesn't have access to read integrations data, this would be a pretty big widening of privileges to enable. (it was discussed before here: elastic/elasticsearch#132501)

The alternative is to use field-presence checks: view_index_metadata grants access to do _field_caps on logs-generic.otel-*, so we can check the presence of mapped fields. However searching only on field names can't cover all use cases if we have to distinguish based on attribute value.

[nimarezainia]

However searching only on field names can't cover all use cases if we have to distinguish based on attribute value.

@juliaElastic could you explain this a bit more please?

I think we need to have value matching in these cases and presence is not enough. @mrodm I don't fully follow why that issue was reverted. Can we enable that?

[juliaElastic]

could you explain this a bit more please?

I think we need to have value matching in these cases and presence is not enough.

I meant the same thing, in the example that Mario described, two conditions use resource.attributes.telemetry.sdk.language field with different values, so we have to match on the value, not just the presence of a field.

[mrodm]

@mrodm I don't fully follow why that issue was reverted. Can we enable that?

IIRC here it was discussed some options to include datasets https://github.com/elastic/ingest-dev/issues/5684#issuecomment-3044406932. As it was a bit different the logic between the discovery.fields (it must match all attributes) and the requirement for datasets (that matching at least one is enough), it was decided to move to a specific field for the datasets with its own semantics discovery.datasets.

I guess if needed it could be enabled again a similar approach. Do you mean to use discovery.fields adding the valuefield ? or discovery.datasets adding the attributes field ?

For the latter option, if it is chosen, probably it would be better (for consistency with the other discovery objects in the spec) to use name and value fields to define each attribute. Example (TBD):

discovery:
  datasets:
    - name: generic.otel
      attributes:
        - name: resource.attributes.telemetry.sdk.language
          value: java
        - name: scope.name
          value: io.opentelemetry.okhttp-3.0

According to @juliaElastic's comment #1202 (comment) there is a blocker for checking attribute values in documents, that could affect both options.

[hegerchr]

it would be better (for consistency with the other discovery objects in the spec) to use name and value fields to define each attribute.

@mrodm that's fine for me. I like it even more compared to my initial proposal.