Skip to content

Support configuring dynamic dataset and namespace in input packages #1140

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jsoriano wants to merge 2 commits into from
Closed

Support configuring dynamic dataset and namespace in input packages #1140

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5f21add
Support configuring dynamic dataset and namespace in input packages
jsoriano Apr 15, 2026
13ceb99
Fix changelog
jsoriano Apr 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions spec/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@
- description: Add validation for Fleet-reserved variables (use_apm, data_stream.dataset) to enforce correct types and constraints when explicitly overridden in package manifests.
type: enhancement
link: https://github.com/elastic/package-spec/pull/1134
- description: Add "elasticsearch.dynamic_dataset" and "elasticsearch.dynamic_namespace" to input package manifest.
type: enhancement
link: https://github.com/elastic/package-spec/pull/1140
- version: 3.6.0
changes:
- description: Add pipeline tag validations.
Expand Down
4 changes: 4 additions & 0 deletions spec/input/manifest.spec.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,10 @@ spec:
$ref: "../integration/data_stream/manifest.spec.yml#/definitions/elasticsearch_index_mode"
index_template:
$ref: "../integration/data_stream/manifest.spec.yml#/definitions/elasticsearch_index_template"
dynamic_dataset:
$ref: "../integration/data_stream/manifest.spec.yml#/definitions/elasticsearch_dynamic_dataset"
dynamic_namespace:
$ref: "../integration/data_stream/manifest.spec.yml#/definitions/elasticsearch_dynamic_namespace"
Comment on lines +158 to +161

@coderabbitai coderabbitai Bot Apr 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Add a version patch to gate the new input fields by spec version.

dynamic_dataset and dynamic_namespace are added, but older versions are not patched to remove them. This leaks new schema surface into legacy format_versions.

Proposed patch 
 versions:
+  - before: 3.6.1
+    patch:
+      - op: remove
+        path: "/properties/elasticsearch/properties/dynamic_dataset"
+      - op: remove
+        path: "/properties/elasticsearch/properties/dynamic_namespace"
   - before: 3.6.0
     patch:
       - op: remove
         path: "/properties/var_groups" # removes var_groups from package root

As per coding guidelines, "Version patches enable backward compatibility by removing features from older spec versions."

Also applies to: 173-220

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@spec/input/manifest.spec.yml` around lines 158 - 161, Add a version patch to
gate the newly added fields dynamic_dataset and dynamic_namespace so they are
removed for older spec format versions; update the manifest.spec.yml patching
logic (the elasticsearch_dynamic_dataset / elasticsearch_dynamic_namespace
additions under dynamic_dataset and dynamic_namespace refs) to include a
backward-compatibility version patch that deletes those properties for legacy
format_version values (also apply same patching for the related additions in the
173-220 range) so older spec versions do not expose the new schema surface.

deprecated:
$ref: "../integration/manifest.spec.yml#/definitions/deprecated"
required:
Expand Down
12 changes: 8 additions & 4 deletions spec/integration/data_stream/manifest.spec.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -247,6 +247,12 @@ spec:
- "time_series" # Enables time series data streams https://www.elastic.co/guide/en/elasticsearch/reference/master/tsds.html
examples:
- "time_series"
elasticsearch_dynamic_dataset:
description: When set to true, agents running this integration are granted data stream privileges for all datasets of its type
type: boolean
elasticsearch_dynamic_namespace:
description: When set to true, agents running this integration are granted data stream privileges for all namespaces of its type
type: boolean
Comment on lines +250 to +255

@coderabbitai coderabbitai Bot Apr 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Move shared dynamic field definitions to the integration manifest definitions file.

These definitions are now shared by multiple specs (spec/integration/data_stream/manifest.spec.yml and spec/input/manifest.spec.yml), but they are defined in the data_stream spec file instead of spec/integration/manifest.spec.yml.

As per coding guidelines, "When a field is used in multiple locations, define it once in spec/integration/manifest.spec.yml and reference it using relative paths."

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@spec/integration/data_stream/manifest.spec.yml` around lines 250 - 255, Move
the shared field definitions for elasticsearch_dynamic_dataset and
elasticsearch_dynamic_namespace out of the data_stream spec and into the
integration manifest's shared definitions section; then replace the local
definitions with $ref entries that point (relatively) to those centralized
definitions so both the data_stream and input specs reuse the same definition.
Ensure the centralized definitions include the same description and type
boolean, remove the duplicate blocks from the original file (the
elasticsearch_dynamic_dataset and elasticsearch_dynamic_namespace blocks), and
update any occurrences to reference the manifest definitions via relative $ref
paths.

elasticsearch_index_template:
description: Index template definition
type: object
Expand Down Expand Up @@ -685,11 +691,9 @@ spec:
examples:
- "synthetic"
dynamic_dataset:
description: When set to true, agents running this integration are granted data stream privileges for all datasets of its type
type: boolean
$ref: "#/definitions/elasticsearch_dynamic_dataset"
dynamic_namespace:
description: When set to true, agents running this integration are granted data stream privileges for all namespaces of its type
type: boolean
$ref: "#/definitions/elasticsearch_dynamic_namespace"
categories:
description: >
Optional categories that describe the type of data collected by this data stream.
Expand Down
2 changes: 2 additions & 0 deletions test/packages/good_input/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,8 @@ agent:
privileges:
root: true
elasticsearch:
dynamic_dataset: true
dynamic_namespace: true
index_template:
mappings:
properties:
Expand Down