From 9239d739b5f5bd89338d2ca605dd8e10b51c7131 Mon Sep 17 00:00:00 2001 From: May Lee Date: Thu, 31 Jul 2025 14:16:17 -0400 Subject: [PATCH] add process groups --- .../observability_pipelines/processors/_index.md | 10 ++++++++++ .../set_up_pipelines/_index.md | 14 ++++++++++++++ 2 files changed, 24 insertions(+) diff --git a/content/en/observability_pipelines/processors/_index.md b/content/en/observability_pipelines/processors/_index.md index b420aa91e6a08..9e4d73acda040 100644 --- a/content/en/observability_pipelines/processors/_index.md +++ b/content/en/observability_pipelines/processors/_index.md @@ -17,6 +17,16 @@ Processor groups are executed from top to bottom. The order of the processors is Select a processor in the left navigation menu to see more information about it. +## Processor groups + +
Configuring a pipeline with processor groups is only available for Worker versions 2.7 and later.
+ +You can organize your processors into logical groups to help you manage them. Each processor group has a Group Filter so that those processors are only applied to specific logs. For example, if you want the group processors to only process logs coming from `vpc`, then use the group filter `source:vpc`. You can also add filters for each individual processor. + +Processor groups and the processors within each group are executed from top to bottom. The order of the processors is important because logs are checked by each processor, but only logs that match the processor's filters are processed. To change the order of the processors, use the drag handle on the top left corner of the processor you want to move. + +**Note**: There is a limit of 10 processor groups for a pipeline canvas. For example, if you have a dual ship pipeline, where there are two destinations and each destination has its own set of processor groups, the combined number of processor groups from both sets is limited to 10. + {{% observability_pipelines/processors/filter_syntax %}} [1]: https://app.datadoghq.com/observability-pipelines diff --git a/content/en/observability_pipelines/set_up_pipelines/_index.md b/content/en/observability_pipelines/set_up_pipelines/_index.md index 08596d2f65786..c712ade38f3a4 100644 --- a/content/en/observability_pipelines/set_up_pipelines/_index.md +++ b/content/en/observability_pipelines/set_up_pipelines/_index.md @@ -43,6 +43,20 @@ Set up your pipelines and its sources, processors, and destinations in the Obser 1. Select and set up your [source][1]. 1. Select and set up your [destinations][2]. 1. Set up your [processors][3]. + - If you want to copy a processor, click the copy icon for that processor and then use `command-v` to paste it. +1. If you want to add another group of processors for a destination: + 1. Click the plus sign (**+**) at the bottom of the existing processor group. + 1. Click the name of the processor group to update it. + 1. Optionally, enter a group filter. See [Filter Syntax](#filter-query-syntax) for more information. + 1. Click **Add** to add processors to the group. + 1. If you want to copy all the processors in a group and paste them into the same processor group or a different group: + 1. Click the three dots on the processor group. + 1. Select **Copy all processors**. + 1. Select the processor group you want to paste the processors into and then use `command-v` to paste them. + 1. You can toggle the switch to enable and disable the processor group and also each individual processor. +
**Notes**: +
- Configuring a pipeline with processor groups is available for Worker versions 2.7 and later. +
- There is a limit of 10 processor groups for a pipeline canvas. 1. If you want to add another set of processors and destinations, click the plus sign (**+**) to the left of the processor group to add another set of processors and destinations to the source. - To delete a processor group, you need to delete all destinations linked to that processor group. When the last destination is deleted, the processor group is removed with it. 1. If you want to add an additional destination to a processor group, click the plus sign (**+**) to the right of the processor group.