-
Notifications
You must be signed in to change notification settings - Fork 770
Add Dynatrace OpenTelemetry Collector integration guide #4739
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
e1c652e
Add Dynatrace OpenTelemetry Collector integration guide
MyMirelHub f05f71d
Merge branch 'dapr:v1.15' into dynatrace-tracing
MyMirelHub 8f32873
Update daprdocs/content/en/operations/observability/tracing/otel-coll…
MyMirelHub 78ea6b9
Update daprdocs/content/en/operations/observability/tracing/otel-coll…
MyMirelHub 49d2873
Update daprdocs/content/en/operations/observability/tracing/otel-coll…
MyMirelHub 29a69cf
Update daprdocs/content/en/operations/observability/tracing/otel-coll…
MyMirelHub adbdb8a
Update daprdocs/content/en/operations/observability/tracing/otel-coll…
MyMirelHub adbb909
Update daprdocs/content/en/operations/observability/tracing/otel-coll…
MyMirelHub 2c4b99b
Update daprdocs/content/en/operations/observability/tracing/otel-coll…
MyMirelHub 2fa89d0
Update daprdocs/content/en/operations/observability/tracing/otel-coll…
MyMirelHub 25f5962
Update daprdocs/content/en/operations/observability/tracing/otel-coll…
MyMirelHub 32bcc2c
Apply suggestion from @alicejgibbons
MyMirelHub 65b7cf8
Update daprdocs/content/en/operations/observability/tracing/otel-coll…
MyMirelHub 9b0a470
Enhance documentation for Dynatrace OpenTelemetry Collector setup wit…
MyMirelHub 3a054e5
Fix formatting and indentation in Dynatrace OpenTelemetry Collector s…
MyMirelHub File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
139 changes: 139 additions & 0 deletions
139
...ions/observability/tracing/otel-collector/open-telemetry-collector-dynatrace.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,139 @@ | ||
--- | ||
type: docs | ||
title: "Using Dynatrace OpenTelemetry Collector to collect traces to send to Dynatrace" | ||
linkTitle: "Using the Dynatrace OpenTelemetry Collector" | ||
weight: 1000 | ||
description: "How to push trace events to Dynatrace, using the Dynatrace OpenTelemetry Collector." | ||
--- | ||
|
||
Dapr integrates with the [Dynatrace Collector](https://docs.dynatrace.com/docs/ingest-from/opentelemetry/collector) using the OpenTelemetry protocol (OTLP). This guide walks through an example using Dapr to push traces to Dynatrace, using the Dynatrace version of the OpenTelemetry Collector. | ||
|
||
{{% alert title="Note" color="primary" %}} | ||
This guide refers to the Dynatrace OpenTelemetry Collector, which uses the same Helm chart as the open-source collector but overridden with the Dynatrace-maintained image for better support and Dynatrace-specific features. | ||
{{% /alert %}} | ||
|
||
## Prerequisites | ||
|
||
- [Install Dapr on Kubernetes]({{< ref kubernetes >}}) | ||
- Access to a Dynatrace tenant and an API token with `openTelemetryTrace.ingest`, `metrics.ingest`, and `logs.ingest` scopes | ||
- Helm | ||
|
||
## Set up Dynatrace OpenTelemetry Collector to push to your Dynatrace instance | ||
|
||
MyMirelHub marked this conversation as resolved.
Show resolved
Hide resolved
|
||
To push traces to your Dynatrace instance, install the Dynatrace OpenTelemetry Collector on your Kubernetes cluster. | ||
|
||
1. Create a Kubernetes secret with your Dynatrace credentials: | ||
|
||
```sh | ||
kubectl create secret generic dynatrace-otelcol-dt-api-credentials \ | ||
--from-literal=DT_ENDPOINT=https://YOUR_TENANT.live.dynatrace.com/api/v2/otlp \ | ||
--from-literal=DT_API_TOKEN=dt0s01.YOUR_TOKEN_HERE | ||
``` | ||
|
||
Replace `YOUR_TENANT` with your Dynatrace tenant ID and `YOUR_TOKEN_HERE` with your Dynatrace API token. | ||
|
||
1. Use the Dynatrace OpenTelemetry Collector distribution for better defaults and support than the open source version. Download and inspect the [`collector-helm-values.yaml`](https://github.com/Dynatrace/dynatrace-otel-collector/blob/main/config_examples/collector-helm-values.yaml) file. This is based on the [k8s enrichment demo](https://docs.dynatrace.com/docs/ingest-from/opentelemetry/collector/use-cases/kubernetes/k8s-enrich#demo-configuration) and includes Kubernetes metadata enrichment for proper pod/namespace/cluster context. | ||
|
||
|
||
1. Deploy the Dynatrace Collector with Helm. | ||
|
||
```sh | ||
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts | ||
helm repo update | ||
helm upgrade -i dynatrace-collector open-telemetry/opentelemetry-collector -f collector-helm-values.yaml | ||
``` | ||
|
||
## Set up Dapr to send traces to the Dynatrace Collector | ||
|
||
MyMirelHub marked this conversation as resolved.
Show resolved
Hide resolved
|
||
Create a Dapr configuration file to enable tracing and send traces to the OpenTelemetry Collector via [OTLP](https://opentelemetry.io/docs/specs/otel/protocol/). | ||
|
||
|
||
1. Update the following file to ensure the `endpointAddress` points to your Dynatrace OpenTelemetry Collector service in your Kubernetes cluster. If deployed in the `default` namespace, it's typically `dynatrace-collector.default.svc.cluster.local`. | ||
|
||
**Important:** Ensure the `endpointAddress` does NOT include the `http://` prefix to avoid URL encoding issues: | ||
|
||
```yaml | ||
apiVersion: dapr.io/v1alpha1 | ||
kind: Configuration | ||
metadata: | ||
name: tracing | ||
spec: | ||
tracing: | ||
samplingRate: "1" | ||
otel: | ||
endpointAddress: "dynatrace-collector.default.svc.cluster.local:4318" # Update with your collector's service address | ||
``` | ||
|
||
1. Apply the configuration with: | ||
|
||
```sh | ||
kubectl apply -f collector-config-otel.yaml | ||
``` | ||
|
||
## Deploy your app with tracing | ||
|
||
Apply the `tracing` configuration by adding a `dapr.io/config` annotation to the Dapr applications that you want to include in distributed tracing, as shown in the following example: | ||
|
||
```yaml | ||
apiVersion: apps/v1 | ||
kind: Deployment | ||
metadata: | ||
... | ||
spec: | ||
... | ||
template: | ||
metadata: | ||
... | ||
annotations: | ||
dapr.io/enabled: "true" | ||
dapr.io/app-id: "MyApp" | ||
dapr.io/app-port: "8080" | ||
dapr.io/config: "tracing" | ||
``` | ||
|
||
{{% alert title="Note" color="primary" %}} | ||
If you are using one of the Dapr tutorials, such as [distributed calculator](https://github.com/dapr/quickstarts/tree/master/tutorials/distributed-calculator), you will need to update the `appconfig` configuration to `tracing`. | ||
{{% /alert %}} | ||
|
||
You can register multiple tracing exporters at the same time, and the tracing logs are forwarded to all registered exporters. | ||
|
||
That's it! There's no need to include any SDKs or instrument your application code. Dapr automatically handles the distributed tracing for you. | ||
|
||
## View traces | ||
|
||
Deploy and run some applications. After a few minutes, you should see traces appearing in your Dynatrace tenant: | ||
|
||
1. Navigate to **Search > Distributed tracing** in your Dynatrace UI. | ||
2. Filter by service names to see your Dapr applications and their associated tracing spans. | ||
|
||
<img src="/images/open-telemetry-collector-dynatrace-traces.png" width=1200 alt="Dynatrace showing tracing data."> | ||
|
||
{{% alert title="Note" color="primary" %}} | ||
Only operations going through Dapr API exposed by Dapr sidecar (for example, service invocation or event publishing) are displayed in Dynatrace distributed traces. | ||
{{% /alert %}} | ||
|
||
|
||
{{% alert title="Disable OneAgent daprd monitoring" color="warning" %}} | ||
If you are running Dynatrace OneAgent in your cluster, you should exclude the `daprd` sidecar container from OneAgent monitoring to prevent interferences in this configuration. Excluding it prevents any automatic injection attempts that could break functionality or result in confusing traces. | ||
|
||
|
||
Add this annotation to your application deployments or globally in your dynakube configuration file: | ||
|
||
```yaml | ||
metadata: | ||
annotations: | ||
dapr.io/enabled: "true" | ||
dapr.io/app-id: "MyApp" | ||
dapr.io/app-port: "8080" | ||
dapr.io/config: "tracing" | ||
container.inject.dynatrace.com/daprd: "false" # Exclude dapr sidecar from being auto-monitored by OneAgent | ||
|
||
``` | ||
{{% /alert %}} | ||
|
||
## Related links | ||
- Try out the [observability quickstart](https://github.com/dapr/quickstarts/tree/master/tutorials/observability/README.md) | ||
- Learn how to set [tracing configuration options]({{< ref "configuration-overview.md#tracing" >}}) | ||
- [Dynatrace OpenTelemetry documentation](https://docs.dynatrace.com/docs/ingest-from/opentelemetry) | ||
- Enrich OTLP telemetry data [with Kubernetes metadata | ||
](https://docs.dynatrace.com/docs/ingest-from/opentelemetry/collector/use-cases/kubernetes/k8s-enrich) |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.