OpenTelemetry Integration #370

saada · 2020-10-22T14:47:31Z

saada
Oct 22, 2020

We would like to get traces out of Flux to help debug issues with Flux and get visibility on what's happening. This is even more valuable now that Flux is broken up into different controllers. Would it be possible to integrate with OpenTelemetry?

Example use case:

I want to see everything that happened from when a git change occurred in a single trace. All the applied yaml files in Source Controller, all the notifications that went out to Notifications Controller, all the HelmReleases that were applied in Helm-Controller, Kustomize Controller, etc...
I want to build a UI using trace data to track release changes and integrate deeply with a Git commit/tag, a Docker image tag, and the GOTK flow of operations

Answered by stefanprodan

Oct 29, 2020

Similar to to how we defile alerts we could extend notification-controller to collect and push traces to an OpenTelemetry provider.

Define a tracing provider:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
 kind: Provider
 metadata:
   name: jaeger
   namespace: flux-system
 spec:
   type: otel
   address: http://jaeger-collector.jaeger:4318/v1/traces

Enable tracing for specific reconciliations:

 apiVersion: notification.toolkit.fluxcd.io/v1beta3
 kind: Alert
 metadata:
   name: alert-test-1
   namespace: flux-system
 spec:
   providerRef:
     name: jaeger
   eventSeverity: info
   eventSources:
     - kind: GitRepository
       name: '*'
       namespace: test-1
     - kind: Kustomi…

View full answer

stefanprodan · 2020-10-29T10:01:04Z

stefanprodan
Oct 29, 2020
Maintainer

Similar to to how we defile alerts we could extend notification-controller to collect and push traces to an OpenTelemetry provider.

Define a tracing provider:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
 kind: Provider
 metadata:
   name: jaeger
   namespace: flux-system
 spec:
   type: otel
   address: http://jaeger-collector.jaeger:4318/v1/traces

Enable tracing for specific reconciliations:

 apiVersion: notification.toolkit.fluxcd.io/v1beta3
 kind: Alert
 metadata:
   name: alert-test-1
   namespace: flux-system
 spec:
   providerRef:
     name: jaeger
   eventSeverity: info
   eventSources:
     - kind: GitRepository
       name: '*'
       namespace: test-1
     - kind: Kustomization
       name: '*'
       namespace: test-1
   eventMetadata:
     env: staging
     cluster: cluster-1
     region: eu-west-1