|
1 | | -# Profiling Instrumentation |
| 1 | +# Profiling Instrumentation for OpenTelemetry Go SDK |
2 | 2 |
|
3 | 3 | **NOTE**: This is an experimental package -- and will be officially supported in future versions of Pyroscope |
4 | 4 |
|
5 | 5 | The package provides means to integrate tracing with profiling. More specifically, a `TracerProvider` implementation, |
6 | | -that annotates profiling data with span IDs: when a new trace span emerges, the tracer adds a `profile_id` [pprof tag](https://github.com/google/pprof/blob/master/doc/README.md#tag-filtering) |
| 6 | +that annotates profiling data with span IDs: when a new trace span emerges, the tracer adds a `span_id` [pprof tag](https://github.com/google/pprof/blob/master/doc/README.md#tag-filtering) |
7 | 7 | that points to the span. This makes it possible to filter out a profile of a particular trace span in [Pyroscope](https://pyroscope.io). |
8 | 8 |
|
9 | | -### Jaeger Example |
10 | | -You can find a full Jaeger example (with custom Jaeger UI) in the [tracing/jaeger](https://github.com/pyroscope-io/pyroscope/tree/main/examples/tracing/jaeger) folder in the Pyroscope repository. |
11 | | - |
| 9 | +Note that the module does not control `pprof` profiler itself – it still needs to be started for profiles to be |
| 10 | +collected. This can be done either via `runtime/pprof` package, or using the [Pyroscope client](https://github.com/grafana/pyroscope-go). |
12 | 11 |
|
13 | | -#### Baseline Diffs |
14 | | -We also added functionality where each individual span is compared to a baseline of spans with similar properties and the diff can be shown in the UI: |
| 12 | +By default, only the root span gets labeled (the first span created locally): such spans are marked with the |
| 13 | +`pyroscope.profile.id` attribute set to the span ID. Please note that presence of the attribute does not necessarily |
| 14 | +indicate that the span has a profile: stack trace samples might not be collected, if the utilized CPU time is |
| 15 | +less than the sample interval (10ms). |
15 | 16 |
|
16 | | -[](https://user-images.githubusercontent.com/23323466/165633049-9591b0fd-b8be-4fbd-a0af-d90a1dd89b7b.mov) |
| 17 | +Limitations: |
| 18 | +- Only CPU profiling is fully supported at the moment. |
17 | 19 |
|
| 20 | +### Trace spans profiles |
18 | 21 |
|
19 | | -### Grafana Example |
20 | | -For another example of what this package allows you to do you can see with Grafana the ability to link between logs, traces and profiles in the following video ([source](https://github.com/pyroscope-io/pyroscope/tree/main/examples/tracing/jaeger)): |
| 22 | +To start profiling trace spans, you need to include our go module in your app: |
21 | 23 |
|
22 | | -[](https://user-images.githubusercontent.com/23323466/172881613-842f67f0-6bfa-4671-a44a-e966d5ca67a4.mov) |
| 24 | +``` |
| 25 | +go get github.com/grafana/otel-profiling-go |
| 26 | +``` |
23 | 27 |
|
| 28 | +Then add the pyroscope tracer provider: |
24 | 29 |
|
25 | | -### Other Notes |
26 | | -Note that the module does not control `pprof` profiler itself – it still needs to be started for profiles to be |
27 | | -collected. This can be done either via `runtime/pprof` package, or using the [Pyroscope client](https://github.com/pyroscope-io/client). |
| 30 | +```go |
| 31 | +package main |
| 32 | + |
| 33 | +import ( |
| 34 | + otelpyroscope "github.com/grafana/otel-profiling-go" |
| 35 | + "github.com/grafana/pyroscope-go" |
| 36 | +) |
| 37 | + |
| 38 | +func main() { |
| 39 | + // Initialize your tracer provider as usual. |
| 40 | + tp := initTracer() |
| 41 | + |
| 42 | + // Wrap it with otelpyroscope tracer provider. |
| 43 | + otel.SetTracerProvider(otelpyroscope.NewTracerProvider(tp)) |
| 44 | + |
| 45 | + // If you're using Pyroscope Go SDK, initialize pyroscope profiler. |
| 46 | + _, _ = pyroscope.Start(pyroscope.Config{ |
| 47 | + ApplicationName: "my-service", |
| 48 | + ServerAddress: "http://localhost:4040", |
| 49 | + }) |
| 50 | + |
| 51 | + // Your code goes here. |
| 52 | +} |
| 53 | +``` |
| 54 | + |
| 55 | +Tracing integration is supported in pull mode as well: if you scrape profiles using Grafana Agent, you should |
| 56 | +make sure that the pyroscope `service_name` label matches `service.name` attribute specified in the OTel SDK configuration. |
| 57 | +Please refer to the [Grafana Agent](https://grafana.com/docs/pyroscope/latest/configure-client/grafana-agent/go_pull/) |
| 58 | +documentation to learn more. |
| 59 | + |
| 60 | +## Example |
28 | 61 |
|
29 | | -By default, only the root span gets annotated (the first span created locally), this is done to circumvent the fact that the profiler records only the time spent on CPU. Otherwise, all the children profiles should be merged to get the full representation of the root span profile. |
| 62 | +You can find a complete example setup with Grafana Tempo in the [Pyroscope repository](https://github.com/grafana/pyroscope/tree/main/examples/tracing/tempo). |
30 | 63 |
|
31 | | -There are few limitations: |
32 | | - - Only Go CPU profiling is fully supported at the moment. |
33 | | - - Due to the very idea of the sampling profilers, spans shorter than the sample interval may not be captured. For example, Go CPU profiler probes stack traces 100 times per second, meaning that spans shorter than 10ms may not be captured. |
| 64 | + |
0 commit comments