Setting sampling rate across microservices shows inconsistent sampling

[paleboot]

Hi folks, I'd like to ask a question regarding sampling rate, and whether or not setting it across microservices can result in inconsistent trace.

Hypothesis

We believe that when a downstream service defines its own tracer.RateRule, it overrides the sampling decision made by the root service (our Gateway), leading to inconsistent and incomplete traces.

Background

Consider the following architecture:

flowchart LR
  g[Gateway] --> a[Authn/Authz]
  g[Gateway] --> u[User]
  g[Gateway] --> t[Transaction]

Loading

For each services, we would have a wrapper on top of [email protected], with the following minimum config as an example:

func initializeTracer(rate float64) {
    var samplingRate = 1.0

    if rate != 0 {
        samplingRate = rate
    }

    tracer.Start(
        // Agent address and other service-specific tags have been omitted for clarity.
        // ...

        tracer.WithSamplingRules([]tracer.SamplingRule{
            tracer.RateRule(samplingRate),
        }),
        tracer.WithAnalytics(true),
        tracer.WithAnalyticsRate(samplingRate),

        // ...
    )
}

Which basically means that each service can define their own sampling rate, and when it's not, it defaults to 1.0

Scenario

Given

1. Gateway has 0.3 sampling rate
2. User has 1.0 sampling rate

Expected

When a Gateway trace get's dropped, User trace and all subsequent trace should get dropped.

Actual

1. The Gateway service correctly drops its root span.
2. The User service, however, does not respect the propagated DROP decision. Its local RateRule(1.0) takes precedence, and it generates and sends its own span.
3. This results in an "orphaned" trace in the backend, where we have a span for the User service but the corresponding root span from the Gateway is missing.

Question

Is this behavior intended? Is the tracer.RateRule designed to always override any inherited sampling decision? If so, what is the recommended approach to enforce a unified, head-based sampling decision made at the Gateway across all downstream microservices?

[darccio]

@paleboot Thanks for the detailed message. You are right, it wasn't an intended behaviour and we already fixed it in #3856, which is going to be released as part of v2.2.3 (also available as v1.74.6) next week.

[paleboot]

Thanks for the quick response, looking forward for the next release!

[darccio]

@paleboot v2.2.3 and v1.74.6 are already out. Please ping us if you see this issue again.