Unable to export Micrometer traces via OpenTelemetry - (Spring Boot 3.5.5, Spring Cloud 2025.0.0) #3904
Description
Issue
I am trying to trace my Spring Cloud Gateway application and have those traces exported via OpenTelemetry, however I have failed to get anything useful successfully exported.
I can see that my request has been given a trace id, and the trace is propagated to downstream services. My only issue is that Spring Cloud Gateway is not exporting the spans I would expect.
There have been similar issues raised (e.g. #3184) but I the solutions there do not seem to work for me. I am sure I am missing something; I just need some help identifying what that is.
Any help would be greatly appreciated!
Sample Application
I have created a small application below to reproduce this issue.
.
├── pom.xml
└── src
└── main
├── java
│ └── com
│ └── example
│ ├── CloudGatewayApplication.java
│ └── CustomGatewayInterceptor.java
└── resources
└── application.yml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://maven.apache.org/POM/4.0.0"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>3.5.5</version>
<relativePath /> <!-- Force parent lookup from Maven repositories -->
</parent>
<groupId>com.example</groupId>
<artifactId>tracing-example</artifactId>
<version>0.0.1</version>
<packaging>jar</packaging>
<name>Tracing Example</name>
<dependencyManagement>
<dependencies>
<!-- Spring Cloud bill of materials -->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>2025.0.0</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<!-- Spring Boot dependencies -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<!-- Spring Cloud dependencies -->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-gateway-server-webflux</artifactId>
</dependency>
<!-- Tracing dependencies -->
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>
</dependencies>
<build>
<plugins>
<!-- SpringFramework - Creates an executable jar -->
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>package com.example;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
@SpringBootApplication
public class CloudGatewayApplication {
public static void main(final String[] args) {
SpringApplication.run(CloudGatewayApplication.class, args);
}
}package com.example;
import io.micrometer.observation.annotation.Observed;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.springframework.cloud.gateway.filter.GatewayFilterChain;
import org.springframework.cloud.gateway.filter.GlobalFilter;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.Ordered;
import org.springframework.web.server.ServerWebExchange;
import reactor.core.publisher.Mono;
@Configuration
public class CustomGatewayInterceptor {
@Bean
public GlobalFilter customFilter() {
return new CustomGlobalFilter();
}
public class CustomGlobalFilter implements GlobalFilter, Ordered {
private static final Logger logger = LogManager.getLogger(CustomGlobalFilter.class);
@Override
@Observed(name = "com.example.CustomGlobalFilter", contextualName = "CustomGlobalFilter")
public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
logger.info("Inside CustomGlobalFilter");
return chain.filter(exchange);
}
@Override
public int getOrder() {
return 1;
}
}
}server:
port: 8443
http2:
enabled: true
ssl:
enabled: false
spring:
application:
name: Test Tracing
reactor:
context-propagation: AUTO
cloud:
gateway:
server:
webflux:
x-forwarded:
enabled: false
httpserver:
wiretap: false
routes:
- id: TestRoute
uri: https://echo.free.beeceptor.com
predicates:
- Path=/test/**
filters:
- RewritePath=/test/?(?<segment>.*), /$\{segment}
logging:
level:
io.micrometer: DEBUG
io.opentelemetry: DEBUG
management:
tracing:
enabled: true
propagation:
type: w3c
sampling:
probability: 1
otlp:
tracing:
endpoint: http://127.0.0.1:4318/v1/tracesOTEL Collector
This is the OTEL Collector configuration I am running via docker run -it --rm -p 4318:4318 -v $(pwd)/otel-collector.yml:/otel-collector.yml:ro otel/opentelemetry-collector --config=/otel-collector.yml.
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
exporters:
debug:
verbosity: normal
service:
pipelines:
traces:
receivers: [otlp]
exporters: [debug]Results
I invoke the gateway using curl, which results in the following logs, showing that the custom filter is being executed, but none of the tracing is exported to OpenTelemetry.
You can see that:
- The incoming request from
curldidn't have any trace details, so a new trace id6c2cfbb7931efae19359ef15872f53e0was generated within Spring Cloud Gateway. - The custom global filter was executed and the trace details ware included on the log printed (
6c2cfbb7931efae19359ef15872f53e0-b1a77d955987ef1b). - The trace details were propagated to the downstream service correctly in the header
Traceparent: 00-6c2cfbb7931efae19359ef15872f53e0-7f2a73ad786de5e6-01. - The tracing within my custom global filter does not show in the OTEL debug logs.
$ curl http://localhost:8443/test/tracing
{
"method": "GET",
"protocol": "https",
"host": "echo.free.beeceptor.com",
"path": "/tracing",
"ip": "91.237.176.102:60509",
"headers": {
"Host": "echo.free.beeceptor.com",
"User-Agent": "curl/7.61.1",
"Accept": "*/*",
"Traceparent": "00-6c2cfbb7931efae19359ef15872f53e0-7f2a73ad786de5e6-01",
"Via": "2.0 Caddy",
"Accept-Encoding": "gzip"
},
"parsedQueryParams": {}
}
$ java -jar target/tracing-example-0.0.1.jar
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v3.5.5)
2025-09-04T10:29:54.384+01:00 INFO 30669 --- [Test Tracing] [ main] [ ] com.example.CloudGatewayApplication : Starting CloudGatewayApplication v0.0.1 using Java 21.0.8 with PID 30669 (/ems/users/njames/tmp/CloudGatewayTrace/target/tracing-example-0.0.1.jar started by njames in /ems/users/njames/tmp/CloudGatewayTrace)
2025-09-04T10:29:54.393+01:00 INFO 30669 --- [Test Tracing] [ main] [ ] com.example.CloudGatewayApplication : No active profile set, falling back to 1 default profile: "default"
2025-09-04T10:29:57.689+01:00 INFO 30669 --- [Test Tracing] [ main] [ ] o.s.cloud.context.scope.GenericScope : BeanFactory id=2d96c5cc-560a-39cd-9cd6-7fe5d0343ad3
2025-09-04T10:30:00.460+01:00 INFO 30669 --- [Test Tracing] [ main] [ ] o.s.c.g.r.RouteDefinitionRouteLocator : Loaded RoutePredicateFactory [After]
2025-09-04T10:30:00.461+01:00 INFO 30669 --- [Test Tracing] [ main] [ ] o.s.c.g.r.RouteDefinitionRouteLocator : Loaded RoutePredicateFactory [Before]
2025-09-04T10:30:00.461+01:00 INFO 30669 --- [Test Tracing] [ main] [ ] o.s.c.g.r.RouteDefinitionRouteLocator : Loaded RoutePredicateFactory [Between]
2025-09-04T10:30:00.461+01:00 INFO 30669 --- [Test Tracing] [ main] [ ] o.s.c.g.r.RouteDefinitionRouteLocator : Loaded RoutePredicateFactory [Cookie]
2025-09-04T10:30:00.462+01:00 INFO 30669 --- [Test Tracing] [ main] [ ] o.s.c.g.r.RouteDefinitionRouteLocator : Loaded RoutePredicateFactory [Header]
2025-09-04T10:30:00.462+01:00 INFO 30669 --- [Test Tracing] [ main] [ ] o.s.c.g.r.RouteDefinitionRouteLocator : Loaded RoutePredicateFactory [Host]
2025-09-04T10:30:00.462+01:00 INFO 30669 --- [Test Tracing] [ main] [ ] o.s.c.g.r.RouteDefinitionRouteLocator : Loaded RoutePredicateFactory [Method]
2025-09-04T10:30:00.463+01:00 INFO 30669 --- [Test Tracing] [ main] [ ] o.s.c.g.r.RouteDefinitionRouteLocator : Loaded RoutePredicateFactory [Path]
2025-09-04T10:30:00.463+01:00 INFO 30669 --- [Test Tracing] [ main] [ ] o.s.c.g.r.RouteDefinitionRouteLocator : Loaded RoutePredicateFactory [Query]
2025-09-04T10:30:00.463+01:00 INFO 30669 --- [Test Tracing] [ main] [ ] o.s.c.g.r.RouteDefinitionRouteLocator : Loaded RoutePredicateFactory [ReadBody]
2025-09-04T10:30:00.464+01:00 INFO 30669 --- [Test Tracing] [ main] [ ] o.s.c.g.r.RouteDefinitionRouteLocator : Loaded RoutePredicateFactory [RemoteAddr]
2025-09-04T10:30:00.464+01:00 INFO 30669 --- [Test Tracing] [ main] [ ] o.s.c.g.r.RouteDefinitionRouteLocator : Loaded RoutePredicateFactory [XForwardedRemoteAddr]
2025-09-04T10:30:00.464+01:00 INFO 30669 --- [Test Tracing] [ main] [ ] o.s.c.g.r.RouteDefinitionRouteLocator : Loaded RoutePredicateFactory [Weight]
2025-09-04T10:30:00.465+01:00 INFO 30669 --- [Test Tracing] [ main] [ ] o.s.c.g.r.RouteDefinitionRouteLocator : Loaded RoutePredicateFactory [CloudFoundryRouteService]
2025-09-04T10:30:01.232+01:00 INFO 30669 --- [Test Tracing] [ main] [ ] o.s.b.a.e.web.EndpointLinksResolver : Exposing 1 endpoint beneath base path '/actuator'
2025-09-04T10:30:02.018+01:00 DEBUG 30669 --- [Test Tracing] [ main] [ ] i.o.e.internal.http.HttpExporterBuilder : Using HttpSender: io.opentelemetry.exporter.sender.okhttp.internal.OkHttpHttpSender
2025-09-04T10:30:02.161+01:00 DEBUG 30669 --- [Test Tracing] [ main] [ ] i.o.sdk.internal.JavaVersionSpecific : Using the APIs optimized for: Java 9+
2025-09-04T10:30:02.871+01:00 INFO 30669 --- [Test Tracing] [ main] [ ] o.s.b.web.embedded.netty.NettyWebServer : Netty started on port 8443 (http)
2025-09-04T10:30:03.088+01:00 INFO 30669 --- [Test Tracing] [ main] [ ] com.example.CloudGatewayApplication : Started CloudGatewayApplication in 9.971 seconds (process running for 11.597)
2025-09-04T10:32:47.166+01:00 DEBUG 30669 --- [Test Tracing] [or-http-epoll-2] [ ] i.m.t.o.p.BaggageTextMapPropagator : Will propagate new baggage context for entries {}
2025-09-04T10:32:47.265+01:00 INFO 30669 --- [Test Tracing] [or-http-epoll-2] [6c2cfbb7931efae19359ef15872f53e0-b1a77d955987ef1b] c.e.C.CustomGlobalFilter : Inside CustomGlobalFilter
$ docker run -it --rm -p 4318:4318 -v $(pwd)/otel-collector.yml:/otel-collector.yml:ro otel/opentelemetry-collector --config=/otel-collector.yml
2025-09-04T09:31:35.671Z info [email protected]/service.go:211 Starting otelcol... {"resource": {"service.instance.id": "edc91222-b16d-4987-a9d9-405526bfcbef", "service.name": "otelcol", "service.version": "0.134.0"}, "Version": "0.134.0", "NumCPU": 4}
2025-09-04T09:31:35.671Z info extensions/extensions.go:41 Starting extensions... {"resource": {"service.instance.id": "edc91222-b16d-4987-a9d9-405526bfcbef", "service.name": "otelcol", "service.version": "0.134.0"}}
2025-09-04T09:31:35.672Z info [email protected]/otlp.go:179 Starting HTTP server {"resource": {"service.instance.id": "edc91222-b16d-4987-a9d9-405526bfcbef", "service.name": "otelcol", "service.version": "0.134.0"}, "otelcol.component.id": "otlp", "otelcol.component.kind": "receiver", "endpoint": "[::]:4318"}
2025-09-04T09:31:35.673Z info [email protected]/service.go:234 Everything is ready. Begin running and processing data. {"resource": {"service.instance.id": "edc91222-b16d-4987-a9d9-405526bfcbef", "service.name": "otelcol", "service.version": "0.134.0"}}
2025-09-04T09:32:52.434Z info Traces {"resource": {"service.instance.id": "edc91222-b16d-4987-a9d9-405526bfcbef", "service.name": "otelcol", "service.version": "0.134.0"}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "resource spans": 2, "spans": 2}
2025-09-04T09:32:52.436Z info ResourceTraces #0 service.name=Test Tracing telemetry.sdk.language=java telemetry.sdk.name=opentelemetry telemetry.sdk.version=1.49.0
ScopeTraces #0 [email protected]
HTTP GET 6c2cfbb7931efae19359ef15872f53e0 7f2a73ad786de5e6 http.method=GET http.status_code=200 http.uri=http://localhost:8443/tracing spring.cloud.gateway.route.id=TestRoute spring.cloud.gateway.route.uri=https://echo.free.beeceptor.com:443
ResourceTraces #1 service.name=Test Tracing telemetry.sdk.language=java telemetry.sdk.name=opentelemetry telemetry.sdk.version=1.49.0
ScopeTraces #1 [email protected]
http get 6c2cfbb7931efae19359ef15872f53e0 b1a77d955987ef1b exception=none http.url=/test/tracing method=GET outcome=SUCCESS status=200 uri=UNKNOWN
{"resource": {"service.instance.id": "edc91222-b16d-4987-a9d9-405526bfcbef", "service.name": "otelcol", "service.version": "0.134.0"}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "traces"}