Armeria integration with Spring Boot takes forever to shutdown

[d3bt3ch]

Armeria integration with Spring Boot takes forever to shutdown. What could be wrong with my implementation. The actual implementation is as mentiontioned below

    @Bean
    public ArmeriaServerConfigurator armeriaServerConfigurator(@NonNull BlockingTaskExecutor blockingTaskExecutor,
                                                               @NonNull EventLoopGroup workerGroup,
                                                               @NonNull EventLoopGroup serviceWorkerGroup,
                                                               @NonNull GrpcService grpcService,
                                                               @NonNull DocService docService
    ) {

        return builder -> builder
                .serviceUnder("/", grpcService)
                .serviceUnder("/docs", docService)
                .channelOption(SO_BACKLOG, 10000)
                .channelOption(SO_REUSEADDR, true)
                .childChannelOption(SO_SNDBUF, 8192)
                .childChannelOption(SO_RCVBUF, 8192)
                .childChannelOption(SO_KEEPALIVE, true)
                .childChannelOption(TCP_NODELAY, true)
                // .childChannelOption(EPOLL_MODE, EpollMode.LEVEL_TRIGGERED)
                // .decorator(DecodingService.newDecorator())
                // .decorator(ThrottlingService.newDecorator(ThrottlingStrategy.rateLimiting(10000)))
                // .maxNumConnections(1000)
                // .maxRequestLength(10485760)
                // .requestTimeout(Duration.ofSeconds(10))
                // .blockingTaskExecutor(scheduledExecutorService, true)
                // .gracefulShutdownTimeout(Duration.ofSeconds(10), Duration.ofSeconds(15))
                // .pingInterval(Duration.ofMillis(1000))
                // .blockingTaskExecutor(blockingTaskExecutor, false)
                // .tlsSelfSigned()
                .disableServerHeader()
                .disableDateHeader()
                // .workerGroup(workerGroup, true)
                // .serviceWorkerGroup(serviceWorkerGroup, true)
                .accessLogger(LoggerFactory.getLogger(Application.class))
                .errorHandler((ctx, cause) -> {
                    if (log.isErrorEnabled()) {
                        log.error("Error occurred during serving request", cause);
                    }
                    return HttpResponse.ofFailure(cause);
                });

        // armeriaServerGracefulShutdownLifecycle
    }

    @Bean(destroyMethod = "close")
    public BlockingTaskExecutor blockingTaskExecutor() {
        return BlockingTaskExecutor.builder()
                .numThreads(200)
                .build();
    }

    @Bean(value = "workerGroup")
    public EventLoopGroup workerGroup() {
        return EventLoopGroups.newEventLoopGroup(100, ExtraExecutors.platformThreadFactory());
    }

    @Bean(value = "serviceWorkerGroup")
    public EventLoopGroup serviceWorkerGroup() {
        return EventLoopGroups.newEventLoopGroup(100, ExtraExecutors.platformThreadFactory());
    }

    @Bean
    public GrpcService grpcService(@NonNull UserService userService) {

        return GrpcService.builder()
                .addService(userService)
                .autoCompression(true)
                .enableHttpJsonTranscoding(false)
                .enableUnframedRequests(false)
                .useBlockingTaskExecutor(false)
                .enableHealthCheckService(true)
                // .jsonMarshallerFactory(descriptor -> GrpcJsonMarshaller.ofGson())
                .exceptionHandler((ctx, status, cause, metadata) -> {
                    if (log.isErrorEnabled()) {
                        log.error("Exception occurred during serving grpc request", cause);
                    }
                    return status;
                })
                //                .unframedGrpcErrorHandler((ctx, status, response) -> {
                //                    if (log.isErrorEnabled()) {
                //                        log.error("Error occurred during serving unframed grpc request: {}", status);
                //                    }
                //                    return response.toHttpResponse();
                //                })
                .build();
    }

    @Bean
    public DocService docService() {
        return DocService.builder()
                .build();
    }

[ikhoon]

I didn't see any wrong with your configuration except for ExtraExecutors.platformThreadFactory(). Would you provide a reproducible example?

[d3bt3ch]

The following is the code for ExtraExecutors.platformThreadFactory()

public final class ExtraExecutors {

    ......

    public static ThreadFactory platformThreadFactory() {
        return platformThreadFactory("exec-" + PLATFORM_INT_COUNTER.getAndIncrement(), false);
    }

    public static ThreadFactory platformThreadFactory(String namePrefix) {
        return platformThreadFactory(namePrefix, false);
    }

    public static ThreadFactory platformThreadFactory(boolean daemon) {
        return platformThreadFactory("exec-" + PLATFORM_INT_COUNTER.getAndIncrement(), daemon);
    }

    public static ThreadFactory platformThreadFactory(String namePrefix, boolean daemon) {
        return Thread.ofPlatform()
                .name(namePrefix, 1)
                .daemon(daemon)
                .priority(NORM_PRIORITY)
                .inheritInheritableThreadLocals(true)
                .uncaughtExceptionHandler(new UncaughtExceptionHandlerImpl())
                .factory();
    }

    public static ThreadFactory virtualThreadFactory() {
        return virtualThreadFactory("virt-" + VIRTUAL_INT_COUNTER.getAndIncrement());
    }

    public static ThreadFactory virtualThreadFactory(String namePrefix) {
        return Thread.ofVirtual()
                .name(namePrefix, 1)
                .inheritInheritableThreadLocals(true)
                .uncaughtExceptionHandler(new UncaughtExceptionHandlerImpl())
                .factory();
    }

    ......
}

[ikhoon]

Would you try to use Armeria ThreadFactories instead?

armeria/core/src/main/java/com/linecorp/armeria/common/util/ThreadFactories.java

Lines 24 to 26 in e649318

	* Provides a builder for {@link ThreadFactory}.
	*/
	public final class ThreadFactories {

I have no time to investigate the compatibility with the JDK Platform thread. 😅 If you find the root cause, would you share it with us?

[d3bt3ch]

I could not yet. But I noticed a surprising behaviour. If I have a few connections, the server shuts down normally. But if there too many concurrent connections the server takes forever to shutdown, like when doing a load test using ghz with many concurrent connections and requests.