Bug report
What Debezium connector do you use and what version?
Debezium Server 3.6.0.Final (quay.io/debezium/server:3.6.0.Final), Amazon Kinesis sink.
Source connector: SQL Server. Running on Quarkus 3.33.1.1, JVM Java 21.
What is the connector configuration?
Kinesis sink using the AWS SDK default credential chain (no debezium.sink.kinesis.credentials.profile set), region only:
debezium.sink.type = kinesisdebezium.sink.kinesis.region = us-east-1- offset/schema-history storage on file, region
us-east-1
Deployed on EKS with IRSA: the service account is annotated with eks.amazonaws.com/role-arn, so the pod has AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE injected. IMDS is disabled.
What is the captured database version and mode of deployment?
SQL Server, containerized deployment on EKS (Kubernetes), snapshot.mode = initial.
What behavior do you expect?
Without credentials.profile, the Kinesis sink uses the AWS SDK default credential chain, which includes web identity token credentials. On EKS/IRSA this should resolve credentials via sts:AssumeRoleWithWebIdentity so PutRecords succeeds out of the box.
The web-identity path is only reliably supported in recent AWS SDK for Java v2 releases. The official image should therefore ship a current AWS SDK that includes the sts module so IRSA/web-identity works without any custom image changes.
What behavior do you see?
The snapshot starts and reads rows, but the first Kinesis PutRecords fails because the credential chain cannot use the injected web-identity token — the sts module is not on the class path. The engine then fails and stops.
Do you see the same behaviour using the latest released Debezium version?
The image bundles AWS SDK for Java v2 2.17.241 (from 2021) for auth/kinesis/regions, and does not include the sts module, so the web-identity provider cannot call sts:AssumeRoleWithWebIdentity. Suggest bundling a latest AWS SDK for Java v2 (including the sts module) in the Kinesis sink so IRSA/web-identity resolves out of the box.
Do you have the connector logs, ideally from start till finish?
ERROR io.debezium.embedded.async.AsyncEmbeddedEngine - Engine has failed with java.util.concurrent.ExecutionException: io.debezium.DebeziumException: Error while executing batch at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:191) at io.debezium.embedded.async.AsyncEmbeddedEngine.runTasksPolling(AsyncEmbeddedEngine.java:550) at io.debezium.embedded.async.AsyncEmbeddedEngine.run(AsyncEmbeddedEngine.java:232) at io.debezium.server.DebeziumServer.lambda$start$0(DebeziumServer.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.lang.Thread.run(Thread.java:1583) Caused by: io.debezium.DebeziumException: Error while executing batch at io.debezium.server.DefaultChangeConsumer.handleBatch(DefaultChangeConsumer.java:61) at io.debezium.embedded.async.AbstractRecordProcessor$BatchProcessor$DirectProcessor.process(AbstractRecordProcessor.java:76) at io.debezium.embedded.async.ParallelSmtAndConvertBatchProcessor.processRecords(ParallelSmtAndConvertBatchProcessor.java:54) at io.debezium.embedded.async.AsyncEmbeddedEngine$PollRecords.doCall(AsyncEmbeddedEngine.java:1337) at io.debezium.embedded.async.AsyncEmbeddedEngine$PollRecords.doCall(AsyncEmbeddedEngine.java:1311) at io.debezium.embedded.async.RetryingCallable.call(RetryingCallable.java:47) at java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) at java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.lang.Thread.run(Thread.java:1583) Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(profiles=[])), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): To use web identity tokens, the 'sts' service module must be on the class path., ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(profiles=[])): Profile file contained no credentials for profile 'default', ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): IMDS credentials have been disabled by environment variable or system property.] at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:102) at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:117) at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:45) at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:123) at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.resolveCredentials(AwsExecutionContextBuilder.java:184) at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:120) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:69) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:78) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:175) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76) at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56) at software.amazon.awssdk.services.kinesis.DefaultKinesisClient.putRecords(DefaultKinesisClient.java:1969) at io.debezium.server.kinesis.KinesisChangeConsumer.recordsSent(KinesisChangeConsumer.java:221) at io.debezium.server.kinesis.KinesisChangeConsumer.handleBatch(KinesisChangeConsumer.java:174) at io.debezium.server.DefaultChangeConsumer.handleBatch(DefaultChangeConsumer.java:58) at io.debezium.embedded.async.AbstractRecordProcessor$BatchProcessor$DirectProcessor.process(AbstractRecordProcessor.java:76) at io.debezium.embedded.async.ParallelSmtAndConvertBatchProcessor.processRecords(ParallelSmtAndConvertBatchProcessor.java:54) at io.debezium.embedded.async.AsyncEmbeddedEngine$PollRecords.doCall(AsyncEmbeddedEngine.java:1337) at io.debezium.embedded.async.AsyncEmbeddedEngine$PollRecords.doCall(AsyncEmbeddedEngine.java:1311) at io.debezium.embedded.async.RetryingCallable.call(RetryingCallable.java:47) at java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) at java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.lang.Thread.run(Thread.java:1583)
How to reproduce the issue using our tutorial deployment?
- Deploy
quay.io/debezium/server:3.6.0.Final with the Amazon Kinesis sink on EKS.
- Configure the sink with only
debezium.sink.kinesis.region (no credentials.profile), so the AWS SDK default credential chain is used.
- Grant AWS access via IRSA (annotate the service account with
eks.amazonaws.com/role-arn); do not provide static keys, and keep IMDS disabled.
- Start any source connector so records are produced. The snapshot reads rows, then the first
PutRecords fails with WebIdentityTokenCredentialsProvider(): To use web identity tokens, the 'sts' service module must be on the class path.
Bug report
What Debezium connector do you use and what version?
Debezium Server 3.6.0.Final (
quay.io/debezium/server:3.6.0.Final), Amazon Kinesis sink.Source connector: SQL Server. Running on Quarkus 3.33.1.1, JVM Java 21.
What is the connector configuration?
Kinesis sink using the AWS SDK default credential chain (no
debezium.sink.kinesis.credentials.profileset), region only:debezium.sink.type = kinesisdebezium.sink.kinesis.region = us-east-1us-east-1Deployed on EKS with IRSA: the service account is annotated with
eks.amazonaws.com/role-arn, so the pod hasAWS_ROLE_ARNandAWS_WEB_IDENTITY_TOKEN_FILEinjected. IMDS is disabled.What is the captured database version and mode of deployment?
SQL Server, containerized deployment on EKS (Kubernetes),
snapshot.mode = initial.What behavior do you expect?
Without
credentials.profile, the Kinesis sink uses the AWS SDK default credential chain, which includes web identity token credentials. On EKS/IRSA this should resolve credentials viasts:AssumeRoleWithWebIdentitysoPutRecordssucceeds out of the box.The web-identity path is only reliably supported in recent AWS SDK for Java v2 releases. The official image should therefore ship a current AWS SDK that includes the
stsmodule so IRSA/web-identity works without any custom image changes.What behavior do you see?
The snapshot starts and reads rows, but the first Kinesis
PutRecordsfails because the credential chain cannot use the injected web-identity token — thestsmodule is not on the class path. The engine then fails and stops.Do you see the same behaviour using the latest released Debezium version?
The image bundles AWS SDK for Java v2
2.17.241(from 2021) for auth/kinesis/regions, and does not include thestsmodule, so the web-identity provider cannot callsts:AssumeRoleWithWebIdentity. Suggest bundling a latest AWS SDK for Java v2 (including thestsmodule) in the Kinesis sink so IRSA/web-identity resolves out of the box.Do you have the connector logs, ideally from start till finish?
ERROR io.debezium.embedded.async.AsyncEmbeddedEngine - Engine has failed with java.util.concurrent.ExecutionException: io.debezium.DebeziumException: Error while executing batch at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:191) at io.debezium.embedded.async.AsyncEmbeddedEngine.runTasksPolling(AsyncEmbeddedEngine.java:550) at io.debezium.embedded.async.AsyncEmbeddedEngine.run(AsyncEmbeddedEngine.java:232) at io.debezium.server.DebeziumServer.lambda$start$0(DebeziumServer.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.lang.Thread.run(Thread.java:1583) Caused by: io.debezium.DebeziumException: Error while executing batch at io.debezium.server.DefaultChangeConsumer.handleBatch(DefaultChangeConsumer.java:61) at io.debezium.embedded.async.AbstractRecordProcessor$BatchProcessor$DirectProcessor.process(AbstractRecordProcessor.java:76) at io.debezium.embedded.async.ParallelSmtAndConvertBatchProcessor.processRecords(ParallelSmtAndConvertBatchProcessor.java:54) at io.debezium.embedded.async.AsyncEmbeddedEngine$PollRecords.doCall(AsyncEmbeddedEngine.java:1337) at io.debezium.embedded.async.AsyncEmbeddedEngine$PollRecords.doCall(AsyncEmbeddedEngine.java:1311) at io.debezium.embedded.async.RetryingCallable.call(RetryingCallable.java:47) at java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) at java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.lang.Thread.run(Thread.java:1583) Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(profiles=[])), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): To use web identity tokens, the 'sts' service module must be on the class path., ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(profiles=[])): Profile file contained no credentials for profile 'default', ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): IMDS credentials have been disabled by environment variable or system property.] at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:102) at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:117) at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:45) at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:123) at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.resolveCredentials(AwsExecutionContextBuilder.java:184) at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:120) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:69) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:78) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:175) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76) at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56) at software.amazon.awssdk.services.kinesis.DefaultKinesisClient.putRecords(DefaultKinesisClient.java:1969) at io.debezium.server.kinesis.KinesisChangeConsumer.recordsSent(KinesisChangeConsumer.java:221) at io.debezium.server.kinesis.KinesisChangeConsumer.handleBatch(KinesisChangeConsumer.java:174) at io.debezium.server.DefaultChangeConsumer.handleBatch(DefaultChangeConsumer.java:58) at io.debezium.embedded.async.AbstractRecordProcessor$BatchProcessor$DirectProcessor.process(AbstractRecordProcessor.java:76) at io.debezium.embedded.async.ParallelSmtAndConvertBatchProcessor.processRecords(ParallelSmtAndConvertBatchProcessor.java:54) at io.debezium.embedded.async.AsyncEmbeddedEngine$PollRecords.doCall(AsyncEmbeddedEngine.java:1337) at io.debezium.embedded.async.AsyncEmbeddedEngine$PollRecords.doCall(AsyncEmbeddedEngine.java:1311) at io.debezium.embedded.async.RetryingCallable.call(RetryingCallable.java:47) at java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) at java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.lang.Thread.run(Thread.java:1583)How to reproduce the issue using our tutorial deployment?
quay.io/debezium/server:3.6.0.Finalwith the Amazon Kinesis sink on EKS.debezium.sink.kinesis.region(nocredentials.profile), so the AWS SDK default credential chain is used.eks.amazonaws.com/role-arn); do not provide static keys, and keep IMDS disabled.PutRecordsfails withWebIdentityTokenCredentialsProvider(): To use web identity tokens, the 'sts' service module must be on the class path.