Intermittent stall and failure

[Quantumplation]

Hello,

When requesting many dynamodb items in parallel (in my case, 50) through the DAX client, the DAX client seemingly deadlocks and ultimately fails with the following error:

UnknownError: unknown error
caused by: file already closed

I did some initial investigation, and it appears this error is thrown here:

aws-dax-go/dax/internal/client/tubepool.go

Line 123 in cb4d531

return nil, os.ErrClosed

My hypothesis is that there is a deadlock, the tubepool gets closed because of some kind of idle check, and then the deadlock times out and fails on this error; but I'm not familiar enough with the code base to investigate further.

[Quantumplation]

This behavior appears to disappear somewhere between 8 and 16 requests, but that might just be because it comes rarer and rarer.

[stefan-cretu-endava]

@Quantumplation could you share the config details of your DAX client and of your cluster? I Used 25 requests and a 3-node cluster, r5.large and couldn't reproduce the issue. My dax config is:

daxCfg.LogLevel = utils.LogDebugWithRequestRetries
	daxCfg.Config.SkipHostnameVerification = true
	daxCfg.Config.RouteManagerEnabled = true
	daxCfg.Config.ClientHealthCheckInterval = time.Second * 5
	daxCfg.Config.ClusterUpdateInterval = time.Second * 5
	daxCfg.Config.DialContext = (&net.Dialer{
		Timeout:   time.Millisecond * time.Duration(5000),
		KeepAlive: time.Minute,
	}).DialContext

[stefan-cretu-endava]

@Quantumplation The issue occurs especially when the number of concurrent requests (goroutines executing concurrent DB access through DAX) increases. For clusters having 2-3 nodes, it was noticeable when the number of goroutines was around 40-50, even greater.

It's default values is 10: https://github.com/aws/aws-dax-go-v2/blob/bafd9a115399ddc4c83db37e05594d91b0735b17/dax/internal/client/cluster.go#L125

The recommendation is to set it to the number of concurrent accesses that is targeted for the application. For example: cfg.Config.MaxPendingConnectionsPerHost = 20

Additionally, when the goroutines are executed, their concurrency can be limited by using a semaphore. The recommendation is to have identical values for semaphore count and MaxPendingConnectionsPerHost . Example:

wg := sync.WaitGroup{}
sem := make(chan struct{}, 20)

for idx :=range 50 {
    wg.Add(1)
    go func() {
        defer wg.Done()
        //increase sem count: acquire
        sem <- struct{}{}
        daxClient.GetItem(context, item)
        //decrease sem count: release
        <-sem
    }
}

wg.Wait()

[Quantumplation]

Oh gosh, this was over a year ago, I'll have to dig into and see if I can find what my config was.

Regardless of the concurrency settings, ideally the client wouldn't deadlock 😅

[stefan-cretu-endava]

I wouldn't say it is a deadlock.

When the connection pool is too small, requests may queue up or fail, causing the SDK to retry. Excessive retries can lead the SDK to temporarily mark a node as unhealthy, triggering health check failures, consequently closing the tubepool. A larger pool allows better distribution of requests across nodes. This can prevent overloading a single node, which might otherwise fail health checks due to high latency or throttling.

Test scenario:
I tested with a cluster configured to 3 DAX nodes. My DAX client used MaxPendingConnectionsPerHost = 10. Therefore, a total of 30 possible concurrent pending connections was possible If one node fails health checks, then only 20 connections remain usable.

Then the app spawns 50 goroutines. When only 20 connections are usable, maximum 30 goroutines will be blocked, waiting for a connection, leading to increased latency, timeouts, or throttling. If retries are enabled, this can further stress the remaining healthy nodes.

Due to connection pool saturation, if the remaining nodes are already at their MaxPendingConnectionsPerHost limit, new requests will be blocked until a connection becomes available. This can cause goroutines to stall or timeout, even though other nodes are technically healthy. In addition, the SDK may take a short time to detect and mark a node as unhealthy.
During this window, requests may still be routed to the failing node and fail or retry, causing delays.

runtime/debug.Stack()
/home/ubuntu/.gvm/pkgsets/go1.24.0/global/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-amd64/src/runtime/debug/stack.go:26 +0x5e
github.com/aws/aws-dax-go-v2/dax/internal/client.(*tubePool).Close(0xc00020a510)
aws-dax-go-v2/dax/internal/client/tubepool.go:285 +0x65
github.com/aws/aws-dax-go-v2/dax/internal/client.(*SingleDaxClient).Close(0xc0001acae0)
aws-dax-go-v2/dax/internal/client/single.go:162 +0x70
github.com/aws/aws-dax-go-v2/dax/internal/client.(*cluster).closeClient(...)
aws-dax-go-v2/dax/internal/client/cluster.go:847
github.com/aws/aws-dax-go-v2/dax/internal/client.(*cluster).onHealthCheckFailed(0xc000206160, {{0xc000250bb0, 0xc}, 0x1faf})
aws-dax-go-v2/dax/internal/client/cluster.go:682 +0x304
github.com/aws/aws-dax-go-v2/dax/internal/client.(*SingleDaxClient).startHealthChecks.func1()
aws-dax-go-v2/dax/internal/client/single.go:179 +0x16a
github.com/aws/aws-dax-go-v2/dax/internal/client.(*taskExecutor).start.func1()
aws-dax-go-v2/dax/internal/client/cluster.go:914 +0x39
created by github.com/aws/aws-dax-go-v2/dax/internal/client.(*taskExecutor).start in goroutine 1
aws-dax-go-v2/dax/internal/client/cluster.go:910 +0x99