SIGSEGV with S3CrtClient during CopyObjectRequest on large files

[lybeck]

Describe the bug

S3CrtClient crashes with SIGSEGV when running a CopyObjectRequest on large files, either within the bucket or between two different buckets. The object is copied succesfully to the target destination.

The copy object request works as expected with relatively small files, but fails on large files. For example, the request crashes the application with part size 64MB and a file size of about 600MB. The exact threshold is not known at this time.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

The copy object request doesn't cause a crash.

Current Behavior

When running a copy object request with a large enough file the S3-CRT client crashes with the following error (built with AWS SDK 1.11.706 and ASan (address sanitizer), see reproduction steps for more information):

[entrypoint] Building s3crt_copy_test
[entrypoint] Running test with:
  AWS_REGION=eu-central-1
  SRC_BUCKET=SOURCE_BUCKET
  SRC_KEY=PATH/TO/SOURCE_FILE
  DEST_BUCKET=DEST_BUCKET
  DEST_KEY=PATH/TO/DEST_FILE
Initializing AWS SDK
Source: SOURCE_BUCKET/PATH/TO/SOURCE_FILE
Dest  : DEST_BUCKET/PATH/TO/DEST_FILE
Creating S3CrtClient (partSize = 67108864 bytes)
Building CopyObjectRequest
Starting CopyObject from 'SOURCE_BUCKET/PATH/TO/SOURCE_FILE' to 'DEST_BUCKET/PATH/TO/DEST_FILE'
AddressSanitizer:DEADLYSIGNAL
=================================================================
==12==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000018 (pc 0x7744a271e490 bp 0x000000000000 sp 0x774495ac58a8 T12)
==12==The signal is caused by a READ memory access.
==12==Hint: address points to the zero page.
    #0 0x7744a271e490 in aws_http_headers_count (/opt/aws/lib64/libaws-c-http.so.1.0.0+0x3d490) (BuildId: fb4f4764a839d3efe26c6c2103d94dec900edec9)
    #1 0x7744a2fd6b3b in Aws::S3Crt::S3CrtClient::S3CrtRequestHeadersCallback(aws_s3_meta_request*, aws_http_headers const*, int, void*) (/opt/aws/lib64/libaws-cpp-sdk-s3-crt.so+0xedb3b) (BuildId: 139b6a30681c865690dafc483525762c1c94dc96)
    #2 0x7744a27a4633 in s_s3_copy_object_request_finished (/opt/aws/lib64/libaws-c-s3.so.0unstable+0x1d633) (BuildId: 50577b63226e4a0c0a015bf20673e80494866fac)
    #3 0x7744a27a2540 in aws_s3_client_notify_connection_finished (/opt/aws/lib64/libaws-c-s3.so.0unstable+0x1b540) (BuildId: 50577b63226e4a0c0a015bf20673e80494866fac)
    #4 0x7744a26f856e in s_stream_complete (/opt/aws/lib64/libaws-c-http.so.1.0.0+0x1756e) (BuildId: fb4f4764a839d3efe26c6c2103d94dec900edec9)
    #5 0x7744a26fa31e in s_handler_shutdown (/opt/aws/lib64/libaws-c-http.so.1.0.0+0x1931e) (BuildId: fb4f4764a839d3efe26c6c2103d94dec900edec9)
    #6 0x7744a1f99d32 in s_run_all (/opt/aws/lib64/libaws-c-common.so.1+0x35d32) (BuildId: 5ab2de0279517572617b88ce226534ab2258104c)
    #7 0x7744a26ace53 in aws_event_loop_thread (/opt/aws/lib64/libaws-c-io.so.1.0.0+0x1ee53) (BuildId: a81a8a2fa914f8dca553e1d2f935b66105ccfe74)
    #8 0x7744a1f97958 in thread_fn (/opt/aws/lib64/libaws-c-common.so.1+0x33958) (BuildId: 5ab2de0279517572617b88ce226534ab2258104c)
    #9 0x7744a33cc259 in asan_thread_start(void*) (/lib64/libasan.so.8+0x5e259) (BuildId: ad7620e2d73aec35a93d534569dfc5af69229262)
    #10 0x7744a2a40127 in start_thread (/lib64/libc.so.6+0x95127) (BuildId: a0fecbb478ee32896bfb8b799a0e09a376f95939)
    #11 0x7744a2ab0923 in clone (/lib64/libc.so.6+0x105923) (BuildId: a0fecbb478ee32896bfb8b799a0e09a376f95939)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/opt/aws/lib64/libaws-c-http.so.1.0.0+0x3d490) (BuildId: fb4f4764a839d3efe26c6c2103d94dec900edec9) in aws_http_headers_count
Thread T12 created by T0 here:
    #0 0x7744a34620a3 in pthread_create (/lib64/libasan.so.8+0xf40a3) (BuildId: ad7620e2d73aec35a93d534569dfc5af69229262)
    #1 0x7744a1f975ce in aws_thread_launch (/opt/aws/lib64/libaws-c-common.so.1+0x335ce) (BuildId: 5ab2de0279517572617b88ce226534ab2258104c)
    #2 0x7744a26ac2a7 in s_run (/opt/aws/lib64/libaws-c-io.so.1.0.0+0x1e2a7) (BuildId: a81a8a2fa914f8dca553e1d2f935b66105ccfe74)

==12==ABORTING

Reproduction Steps

The following C++ file reproduces the issue. The C++ code, a docker file, and an entrypoint shell script are also attached below.

#include <aws/core/Aws.h>
#include <aws/core/utils/logging/DefaultLogSystem.h>
#include <aws/core/utils/logging/LogLevel.h>
#include <aws/s3-crt/S3CrtClient.h>
#include <aws/s3-crt/model/CopyObjectRequest.h>

#include <iostream>
#include <memory>

class AwsApiWrapper {
public:
    AwsApiWrapper() {
        // _options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Trace;
        // _options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Debug;
        _options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Info;
        _options.loggingOptions.defaultLogPrefix = "aws_sdk_";

        std::cout << "Initializing AWS SDK" << std::endl;
        Aws::InitAPI(_options);
        _initialized = true;
    }

    ~AwsApiWrapper() {
        if (_initialized) {
            std::cout << "Shutting down AWS SDK" << std::endl;
            Aws::ShutdownAPI(_options);
        }
    }

    AwsApiWrapper(const AwsApiWrapper&) = delete;
    AwsApiWrapper& operator=(const AwsApiWrapper&) = delete;

private:
    Aws::SDKOptions _options;
    bool _initialized = false;
};

int main(int argc, char** argv) {
    if (argc != 5) {
        std::cerr << "Usage: " << argv[0] << " <source-bucket> <source-key> <dest-bucket> <dest-key>" << std::endl;
        return 1;
    }

    const Aws::String sourceBucket = argv[1];
    const Aws::String sourceKey = argv[2];
    const Aws::String destBucket = argv[3];
    const Aws::String destKey = argv[4];
    
    AwsApiWrapper awsApiWrapper;

    std::cout << "Source: " << sourceBucket << "/" << sourceKey << "" << std::endl;
    std::cout << "Dest  : " << destBucket << "/" << destKey << "" << std::endl;

    Aws::Client::ClientConfiguration config;
    Aws::S3Crt::ClientConfiguration s3CrtConfig(config);

    // Set 64 MiB part size
    static const uint64_t PART_SIZE_64_MB = 64ULL * 1024ULL * 1024ULL;
    s3CrtConfig.throughputTargetGbps = 10.0;
    s3CrtConfig.partSize = PART_SIZE_64_MB;

    std::cout << "Creating S3CrtClient (partSize = " << s3CrtConfig.partSize << " bytes)" << std::endl;

    std::shared_ptr<Aws::S3Crt::S3CrtClient> s3Client =
            Aws::MakeShared<Aws::S3Crt::S3CrtClient>("S3CrtClient", s3CrtConfig);

    if (!s3Client) {
        std::cerr << "Failed to create S3CrtClient" << std::endl;
        return 1;
    }

    std::cout << "Building CopyObjectRequest" << std::endl;

    Aws::S3Crt::Model::CopyObjectRequest request;
    request.SetBucket(destBucket);
    request.SetKey(destKey);

    Aws::String sourcePath = sourceBucket + "/" + sourceKey;
    Aws::String destPath = destBucket + "/" + destKey;

    request.SetCopySource(sourcePath);

    std::cout << "Starting CopyObject from '" << sourcePath << "' to '" << destPath << "'" << std::endl;

    Aws::S3Crt::Model::CopyObjectOutcome outcome = s3Client->CopyObject(request);

    std::cout << "CopyObject returned" << std::endl;

    if (!outcome.IsSuccess()) {
        const auto& err = outcome.GetError();
        std::cerr << "CopyObject failed:" << std::endl
                  << "  ExceptionName: " << err.GetExceptionName() << std::endl
                  << "  Message      : " << err.GetMessage() << std::endl;
        return 2;
    }

    std::cout << "CopyObject succeeded" << std::endl;
    return 0;
}

entrypoint.sh
s3_crt_copy_object_test.cpp
Dockerfile.txt

Possible Solution

No response

Additional Information/Context

No response

AWS CPP SDK version used

1.11.706

Compiler and Version used

g++ (GCC) 14.3.1 20250617 (Red Hat 14.3.1-2)

Operating System and version

Rocky Linux 10.1 (Red Quartz)

[sbiscigl]

dug into this a little bit, and while it is a segfault the original request will still fail.

In our callback we expect the header that we receive to be non-null headers is null, and response body is empty. If you look at the error logs you will see

[ERROR] 2025-12-10 15:40:45.107 S3MetaRequest [0x16df4f000] id=0x61a000002480 Request failed from error 2073 (Http connection channel shut down due to failure to meet throughput minimum). (request=0x61300000e200, response status=0). Try to setup a retry.

and debugging through the stack i can see that we encounter this error then call the headers callback. so that really boils down to the question

Should CRT ever pass us null headers, or is that a invariant?

If it is a invariant, they likely shouldn't call the headers callback if they want us to handle null headers we can just check for that and return a error. however either way your request will fail because it hits this AWS_ERROR_HTTP_CHANNEL_THROUGHPUT_FAILURE which means it has transferred 0 bytes for 3 seconds. this is because the transfer does actually hit that because of size considerations. setting the configuration to

s3CrtConfig.lowSpeedLimit = 0;

will allow the request to complete as expected.

So to summarize

to fix the segfault either:

• we need to handle the null header
• crt should not call the headers callback when a transfer error has occured

to fix the acutual failure in the transfer you need will need to add

s3CrtConfig.lowSpeedLimit = 0;

once the segfault fix is pushed, will talk to CRT about what to do with the segfault.