S3 Compability Tests

A small project with bash scripts for testing S3 API compability:

• test_put_if_match.sh - tests If-Match
• test_put_if_not_match.sh - tests If-None-Match
• test_list_order.sh - tests global LIST order across pagination for WAL keys
• test_list_perf.sh - uploads many objects in parallel and measures 3 plain LIST calls speed
• test_put_perf.sh - uploads many small objects in parallel and measures average PUT time per file
• test_put_perf_large.sh - sequentially uploads 5x10 MiB and 5x100 MiB files and measures average upload speed

Works with both AWS S3 and S3-compatible providers (MinIO, Ceph RGW, etc.).

Requirements

• aws CLI v2
• S3 access (access key, secret key, region)

AWS CLI installation:

• Official install guide (all platforms): https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
• Linux: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html#cliv2-linux-install
• macOS: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html#cliv2-mac-install
• Windows: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html#cliv2-windows-install

Setup

1. Create .env from the example:

cp .env.example .env

2. Fill required fields in .env:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_DEFAULT_REGION

3. ENDPOINT_URL:

• keep empty for AWS S3
• set URL for S3-compatible service, for example http://127.0.0.1:9000

Run

Using make:

make test-if-match
make test-if-not-match
make test-list-order
make test-list-perf
make test-put-perf
make test-put-perf-large
make s3-delete-bucket

Or directly:

bash test_put_if_match.sh
bash test_put_if_not_match.sh
bash test_list_order.sh
bash test_list_perf.sh
bash test_put_perf.sh
bash test_put_perf_large.sh
bash ./remove_bucket.sh

Bucket Cleanup

make s3-delete-bucket or direct bash ./remove_bucket.sh deletes a bucket in 4 steps:

• removes current objects
• removes all object versions and delete markers
• aborts unfinished multipart uploads
• deletes the bucket itself

This is safer than plain aws s3 rb --force, because versioned buckets and unfinished multipart uploads often block bucket deletion.

S3 API Tests

Pagination Order Test (WAL)

test_list_order.sh creates WAL objects with lexicographic keys:

• wal/0000001 ... wal/0000201

Then it lists objects with page size 100 (list-objects-v2 --max-keys 100) and checks:

• pagination is real (page_count >= 2)
• first key on each next page is strictly greater than the previous page last key
• global last key from paginated traversal is exactly wal/0000201

This verifies that the last WAL key is found globally across pages, not only inside one page.

LIST Performance Test

test_list_perf.sh loads a large amount of objects (default: 10000) with parallel PUT (default workers: 16) and then runs 3 regular LIST requests (list-objects-v2 without pagination loop).

If you want to measure only LIST on existing data, set SKIP_PUT_STAGE=true.

For each LIST run it prints:

• listed objects count from the single response page
• elapsed time in seconds
• speed in objects/sec
• IsTruncated flag

This helps detect if LIST latency grows noticeably when bucket prefix contains many objects.

PUT Performance Tests

test_put_perf.sh uploads many small objects in parallel and prints average PUT time per file.

test_put_perf_large.sh runs two sequential series:

• 5 files by 10 MiB
• 5 files by 100 MiB

For each series it prints per-file progress, average time per file and average throughput in MiB/s. Payload files are generated once using sparse zero-filled files when supported, which is the fastest local preparation path.

Useful .env Variables

See in .env.example.

Key variables for list tests:

• CATALOG_PREFIX
• OBJECTS_COUNT
• OBJECT_KEY_WIDTH
• PUT_PARALLELISM
• LIST_RUNS
• SKIP_PUT_STAGE
• CREATE_BUCKET
• CLEANUP_BUCKET