GH-47349: [C++] Include request ID in AWS S3 Error

[wingkitlee0]

Rationale for this change

It is a uuid useful for debugging with AWS support team.

Indeed, minio errors will print out a request ID by default.

What changes are included in this PR?

The request ID is appended to the end of the error message.

Are these changes tested?

No new test added yet.

I will try to test it on a real S3 system. Minio seems to return empty string only.

Are there any user-facing changes?

No

• GitHub Issue: [C++][Python] Add access to Request ID in AWS client error #47349

[github-actions]

⚠️ GitHub issue #47349 has been automatically assigned in GitHub to PR creator.

[raulcd]

The only E2E test I can seem to find that reuses this error message is:

arrow/python/pyarrow/tests/test_fs.py

Lines 1431 to 1452 in 71a7b55

	@pytest.mark.s3
	def test_s3fs_wrong_region():
	from pyarrow.fs import S3FileSystem
	# wrong region for bucket
	# anonymous=True incase CI/etc has invalid credentials
	fs = S3FileSystem(region='eu-north-1', anonymous=True)
	msg = ("When getting information for bucket 'voltrondata-labs-datasets': "
	r"AWS Error UNKNOWN \(HTTP status 301\) during HeadBucket "
	"operation: No response body. Looks like the configured region is "
	"'eu-north-1' while the bucket is located in 'us-east-2'."
	"|NETWORK_CONNECTION")
	with pytest.raises(OSError, match=msg) as exc:
	fs.get_file_info("voltrondata-labs-datasets")
	# Sometimes fails on unrelated network error, so next call would also fail.
	if 'NETWORK_CONNECTION' in str(exc.value):
	return
	fs = S3FileSystem(region='us-east-2', anonymous=True)
	fs.get_file_info("voltrondata-labs-datasets")

Maybe you can reuse it? I haven't tested so I am unsure

[wingkitlee0]

@raulcd Thanks for looking at it! It turns out get_file_info has a little bit different extra error handling.

For other APIs, I was able to use minio locally get the expected message:

In [6]: fs = S3FileSystem(endpoint_override="localhost:9000", scheme="http", region="us-east-1", access_key="minioadmin", secret_key="minioadmin")

In [7]: fs.copy_file("test-bucket/a", "test-bucket/b")
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
Cell In[7], line 1
----> 1 fs.copy_file("test-bucket/a", "test-bucket/b")

File ~/.pyenv/versions/3.12.10/envs/py312-arrow-dev/lib/python3.12/site-packages/pyarrow/_fs.pyx:754, in pyarrow._fs.FileSystem.copy_file()

File ~/.pyenv/versions/3.12.10/envs/py312-arrow-dev/lib/python3.12/site-packages/pyarrow/error.pxi:92, in pyarrow.lib.check_status()

OSError: When copying key 'a' in bucket 'test-bucket' to key 'b' in bucket 'test-bucket': AWS Error NO_SUCH_BUCKET during CopyObject operation: The specified bucket does not exist request ID: 185CFDB6B0C338B1

So I should be able to add a test for this..

[raulcd]

We use minio for several tests, see this fixture:

arrow/python/pyarrow/tests/conftest.py

Lines 210 to 245 in 69d2487

	def s3_server(s3_connection, tmpdir_factory):
	@retry(attempts=5, delay=1, backoff=2)
	def minio_server_health_check(address):
	resp = urllib.request.urlopen(f"http://{address}/minio/health/live")
	assert resp.getcode() == 200
	tmpdir = tmpdir_factory.getbasetemp()
	host, port, access_key, secret_key = s3_connection
	address = f'{host}:{port}'
	env = os.environ.copy()
	env.update({
	'MINIO_ACCESS_KEY': access_key,
	'MINIO_SECRET_KEY': secret_key
	})
	args = ['minio', '--compat', 'server', '--quiet', '--address',
	address, tmpdir]
	proc = None
	try:
	proc = subprocess.Popen(args, env=env)
	except OSError:
	pytest.skip('`minio` command cannot be located')
	else:
	# Wait for the server to startup before yielding
	minio_server_health_check(address)
	yield {
	'connection': s3_connection,
	'process': proc,
	'tempdir': tmpdir
	}
	finally:
	if proc is not None:
	proc.kill()
	proc.wait()

Could we replicate your local validation with a test?

[wingkitlee0]

Added some tests.