[FEATURE] Add a great-expectations user-agent suffix to S3 clients

[goanpeca]

Why

Great Expectations already lets a datasource pass boto3_options (including endpoint_url and a custom botocore.config.Config) to the underlying boto3 S3 client, so it works with Amazon S3 and any S3-compatible object store. Two gaps today: the S3 client goes out with only the default boto3/botocore user agent, so operators and providers cannot see which requests originate from Great Expectations; and the docs document endpoint_url but do not make clear that setting it is how you connect to a non-AWS S3-compatible store.

This change adds a great-expectations/<version> suffix to the S3 client user agent and clarifies that docs option. It follows the common library/version convention that libraries use to attribute their S3 traffic, and helps with usage attribution, support, and debugging on Amazon S3 as well as on any S3-compatible object store (for example Backblaze B2, Cloudflare R2, or MinIO). The change is intentionally small and additive: it does not alter datasource configuration, credentials handling, or request behavior.

What

• Added get_s3_boto3_options(boto3_options) to great_expectations/compatibility/aws.py. It returns the caller's options with a user_agent_extra of great-expectations/<distribution version>. If the caller already supplied a Config with a user_agent_extra, the suffix is appended rather than replacing it (via Config.merge), so a user's own agent string and any endpoint_url are preserved.
• Falls back to a "dev" version when the installed distribution metadata is unavailable, and is a no-op that returns the options unchanged when botocore's Config is not importable, so environments without the AWS extras are unaffected.
• Routed the three existing S3 client construction sites through the helper: PandasS3Datasource, SparkS3Datasource, and PandasExecutionEngine._instantiate_s3_client. No call signatures or public APIs change.
• Docs: clarified the endpoint_url option on the S3 Data Source page (docs/docusaurus/docs/core/connect_to_data/filesystem_data/_create_a_data_source/_s3/_s3.md) to note that it connects Great Expectations to an S3-compatible object store (Amazon S3, or a compatible provider such as Backblaze B2, Cloudflare R2, or MinIO), and to leave it unset for Amazon S3.

Scope

This is a minimal, additive change: the AWS compatibility module, the three call sites that build a boto3 S3 client, and a one-line docs clarification. There is no change to configuration schema, type stubs, credentials handling, or the requests that Great Expectations issues. The framing is deliberately generic: Amazon S3 is named first and S3-compatible providers appear only as alphabetical examples; the user-agent suffix is appended to, never a replacement for, a user-provided agent string.

Test plan

Locally, against the unit marker:

• get_s3_boto3_options({}) returns options whose config.user_agent_extra equals great-expectations/<version>.
• When a Config(user_agent_extra="my-app/1.0") is passed in, the result is my-app/1.0 great-expectations/<version> (append, not overwrite), and an accompanying endpoint_url in the options is preserved.
• The datasource/engine call sites still build a client when boto3 and botocore are available, matching the existing mock_boto3_options unit fixtures in tests/datasource/fluent/test_pandas_s3_datasource.py.

I have not committed a new test in this branch yet. If the reviewers would like the behavior locked in, I am happy to add a small @pytest.mark.unit test for get_s3_boto3_options in tests/ (append vs set, and endpoint_url preservation) before merge.

---

• Description of PR changes above includes a link to an existing GitHub issue - no existing issue; this is a small, additive attribution and docs change. Happy to open a tracking issue or discuss in Slack first if the team prefers.
• PR title is prefixed with one of: [BUGFIX], [FEATURE], [DOCS], [MAINTENANCE], [CONTRIB], [MINORBUMP]
• Code is linted - run invoke lint (uses ruff format + ruff check)
• Appropriate tests and docs have been updated - docs updated (the S3 Data Source endpoint_url clarification); no unit test committed yet, see the Test plan above. I can add a @pytest.mark.unit test before merge if desired.

[netlify]

👷 Deploy request for niobium-lead-7998 pending review.

Visit the deploys page to approve it

Name	Link
🔨 Latest commit	50a7c36

[Copilot]

Pull request overview

This PR adds a Great Expectations–specific great-expectations/<version> suffix to the boto3 S3 client user agent to make S3 traffic attributable (and easier to support/debug) across AWS S3 and S3-compatible object stores, while preserving any caller-provided boto3 options and agent strings.

Changes:

• Added get_s3_boto3_options(...) in great_expectations/compatibility/aws.py to inject/append user_agent_extra using botocore Config.
• Routed S3 client construction in the Pandas and Spark S3 datasources through the new helper.
• Routed PandasExecutionEngine S3 client instantiation through the new helper.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
great_expectations/compatibility/aws.py	Adds helper that appends great-expectations/<version> to botocore Config.user_agent_extra for S3 clients.
great_expectations/execution_engine/pandas_execution_engine.py	Uses the helper when instantiating the execution engine’s internal S3 client.
great_expectations/datasource/fluent/pandas_s3_datasource.py	Uses the helper when constructing the datasource S3 client from boto3_options.
great_expectations/datasource/fluent/spark_s3_datasource.py	Uses the helper when constructing the datasource S3 client from boto3_options.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.