Deploy to the Web fails at 'Upload PostSharp' (aws s3 cp connection drop) on every build since 2026-06-02

[gfraiteur]

Summary

The Deploy to the Web TeamCity build (PostSharpGitHub_PostSharp20260_DeployToTheWeb) fails at the "Upload PostSharp" step (4/5) on every run since 2026-06-02. The aws s3 cp upload of distribution files to s3://download-sharpcrafters-com/... aborts part-way through the batch with a connection-level error, failing the whole deploy. As a result the direct-download site is missing complete uploads for 2026.0.10, 2026.0.11 and 2026.0.12, and the "Refresh Web Download Manager" step (5/5) never runs.

Error

Each build dies on a different file with the same low-level error:

upload failed: <file> to s3://download-sharpcrafters-com/postsharp/postsharp-2026.0/<ver>/<file>
Connection was closed before we received a valid response from endpoint URL:
"https://download-sharpcrafters-com.s3.amazonaws.com/postsharp/postsharp-2026.0/<ver>/<file>"

Wrapped by MSBuild as error MSB3073 ... aws s3 cp ... exited with code 1 (target UploadWeb, Build/Distribution/Distribution.proj:527).

Affected builds

Build	Version	Files uploaded OK	Failed on
325096	2026.0.10	~36	PostSharp.Redist.*.nupkg
325788	2026.0.11	16	PostSharp.Patterns.Common.Redist.*.nupkg
326000	2026.0.12	7	PostSharp.Compiler.Engine.*.nupkg

Last green deploy: 324475 (2026-05-15). All runs (green and failed) used agent vanocka-2.

Analysis

• Not file/size specific — fails at a random point in the ~37-file sequential batch (the large engine package uploaded fine in build 325788).
• Not a code regression — the UploadWeb target is unchanged since March 2026; AWS CLI is pinned to 2.31.10 since Oct 2025 (the last green build ran the same version and target).
• Not an agent switch — same agent (vanocka-2) for both the last success and all failures.
• Most likely an intermittent network-level connection reset on the agent's path to *.s3.amazonaws.com, degraded from occasional (historically a manual retry went green) to near-constant since ~2026-06-02. Each file is a separate aws process/TLS connection, and one mid-batch drop fails the entire deploy because UploadWeb has no retry.

Suggested fixes

1. Resilience (in-repo): wrap each aws s3 cp in Build/Distribution/Distribution.proj (UploadWeb, line ~527) in a retry loop (3-5 attempts, backoff); set AWS_MAX_ATTEMPTS=10, AWS_RETRY_MODE=adaptive; consider lowering default.s3.max_concurrent_requests.
2. Environment: investigate what changed for agent vanocka-2 / its network path to AWS around 2026-06-02 (firewall/proxy/MTU/NAT, AWS-side throttling); test on a different agent.
3. Recover web releases: re-run/complete the uploads for 2026.0.10/.11/.12 and refresh the Web Download Manager.

Secondary (pre-existing)

The upload step echoes set AWS_ACCESS_KEY_ID=<key id> in clear text into the build log (key id exposed; secret value not). Consider masking / env-based credentials.