The PostRelease Nightly Snapshot job is flaky

[github-actions]

The PostRelease Nightly Snapshot is failing over 50% of the time
Please visit https://github.com/apache/beam/actions/workflows/beam_PostRelease_NightlySnapshot.yml?query=is%3Afailure+branch%3Amaster to see the logs.

[shunping]

Related to ##30447

[Abacn]

Still failing:

Container image gcr.io/cloud-dataflow/v1beta3/beam_java8_sdk:beam-master-20240306 not downloaded yet.

It is strange that the container gets resolved to "beam_java8_sdk:beam-master-20240306". What happens is it picks the label for legacy runner but actually trying to pull runner v2 image. This is likely due to Dataflow switched to runner v2 by default in Beam 2.55.0+

beam/runners/google-cloud-dataflow-java/build.gradle

Lines 54 to 55 in ef919e2

	ext.dataflowLegacyContainerVersion = 'beam-master-20240306'
	ext.dataflowFnapiContainerVersion = 'beam-master-20240125'

entered #30634

[liferoad]

https://github.com/apache/beam/actions/runs/8619063045

java.lang.RuntimeException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
POST https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DataflowRunner_team/insertAll?prettyPrint=false
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_team",
    "reason" : "notFound"
  } ],
  "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_team",
  "status" : "NOT_FOUND"
}

[liferoad]

Looks much better. Close this now.