[Feature]: end to end tests during UAT functional validation run on real GPU cluster in GCP

[lalitadithya]

Prerequisites

• I searched existing issues

Feature Summary

Currently the release is being manually tested by the maintainers before a tag is cut. The is very time consuming as the cluster takes close to an hour to bring up, capacity might not always be available and in general we can't keep the cluster running. We need a way to bring up clusters on a cron schedule and run our UAT tests

Problem/Use Case

Continuously test main on a real cluster

Proposed Solution

TBD

Component

Deployment/Config

[mchmarny]

Closed with #355