Skip to content

[WIP] Co-execution client for Google Cloud Platform Batch v1 #404

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

jmchilton
Copy link
Member

@jmchilton jmchilton commented Jul 11, 2025

Implements #390.

Co-execution Diagram

From https://pulsar.readthedocs.io/en/latest/containers.html#galaxy-and-shared-file-systems about the advantages of a Pulsar co-execution client (a container native paradigm) over implementing a Galaxy job runner (an interface designed for shared file systems and DRMAA-style batch of jobs, not containers):

The most glaring disadvantage of not using Pulsar in the above scenarios is that Galaxy must be deployed in the same container with the same mounts as the job execution environment. This prevents leveraging external cloud compute, multi-cloud compute, and makes it unsuitable for common Galaxy use cases such as large public instances, Galaxy’s leveraging institution non-cloud storage, etc… Even within the same cloud - a large shared file system can be an expensive prospect and Pulsar may allow making use of buckets and such more tractable.

@jmchilton jmchilton changed the title [WIP] Co-execution client Google Cloud Platform Batch v1 [WIP] Co-execution client for Google Cloud Platform Batch v1 Jul 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant