Skip to content

Conversation

talsperre
Copy link
Collaborator

Bug Description

Currently, runner/deployer always users the environment of the context in which it was launched as the base environment, and updates it with the environment provided by the user.

This causes weird issues like the following:

When using deployer/runner to deploy a flow from a step within another flow, it automatically sets environment=conda for the deployed flow even if we didn't set it explicitly.

Bug:

hello_deployer.py

from metaflow import FlowSpec, conda_base, step, Deployer

@conda_base(libraries={"pandas": ""})
class HelloDeployer(FlowSpec):
    @step
    def start(self):
        deployed_flow = Deployer('hello_flow.py', pylint=False).argo_workflows().create()
        triggered_run = deployed_flow.run()
        self.next(self.end)

    @step
    def end(self):
        print("HelloFlow is all done.")

if __name__ == "__main__":
    HelloDeployer()

hello_flow.py

from metaflow import FlowSpec, step

class HelloFlow(FlowSpec):
    @step
    def start(self):
        self.next(self.end)

    @step
    def end(self):
        print("HelloFlow is all done.")

if __name__ == "__main__":
    HelloFlow()

Fix:

Allow users to specify the appropriate behavior for their context via an argument: extend_existing_env.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants