Update submitter for pre/post tasks by Alxiice · Pull Request #8 · meshroomHub/mrSubmitters

Alxiice · 2026-04-27T07:45:49Z

Description

This PR aims to update the submitter so that it works with the pre/post task feature in Meshroom.
It requires this PR in order to work.
Make sure to follow these steps :

merge [tractor] Override submit method from BaseSubmitter #4
merge Meshroom!2984
merge the current PR

Content of the PR

Changes :

All the task chain logic is moved in Meshroom.
New structures are used to store a concept of "OrderedTask" in Meshroom.
Meshroom handles the "cooking" part that was here previously : it will build a Graph-like object that holds all the tasks & dependencies. All the logic concerning expanding/chunk/preprocess/postprocess/placeholder tasks already exist in this new OrderedTasks object.
Now the submitter "simply" has to take this object and straightforwardly create tasks one by one, then add dependencies, then submit. No complex cooking/reordering task is involved here anymore

Additions :

The concept of "task type" has been fully implemented.
We also added the handling of preprocess/postprocess tasks.
Also there is the concept of "placeholder" tasks : tasks that don't have any command but are here to create nodes in the graph
The tractorSubtaskWrapper script has been renamed to tractorExpander to make it more general-purpose.

Removal :

For now we removed the chunk grouping : every chunk will have it's own task. Grouping chunks on multiple tasks have no real purpose here as we already have a size concept in the chunks. That was also a feature that we didn't use so there was no reason to keep it.

…s after adding pre/post tasks on Meshroom

Copilot

Pull request overview

This PR updates the Tractor submitter integration to work with Meshroom’s new pre/post-task “OrderedTasks” submission model, shifting graph “cooking” to Meshroom and simplifying the submitter to straightforward task creation + dependency wiring.

Changes:

Renames the Tractor stdout-wrapper script from tractorSubtaskWrapper to the more general tractorExpander and updates related references/env vars.
Refactors Tractor submission to build tasks from Meshroom OrderedTasks and introduces task-type handling (expanding/chunk/preprocess/postprocess/placeholder).
Removes the old tractorJobCreation.py implementation and adapts TaskInfo/chunk queuing to the new task-type approach.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
`test/createJob.py`	Updates test job creation to call the renamed `tractorExpander`.
`script/tractorExpander.py`	Renamed/generalized wrapper that reserves stdout for Tractor expansion commands and forwards normal output to stderr.
`meshroom/tractorSubmitter/tractorSubmitter.py`	Major refactor: constructs Tractor tasks from Meshroom `OrderedTasks`, adds task-type mapping, and introduces new in-file `Job`/`Task` wrappers.
`meshroom/tractorSubmitter/api/tractorJobCreation.py`	Removed legacy job/task graph cooking implementation (now handled upstream by Meshroom).
`meshroom/tractorSubmitter/api/subtaskCreator.py`	Updates expander integration + rewrites chunk subtask emission; introduces a currently broken `getChunks` helper.
`meshroom/tractorSubmitter/api/base.py`	Reworks `TaskInfo` to use `reqPackages` + task-type flags (expanding/pre/post/chunk/placeholder) and updates command cooking accordingly.

Comments suppressed due to low confidence (1)

meshroom/tractorSubmitter/tractorSubmitter.py:121

serialsubtasks is computed from rootTasks, but when serialsubtasks is True, no root task is created and rootTasks is otherwise unused. If you intend a consistent graph shape, it may be simpler/safer to always create a single job root task and attach only true roots beneath it.

        rootTasks = self.getRootTasks()
        serialsubtasks = len(rootTasks) == 1
        if not serialsubtasks:
            rootTaskName = self.jobInfo.name + " (root)"
            rootTask = tractorJob.newTask(title=rootTaskName, argv=None, serialsubtasks=serialsubtasks)
            for task in rootTasks:
                rootTask.addChild(task.tractorTask)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+def getChunks(chunkParams):
+    it = None
+    ignoreIterations = chunkParams.get("ignoreIterations", [])
+    if chunkParams:
+        start, end = chunkParams.get("start", -1), chunkParams.get("end", -2)
+        size = 1
+        frameRange = list(range(start, end+1, 1))
+        if frameRange:
+            it = [
+                Chunk(i, )
+            ]
+            slices = [frameRange[i : i+1] for i in range(0, len(frameRange))]
+            it = [Chunk(i, item[0], item[-1]) for i, item in enumerate(slices)
+                    if i not in ignoreIterations]
+    return it


+        self.taskTependencies = {}  # task: [tasks that the task depends on]
+
+    def addTask(self, task: Task):
+        self.tasks.append(task)
+        self.taskTependencies[task] = []
+
+    def addTaskDependency(self, parentTask: Task, childTask: Task):
+        parentTask.tractorTask.addChild(childTask.tractorTask)
+        self.taskTependencies[parentTask].append(childTask)
+
+    def getRootTasks(self):
+        """ Get all tasks that are not children of other tasks """
+        tasksWithoutDeps = set(self.tasks)
+        for _, childTasks in self.taskTependencies.items():


+        # Create the job task (no command, at the graph root)
+        rootTasks = self.getRootTasks()
+        serialsubtasks = len(rootTasks) == 1
+        if not serialsubtasks:
+            rootTaskName = self.jobInfo.name + " (root)"
+            rootTask = tractorJob.newTask(title=rootTaskName, argv=None, serialsubtasks=serialsubtasks)
+            for task in rootTasks:
+                rootTask.addChild(task.tractorTask)
+        # Cook tasks
+        taskToTractorTask = {}
+        for task in self.tasks:
+            tractorTask = task.tractorTask
+            tractorJob.addChild(tractorTask)
+            taskToTractorTask[task] = tractorTask


+        elif orderedTask.taskType == OrderedTaskType.PLACEHOLDER:
+            taskType = (None, None)


+    def createJob(self, orderedTasks: OrderedTasks, filepath, submitLabel="{projectName}") -> TractorJob:
+        # Create job
        projectName = os.path.splitext(os.path.basename(filepath))[0]
        name = submitLabel.format(projectName=projectName)
        comment = filepath
-        maxNodeSize = max([node.size for node in nodes])
        mainTags = {


        res = job.submit(share=self.share, dryRun=self.dryRun)
        if self.dryRun:
            return True


cbentejac

Minor comments, mostly for code readability. Functionally speaking, this looks good to me.

cbentejac · 2026-05-13T15:49:37Z

+            # if taskInfos.expandingTask:
+            #     cmd.expand = taskInfos.expandingFile


Suggested change

# if taskInfos.expandingTask:

# cmd.expand = taskInfos.expandingFile

cbentejac · 2026-05-13T15:51:29Z

+
+    @staticmethod
+    def createDummyTask(tractorJob: tractorAuthor.Job):
+        """ tractor API will raise a RequiredValueError if no task are 


Suggested change

""" tractor API will raise a RequiredValueError if no task are

""" Tractor API will raise a RequiredValueError if no task is

cbentejac · 2026-05-13T15:52:07Z

+        tractorJob.priority = PRIORITY_DICT.get(priority, PRIORITY_DICT["normal"])
+
+        if dryRun:
+            logging.info("TractorSubmitter: Job in TCL format :")


Suggested change

logging.info("TractorSubmitter: Job in TCL format :")

logging.info("TractorSubmitter: Job in TCL format:")

cbentejac · 2026-05-13T15:55:21Z

+        self.placeholderTask = (taskType_=="placeholder")
+        self.expandingTask = (taskType_=="expanding")
+        self.preprocessTask = (taskType_=="preprocess")
+        self.postprocessTask = (taskType_=="postprocess")
+        self.chunkTask = (taskType_=="chunk")


Nitpicking comment, for readability purposes:

Suggested change

self.placeholderTask = (taskType_=="placeholder")

self.expandingTask = (taskType_=="expanding")

self.preprocessTask = (taskType_=="preprocess")

self.postprocessTask = (taskType_=="postprocess")

self.chunkTask = (taskType_=="chunk")

self.placeholderTask = (taskType_ == "placeholder")

self.expandingTask = (taskType_ == "expanding")

self.preprocessTask = (taskType_ == "preprocess")

self.postprocessTask = (taskType_ == "postprocess")

self.chunkTask = (taskType_ == "chunk")

cbentejac · 2026-05-13T16:02:10Z

-                 service=None, licenses=None, tags=None,
-                 expandingTask=False, chunkParams=None):
+                 environment=None, reqPackages=None, service=None, 
+                 licenses=None, taskType=None, tags=None):


It'd be useful to mention somewhere that taskType is either None or a tuple, either here or at line 224 (which very quickly made me question my understanding of what was happening here 👼 ).

cbentejac · 2026-05-13T16:09:24Z

 currentDir = os.path.dirname(os.path.realpath(__file__))
 binDir = os.path.dirname(os.path.dirname(os.path.dirname(currentDir)))

+CreatedTask = namedtuple("task", ["task", "chunkParams"])


Suggested change

CreatedTask = namedtuple("task", ["task", "chunkParams"])

This is not used anywhere else in the code, it can be removed (and so can the from collections import namedtuple statement).

Alxiice added 3 commits April 21, 2026 11:39

Temporary correction to make sure we don't break the submitting poces…

1461077

…s after adding pre/post tasks on Meshroom

Rename tractorSubtaskWrapper to tractorExpander

f316d2f

Finalize the new tractor submitter

c444110

Alxiice requested a review from cbentejac April 27, 2026 07:45

Alxiice assigned fabiencastan and servantftransperfect Apr 27, 2026

Alxiice requested review from fabiencastan and servantftransperfect April 27, 2026 07:46

Alxiice assigned Alxiice and unassigned fabiencastan and servantftransperfect Apr 27, 2026

Alxiice added 2 commits May 6, 2026 10:27

Remove the duplicate root task + better task naming

94322ee

Get _MESHROOM_ROOT from meshroom.__init__

8512d3f

cbentejac requested a review from Copilot May 13, 2026 16:04

Copilot started reviewing on behalf of cbentejac May 13, 2026 16:05 View session

Copilot AI reviewed May 13, 2026

View reviewed changes

cbentejac requested changes May 13, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update submitter for pre/post tasks#8

Update submitter for pre/post tasks#8
Alxiice wants to merge 5 commits into
masterfrom
feature/update_submitter_with_new_api

Alxiice commented Apr 27, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

cbentejac left a comment

Uh oh!

cbentejac May 13, 2026

Uh oh!

cbentejac May 13, 2026

Uh oh!

cbentejac May 13, 2026

Uh oh!

cbentejac May 13, 2026

Uh oh!

cbentejac May 13, 2026

Uh oh!

cbentejac May 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

		elif orderedTask.taskType == OrderedTaskType.PLACEHOLDER:
		taskType = (None, None)

		# if taskInfos.expandingTask:
		# cmd.expand = taskInfos.expandingFile

	""" tractor API will raise a RequiredValueError if no task are
	""" Tractor API will raise a RequiredValueError if no task is

	logging.info("TractorSubmitter: Job in TCL format :")
	logging.info("TractorSubmitter: Job in TCL format:")

Conversation

Alxiice commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Content of the PR

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

cbentejac left a comment

Choose a reason for hiding this comment

Uh oh!

cbentejac May 13, 2026

Choose a reason for hiding this comment

Uh oh!

cbentejac May 13, 2026

Choose a reason for hiding this comment

Uh oh!

cbentejac May 13, 2026

Choose a reason for hiding this comment

Uh oh!

cbentejac May 13, 2026

Choose a reason for hiding this comment

Uh oh!

cbentejac May 13, 2026

Choose a reason for hiding this comment

Uh oh!

cbentejac May 13, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Alxiice commented Apr 27, 2026 •

edited

Loading