Layer needs main thread, how do I make dependent layers wait for it?

[Dekker3D]

Title basically says it all. My terrain layer can't actually add the generated terrain to the scene from another thread, so it has to defer to the main thread (using MainThreadActionQueue), and the navigation layer then needs to generate a navigation map from the actual terrain object (plus any buildings spawned on it, etc). Is there any way to make the navigation layer wait until the deferred actions have taken place?

[runevision]

In the framework, layers depend on other layers, and a chunk from one layer only starts generating when the required chunks in the layers it depends on are done generating. This means that to incorporate dependencies on main thread actions cleanly into this conceptual framework, some chunks should not finish generating until the required main thread actions are done.

There is nothing built-in specifically for this, but you can get the functionality by having a bool property "completed" which the main thread actions sets when it's done. The bool could be either on the main thread action or on the chunk; a place where both the action and the chunk can access it. Then in the chunk you can wait for this bool to be true:

while (!mainThreadAction.completed)
	System.Threading.Thread.Sleep(1);

This way the chunk that enqueued the actions won't be done generating until the actions have been performed too.

However, note that waiting for a main thread action in the threaded generation code can have major impacts on performance. The generation effectively becomes single-threaded while waiting for these actions, as all the chunks generating in parallel are waiting for the main thread actions to be completed one by one.

If you can find a way to generate the navigation map based directly on the other data you generate, without needing the actual terrain objects or spawned objects, that would likely be much more efficient.

If this is not an option, another approach I can think of is to not have the chunks that enqueue the main thread actions wait for them, but instead implement the waiting in the methods on the layer which your navigation layer uses to access the terrain data. If your terrain layer can query whether the terrain object of a given chunk has completed spawning or not, it can halt a method trying to retrieve the data until the data is actually available. This might have slightly better performance characteristics than waiting in the terrain layer chunks. However, I've never implemented this pattern, so I'm not aware of potential pitfalls or difficulties.

[Dekker3D]

Personally, I'd just use a threadpool, and just add new tasks for dependent layers when the lower layer chunks finish generating, probably with some "finished generating" function call for this use case. That way, work can just continue between frames. As far as I understand your current code, it spawns one thread per chunk that needs to be generated, and just waits until they're all done before continuing the rest of the frame?

[runevision]

Personally, I'd just use a threadpool, and just add new tasks for dependent layers when the lower layer chunks finish generating, probably with some "finished generating" function call for this use case.

A chunk in a dependent layer often depends on multiple chunks in the depended on layer. Only when everything a chunk depends on has finished generating can the chunk itself be generated. Keeping track of these many-to-many dependencies is the core of what the LayerProcGen system does.

That way, work can just continue between frames.

Work already continues between frames. The generation code is not tied to frames in any way. It happens in the background, independent of the game's frame loop. Only the main thread actions are by necessity tied to frames.

As far as I understand your current code, it spawns one thread per chunk that needs to be generated, and just waits until they're all done before continuing the rest of the frame?

I'm confused why you talk about frames here at all. In any case, the framework does not only handle one layer at a time. It starts generating chunks in higher layers once everything those chunks depend on has finished. You can see in the video here (where each thread is represented as a dot) that some threads start in new layers when other threads are still working on other layers:
https://mastodon.gamedev.place/@runevision/111999978778863849

When I said that depending on main thread actions will effectively make the execution single-threaded, it's not entirely accurate. If there's other chunks that don't depend on the main thread actions in any direct or indirect way, those could still run. However, the main thread actions will likely become a bottleneck since they can only run on one thread. So all work that can still run on multiple threads is likely to finish before the main thread work is done, making the main thread work a bottleneck. No changing of the multithreading architecture can change this. This is why I encourage treating the main threat actions as output only and not have the generation code depend on them.