Handle exceptions from parallel ReadyToRun compilation

[agocke]

Summary

• encapsulate ReadyToRun work distribution and persistent worker lifetime in CompilationWorklist
• capture the first unexpected compilation exception, stop issuing new work, join all participating workers, and rethrow on the coordinating thread
• make compilation disposal an exception-safe worker cleanup backstop while releasing per-worker JIT state before object emission
• simplify crossgen2 compilation ownership with scoped disposal

Testing

• checked crossgen2 build: succeeded with 0 warnings and 0 errors
• ILCompiler.ReadyToRun.Tests: 57 passed, 2 skipped, 0 failed
• parallel CoreLib compilation validated at parallelism 1 and 4

Note

This pull request description was generated with GitHub Copilot.

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[agocke]

GH diff is kind of failing here -- VS Code is better. It shows that I just extracted the real work into the new BuildCompilation method so that I could using using var compilation instead of manual Dispose and try-finally

[agocke]

This was the actual source of an exception. It should now be handled by the Dispose on this class, which is driven by the using in Program.cs.

[Copilot]

Pull request overview

This PR refactors parallel ReadyToRun compilation to centralize work distribution/worker lifetime management in a new CompilationWorklist<TWorkerState>, improving exception handling and cleanup behavior, and adjusts crossgen2 compilation ownership to ensure deterministic disposal.

Changes:

• Introduces CompilationWorklist<TWorkerState> to coordinate parallel work and propagate the first unexpected worker exception to the coordinating thread.
• Refactors ReadyToRunCodegenCompilation to use per-worker state and dispose worker/JIT state earlier to reduce peak memory during object emission.
• Refactors crossgen2 to build compilations via a helper and use scoped (using var) disposal for safer cleanup.

Show a summary per file

File	Description
src/coreclr/tools/aot/ILCompiler.ReadyToRun/ILCompiler.ReadyToRun.csproj	Adds the new CompilationWorklist.cs to the build.
src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunCodegenCompilation.cs	Replaces bespoke thread/semaphore orchestration with CompilationWorklist, adds per-worker state tracking, and disposes worklist before emission.
src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/CompilationWorklist.cs	New helper to manage parallel work distribution, worker lifetime, and exception capture/rethrow.
src/coreclr/tools/aot/crossgen2/Program.cs	Refactors compilation creation into BuildCompilation and uses scoped disposal to simplify ownership and cleanup.

Review details

Suppressed comments (1)

src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/CompilationWorklist.cs:60

• Run(...) should validate that the provided items list is non-null before storing it and using items.Count in TryTake; otherwise a null argument would crash later with a NullReferenceException.

        public void Run(IReadOnlyList<DependencyNodeCore<NodeFactory>> items)
        {
            EnsureWorkers();
            Debug.Assert(_items is null);

            _items = items;

• Files reviewed: 4/4 changed files
• Comments generated: 1
• Review effort level: Lite

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[jkotas]

NativeAOT has equivalent logic implemented using Parallel.ForEach:

runtime/src/coreclr/tools/aot/ILCompiler.RyuJit/Compiler/RyuJitCompilation.cs

Lines 166 to 169 in 2580b59

	Parallel.ForEach(
	methodsToCompile,
	new ParallelOptions { MaxDegreeOfParallelism = _parallelism },
	CompileSingleMethod);

Can we use that instead of implementing our own custom MT scheduler?

In any case, both R2R and NAOT should be on the same plan if we are refactoring this.

[agocke]

Makes sense to me. I was starting with refactoring, but keeping the existing scheduler, but I can go further and try to get us down to just Parallel.ForEach.