.NET 10: Access violation in Tiered JIT Compilation

[OliverHallam]

Description

We've been seeing some nasty crashes after upgrading to .NET 10 - Its taken a long time to track it down but I've narrowed down to the attached fairly minimal repro, which is fully managed code.

When compiled in release mode (without a debugger attached), the program fails - In our full application the crash manifests itself in a number of different forms from access violation errors to heap corruption.

In the test attached this failure this always happens on the 4097th call to Add which is presumably when tiered compilation kicks in to recomplile the method.

Reproduction Steps

class Program
{
    public static void Main()
    {
        var map = new UInt64Map_String();

        for(var x = 0; x < 5000; x++)
        {
            map.Add((uint)x, "123");
        }

        Console.WriteLine("Success");
    }
}


public sealed class UInt64Map_String
{
    private const int BitsInULong = 64;
    private const int DefaultBitCount = 3; // Default initial capacity is 2^3.
    private Entry[] fData;
    private int fNumBitsInHashcode;
    private int fCount; // Number of slots in use.
    private int fModuloMask;

    public UInt64Map_String()
    {
        fNumBitsInHashcode = DefaultBitCount;
        fData = new Entry[1 << fNumBitsInHashcode]; // = 2 ^ numBits
        fModuloMask = fData.Length - 1;
    }

    public bool Add(ulong key, string value)
    {
        if(fCount * 2 >= fData.Length) // Max load factor of 0.5
            Expand();

        int iTarget = GetFibHashCode(key, BitsInULong - fNumBitsInHashcode);

        for(int i = 0; i < fData.Length; i++)
        {
            int probeDistance = i;
            int index = GetBucketIndex((uint)iTarget + (uint)probeDistance);
            ref var entry = ref fData[index];
            if(!entry.Used) // Reached end of used slots.
            {
                entry = new Entry(key, value, used: true);
                fCount++;
                return true;
            }

            if(entry.Key == key)
            {
                return false;
            }
        }

        throw new InvalidOperationException(); // Should never get here.
    }

    private void Expand()
    {
        var oldData = fData;
        fNumBitsInHashcode++;
        if(fData.Length < 1 << 30)
        {
            fData = new Entry[1 << fNumBitsInHashcode]; // = 2 ^ numBits
            fModuloMask = fData.Length - 1;
        }
        else
        {
            fData = new Entry[Array.MaxLength];
            fModuloMask = int.MaxValue;
        }

        // Copy over old entries to new array.
        // Note that positions will change, so we need to explicitly re-add each entry.
        fCount = 0;
        for(int i = 0; i < oldData.Length; i++)
        {
            ref var oldEntry = ref oldData[i];
            if(oldEntry.Used)
                _ = Add(oldEntry.Key, oldEntry.Value);
        }
    }

    private readonly struct Entry
    {
        public readonly ulong Key;
        public readonly string Value;
        public readonly bool Used;

        public Entry(ulong key, string value, bool used)
        {
            Key = key;
            Value = value;
            Used = used;
        }
    }

    private int GetBucketIndex(uint target)
    {
        // This method is equivalent to target % fData.Lengrh
        var bucketIndex = target & fModuloMask;

        if(bucketIndex > Array.MaxLength)
            bucketIndex -= Array.MaxLength;

        return (int)bucketIndex;
    }
    
    public static int GetFibHashCode(ulong key, int numBitsDiscarded)
    {
        const ulong GoldenMultiplier = 11400714819323198485; // Roughly equal to (2^64 / phi)
        return (int)((key * GoldenMultiplier) >> numBitsDiscarded); // Take the uppermost bits to map into target range.
    }
}

Expected behavior

Application prints Success and exits

Actual behavior

Fatal error.
0x80131506

Process finished with exit code -2,146,233,082.

Regression?

Works fine in .NET 8.0; regression in 10.0

Known Workarounds

No response

Configuration

No response

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[AndyAyersMS]

Thank you for reporting this, we'll investigate.

[AndyAyersMS]

This seems to be fixed in .NET 11. We may just be able to backport an existing fix.

[OliverHallam]

That's good news if it is.

I'll try testing our full codebase on .NET 11 tomorrow to confirm (I hadn't tried this yet).

[AndyAyersMS]

Same issue as #123361, which was fixed in #123362 but never backported. I will get that going.

As a workaround you can change the else clause in Expand to outline the contents:

[MethodImpl(MethodImplOptions.NoInlining)]
private void GrowToMax()
{
    fData = new Entry[Array.MaxLength];
    fModuloMask = int.MaxValue;
}

[OliverHallam]

That makes sense, and completely explains the crashes we have been seeing. We have tried the workaround you suggested and it completely resolves the issue for us.

Thanks for your help - that's an impressive turnaround time!