Intel CPU bug

[glandium]

After enabling zlib-rs/libz-rs-sys 0.4.1 in Firefox nightly, we've started receiving crash reports for out of bound accesses in State::d_code:
https://crash-stats.mozilla.org/report/index/113ebfb1-9197-44e2-88ce-322b20250223
8 xul.dll core::panicking::panic_bounds_check() library/core/src/panicking.rs:273 cfi
9 xul.dll zlib_rs::deflate::State::d_code(unsigned long long) third_party/rust/zlib-rs/src/deflate.rs:1063 inlined
9 xul.dll zlib_rs::deflate::BitWriter::emit_dist(ref$<slice2$<zlib_rs::deflate::Value> >, ref$<slice2$<zlib_rs::deflate::Value> >, unsigned char, unsigned long long) third_party/rust/zlib-rs/src/deflate.rs:1077 inlined
9 xul.dll zlib_rs::deflate::BitWriter::compress_block_help(ref$<slice2$ >, ref$<slice2$<zlib_rs::deflate::Value> >, ref$<slice2$<zlib_rs::deflate::Value> >) third_party/rust/zlib-rs/src/deflate.rs:1105 cfi
10 xul.dll zlib_rs::deflate::State::compress_block_dynamic_trees() third_party/rust/zlib-rs/src/deflate.rs:1469 inlined
10 xul.dll zlib_rs::deflate::zng_tr_flush_block(zlib_rs::deflate::DeflateStream*, enum2$<core::option::Option >, unsigned int, bool) third_party/rust/zlib-rs/src/deflate.rs:2327 inlined
10 xul.dll zlib_rs::deflate::flush_block_only(zlib_rs::deflate::DeflateStream*, bool) third_party/rust/zlib-rs/src/deflate.rs:2343 cfi
11 xul.dll zlib_rs::deflate::algorithm::medium::deflate_medium(zlib_rs::deflate::DeflateStream*, zlib_rs::DeflateFlush) third_party/rust/zlib-rs/src/deflate/algorithm/mod.rs:19 cfi
12 xul.dll zlib_rs::deflate::algorithm::run(zlib_rs::deflate::DeflateStream*, zlib_rs::DeflateFlush) third_party/rust/zlib-rs/src/deflate.rs:2706 inlined
12 xul.dll zlib_rs::deflate::deflate(zlib_rs::deflate::DeflateStream*, zlib_rs::DeflateFlush) third_party/rust/zlib-rs/src/deflate.rs:2612 cfi
13 xul.dll MOZ_PNG_compress_IDAT(png_struct_def*, unsigned char const*, unsigned long long, int)

I unfortunately don't have more information than the stack trace that says it starts from compressing data while creating a PNG IDAT section (so I can't give an example of what specific data leads to this), and we have yet to update to 0.4.2, but it doesn't look like the relevant code was updated since 0.4.1.

[glandium]

This could be a CPU bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1950764#c3

[folkertdev]

This is still looking like a hardware bug, right?

We could provide a branch with some additional assertions if that is helpful for tracking down where the error originates.

[gabrielesvelto]

Yes, this is definitely a hardware bug in Intel Raptor Lake CPU. Tracking down the instruction sequence leading to this is going to be very complicated. I'll see if we have contacts at Intel to help with this because I doubt it's easy to work around this issue effectively.

[bdaehlie]

Is it possible to detect this particular CPU and disable optimizations for it for now, assuming optimizations are triggering the problem? That might be the fastest path to moving forward here.

[erikjee]

We are considering creating a version that disables stuff for this CPU. Do you think that makes sense @glandium?
Is it even feasible for you to run your tests again on a patched version?

We looked at buying an affected CPU, but there is probably a high likelihood of not being able to reproduce the crash, so we haven't proceeded with that.

[gabrielesvelto]

I've reached out to Intel about this but haven't heard back yet. There's another path forward: we don't have crashes on file for CPUs using microcode version 0x12c which was released in mid-February. That could mean that the issue might have been fixed in that version, but given the short timespan we can't be sure if users are already running it or not. Waiting a couple of weeks should give us some insight about that. If no crashes crop up with that microcode version then we can reasonably assume that the bug has been fixed by Intel.

[erikjee]

Thanks for the update! A couple of weeks is not that long. Let's wait and see.

[erikjee]

@gabrielesvelto I'm not entirely sure what this means, but this might be good news?

Based on the topcrash criteria, the crash signature linked to this bug is not a topcrash signature anymore.

See https://bugzilla.mozilla.org/show_bug.cgi?id=1950764#c14

[gabrielesvelto]

@gabrielesvelto I'm not entirely sure what this means, but this might be good news?

Yes, the number of crashes we're experiencing in the wild has decreased significantly and the decrease matched the release of a new microcode version from Intel that likely addressed the issue.

[folkertdev]

That's good to hear! Do you have any indication of when the number of reports will drop below the threshold that is acceptable for you?

[gabrielesvelto]

That's hard to tell, but we're currently shipping zlib-rs only in nightly builds which are used by 10s of thousands of users. We'll know if things are under control only when we attempt to ship to a larger population which is going to happen incrementally.

[gabrielesvelto]

FYI someone probably found what is the root cause of the CPU bug causing this issue in a different piece of code: https://fosstodon.org/@rygorous@mastodon.gamedev.place/114550012226316857

In case we need to mitigate it there's information about that too.

[glandium]

FWIW, I didn't find a sequence of shr/shl+mov applying on a register and its bits[8:15] subpart in one of the affected versions of Firefox. So even with the extra information from the blog post, we'd still need to identify the exact sequence of code that causes the problem in order to mitigate if needed.

[gabrielesvelto]

I have bad news: Intel released a new version of the microcode for this family of CPUs (0x12f) and it seems to be affected by this bug. Given the evolution of the crash volume this appears to be a regression on Intel's part though it's hard to be sure.