Add fix for https://github.com/nothings/stb/issues/1860 by checking malloc multiplication before allocating

[NBickford-NV]

This should fix #1860 .

Previously, stbi__convert_format16 didn't check if the multiplication req_comp * x * y * 2 would overflow before calling stbi__malloc. Thankfully, the fix is a quick change to use stbi__malloc_mad4 instead of a raw stbi__malloc. (stbi__convert_format8 already checked this. The floating-point path checks this inside stbi__hdr_load.)

I also added a safety check to code in stbi__convert_8_to_16 that looks like it could run into a similar issue. (We know that req_comp * x * y is valid here because we've been passed a valid buffer. That does imply that we could save a few operations by doing an stbi__malloc_mad2 inside stbi__convert_format16, though.)

Thanks!

[NBickford-NV]

(It looks like the fix in stbi__convert_8_to_16 is another way of fixing #1529; @JarLob previously wrote nvpro-samples@f100bfc .)

[sezero]

First patch requires adjusting restrictions on stbi__malloc_mad4() and
stbi__mad4sizes_valid(), like:

diff --git a/src/stb_image.h b/src/stb_image.h
index 99115a6..03c22e0 100644
--- a/src/stb_image.h
+++ b/src/stb_image.h
@@ -1110,7 +1110,7 @@ static int stbi__mad3sizes_valid(int a, int b, int c, int add)
 }
 
 // returns 1 if "a*b*c*d + add" has no negative terms/factors and doesn't overflow
-#if !defined(STBI_NO_LINEAR) || !defined(STBI_NO_HDR) || !defined(STBI_NO_PNM)
+#if !defined(STBI_NO_LINEAR) || !defined(STBI_NO_HDR) || !defined(STBI_NO_PNM) || !defined(STBI_NO_PNG) || !defined(STBI_NO_PSD)
 static int stbi__mad4sizes_valid(int a, int b, int c, int d, int add)
 {
    return stbi__mul2sizes_valid(a, b) && stbi__mul2sizes_valid(a*b, c) &&
@@ -1133,7 +1133,7 @@ static void *stbi__malloc_mad3(int a, int b, int c, int add)
    return stbi__malloc(a*b*c + add);
 }
 
-#if !defined(STBI_NO_LINEAR) || !defined(STBI_NO_HDR) || !defined(STBI_NO_PNM)
+#if !defined(STBI_NO_LINEAR) || !defined(STBI_NO_HDR) || !defined(STBI_NO_PNM) || !defined(STBI_NO_PNG) || !defined(STBI_NO_PSD)
 static void *stbi__malloc_mad4(int a, int b, int c, int d, int add)
 {
    if (!stbi__mad4sizes_valid(a, b, c, d, add)) return NULL;

[NBickford-NV]

Thank you @sezero ! I've added that change to this pull request.