Integer overflow in stb_image_write.h leads to heap buffer overflow in PNG encoding and out-of-bounds reads in JPEG/TGA encoding

[JerryGW]

Environment: x86_64 Linux, GCC 13/Clang 16, ASAN, 64-bit build (int is 32-bit)
Commit: 31c1ad3 (line numbers relative to this commit)

stb_image_write.h performs several size calculations using int arithmetic, which can overflow when processing large images. When the overflow results in a negative or truncated value being passed to STBIW_MALLOC or used as a memory offset, it leads to memory corruption.

Three cases were identified:

Case 1: PNG filter buffer allocation overflow (heap buffer overflow)

In stbi_write_png_to_mem() at line 1144:

filt = (unsigned char *) STBIW_MALLOC((x*n+1) * y);

When x=50000, n=1, y=85899, the computation (x*n+1) * y = 50001 * 85899 = 4,295,035,899 overflows a 32-bit int and wraps to 68,603. The 68KB buffer is far too small for the actual filter data. During encoding, memmove at line 1173 writes up to 50,000 bytes past the end of this undersized buffer.

ASAN output:

heap-buffer-overflow WRITE of size 50000
  allocated by malloc at stbi_write_png_to_mem:1144 (68,603 bytes)
  WRITE at stbi_write_png_to_mem:1173

Reproduce:

#define STB_IMAGE_WRITE_IMPLEMENTATION
#include "stb_image_write.h"
int main() {
    int w = 50000, h = 85899, n = 1;
    unsigned char *buf = (unsigned char *)malloc(w * h * n);
    stbi_write_png("out.png", w, h, n, buf, w * n);
    free(buf);
    return 0;
}

Compile with: gcc -fsanitize=address -g poc.c -o poc && ./poc

Case 2: JPEG row offset calculation overflow (segfault)

In stbi_write_jpg_core() at line 1540:

int base_p = (stbi__flip_vertically_on_write ? (height-1-clamped_row) : clamped_row) * width * comp;

When width=500000, comp=3, height=1433, the computation 1432 * 500000 * 3 = 2,148,000,000 overflows INT_MAX (2,147,483,647), producing a negative base_p. This causes the subsequent dataR[p] access to read from invalid memory.

ASAN output:

SEGV on READ at stbi_write_jpg_core:1582

Reproduce:

#define STB_IMAGE_WRITE_IMPLEMENTATION
#include "stb_image_write.h"
int main() {
    int w = 500000, h = 1433, comp = 3;
    unsigned char *buf = (unsigned char *)malloc(w * h * comp);
    stbi_write_jpg("out.jpg", w, h, comp, buf, 100);
    free(buf);
    return 0;
}

Case 3: TGA RLE row pointer calculation overflow (segfault)

In stbi_write_tga_core() at line 560:

unsigned char *row = (unsigned char *) data + j * x * comp;

When stbi_write_tga_with_rle is enabled and width=500000, comp=4, height=1075, the computation 1074 * 500000 * 4 = 2,148,000,000 overflows INT_MAX, producing a negative offset. The resulting row pointer points before the image buffer, and subsequent memcmp at line 570 crashes on invalid memory access.

ASAN output:

SEGV on READ in memcmp at stbi_write_tga_core:570

Reproduce:

#define STB_IMAGE_WRITE_IMPLEMENTATION
#include "stb_image_write.h"
int stbi_write_tga_with_rle = 1;
int main() {
    int w = 500000, h = 1075, comp = 4;
    unsigned char *buf = (unsigned char *)malloc(w * h * comp);
    stbi_write_tga("out.tga", w, h, comp, buf);
    free(buf);
    return 0;
}

Root cause

The core issue is that all three functions use int arithmetic for size/offset calculations that can reach into the multi-billion range when processing large images. A single int overflow guard at the entry of each function (or switching to size_t arithmetic internally) would prevent all three.

The same pattern of unchecked int multiplications likely exists in other encoders (BMP, HDR) in the same file.

Suggested fix

Add an overflow check at the beginning of each affected function:

1. For PNG: before STBIW_MALLOC((x*n+1) * y), check if x > 0 && y > 0 && (x*n+1) > 0 && (x*n+1) <= INT_MAX / y
2. For JPEG: before the row loop, check if width * comp > 0 && height <= INT_MAX / (width * comp)
3. For TGA: same as JPEG

Alternatively, promote intermediate computations to size_t or ptrdiff_t to avoid signed integer overflow on 64-bit platforms.