C Extension Uses Static Buffer for String Return, Returns Wrong Data Under Concurrency

[VyCen]

Many C extensions use a static char[] buffer to temporarily build strings before returning via PyUnicode_FromString(buf). In the GIL-enabled build, this pattern is safe because only one thread executes C code at a time.

In free-threaded Python (GIL disabled), two threads can enter the same C function simultaneously and write to the same static buffer. Result:

1. Thread A's data is overwritten by Thread B
2. Thread A returns Thread B's data
3. If snprintf execution interleaves, the returned data is a garbled mix

Code Example

static char shared_buffer[1024];

PyObject *format_result(PyObject *self, PyObject *args) {
    int code;
    const char *message;
    if (!PyArg_ParseTuple(args, "is", &code, &message))
        return NULL;
    // Thread A: snprintf(shared_buffer, "Code 42: OK")
    // Thread B: snprintf(shared_buffer, "Code 99: ERROR")  ← overwrites A
    snprintf(shared_buffer, 1024, "Code %d: %s", code, message);
    return PyUnicode_FromString(shared_buffer);
}

Recommended Fixes

Option 1: Dynamic allocation

PyObject *format_result(int code, const char *msg) {
    return PyUnicode_FromFormat("Code %d: %s", code, msg);
}

Option 2: Thread-local storage

static THREAD_LOCAL char buffer[1024];
PyObject *format_result(int code, const char *msg) {
    snprintf(buffer, 1024, "Code %d: %s", code, msg);
    return PyUnicode_FromString(buffer);
}

Reproduction

import threading
import _static_buffer_string

def worker():
    for i in range(1000):
        result = _static_buffer_string.format_result(i % 100, f"msg_{i}")
        expected = f"Code {i % 100}: msg_{i}"
        if result != expected:
            print(f"CORRUPTION! Expected: {expected}, Got: {result}")
            return

threads = [threading.Thread(target=worker) for _ in range(8)]
for t in threads: t.start()
for t in threads: t.join()

[ngoldbaum]

Is there a real extension you ran into that does this?

Was an AI model used to create this issue?