Add multi-threaded compression via SQ01 multi-block format

The model pipeline is bit-serial within a stream, so parallelism comes
from chunking: the SQ01 container wraps independent SQ02 chunk streams
that compress and decompress concurrently, one model per worker.

- New API: squish_compress_mt / squish_decompress_mt, _alloc_mt and
  _file_mt variants, squish_threads(); progress callbacks are
  aggregated across chunks and stay monotonic
- Output depends only on the chunk size (default 16 MiB, min 64 KiB),
  never on the thread count; inputs no larger than one chunk emit a
  plain SQ02 stream, and the compress_bound guarantee is preserved via
  a whole-input stored-mode fallback
- Existing decompression entry points read SQ01 transparently; nested
  SQ01 containers are rejected
- CLI: -t/--threads (compress defaults to 1, keeping the ratio-optimal
  single-block format; decompress uses all cores) and -b/--block MiB;
  status line/summary now use a wall clock instead of clock()
- Threads: pthreads on POSIX (-pthread added to Makefile and squish.pc),
  native Win32 threads on Windows
- The SQ01 magic previously named an unreleased pre-1.0 draft that no
  reader accepted; FORMAT.md documents the reassignment

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: paigejulianne

CHANGELOG.md

@@ -11,6 +11,18 @@ Per [CONTRIBUTING.md](CONTRIBUTING.md), any change to the model constants in
 
 ### Added
 
+- Multi-threaded compression and decompression: new `SQ01` multi-block
+  container (independent `SQ02` chunk streams; see docs/FORMAT.md §1b) and
+  library functions `squish_compress_mt` / `squish_decompress_mt`, their
+  `_alloc_mt` / `_file_mt` variants, and `squish_threads()`. Output depends
+  only on the chunk size (default 16 MiB), never on the thread count, and
+  the `squish_compress_bound` guarantee is preserved. Existing
+  decompression entry points read `SQ01` streams transparently. (The
+  `SQ01` magic previously named an unreleased pre-1.0 draft no reader
+  accepted; it has been reassigned to the multi-block container.)
+- CLI: `-t N` / `--threads N` — 0 = all cores; compression defaults to 1,
+  keeping the ratio-optimal single-block format, decompression to all
+  cores — and `-b N` / `--block N` block size in MiB for `-t`
 - CLI: live status line on stderr (percent, bytes, throughput) while
   compressing/decompressing when stderr is a terminal; `-q`/`--quiet`
   suppresses the status line and the final summary (errors still print)
@@ -20,6 +32,13 @@ Per [CONTRIBUTING.md](CONTRIBUTING.md), any change to the model constants in
 - `build-windows.bat`: builds `squish.dll` + `squish.exe` with MSVC from a
   plain command prompt (locates Visual Studio via vswhere; no make needed)
 
+### Changed
+
+- CLI: the status line and summary now measure throughput on a wall clock
+  (was CPU time, which over-counts when threads are in play)
+- Building now requires a threads library: `-pthread` outside Windows
+  (added to the Makefile and pkg-config file), Win32 threads on Windows
+
 ## [1.0.0]
 
 ### Added

Makefile

@@ -12,6 +12,7 @@
 CC       ?= gcc
 OPT      ?= -O3 -funroll-loops
 WARN      = -Wall -Wextra
+THREADS  ?= -pthread
 CFLAGS   ?= $(OPT)
 PREFIX   ?= /usr/local
 DESTDIR  ?=
@@ -30,7 +31,7 @@ all: libsquish.so squish
 
 # ---- shared library ---------------------------------------------------------
 libsquish.so.$(VERSION): squish.c squish.h
-	$(CC) $(CFLAGS) $(WARN) -fPIC -fvisibility=hidden -DSQUISH_BUILD \
+	$(CC) $(CFLAGS) $(THREADS) $(WARN) -fPIC -fvisibility=hidden -DSQUISH_BUILD \
 	    -shared -Wl,-soname,$(SONAME) -o $@ squish.c -lm
 
 libsquish.so: libsquish.so.$(VERSION)
@@ -39,14 +40,14 @@ libsquish.so: libsquish.so.$(VERSION)
 
 # ---- static library ---------------------------------------------------------
 squish.o: squish.c squish.h
-	$(CC) $(CFLAGS) $(WARN) -c -o $@ squish.c
+	$(CC) $(CFLAGS) $(THREADS) $(WARN) -c -o $@ squish.c
 
 libsquish.a: squish.o
 	ar rcs $@ $^
 
 # ---- CLI (statically linked against the lib) ---------------------------------
 squish: squish_cli.c libsquish.a
-	$(CC) $(CFLAGS) $(WARN) -o $@ squish_cli.c libsquish.a -lm
+	$(CC) $(CFLAGS) $(THREADS) $(WARN) -o $@ squish_cli.c libsquish.a -lm
 
 # ---- Windows DLL + CLI (cross-compile; or use cl.exe /DSQUISH_BUILD_DLL) ------
 dll: squish.dll squish.exe
@@ -78,11 +79,11 @@ test: tests/test_squish
 	./tests/test_squish
 
 tests/test_squish: tests/test_squish.c libsquish.a
-	$(CC) $(CFLAGS) $(WARN) -I. -o $@ tests/test_squish.c libsquish.a -lm
+	$(CC) $(CFLAGS) $(THREADS) $(WARN) -I. -o $@ tests/test_squish.c libsquish.a -lm
 
 example: examples/example
 examples/example: examples/example.c libsquish.so
-	$(CC) $(CFLAGS) $(WARN) -I. -o $@ examples/example.c -L. -lsquish -lm \
+	$(CC) $(CFLAGS) $(THREADS) $(WARN) -I. -o $@ examples/example.c -L. -lsquish -lm \
 	    -Wl,-rpath,'$$ORIGIN/..'
 
 # ---- install ------------------------------------------------------------------
@@ -95,7 +96,7 @@ install: libsquish.so libsquish.a squish
 	install -m 644 libsquish.a $(DESTDIR)$(PREFIX)/lib/
 	install -m 644 squish.h $(DESTDIR)$(PREFIX)/include/
 	install -m 755 squish $(DESTDIR)$(PREFIX)/bin/
-	printf 'prefix=%s\nlibdir=$${prefix}/lib\nincludedir=$${prefix}/include\n\nName: squish\nDescription: context-mixing compressor\nVersion: %s\nLibs: -L$${libdir} -lsquish -lm\nCflags: -I$${includedir}\n' \
+	printf 'prefix=%s\nlibdir=$${prefix}/lib\nincludedir=$${prefix}/include\n\nName: squish\nDescription: context-mixing compressor\nVersion: %s\nLibs: -L$${libdir} -lsquish -lm -pthread\nCflags: -I$${includedir}\n' \
 	    "$(PREFIX)" "$(VERSION)" > $(DESTDIR)$(PREFIX)/lib/pkgconfig/squish.pc
 
 clean:

README.md

@@ -46,10 +46,22 @@ No dependencies beyond libc/libm.
 ## CLI
 
 ```sh
-./squish c bigfile bigfile.sq     # compress
-./squish d bigfile.sq restored    # decompress (checksum-verified)
+./squish c bigfile bigfile.sq          # compress
+./squish d bigfile.sq restored         # decompress (checksum-verified)
+./squish -t 0 c bigfile bigfile.sq     # compress on all cores (multi-block)
+./squish -t 0 -b 4 c big big.sq        # ... with 4 MiB blocks (more parallel,
+                                       #     slightly worse ratio)
 ```
 
+`-t N` compresses with N threads (`0` = all cores) by splitting the input
+into independently modeled blocks — near-linear speedup, at a small ratio
+cost because each block's model starts cold (about 1–2% at the default
+16 MiB blocks, more at smaller `-b` sizes). The default `-t 1` keeps the
+ratio-optimal single-block format used for the results table above.
+Decompression always uses all cores when the stream allows it (`-t` caps
+it) and reads both formats transparently. Budget ~150 MB of model memory
+per thread.
+
 When stderr is a terminal, a live status line (percent, bytes, throughput)
 is shown while working, followed by a one-line summary. Pass `-q` /
 `--quiet` to suppress both; errors are still reported.
@@ -68,8 +80,16 @@ squish_decompress_alloc(c, cn, &d, &dn);   /* integrity-checked */
 squish_free(c);  squish_free(d);
 ```
 
-Link with `-lsquish -lm` (or `pkg-config --cflags --libs squish` after
-`make install`).
+Multi-threaded variants take a thread count (0 = all cores) and, for
+compression, a chunk size (0 = 16 MiB default):
+
+```c
+squish_compress_alloc_mt(data, n, &c, &cn, 0, 0, NULL, NULL);
+squish_decompress_alloc_mt(c, cn, &d, &dn, 0, NULL, NULL);
+```
+
+Link with `-lsquish -lm -pthread` (or `pkg-config --cflags --libs squish`
+after `make install`).
 
 Python needs no wrapper — `libsquish.so` loads directly with ctypes; see
 [examples/example.py](examples/example.py).

SQUISH.md

@@ -72,6 +72,13 @@ payload · 32-bit checksum of the original data. Incompressible inputs fall
 back to stored mode, so output never exceeds input + 17 bytes; decompression
 verifies the checksum. Full byte-level spec: [docs/FORMAT.md](docs/FORMAT.md).
 
+The model pipeline is strictly sequential within a stream — every bit's
+prediction depends on the previous bit's update — so multi-core operation
+comes from cutting the input into chunks instead: an `"SQ01"` multi-block
+container wraps independent `SQ02` chunk streams that compress and
+decompress in parallel (`squish -t`, `squish_*_mt`), trading ~1–2% of
+ratio for near-linear speedup.
+
 ## Usage
 
 ```

docs/API.md

@@ -128,6 +128,59 @@ from the coding loop, so keep it cheap and do not call back into the
 library. `user` is passed through untouched. With `progress == NULL` these
 are identical to the plain versions.
 
+## Multi-threaded variants
+
+```c
+int squish_threads(void);
+```
+Number of processors online (>= 1); what `nthreads = 0` selects below.
+
+```c
+int squish_compress_mt(const void *src, size_t src_len,
+                       void *dst, size_t *dst_len,
+                       int nthreads, size_t chunk_size,
+                       squish_progress_fn progress, void *user);
+int squish_decompress_mt(const void *src, size_t src_len,
+                         void *dst, size_t *dst_len, int nthreads,
+                         squish_progress_fn progress, void *user);
+
+int squish_compress_alloc_mt(const void *src, size_t src_len,
+                             void **dst, size_t *dst_len,
+                             int nthreads, size_t chunk_size,
+                             squish_progress_fn progress, void *user);
+int squish_decompress_alloc_mt(const void *src, size_t src_len,
+                               void **dst, size_t *dst_len, int nthreads,
+                               squish_progress_fn progress, void *user);
+
+int squish_compress_file_mt(const char *src_path, const char *dst_path,
+                            int nthreads, size_t chunk_size,
+                            squish_progress_fn progress, void *user);
+int squish_decompress_file_mt(const char *src_path, const char *dst_path,
+                              int nthreads,
+                              squish_progress_fn progress, void *user);
+```
+
+Parallel counterparts of the functions above, with the same buffer,
+allocation, file, and progress contracts. Compression splits the input
+into `chunk_size`-byte chunks (`0` = `SQUISH_DEFAULT_CHUNK`, 16 MiB;
+minimum `SQUISH_MIN_CHUNK`, 64 KiB) and codes each independently on a pool
+of `nthreads` workers (`0` = all cores), emitting a multi-block `SQ01`
+stream — see [FORMAT.md](FORMAT.md) §1b. Points worth knowing:
+
+- **Output is deterministic**: it depends on the input and `chunk_size`
+  only, never on `nthreads`.
+- **Ratio cost**: each chunk's model starts cold, costing roughly 1–2% at
+  the 16 MiB default (more at smaller chunks). Inputs no larger than one
+  chunk produce a plain `SQ02` stream, bit-identical to `squish_compress`.
+- **Bound preserved**: `squish_compress_bound` still holds — inputs that
+  don't benefit fall back to a single stored-mode `SQ02` stream.
+- The decompression functions read both formats; parallelism applies to
+  `SQ01` streams (`SQ02` has a single sequential model). The plain
+  (non-`_mt`) decompression functions also accept `SQ01`, serially.
+- **Memory**: ~150 MB of model state per active worker.
+- `progress` calls are serialized by the library but may arrive on worker
+  threads; `processed` stays monotonic across the whole input.
+
 ---
 
 ## Linking recipes
@@ -150,7 +203,7 @@ From Python, load `libsquish.so` with `ctypes` directly —
 
 - The shared library uses semantic versioning; soname `libsquish.so.1`.
   Functions are added, never changed, within a major version.
-- The bitstream format is identified by the magic (`SQ02`). The model
-  constants in `squish.c` **are** the format: streams are only decodable by
-  a build with identical constants. `SQ02` readers reject other magics with
-  `SQUISH_E_FORMAT`.
+- The bitstream format is identified by the magic (`SQ02` single stream,
+  `SQ01` multi-block). The model constants in `squish.c` **are** the
+  format: streams are only decodable by a build with identical constants.
+  Readers reject other magics with `SQUISH_E_FORMAT`.

docs/FORMAT.md

@@ -1,11 +1,15 @@
-# SQUISH stream format "SQ02"
+# SQUISH stream formats "SQ02" and "SQ01"
 
 This document specifies the byte format completely enough to write an
 independent decoder. Because SQUISH is a context-mixing design, the
 container is trivial and the real specification is the **predictor**: the
 decoder must reproduce the encoder's probability for every bit exactly,
 which means every constant and update rule below is normative.
 
+There are two containers: `SQ02`, a single model stream, and `SQ01`
+(§1b), a multi-block wrapper holding independent `SQ02` streams so that
+chunks can be coded in parallel.
+
 ## 1. Container
 
 ```
@@ -24,6 +28,36 @@ last    4     checksum: FNV-1a 64 of the n original bytes, low 32 bits, LE
 - `mode 0`: payload is the arithmetic-coded bitstream described below.
 - FNV-1a 64: `h = 0xcbf29ce484222325; for each byte b: h ^= b; h *= 0x100000001b3`.
 
+## 1b. Multi-block container "SQ01"
+
+Produced by the multi-threaded encoder for inputs larger than one chunk.
+The payload is a sequence of complete, independent `SQ02` streams; models
+never carry state across chunk boundaries, which is what makes parallel
+encode and decode possible.
+
+```
+offset       size  field
+0            4     magic: 'S' 'Q' '0' '1'
+4            8     total original size n, unsigned 64-bit little-endian
+12           1     mode: 2 = multi-block
+13           4     chunk count k >= 1, unsigned 32-bit little-endian
+17           4*k   compressed size of each chunk, u32 LE each
+17 + 4*k     ...   k chunks, each a complete SQ02 stream (§1)
+```
+
+- Chunk i decodes to bytes `[sum of previous chunks' sizes ...)` of the
+  output; the original size of each chunk comes from its own SQ02 header.
+- The compressed sizes must tile the rest of the file exactly, and the
+  per-chunk original sizes must sum to n.
+- Chunks must be `SQ02` streams: nested `SQ01` containers are invalid.
+- There is no container-level checksum; each chunk carries its own (§1).
+- Encoders chunk the input at a fixed chunk size (encoder choice; the
+  reference default is 16 MiB, minimum 64 KiB), so the emitted stream
+  depends only on the input and the chunk size — never on the thread
+  count. An encoder that finds the multi-block stream no smaller than
+  stored mode emits a single stored-mode `SQ02` stream instead, which
+  preserves the n + 17 output bound.
+
 ## 2. Arithmetic coder
 
 Carryless binary range coder over 32-bit registers, coding bits MSB-first
@@ -160,7 +194,10 @@ bit — predict, code, update, in that order, for encoder and decoder alike.
 
 ## 10. Versioning
 
-`SQ02` is the only current version. Any change to a constant, table size,
-initialization value, or update rule in §§2–9 produces incompatible streams
-and requires a new magic. (`SQ01` was a pre-release format without the mode
-byte and checksum; it is not accepted by `SQ02` readers.)
+`SQ02` (single stream) and `SQ01` (multi-block, §1b) are the current
+versions. Any change to a constant, table size, initialization value, or
+update rule in §§2–9 produces incompatible streams and requires a new
+magic. (The magic `SQ01` previously named a pre-release single-stream
+format without the mode byte and checksum; that format was never released
+and no reader accepted it, so the magic has been reassigned to the
+multi-block container, which is distinguishable by its mode byte 2.)