Skip to content

Commit 955dc96

Browse files
authored
build: parallelize native compile with xargs -P, not batch-and-wait (#78)
The previous loop backgrounded JOBS compiles then `wait`ed for the whole batch before starting the next JOBS - one slow TU idles every other core until the batch drains. Worse, a failing background zig cc under `&` was invisible to set -e (bare `wait` returns 0 regardless of child exit status), so a broken compile silently produced no .o and only surfaced later as a cryptic link error. xargs -P keeps all JOBS slots continuously saturated (no batch boundary) and propagates a non-zero exit from any invocation, which set -e now catches immediately.
1 parent 65f48f7 commit 955dc96

1 file changed

Lines changed: 14 additions & 8 deletions

File tree

scripts/build-zstd.sh

Lines changed: 14 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -131,14 +131,20 @@ esac
131131
CFLAGS="-O3 $ARCH_FLAG -DNDEBUG -DZSTD_LEGACY_SUPPORT=4 -DXXH_NAMESPACE=ZSTD_ $VIS_FLAG \
132132
-I$ZSTD_LIB -I$ZSTD_LIB/common -fPIC"
133133

134-
i=0
135-
for src in $SRCS; do
136-
out="$WORK/$(basename "$src").o"
137-
zig cc -target "$ZIG_TARGET" $CFLAGS -c "$src" -o "$out" &
138-
i=$((i + 1))
139-
[ $((i % JOBS)) -eq 0 ] && wait
140-
done
141-
wait
134+
# xargs -P keeps all $JOBS slots saturated (a fixed-size batch-then-wait loop
135+
# idles every other core behind the batch's slowest TU) and, unlike bare `&` +
136+
# `wait`, actually propagates a failing compile's exit status - xargs -P exits
137+
# non-zero if any invocation does, which set -e then catches. A silently
138+
# swallowed failed zig cc would otherwise surface only as a missing .o and a
139+
# cryptic link error, or worse, a link that "succeeds" with a stale .o left
140+
# over from a previous run.
141+
compile_one() {
142+
zig cc -target "$ZIG_TARGET" $CFLAGS -c "$1" -o "$WORK/$(basename "$1").o"
143+
}
144+
export -f compile_one
145+
export ZIG_TARGET CFLAGS WORK
146+
147+
printf '%s\n' $SRCS | xargs -P "$JOBS" -I{} bash -c 'compile_one "$@"' _ {}
142148

143149
# Link the shared library. zstd.h marks the public API with ZSTDLIB_VISIBLE,
144150
# so -fvisibility=hidden keeps everything else internal.

0 commit comments

Comments
 (0)