Low severity, and not a regression. It costs liveness, not correctness: 4 failed
requests in 100 deliberately raced cold activations, zero wrong answers, zero divergence,
zero split-brain. The behaviour predates 7316395 and is already documented.
#putObject writes the object in place (src/s3/session.ts:1429, #writeObject): the
destination is opened w/wx and the body is streamed into it, with no temporary file and
no rename. So a GET that arrives mid-PUT reads a partial object — and for a create the
partial state is a zero-byte file, which is not a truncated document but an empty one.
This is not new and it is not a regression: session.ts:43 already states it ("a reader
arriving mid-PUT can see a partial object"), and it is documented on the docs page as the
write-in-place caveat. What is new is that #19's conditional PUT makes the window easy to
hit, because a CAS loser's very next move is always to read the key it just lost.
Real S3 makes a PUT atomic for readers, so a client written against S3 has no reason to
expect this.
Observed
A celld 0.2.0 fleet (self-hosted Durable Objects) over mountx/s3 on a node-fs driver, two
nodes racing 100 fresh cells. With 7316395 the fence itself is correct — 96 412s, one owner
per epoch, zero "orphaned epoch" trips, zero DurabilityUnproven — and 4 requests in 100
still fail like this:
route failed: ResolveFailed
celld ownership read failed: decode s3://cells/cells/Counter:5a8beec…/own.json:
EOF while parsing a value at line 1 column 0
The loser takes its 412, immediately re-reads own.json, and catches the winner between
open(path, "wx") and its first write. Liveness only: zero wrong answers, zero divergence,
zero split-brain.
The fix, and the part that surprises
Stage the body under the reserved prefix, then commit. .mountx-multipart/ is already hidden
from listings, already answers GET/HEAD/PUT/DELETE as absent, and is already swept by
close(), so a tmp-<hex> file there inherits all of it and crash debris is handled.
rename() alone does not fix the case above. Those races are If-None-Match: * creates,
and rename clobbers unconditionally, so committing a create by rename throws away the CAS
that #19 just bought. It is two commits over one staging path:
| path |
commit |
capability |
memory |
node-fs |
unstorage |
If-None-Match: * create |
link() |
hardlinks |
yes |
yes |
no |
replace (If-Match, plain) |
rename() |
atomicRename |
yes |
yes |
no |
link() is atomic and answers EEXIST when the key is taken, which is the same 412 the
exclusive open produces today. hardlinks is inferred from the presence of link
(src/harness.ts:43), so both real drivers get this without declaring anything, and
unstorage — whose rename is a copy followed by a delete, and which has no link — opts
out of both and keeps today's behaviour, correctly.
#writeObject is the single funnel for PutObject, CopyObject and
CompleteMultipartUpload, so Complete stops being visible half-assembled as a side effect.
UploadPart stays in place: a staged part is already invisible.
What it costs
- Mode and ownership. An in-place write preserves the destination's mode and owner; a
rename replaces the inode, so chmod 600 secret.txt followed by a PUT resets the bits.
That matters because the same tree is served over FUSE and NFS. Needs a stat-then-chmod
of the temp before the commit, gated on permissions — and chown needs root, so the
ownership half may not be recoverable at all.
- Two permanent write paths, which is what the module doc currently talks itself out of.
Every write test doubles: staged for memory/node-fs, in place for unstorage.
- 2× peak space for the length of a large
PUT, and an EXDEV fallback for a node-fs
root that spans a mount point.
- The prose. "
PUT writes the object in place" is stated as a decision in
src/s3/session.ts:43, src/webdav/session.ts:142 and docs/2.transports/5.s3.md; it
becomes capability-conditional in all three.
One thing improves for free: the ETag is sha256(dev:ino:size:mtimeMs), and a new inode per
write means an overwrite can no longer produce the ETag the old content had. Today two
same-size writes inside one millisecond collide.
mountx/webdav has the same PUT shape and would want the same treatment; it is out of
scope here.
Follow-up to #19.
Low severity, and not a regression. It costs liveness, not correctness: 4 failed
requests in 100 deliberately raced cold activations, zero wrong answers, zero divergence,
zero split-brain. The behaviour predates 7316395 and is already documented.
#putObjectwrites the object in place (src/s3/session.ts:1429,#writeObject): thedestination is opened
w/wxand the body is streamed into it, with no temporary file andno rename. So a
GETthat arrives mid-PUTreads a partial object — and for a create thepartial state is a zero-byte file, which is not a truncated document but an empty one.
This is not new and it is not a regression:
session.ts:43already states it ("a readerarriving mid-
PUTcan see a partial object"), and it is documented on the docs page as thewrite-in-place caveat. What is new is that #19's conditional
PUTmakes the window easy tohit, because a CAS loser's very next move is always to read the key it just lost.
Real S3 makes a
PUTatomic for readers, so a client written against S3 has no reason toexpect this.
Observed
A celld 0.2.0 fleet (self-hosted Durable Objects) over
mountx/s3on anode-fsdriver, twonodes racing 100 fresh cells. With 7316395 the fence itself is correct — 96
412s, one ownerper epoch, zero "orphaned epoch" trips, zero
DurabilityUnproven— and 4 requests in 100still fail like this:
The loser takes its
412, immediately re-readsown.json, and catches the winner betweenopen(path, "wx")and its firstwrite. Liveness only: zero wrong answers, zero divergence,zero split-brain.
The fix, and the part that surprises
Stage the body under the reserved prefix, then commit.
.mountx-multipart/is already hiddenfrom listings, already answers
GET/HEAD/PUT/DELETEas absent, and is already swept byclose(), so atmp-<hex>file there inherits all of it and crash debris is handled.rename()alone does not fix the case above. Those races areIf-None-Match: *creates,and
renameclobbers unconditionally, so committing a create by rename throws away the CASthat #19 just bought. It is two commits over one staging path:
If-None-Match: *createlink()hardlinksIf-Match, plain)rename()atomicRenamelink()is atomic and answersEEXISTwhen the key is taken, which is the same412theexclusive open produces today.
hardlinksis inferred from the presence oflink(
src/harness.ts:43), so both real drivers get this without declaring anything, andunstorage— whoserenameis a copy followed by a delete, and which has nolink— optsout of both and keeps today's behaviour, correctly.
#writeObjectis the single funnel forPutObject,CopyObjectandCompleteMultipartUpload, soCompletestops being visible half-assembled as a side effect.UploadPartstays in place: a staged part is already invisible.What it costs
rename replaces the inode, so
chmod 600 secret.txtfollowed by aPUTresets the bits.That matters because the same tree is served over FUSE and NFS. Needs a stat-then-
chmodof the temp before the commit, gated on
permissions— andchownneeds root, so theownership half may not be recoverable at all.
Every write test doubles: staged for
memory/node-fs, in place forunstorage.PUT, and anEXDEVfallback for anode-fsroot that spans a mount point.
PUTwrites the object in place" is stated as a decision insrc/s3/session.ts:43,src/webdav/session.ts:142anddocs/2.transports/5.s3.md; itbecomes capability-conditional in all three.
One thing improves for free: the ETag is
sha256(dev:ino:size:mtimeMs), and a new inode perwrite means an overwrite can no longer produce the ETag the old content had. Today two
same-size writes inside one millisecond collide.
mountx/webdavhas the samePUTshape and would want the same treatment; it is out ofscope here.
Follow-up to #19.