Skip to content

s3: a PUT is visible half-written, so a reader mid-PUT sees a partial object #20

Description

@pi0x

Low severity, and not a regression. It costs liveness, not correctness: 4 failed
requests in 100 deliberately raced cold activations, zero wrong answers, zero divergence,
zero split-brain. The behaviour predates 7316395 and is already documented.

#putObject writes the object in place (src/s3/session.ts:1429, #writeObject): the
destination is opened w/wx and the body is streamed into it, with no temporary file and
no rename. So a GET that arrives mid-PUT reads a partial object — and for a create the
partial state is a zero-byte file, which is not a truncated document but an empty one.

This is not new and it is not a regression: session.ts:43 already states it ("a reader
arriving mid-PUT can see a partial object"), and it is documented on the docs page as the
write-in-place caveat. What is new is that #19's conditional PUT makes the window easy to
hit, because a CAS loser's very next move is always to read the key it just lost.

Real S3 makes a PUT atomic for readers, so a client written against S3 has no reason to
expect this.

Observed

A celld 0.2.0 fleet (self-hosted Durable Objects) over mountx/s3 on a node-fs driver, two
nodes racing 100 fresh cells. With 7316395 the fence itself is correct — 96 412s, one owner
per epoch, zero "orphaned epoch" trips, zero DurabilityUnproven — and 4 requests in 100
still fail like this:

route failed: ResolveFailed
celld ownership read failed: decode s3://cells/cells/Counter:5a8beec…/own.json:
  EOF while parsing a value at line 1 column 0

The loser takes its 412, immediately re-reads own.json, and catches the winner between
open(path, "wx") and its first write. Liveness only: zero wrong answers, zero divergence,
zero split-brain.

The fix, and the part that surprises

Stage the body under the reserved prefix, then commit. .mountx-multipart/ is already hidden
from listings, already answers GET/HEAD/PUT/DELETE as absent, and is already swept by
close(), so a tmp-<hex> file there inherits all of it and crash debris is handled.

rename() alone does not fix the case above. Those races are If-None-Match: * creates,
and rename clobbers unconditionally, so committing a create by rename throws away the CAS
that #19 just bought. It is two commits over one staging path:

path commit capability memory node-fs unstorage
If-None-Match: * create link() hardlinks yes yes no
replace (If-Match, plain) rename() atomicRename yes yes no

link() is atomic and answers EEXIST when the key is taken, which is the same 412 the
exclusive open produces today. hardlinks is inferred from the presence of link
(src/harness.ts:43), so both real drivers get this without declaring anything, and
unstorage — whose rename is a copy followed by a delete, and which has no link — opts
out of both and keeps today's behaviour, correctly.

#writeObject is the single funnel for PutObject, CopyObject and
CompleteMultipartUpload, so Complete stops being visible half-assembled as a side effect.
UploadPart stays in place: a staged part is already invisible.

What it costs

  • Mode and ownership. An in-place write preserves the destination's mode and owner; a
    rename replaces the inode, so chmod 600 secret.txt followed by a PUT resets the bits.
    That matters because the same tree is served over FUSE and NFS. Needs a stat-then-chmod
    of the temp before the commit, gated on permissions — and chown needs root, so the
    ownership half may not be recoverable at all.
  • Two permanent write paths, which is what the module doc currently talks itself out of.
    Every write test doubles: staged for memory/node-fs, in place for unstorage.
  • 2× peak space for the length of a large PUT, and an EXDEV fallback for a node-fs
    root that spans a mount point.
  • The prose. "PUT writes the object in place" is stated as a decision in
    src/s3/session.ts:43, src/webdav/session.ts:142 and docs/2.transports/5.s3.md; it
    becomes capability-conditional in all three.

One thing improves for free: the ETag is sha256(dev:ino:size:mtimeMs), and a new inode per
write means an overwrite can no longer produce the ETag the old content had. Today two
same-size writes inside one millisecond collide.

mountx/webdav has the same PUT shape and would want the same treatment; it is out of
scope here.

Follow-up to #19.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions