Fix problem with large zip downloads

[stevecassidy]

Fix problem with large zip downloads

Ticket

#1983

Description

We are seeing failures with large zip downloads (Full and attachments only). Requests are timing out/failing and the zip file does not complete. Also, once this has happened, subsequent downloads will also fail, even if they are not large. Restarting the server fixes the issue until the next failure.

Proposed Changes

Diagnosed with Claude's help, there is an issue with timeouts of requests to CouchDB and the way that we handle failures. Here is it's summary:

The core false assumption

The code assumes archiver processes entries in parallel — but it doesn't. Archiver serialises its internal queue: it reads one stream fully, writes it into the ZIP, and only then moves to the next. Despite the MAX_CONCURRENT_STREAMS = 15 name, what the code actually does is:

• Call nanoDb.attachment.getAsStream() — this opens an HTTP connection to CouchDB immediately for each of the 15 entries
• Call archive.append(stream) — archiver queues all 15, then processes them one at a time in order

So you have 15 open CouchDB TCP connections, but only 1 is being read. The other 14 sit idle in archiver's queue, their connections open but unconsumed.

Initial trigger 1 — Idle connection timeout on queued streams

For a large file, archiver might spend 30-120 seconds reading a single attachment. While it does, the 14 queued streams are idle: CouchDB sent the first ~16KB (Node.js stream highWaterMark), then TCP flow control paused the rest. The TCP connection is open but silent.

Most CouchDB deployments sit behind nginx, a cloud load balancer, or CouchDB's own built-in idle timeout. Common values are 60 seconds (nginx default proxy_read_timeout) or 75 seconds. After crossing that threshold:

• The connection is reset → the stream emits 'error' (ECONNRESET/ETIMEDOUT)
  With the old code: throw in the archive.on('error', ...) handler leaves archive.finalize() permanently pending → hang
• The pattern scales badly: a dataset with 500 large photos would reliably hit timeouts during every export, while a dataset with 100 small thumbnails would usually finish before the timeout

Initial trigger 2 — A per-record burst past the concurrency limit

There's a subtlety in the loop: the inner while checks activeStreams.size < MAX_CONCURRENT_STREAMS at the top of each iteration, but processRecordAttachments can add N promises in a single iteration without the limit being rechecked mid-record. If a record has 50 photos, activeStreams.size jumps from 10 to 60 in one step — and simultaneously 50 CouchDB HTTP requests fire. For projects with many high-attachment records this compounds the idle timeout problem significantly.

Why small subsequent requests also fail (before our fix)

With the old throw in the error handler: the uncaught exception leaves the Node.js HTTP response socket in a half-closed state that is never released. More critically, the open CouchDB nano HTTP connections hold sockets against the underlying http.Agent pool. If that pool's maxSockets is reached, all new CouchDB requests queue waiting for a free socket — permanently, since the stuck sockets are never freed without a restart.

What the fixes we made actually do

They fix the consequence: once a connection is time-killed, the error is now caught, all open source streams are destroyed (releasing their sockets), and archiver and the Express response are properly wound down. The archive exits cleanly with a 5xx response instead of hanging.

How to Test

Download large zip files - unfortunately the bug is only triggered intermittently so only deployment will test whether we have a fix.

Checklist

• I have confirmed all commits have been signed.
• I have added JSDoc style comments to any new functions or classes.
• Relevant documentation such as READMEs, guides, and class comments are updated.