Skip to content

Commit a82a23a

Browse files
HarperZ9claude
andcommitted
fix(gateway): a ledger that closes while failed tries once more to record the gap
Seen when the reviewer's probe was run against the branch: the gateway stopped inside the one-second retry interval, so the ledger held no `ledger_gap` entry for the append that had failed. `close` now calls `gateway_ledger_health.settle`, which resumes once if the disk allows it and logs what stays unrecorded if it does not. The ledger tests are two files now, each under 300 lines: tests/test_gateway_ledger_write_failure.py (D2) and tests/test_gateway_ledger_live_verify.py (A1, N2, and the cut-line count). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
1 parent 43db9fd commit a82a23a

5 files changed

Lines changed: 150 additions & 84 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -231,7 +231,7 @@ Fixes
231231
- Changed: a model moves from native function calling to the text protocol only when its host's error says tools or function calling are unsupported. Any 400 that mentioned "tool" or "function" used to do it, a context-length error included, and a quoted TOOL line in the next reply then ran. The session prints one notice and records a `native_tools_fallback` entry when a model moves. No provider documents the exact sentence, so the accepted wording is a stated heuristic; `docs/KNOWN-LIMITS-2.0.0.md` has the row.
232232
- New: the request line and headers of a gateway request must arrive within one read timeout, and the body of a call to a model interface within four, with or without a credential. A client that dripped header lines used to hold a thread for as long as it liked. There is still no cap on open connections.
233233
- Changed: `flywheel gateway-ledger show` cuts a field longer than 80 characters and says how much it left out; `--full` prints it whole. The static check of the Lean modules resolves import aliases and also reports `asyncio.create_subprocess_*`, `os.posix_spawn*`, `os.startfile`, `os.fork`, `pty`, `multiprocessing`, `getattr` on those modules and an `env=` that is not exactly `lean_env(...)`; `tests/lean_spawn_scan.py` lists what it cannot see. `docs/KNOWN-LIMITS-2.0.0.md` names `/v1/embeddings` among the routes outside the ledger.
234-
- Tested over real loopback sockets with a stub upstream, and with recorded provider replies: `tests/test_gateway_ledger_write_failure.py`, `tests/test_gateway_round3_small_fixes.py`, `tests/test_code_session_native_fallback_trigger.py`, which run against the built wheel before a release with `tests/test_lean_child_env.py`. The reviewer's probe scripts were run again against the fixed code. These last fixes have had no further independent review. Not run: any hosted provider, a real full disk (the failure is simulated at the segment's file), macOS.
234+
- Tested over real loopback sockets with a stub upstream, and with recorded provider replies: `tests/test_gateway_ledger_write_failure.py`, `tests/test_gateway_ledger_live_verify.py`, `tests/test_gateway_round3_small_fixes.py`, `tests/test_code_session_native_fallback_trigger.py`, which run against the built wheel before a release with `tests/test_lean_child_env.py`. The reviewer's probe scripts were run again against the fixed code. These last fixes have had no further independent review. Not run: any hosted provider, a real full disk (the failure is simulated at the segment's file), macOS.
235235

236236
## 1.9.0 (2026-10-15)
237237

‎harness/gateway_ledger.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -262,6 +262,7 @@ def checkpoint(self) -> str:
262262

263263
def close(self) -> None:
264264
with self._lock:
265+
_health.settle(self) # a ledger that failed gets one more try to record it
265266
self._held.close()
266267

267268

‎harness/gateway_ledger_health.py‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,9 @@
1111
usable is asked by `gateway_ledger.for_home` before every model call. A
1212
failed ledger comes back as None, which every model interface
1313
answers with 503 before it contacts an upstream
14-
(gateway_ledger_gate). At most once a second it tries to resume.
14+
(gateway_ledger_gate). At most once a second it tries to resume,
15+
and once more when the ledger closes (`settle`), so a gateway that
16+
stops while failed still leaves the record when the disk allows it.
1517
resume reads the segment back from the disk. When it verifies and still
1618
begins with the entries this process holds, the gateway appends to
1719
it again, and its first entry is a `ledger_gap` that says what
@@ -114,6 +116,18 @@ def usable(ledger):
114116
return ledger
115117

116118

119+
def settle(ledger) -> None:
120+
"""Before a ledger closes: if its writes had failed, try once more to record the gap."""
121+
if ledger.failed is None:
122+
return
123+
try:
124+
resume(ledger)
125+
except (OSError, ValueError) as exc:
126+
_LOG.error("gateway ledger closed while its writes still failed (%s: %s); %d append(s) "
127+
"were refused and the ledger does not record them", type(exc).__name__, exc,
128+
ledger.failed["refused"])
129+
130+
117131
def resume(ledger) -> None:
118132
"""Append to the ledger again after a failed write, and record the gap."""
119133
from . import gateway_ledger as base
Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
"""`verify` holds on a ledger a gateway is appending to, and ignores stray files.
2+
3+
Independent review of 2.0.0, findings A1 and N2.
4+
5+
A1. `verify` reported "entries were removed" on an intact ledger while a
6+
gateway appended, because it read the segment before the head record. On a
7+
segment of some thousand entries the second read a quarter of a second later
8+
failed the same way. The head record is read first now.
9+
10+
N2. A file named like a segment that is not one (`segment-backup.jsonl`,
11+
`segment-000001 - Copy.jsonl`) stopped the gateway at start with an uncaught
12+
error, or failed `verify` on an intact ledger. Such a file is ignored and
13+
named.
14+
"""
15+
import threading
16+
import time
17+
18+
import pytest
19+
20+
from harness import gateway_ledger, gateway_ledger_cli, gateway_ledger_gate
21+
22+
pytestmark = pytest.mark.release_gate # run against the built wheel before a release
23+
24+
25+
@pytest.fixture
26+
def home(monkeypatch, tmp_path):
27+
where = tmp_path / "home"
28+
where.mkdir()
29+
monkeypatch.setenv("FLYWHEEL_HOME", str(where))
30+
yield where
31+
gateway_ledger.close_all()
32+
33+
34+
@pytest.mark.parametrize("name", ["segment-backup.jsonl", "segment-000001 - Copy.jsonl",
35+
"segment-0000001.jsonl", "segment-000002.jsonl.bak"])
36+
def test_a_stray_file_in_the_ledger_folder_is_ignored_and_named(home, name, capsys):
37+
ledger = gateway_ledger.for_home(home)
38+
ledger.append("endpoint_call", "x", {})
39+
gateway_ledger.close_all()
40+
folder = home / "gateway-ledger"
41+
(folder / name).write_bytes((folder / "segment-000001.jsonl").read_bytes())
42+
found = gateway_ledger.check(home)
43+
assert [(s["name"], s["ok"]) for s in found] == [("segment-000001.jsonl", True)]
44+
assert gateway_ledger_cli.main(["verify"]) == 0
45+
assert f"ignored {name!r}" in capsys.readouterr().out
46+
lines = []
47+
assert gateway_ledger_gate.startup(lambda: gateway_ledger.for_home(home), out=lines.append)
48+
assert any("ignored" in line and name in line for line in lines), lines
49+
gateway_ledger.for_home(home).append("endpoint_call", "y", {}) # and the chain goes on
50+
gateway_ledger.close_all()
51+
assert [s["entries"] for s in gateway_ledger.check(home)] == [2]
52+
53+
54+
def test_verify_holds_when_an_entry_is_appended_between_its_two_reads(home, monkeypatch):
55+
"""The forced order: the head record moves on after the segment was read."""
56+
monkeypatch.setenv(gateway_ledger.SYNC_ENV, "0")
57+
ledger = gateway_ledger.GatewayLedger(home)
58+
for index in range(5):
59+
ledger.append("endpoint_call", "x" * 64, {"n": index})
60+
real = gateway_ledger.read_segment
61+
62+
def read_then_append(path):
63+
found = real(path)
64+
ledger.append("endpoint_call", "y" * 64, {})
65+
return found
66+
monkeypatch.setattr(gateway_ledger, "read_segment", read_then_append)
67+
try:
68+
found = gateway_ledger.check(home)
69+
finally:
70+
ledger.close()
71+
assert [(s["ok"], s["problem"], s["entries"]) for s in found] == [(True, None, 5)]
72+
73+
74+
def test_verify_holds_on_a_ledger_that_a_writer_thread_appends_to(home, monkeypatch):
75+
monkeypatch.setenv(gateway_ledger.SYNC_ENV, "0")
76+
ledger = gateway_ledger.GatewayLedger(home)
77+
for index in range(4000):
78+
ledger.append("endpoint_call", "x" * 64, {"endpoint": "openai", "n": index})
79+
stop = threading.Event()
80+
81+
def writer():
82+
while not stop.is_set():
83+
ledger.append("endpoint_call", "y" * 64, {"endpoint": "openai"})
84+
time.sleep(0.002)
85+
thread = threading.Thread(target=writer)
86+
thread.start()
87+
try:
88+
problems = []
89+
for _ in range(8):
90+
found = gateway_ledger.check(home) # one read, no retry
91+
problems += [s["problem"] for s in found if not s["ok"]]
92+
monkeypatch.setattr(gateway_ledger_cli, "RETRY_AFTER", 0.01)
93+
codes = [gateway_ledger_cli.main(["verify"]) for _ in range(3)]
94+
finally:
95+
stop.set()
96+
thread.join()
97+
ledger.close()
98+
assert [p for p in problems if "removed" in p] == [], problems
99+
assert codes == [0, 0, 0]
100+
assert all(s["ok"] for s in gateway_ledger.check(home))
101+
102+
103+
def test_a_cut_line_is_named_and_the_entries_before_it_still_count(home):
104+
"""`verify` named the line before the cut one and did not count that entry."""
105+
ledger = gateway_ledger.GatewayLedger(home)
106+
for index in range(3):
107+
ledger.append("endpoint_call", f"sha{index}", {})
108+
ledger.close()
109+
path = home / "gateway-ledger" / "segment-000001.jsonl"
110+
path.write_bytes(path.read_bytes()[:-40]) # the third entry is cut
111+
found, = gateway_ledger.check(home)
112+
assert found["problem"] == "line 4 is cut short" and found["entries"] == 2

‎tests/test_gateway_ledger_write_failure.py‎

Lines changed: 21 additions & 82 deletions
Original file line numberDiff line numberDiff line change
@@ -1,35 +1,27 @@
11
"""A ledger that fails after the gateway started stops model calls, and resumes cleanly.
22
3-
Independent review of 2.0.0, findings D2, A1 and N2.
3+
Independent review of 2.0.0, finding D2.
44
5-
D2. The ledger opened at start and a later write failed (a full disk). Every
5+
The ledger opened at start and a later write failed (a full disk). Every
66
call still reached the upstream for as long as the failure lasted, and when
77
writes came back the segment held duplicate sequence numbers. Now the first
88
failed append marks the ledger failed: later calls answer 503 before any
99
upstream call, and a resumed ledger verifies and records the gap. Every model
1010
interface writes its entry after the upstream answered, so the one call that
1111
meets the failure has already reached the provider; these tests count it.
1212
13-
A1. `verify` reported removed entries on an intact ledger while a gateway
14-
appended, because it read the segment before the head record.
15-
16-
N2. A file named like a segment that is not one (`segment-backup.jsonl`,
17-
`segment-000001 - Copy.jsonl`) stopped the gateway at start or failed
18-
`verify`. Such a file is ignored and named.
19-
2013
A real gateway on a loopback port, a stub upstream, the home's on-disk ledger.
2114
The full disk is simulated at the segment's file. No provider is called.
15+
`tests/test_gateway_ledger_live_verify.py` has findings A1 and N2.
2216
"""
2317
import io
2418
import os
2519
import sys
26-
import threading
27-
import time
2820
import types
2921

3022
import pytest
3123

32-
from harness import gateway, gateway_ledger, gateway_ledger_cli, gateway_ledger_gate
24+
from harness import gateway, gateway_ledger, gateway_ledger_cli
3325
from tests import gateway_messages_support as support
3426
from tests.gateway_keys_support import ( # noqa: F401 (fixtures)
3527
BODIES, MESSAGES, keyed, post, served, stream_bytes, turn2)
@@ -206,6 +198,23 @@ def test_a_line_cut_short_by_the_failure_is_recorded_as_a_break(keyed, served, m
206198
assert "records a break" in capsys.readouterr().out
207199

208200

201+
@pytest.mark.parametrize("disk_back", [True, False])
202+
def test_a_gateway_that_stops_while_its_ledger_is_failed_records_the_gap_if_it_can(
203+
keyed, served, monkeypatch, home, disk_back):
204+
keys, _store = keyed
205+
raw, _ = keys.create("ci-bot")
206+
url, upstream, ledger, disk = _gateway(monkeypatch, served, turn2(2), home)
207+
assert post(url, "/v1/messages", raw)[0] == 200
208+
disk.full = True
209+
assert post(url, "/v1/messages", raw)[0] == 500
210+
disk.full = not disk_back
211+
gateway_ledger.close_all() # the gateway stops; no call came in between
212+
disk.full = False
213+
kinds = [kind for _name, _seq, kind in _rows(home)]
214+
assert kinds == (["endpoint_call", "ledger_gap"] if disk_back else ["endpoint_call"])
215+
assert all(segment["ok"] for segment in gateway_ledger.check(home))
216+
217+
209218
def test_the_gateway_reaches_no_provider_when_the_new_module_cannot_be_imported(
210219
keyed, served, monkeypatch, home):
211220
keys, _store = keyed
@@ -224,73 +233,3 @@ def test_the_gateway_reaches_no_provider_when_the_new_module_cannot_be_imported(
224233
status = 0
225234
assert status in (0, 500)
226235
assert upstream.requests == []
227-
228-
229-
@pytest.mark.parametrize("name", ["segment-backup.jsonl", "segment-000001 - Copy.jsonl",
230-
"segment-0000001.jsonl", "segment-000002.jsonl.bak"])
231-
def test_a_stray_file_in_the_ledger_folder_is_ignored_and_named(home, name, capsys):
232-
ledger = gateway_ledger.for_home(home)
233-
ledger.append("endpoint_call", "x", {})
234-
gateway_ledger.close_all()
235-
folder = home / "gateway-ledger"
236-
(folder / name).write_bytes((folder / "segment-000001.jsonl").read_bytes())
237-
found = gateway_ledger.check(home)
238-
assert [(s["name"], s["ok"]) for s in found] == [("segment-000001.jsonl", True)]
239-
assert gateway_ledger_cli.main(["verify"]) == 0
240-
assert f"ignored {name!r}" in capsys.readouterr().out
241-
lines = []
242-
assert gateway_ledger_gate.startup(lambda: gateway_ledger.for_home(home), out=lines.append)
243-
assert any("ignored" in line and name in line for line in lines), lines
244-
gateway_ledger.for_home(home).append("endpoint_call", "y", {}) # and the chain goes on
245-
gateway_ledger.close_all()
246-
assert [s["entries"] for s in gateway_ledger.check(home)] == [2]
247-
248-
249-
def test_verify_holds_when_an_entry_is_appended_between_its_two_reads(home, monkeypatch):
250-
"""The forced order: the head record moves on after the segment was read."""
251-
monkeypatch.setenv(gateway_ledger.SYNC_ENV, "0")
252-
ledger = gateway_ledger.GatewayLedger(home)
253-
for index in range(5):
254-
ledger.append("endpoint_call", "x" * 64, {"n": index})
255-
real = gateway_ledger.read_segment
256-
257-
def read_then_append(path):
258-
found = real(path)
259-
ledger.append("endpoint_call", "y" * 64, {})
260-
return found
261-
monkeypatch.setattr(gateway_ledger, "read_segment", read_then_append)
262-
try:
263-
found = gateway_ledger.check(home)
264-
finally:
265-
ledger.close()
266-
assert [(s["ok"], s["problem"], s["entries"]) for s in found] == [(True, None, 5)]
267-
268-
269-
def test_verify_holds_on_a_ledger_that_a_writer_thread_appends_to(home, monkeypatch):
270-
monkeypatch.setenv(gateway_ledger.SYNC_ENV, "0")
271-
ledger = gateway_ledger.GatewayLedger(home)
272-
for index in range(4000):
273-
ledger.append("endpoint_call", "x" * 64, {"endpoint": "openai", "n": index})
274-
stop = threading.Event()
275-
276-
def writer():
277-
while not stop.is_set():
278-
ledger.append("endpoint_call", "y" * 64, {"endpoint": "openai"})
279-
time.sleep(0.002)
280-
thread = threading.Thread(target=writer)
281-
thread.start()
282-
try:
283-
problems = []
284-
for _ in range(8):
285-
found = gateway_ledger.check(home) # one read, no retry
286-
problems += [s["problem"] for s in found if not s["ok"]]
287-
monkeypatch.setattr(gateway_ledger_cli, "RETRY_AFTER", 0.01)
288-
monkeypatch.setenv("FLYWHEEL_HOME", str(home))
289-
codes = [gateway_ledger_cli.main(["verify"]) for _ in range(3)]
290-
finally:
291-
stop.set()
292-
thread.join()
293-
ledger.close()
294-
assert [p for p in problems if "removed" in p] == [], problems
295-
assert codes == [0, 0, 0]
296-
assert all(s["ok"] for s in gateway_ledger.check(home))

0 commit comments

Comments
 (0)