Skip to content

Explore consensus scenarios with the Lean model - #8778

Open
Amaury Chamayou (achamayou) wants to merge 31 commits into
achamayou-symmetrical-lampfrom
achamayou-model-coverage
Open

Amaury Chamayou (achamayou) wants to merge 31 commits into
achamayou-symmetrical-lampfrom
achamayou-model-coverage

Conversation

@achamayou

Copy link
Copy Markdown
Member

cft_model_driver links the consensus scenarios of #8773 with the Lean model of the protocol. It replays each scenario against the model in the process which runs it, and it explores changes to the scenarios, keeping the runs in which the model or cft.h does something which no scenario makes it do.

It adds 23 scenarios: most came from that exploration or reproduce what it found, and the rest show what the driver could not do before. With them, the scenarios show 78 of the 79 outcomes which replay tells apart in the model, where those of #8773 show 66, and execute 87.8% of the lines and 65.7% of the branches of cft.h, where those of #8773 execute 82.6% and 62.1%.

An exploration of 20,000 changed scenarios now ends without a difference between cft.h and the model. The Lean workflow explores 5,000 changed scenarios with a fixed seed, and fails on such a difference, or if the scenarios stop showing an outcome which they showed. The seed is fixed so that a pull request fails only for what it changes; exploring with new seeds each night is left to a later pull request.

Stacked on #8773.

Changes to consensus behaviour

Five commits change cft.h or the model, and not only tests and tools. Each resolves a disagreement between the two which the exploration found.

  1. A primary refuses a configuration with no nodes (6b7ff8449, cft.h). A transaction which removes the last trusted node of a service leaves a configuration in which nothing can be committed. The primary now throws before it appends that transaction or sends it to another node. The node still stops with a fatal error, as it did before, but the entry no longer reaches its ledger or the other nodes.
  2. The model orders retirement confirmations as CCF does (12908658b, model only). When one commit confirms the retirement of a primary and of other nodes, CCF forgets the nodes with a lower identifier before the primary nominates a successor, and not those with a higher one. The model now nominates among the same nodes. cft.h is unchanged.
  3. A follower acknowledges an AppendEntries which ends before entries of a later term (f77674c5a, cft.h). The follower threw version must not move backwards from its view history, logged it, and sent no response. It now acknowledges, as the model does.
  4. A follower records a new term at the first entry which it receives in it (8dccd7741, cft.h). A follower which held transactions after its last signature recorded a later term as starting at the first of them. It then reported a wrong transaction ID for those transactions, or threw as in 3 and went on reporting the term of an entry which it had replaced. It now records the term where the AppendEntries starts, as it already did when one of the entries was a signature.
  5. A node forgets the nodes which only a rolled back configuration added (d29a76ad2, cft.h). A node kept every node which a configuration had named until the retirement or the deletion of that node was committed, so a rollback removed a configuration and not the nodes which it had added. A primary went on sending AppendEntries to such a node, listed it in GET /node/consensus, and could nominate it as its successor. A rollback now forgets the nodes which no configuration that remains names.

3 and 4 need a node which is elected while it holds a signature and has committed nothing, so a running service should not reach them. 5 needs the transaction which trusts a node to be rolled back on a node which is or becomes primary.

Replay accepts two more differences as they are, each as the loss of messages, which the model allows:

  • A primary which joined from a snapshot sends nothing to a node which retired before that snapshot and whose retirement is not confirmed, where the model does (f951558e0).
  • A node ignores a nomination until its election timer has started, which is when it holds a signature while it is in its latest configuration, where the model has no timer (86b70d659).

Alternatives

  • For 1, the primary could roll the transaction back and stay primary, which would keep a single-node service running. That rollback stays in the same view, so its sequence numbers are used again, and a TxID already returned for a later transaction could report Committed for a different one. Stepping down after the rollback avoids that, at the cost of an election and of a step which the model does not have.
  • For 2, set_retired_committed could forget every node which it is given before the primary nominates, which would make the order of the identifiers irrelevant. That changes cft.h, which this leaves alone.
  • For 5, the model could keep such nodes as peers. The peers would become state which every handler and proof maintains, to describe behaviour which nobody chose.

Safety and compatibility

  • 1 only changes a primary. A node which is not primary still adopts the configuration which its primary sends, so a backup behaves as before beside a primary without this change.
  • 2 changes findSuccessor into bestSuccessor over the eligible peers, with the same meaning. The safety proofs need one new lemma, the proof of deadlock freedom another, and the proof in lean/mathlib-equivalence unfolds one more name.
  • 3 removes an exception on a path where the entries were already handled. The view history is only left unchanged when it already records the term of the last index.
  • 4 changes the index at which a follower records a term only when a primary kept transactions after its last signature when it was elected. The messages of the protocol are unchanged.
  • 5 only removes nodes from all_other_nodes, which no quorum and no vote reads: those count the active configurations. A node which a remaining configuration names, or whose retirement is not confirmed, stays known. No end-to-end test covers it: a unit test and a scenario do.
  • Earlier in this branch, cft.h gained the trace event recv_append_entries_response_unknown_node, and the scenario driver starts the election timer of a node when a real node would.

consensus-replay --outcomes names what each replayed model action
does, as a branch of a handler in Model/Node.lean or an effect which
handlers share, and reports the outcomes which a set of traces shows
and those which none shows.

The Lean workflow adds the report for the scenario traces to the
summary of its replay job.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
dispatchNth and dropNth deliver or drop the message at a position
among those undelivered on a link, where dispatchSingle delivers the
oldest. pendingMessages lists them, as the packets of a trace.

The reordered_messages scenario uses them to reach two cases which
no scenario showed: a follower ignores entries below its commit
index, and a primary ignores an acknowledgement below its match
index.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The driver read the entries of an AppendEntries from the ledger of
the sender when the message was delivered. If the sender had rolled
back by then, the receiver got its newer entries, or the message was
abandoned. The host of a real node reads them when the message is
sent.

The driver now queues each AppendEntries with its entries as it is
sent. No existing trace changes. The delayed_append_entries scenario
delivers such a message late, and fails without this change.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
startCapture starts the driver with its trace kept in memory, and
takeTrace returns the records emitted since the last call, as JSON.
A Lean program can then see what each command did, in the process
which issued it.

FluentdSink exports records only over TCP, so the driver points it
at a local listener, which a thread reads. takeTrace waits for a
marker record, because the sink sends from another thread.

cft_driver takes a file after the scenario name, and writes the
trace to it. For 58 of the 59 scenarios that file equals the trace
which the scenario test collects. The trace of nominate_successor
varies between runs, by either route.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A Session replays the records of each driver command as they
arrive, and keeps the model state between commands, so that a
program which issues commands can follow the model as it goes.

consensus-replay --by-command replays a saved trace in this way,
and exits with status 3 if replaying the whole trace gives another
result. The Lean workflow replays the scenario traces and the
fixtures both ways.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
cft_model_driver links the scenario driver with the Lean model of the
protocol. It runs a scenario and replays its trace, whole and by command,
or runs commands chosen from the model: it replays each command as soon
as the driver has run it, requires each undelivered message of the
driver to be one which the model holds, delivers the messages of the
model newest first, and asserts its commit indexes of the driver.

It is a Lake package, tests/cft_model, which requires the scenarios and
the consensus model. CMake builds it only with -DCFT_MODEL_DRIVER=ON,
because lake then builds the model, and fetches the formatter which the
package of the model requires. cft_model_test runs every scenario and
the run in step, in the Lean workflow.

The main of the scenarios moves to Main.lean, so that another program
can import Scenarios.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Two steps of the consensus scenarios ran no model action in replay, so
no trace showed their outcomes.

A primary which finds no successor traces a nomination and nothing
else. Replay now runs the nomination of the model when the snapshot of
a nomination is of a primary, so the model must find no successor
either.

CCF returned from an AppendEntries response of an unknown node before
tracing it, because the event records the indexes which it keeps for
the node. It now traces recv_append_entries_response_unknown_node, and
replay delivers the response to the model, which decides by itself
whether it knows the sender.

The scenarios now show 69 of the 79 outcomes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A scenario could deliver the messages of a node in order, or pick one
to deliver or drop, but not delay one past a command such as
dispatch_all, which delivers every message.

hold_nth sets an undelivered message aside, where no other command
delivers or drops it, and dispatch_held delivers it, however late. The
model keeps the message in its network until then, so replay needs no
change. held_messages and undelivered_messages list such messages with
their sender and target, for a program which chooses among them.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
cft_model_driver --explore takes the commands of each scenario from its
trace, changes them at random, and runs each changed program in a
process of its own, which checks the run against the model. A change
holds a message until later, loses or delivers one early, advances the
clock of a node, disconnects a node for a while, nominates, replicates,
or leaves out, repeats or swaps commands.

A run is kept, to be changed in turn, if the model shows an outcome, or
the run reaches an edge of cft.h, which no run before it has, and does
so again when it is run a second time. The driver counts the edges of
aft::Aft with -fsanitize-coverage. A run in which the driver fails, or
the model rejects the trace, is a finding, reported once for each
message, numbers aside.

--record writes the commands of a scenario as a program, and --run runs
a program. --shrink leaves out the steps which a finding or an outcome
does not need, and --scenario writes a program as a scenario in Lean.
cft_model_test also runs each scenario from its recorded commands,
which must show the same outcomes, and explores briefly.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Seven scenarios, each of which makes the model show an outcome which
no scenario showed: an acknowledgement, a vote and a request for a vote
from a term which their recipient has left, the answer to a request
for a pre-vote which reaches a candidate, a vote refused only because
the primary of the term is known, a nomination of a node which has no
signed configuration, and an election won by the only member of a
configuration.

cft_model_driver --explore found a run for each, --shrink left out the
steps which the outcome does not need, and --scenario wrote the rest in
Lean, with assertions of the state in which it ends. Two were then
written to end with one primary, so that every run of them gives the
same trace. The scenarios now show 76 of the 79 outcomes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The driver started the election timer of each node as it created the
node. A real node starts it when it first holds a signature while it is
in its latest configuration. A scenario could therefore have a node
campaign which no real node would, and the model rejects such a trace.
An exploration of 20,000 changed scenarios did so in 125 of them.

No scenario changes: 65 of the 66 traces are identical, and that of
nominate_successor varies between runs as it did before.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The signature which follows the removal of a node is rolled back on
that node, and the removal is kept, so its retirement goes from Signed
back to Ordered. No scenario did this, and exploration did not find it
in 80,000 runs: it needs a primary which keeps an unsigned removal, and
so an election before anything has committed.

The scenarios now show 77 of the 79 outcomes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
`--shrink PROGRAM OUTPUT --edges EDGES` leaves out steps of a program for as long as the model accepts its run, and the run reaches every edge which it reached at first and EDGES lack. An exploration writes the edges which the scenarios reach to `scenarios.edges`, for that use.

Compile the model driver without inlining, so that the edges counted are those of cft.h only. With inlining, 8 of the 12 runs which a 20,000-run exploration kept for their edges added no line or branch of cft.h; without it, 1 of 4.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Three runs which an exploration kept for the edges they reach each take a branch of cft.h which no scenario took:

- stale_nack_after_ack: a rejection below the match index, in recv_append_entries_response
- commit_index_beyond_batch: no committable index up to the one given, in commit_if_possible
- nominate_over_retiring_node: a tie in match index broken by configuration, in find_successor

The scenarios now miss 310 of the 859 branches of cft.h, from 313.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A transaction which retires the last trusted node of a service leaves a configuration with no nodes. No quorum of it exists, so nothing can be committed once it is active: update_commit throws std::out_of_range on it. A sole primary threw from replicate after it had appended the entry and passed it to its ledger. A primary with other nodes sent them the entry: in a driver run with three nodes, every node adopted it and none committed anything after it.

add_configuration now throws on a primary which is given an empty configuration, before it changes anything, so replicate fails before the entry is appended or sent. The node still stops, because the KV treats a failure to replicate a transaction which it has applied as fatal. Rolling the transaction back instead would stay in the same view, and let its sequence number be used again. Other nodes still adopt what their primary sends.

A single-node service stopped with 'Failed to serialise committed transaction' when its node was removed, before this change and after it. Before it, the ledger file of the node held the first 4096 bytes of the entry which removes it. After it, the node logs why it refuses, and writes no part of that entry.

The model already disables this append, so an explored run which asks for it is now set aside, where it was a finding.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
When a commit confirms the retirement of a primary, CCF nominates a successor. set_retired_committed takes the nodes which a transaction confirms in the order of their identifiers, and forgets each node other than itself as it does. So the primary no longer knows the nodes which were confirmed before it: by an earlier transaction of the commit, or by the same transaction with a lower identifier. It still knows, and may nominate, a node which the same transaction confirms with a higher identifier.

The model nominated among every peer which the primary knew before the commit. It disagreed with CCF when a node confirmed before the primary had acknowledged more than the nodes which stay. advanceCommit now nominates with nominateOnRetirement, which leaves out retirementsConfirmedBefore. cft.h is unchanged.

findSuccessor is now bestSuccessor over the eligible peers, and means what it did. The proof that it equals its definition with Mathlib unfolds one more name.

There is a scenario for each order of the identifiers. The model rejected the trace of retirement_forgets_lower_node before this change. It accepts that of retirement_nominates_higher_node before and after it.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
After a node has handled an AppendEntries which does not end in a signature, it records that the term of the last index starts after its last signature. If the node still held entries of a later term beyond that index, the view history refused the record with 'version must not move backwards'. recv_message logged the exception, and the node sent no response until an AppendEntries replaced those entries.

The node now records nothing when it already knows the term of the last index, and acknowledges, as the model does. It keeps the later entries until the sender replaces them.

This needs a primary which kept entries after its last signature when it was elected. become_leader only allows that while nothing is committed, so a running service should not get there. It was found by exploring changes to the scenarios, and heartbeat_before_stale_suffix fails without this change.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
cft_model_test runs each scenario again from its recorded commands, and required both runs to show the same model outcomes. The election timeout of a node differs between processes, so they need not: retirement_nominates_most_replicated leaves node 3 without a primary for 60 ms, and node 3 started an election in about 1 pair of runs in 40, which failed the test.

The program must now be accepted by the model and issue every recorded command, the assertions of the scenario among them.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
trustNode, trustNodes, swapNode and swapNodes replicated a configuration of every node which the driver had created, less those which the command removes. The next such command therefore added back a node which an earlier one had removed. CCF does not do that: a node which has retired is never trusted again. These commands now change the latest configuration which the primary holds.

No scenario relied on this: the trace of each has the same commands and configurations as before. A scenario which adds a node after one has retired did, as exploration with joins does.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
trustNodeFromSnapshot creates a node as trustNode does, but the node starts as a real node does from a snapshot which the primary took at its commit index: at that index, in its term, with the configuration of that index. It also holds the entries before the snapshot, like a node which was given the ledger files of the primary, so that the driver can still compare whole ledgers.

The scenario join_from_snapshot shows the outcome joinFromSnapshot of the model, which no trace showed. The scenarios now show 78 of its 79 outcomes.

The explorer adds a node at random, empty or from a snapshot, and changes a trust_node of a scenario into one from a snapshot.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
After a follower has handled an AppendEntries which does not end in a signature, and whose term is new to it, it recorded that term as starting after its last signature. That is wrong when the follower holds transactions after that signature in an earlier term. It then reported those transactions in the new term. If it had just replaced entries of a later term, the view history refused the record with 'version must not move backwards': recv_message logged the exception, and the follower sent no response and went on reporting the replaced term for the new entry.

The follower now records the term as starting at the first entry of the AppendEntries, as it already does when one of its entries is a signature. The two indexes are the same unless a primary kept transactions after its last signature when it was elected.

become_leader only allows that if the node has committed nothing, so a running service should not get there. It was found by exploring changes to the scenarios, and unsigned_entry_replaces_stale_suffix fails without this change. In 10,000 explored runs the indexes differ in 544 of 4,617 such records, each in the term of a primary which kept such transactions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
After each command, the scenario driver now also asserts of every node that:
- its view history gives each entry of its ledger the term which the ledger records for it, and it reports its commit with that transaction ID;
- what it reports of the primary, of whether it can replicate, and of whether it should sign, agrees with its role and its ledger;
- it has notified its commit observer of each advance of its commit index;
- if it is primary, it has asked for retired nodes to be cleaned up since it had a retirement to confirm.

The trace of a scenario does not show these, so replay against the model cannot check them. The driver now sets the two callbacks of a node, and asserts the same once it has confirmed that the network can make progress. The model driver reports a failure of these as an invariant.

The first fails in retirement_signature_rollback without the previous commit: a node reports 4.4 for the entry which its ledger holds as 3.4. Each of them fails in at least two scenarios when cft.h is changed not to do what it checks.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
cleanupRetired replicates what a real primary does when its consensus asks it to clean up retired nodes: it confirms the retirement of each node whose retirement that primary has committed, and has yet to confirm. cleanupNodes still confirms the nodes which a scenario names.

In retirement_confirmed_by_new_primary, a primary commits a retirement and loses its role before it confirms it, and the node which is elected confirms it.

The explorer can now retire a member of the latest configuration, and confirm the retirements which the primary has committed, wherever it changes a scenario. With the choice of adding a node, it can change the membership of any scenario.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
After the last command of a scenario, the driver confirms that the
network can still make progress: it signs on the latest primary, and
runs until the active nodes hold the same ledger. A node which that
primary does not know is sent nothing, and can never catch up.

The driver now leaves such a node out, as it leaves out a node which
has retired. It still waits for a node which has retired without
learning of it, which the primary knows until the retirement is
confirmed.

No scenario changes. In an exploration of 20,000 changed scenarios,
398 runs ended without this confirmation because the nodes did not
agree. With this change and the three which follow, 19 do.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A node keeps the nodes to which a primary sends in all_other_nodes. It
added a node when a configuration named it, and removed one only once
its retirement or its deletion was committed. A rollback removed a
configuration, and not the nodes which that configuration had added.

A node which is elected discards what follows its last signature. If
that includes a configuration which adds a node, the new primary went
on sending AppendEntries to a node which none of its configurations
named, listed it in /node/consensus, and could nominate it as its
successor. Quorums and votes count the configurations only, so what
was committed stayed safe.

A rollback now forgets the nodes which only the configurations rolled
back named. A node whose retirement is not confirmed stays known:
retirement is final, so no later configuration names it.

The consensus model reads the peers of a node from its log, and
cft_model_driver --explore found the difference in 152 of 20,000
changed scenarios. The scenario rollback_forgets_added_node and a unit
test fail without the change.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A node which joins from a snapshot is given the configuration at that
snapshot. Once elected, it sends neither entries nor a nomination to a
node which retired before the snapshot and whose retirement is not
confirmed. Every other node still knows that node, and so does the
consensus model, which reads the peers of a node from its whole log.

Replay now records the index at which each node starts, and treats what
the model sends to such a node as lost, which the model allows. Only
entries and nominations are excused, and only from a node to a node
which a configuration before its snapshot had removed.

cft_model_driver --explore found the difference in 3 of 20,000 changed
scenarios. The scenario join_from_snapshot_after_retirement shows it.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
CCF starts the election timer of a node when it appends or receives a
signature while it is in its latest configuration, and never stops it.
Until then the node ignores a nomination. The consensus model has no
timer: a node campaigns if a signature covers an active configuration
which names it, or if its retirement is committed and not confirmed.

So a node which is added and removed between two signatures, or which
retires before it first signs, ignores a nomination for which the model
campaigns. Replay now treats a nomination which CCF ignores as lost if
the model would campaign and the log of the node holds no signature
which found it in its latest configuration. Otherwise it delivers the
nomination, and the model must ignore it too.

cft_model_driver --explore found the difference in 1 of 20,000 changed
scenarios. The scenarios nominate_node_removed_before_signature and
nominate_node_retired_before_signature show the two cases.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
cft_model_driver --explore changes the consensus scenarios at random,
and reports as a finding a run in which the driver fails or the model
rejects the trace. It exited with status 0 whatever it found, and
cft_model_test explored 40 changed scenarios only to check that
exploring works.

The exploration now exits with status 1 if a run is a finding, and
writes the program of each finding to its error output, so that the
finding can be run again from the log of a test. cft_model_test fails
on a finding as a result.

With a rollback which no longer forgets the nodes of the configurations
which it removes, 2,000 runs with the default seed include 9 of one
finding.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The capture job of the Lean workflow builds cft_model_driver and runs
every scenario with the consensus model. It now also explores 2,000
changed scenarios with a fixed seed, and fails if the implementation
and the model disagree in one of them. It uploads the findings when it
fails.

The same seed gives the same changes, so the step tells whether a
change breaks what those runs reach, and looks for nothing new. In a
Debug build like that of the job, the exploration takes 87 seconds with
8 runs at once.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The replay job reports which outcomes of the model the traces of the
scenarios show. A scenario still passes when a change makes it stop
showing the outcome for which it was written, and 11 outcomes are shown
by one scenario only.

consensus-replay --outcomes takes --not-shown and the names of
outcomes, and fails unless the outcomes which no trace shows are those
named. The replay job names the one outcome which no scenario shows,
so that losing another fails it, and so does showing that one without
removing its name.

30 captures of the 82 scenarios in each of a Debug and a RelWithDebInfo
build all show the same 78 outcomes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
On a runner, the 2,000 changed scenarios which the capture job explored
take 33 seconds, in a job of 4.2 minutes. The longest job of the
workflow takes 5.5 minutes.

The job now explores 5,000. The seed is fixed, so these are the same
runs each time, and more of them check more cases on every pull request
which changes consensus. In a Debug build on a development machine they
take 184 seconds with 8 runs at once, and none is a finding.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@achamayou
Amaury Chamayou (achamayou) requested a review from a team as a code owner November 15, 2026 19:05

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant