Repository navigation
Explore consensus scenarios with the Lean model - #8778
Open
Amaury Chamayou (achamayou) wants to merge 31 commits into
Open
Amaury Chamayou (achamayou) wants to merge 31 commits into
Amaury Chamayou (achamayou) wants to merge 31 commits into
Conversation
consensus-replay --outcomes names what each replayed model action does, as a branch of a handler in Model/Node.lean or an effect which handlers share, and reports the outcomes which a set of traces shows and those which none shows. The Lean workflow adds the report for the scenario traces to the summary of its replay job. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
dispatchNth and dropNth deliver or drop the message at a position among those undelivered on a link, where dispatchSingle delivers the oldest. pendingMessages lists them, as the packets of a trace. The reordered_messages scenario uses them to reach two cases which no scenario showed: a follower ignores entries below its commit index, and a primary ignores an acknowledgement below its match index. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The driver read the entries of an AppendEntries from the ledger of the sender when the message was delivered. If the sender had rolled back by then, the receiver got its newer entries, or the message was abandoned. The host of a real node reads them when the message is sent. The driver now queues each AppendEntries with its entries as it is sent. No existing trace changes. The delayed_append_entries scenario delivers such a message late, and fails without this change. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
startCapture starts the driver with its trace kept in memory, and takeTrace returns the records emitted since the last call, as JSON. A Lean program can then see what each command did, in the process which issued it. FluentdSink exports records only over TCP, so the driver points it at a local listener, which a thread reads. takeTrace waits for a marker record, because the sink sends from another thread. cft_driver takes a file after the scenario name, and writes the trace to it. For 58 of the 59 scenarios that file equals the trace which the scenario test collects. The trace of nominate_successor varies between runs, by either route. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A Session replays the records of each driver command as they arrive, and keeps the model state between commands, so that a program which issues commands can follow the model as it goes. consensus-replay --by-command replays a saved trace in this way, and exits with status 3 if replaying the whole trace gives another result. The Lean workflow replays the scenario traces and the fixtures both ways. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
cft_model_driver links the scenario driver with the Lean model of the protocol. It runs a scenario and replays its trace, whole and by command, or runs commands chosen from the model: it replays each command as soon as the driver has run it, requires each undelivered message of the driver to be one which the model holds, delivers the messages of the model newest first, and asserts its commit indexes of the driver. It is a Lake package, tests/cft_model, which requires the scenarios and the consensus model. CMake builds it only with -DCFT_MODEL_DRIVER=ON, because lake then builds the model, and fetches the formatter which the package of the model requires. cft_model_test runs every scenario and the run in step, in the Lean workflow. The main of the scenarios moves to Main.lean, so that another program can import Scenarios. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Two steps of the consensus scenarios ran no model action in replay, so no trace showed their outcomes. A primary which finds no successor traces a nomination and nothing else. Replay now runs the nomination of the model when the snapshot of a nomination is of a primary, so the model must find no successor either. CCF returned from an AppendEntries response of an unknown node before tracing it, because the event records the indexes which it keeps for the node. It now traces recv_append_entries_response_unknown_node, and replay delivers the response to the model, which decides by itself whether it knows the sender. The scenarios now show 69 of the 79 outcomes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A scenario could deliver the messages of a node in order, or pick one to deliver or drop, but not delay one past a command such as dispatch_all, which delivers every message. hold_nth sets an undelivered message aside, where no other command delivers or drops it, and dispatch_held delivers it, however late. The model keeps the message in its network until then, so replay needs no change. held_messages and undelivered_messages list such messages with their sender and target, for a program which chooses among them. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
cft_model_driver --explore takes the commands of each scenario from its trace, changes them at random, and runs each changed program in a process of its own, which checks the run against the model. A change holds a message until later, loses or delivers one early, advances the clock of a node, disconnects a node for a while, nominates, replicates, or leaves out, repeats or swaps commands. A run is kept, to be changed in turn, if the model shows an outcome, or the run reaches an edge of cft.h, which no run before it has, and does so again when it is run a second time. The driver counts the edges of aft::Aft with -fsanitize-coverage. A run in which the driver fails, or the model rejects the trace, is a finding, reported once for each message, numbers aside. --record writes the commands of a scenario as a program, and --run runs a program. --shrink leaves out the steps which a finding or an outcome does not need, and --scenario writes a program as a scenario in Lean. cft_model_test also runs each scenario from its recorded commands, which must show the same outcomes, and explores briefly. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Seven scenarios, each of which makes the model show an outcome which no scenario showed: an acknowledgement, a vote and a request for a vote from a term which their recipient has left, the answer to a request for a pre-vote which reaches a candidate, a vote refused only because the primary of the term is known, a nomination of a node which has no signed configuration, and an election won by the only member of a configuration. cft_model_driver --explore found a run for each, --shrink left out the steps which the outcome does not need, and --scenario wrote the rest in Lean, with assertions of the state in which it ends. Two were then written to end with one primary, so that every run of them gives the same trace. The scenarios now show 76 of the 79 outcomes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The driver started the election timer of each node as it created the node. A real node starts it when it first holds a signature while it is in its latest configuration. A scenario could therefore have a node campaign which no real node would, and the model rejects such a trace. An exploration of 20,000 changed scenarios did so in 125 of them. No scenario changes: 65 of the 66 traces are identical, and that of nominate_successor varies between runs as it did before. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The signature which follows the removal of a node is rolled back on that node, and the removal is kept, so its retirement goes from Signed back to Ordered. No scenario did this, and exploration did not find it in 80,000 runs: it needs a primary which keeps an unsigned removal, and so an election before anything has committed. The scenarios now show 77 of the 79 outcomes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
`--shrink PROGRAM OUTPUT --edges EDGES` leaves out steps of a program for as long as the model accepts its run, and the run reaches every edge which it reached at first and EDGES lack. An exploration writes the edges which the scenarios reach to `scenarios.edges`, for that use. Compile the model driver without inlining, so that the edges counted are those of cft.h only. With inlining, 8 of the 12 runs which a 20,000-run exploration kept for their edges added no line or branch of cft.h; without it, 1 of 4. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Three runs which an exploration kept for the edges they reach each take a branch of cft.h which no scenario took: - stale_nack_after_ack: a rejection below the match index, in recv_append_entries_response - commit_index_beyond_batch: no committable index up to the one given, in commit_if_possible - nominate_over_retiring_node: a tie in match index broken by configuration, in find_successor The scenarios now miss 310 of the 859 branches of cft.h, from 313. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A transaction which retires the last trusted node of a service leaves a configuration with no nodes. No quorum of it exists, so nothing can be committed once it is active: update_commit throws std::out_of_range on it. A sole primary threw from replicate after it had appended the entry and passed it to its ledger. A primary with other nodes sent them the entry: in a driver run with three nodes, every node adopted it and none committed anything after it. add_configuration now throws on a primary which is given an empty configuration, before it changes anything, so replicate fails before the entry is appended or sent. The node still stops, because the KV treats a failure to replicate a transaction which it has applied as fatal. Rolling the transaction back instead would stay in the same view, and let its sequence number be used again. Other nodes still adopt what their primary sends. A single-node service stopped with 'Failed to serialise committed transaction' when its node was removed, before this change and after it. Before it, the ledger file of the node held the first 4096 bytes of the entry which removes it. After it, the node logs why it refuses, and writes no part of that entry. The model already disables this append, so an explored run which asks for it is now set aside, where it was a finding. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
When a commit confirms the retirement of a primary, CCF nominates a successor. set_retired_committed takes the nodes which a transaction confirms in the order of their identifiers, and forgets each node other than itself as it does. So the primary no longer knows the nodes which were confirmed before it: by an earlier transaction of the commit, or by the same transaction with a lower identifier. It still knows, and may nominate, a node which the same transaction confirms with a higher identifier. The model nominated among every peer which the primary knew before the commit. It disagreed with CCF when a node confirmed before the primary had acknowledged more than the nodes which stay. advanceCommit now nominates with nominateOnRetirement, which leaves out retirementsConfirmedBefore. cft.h is unchanged. findSuccessor is now bestSuccessor over the eligible peers, and means what it did. The proof that it equals its definition with Mathlib unfolds one more name. There is a scenario for each order of the identifiers. The model rejected the trace of retirement_forgets_lower_node before this change. It accepts that of retirement_nominates_higher_node before and after it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
After a node has handled an AppendEntries which does not end in a signature, it records that the term of the last index starts after its last signature. If the node still held entries of a later term beyond that index, the view history refused the record with 'version must not move backwards'. recv_message logged the exception, and the node sent no response until an AppendEntries replaced those entries. The node now records nothing when it already knows the term of the last index, and acknowledges, as the model does. It keeps the later entries until the sender replaces them. This needs a primary which kept entries after its last signature when it was elected. become_leader only allows that while nothing is committed, so a running service should not get there. It was found by exploring changes to the scenarios, and heartbeat_before_stale_suffix fails without this change. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
cft_model_test runs each scenario again from its recorded commands, and required both runs to show the same model outcomes. The election timeout of a node differs between processes, so they need not: retirement_nominates_most_replicated leaves node 3 without a primary for 60 ms, and node 3 started an election in about 1 pair of runs in 40, which failed the test. The program must now be accepted by the model and issue every recorded command, the assertions of the scenario among them. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
trustNode, trustNodes, swapNode and swapNodes replicated a configuration of every node which the driver had created, less those which the command removes. The next such command therefore added back a node which an earlier one had removed. CCF does not do that: a node which has retired is never trusted again. These commands now change the latest configuration which the primary holds. No scenario relied on this: the trace of each has the same commands and configurations as before. A scenario which adds a node after one has retired did, as exploration with joins does. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
trustNodeFromSnapshot creates a node as trustNode does, but the node starts as a real node does from a snapshot which the primary took at its commit index: at that index, in its term, with the configuration of that index. It also holds the entries before the snapshot, like a node which was given the ledger files of the primary, so that the driver can still compare whole ledgers. The scenario join_from_snapshot shows the outcome joinFromSnapshot of the model, which no trace showed. The scenarios now show 78 of its 79 outcomes. The explorer adds a node at random, empty or from a snapshot, and changes a trust_node of a scenario into one from a snapshot. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
After a follower has handled an AppendEntries which does not end in a signature, and whose term is new to it, it recorded that term as starting after its last signature. That is wrong when the follower holds transactions after that signature in an earlier term. It then reported those transactions in the new term. If it had just replaced entries of a later term, the view history refused the record with 'version must not move backwards': recv_message logged the exception, and the follower sent no response and went on reporting the replaced term for the new entry. The follower now records the term as starting at the first entry of the AppendEntries, as it already does when one of its entries is a signature. The two indexes are the same unless a primary kept transactions after its last signature when it was elected. become_leader only allows that if the node has committed nothing, so a running service should not get there. It was found by exploring changes to the scenarios, and unsigned_entry_replaces_stale_suffix fails without this change. In 10,000 explored runs the indexes differ in 544 of 4,617 such records, each in the term of a primary which kept such transactions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
After each command, the scenario driver now also asserts of every node that: - its view history gives each entry of its ledger the term which the ledger records for it, and it reports its commit with that transaction ID; - what it reports of the primary, of whether it can replicate, and of whether it should sign, agrees with its role and its ledger; - it has notified its commit observer of each advance of its commit index; - if it is primary, it has asked for retired nodes to be cleaned up since it had a retirement to confirm. The trace of a scenario does not show these, so replay against the model cannot check them. The driver now sets the two callbacks of a node, and asserts the same once it has confirmed that the network can make progress. The model driver reports a failure of these as an invariant. The first fails in retirement_signature_rollback without the previous commit: a node reports 4.4 for the entry which its ledger holds as 3.4. Each of them fails in at least two scenarios when cft.h is changed not to do what it checks. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
cleanupRetired replicates what a real primary does when its consensus asks it to clean up retired nodes: it confirms the retirement of each node whose retirement that primary has committed, and has yet to confirm. cleanupNodes still confirms the nodes which a scenario names. In retirement_confirmed_by_new_primary, a primary commits a retirement and loses its role before it confirms it, and the node which is elected confirms it. The explorer can now retire a member of the latest configuration, and confirm the retirements which the primary has committed, wherever it changes a scenario. With the choice of adding a node, it can change the membership of any scenario. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
After the last command of a scenario, the driver confirms that the network can still make progress: it signs on the latest primary, and runs until the active nodes hold the same ledger. A node which that primary does not know is sent nothing, and can never catch up. The driver now leaves such a node out, as it leaves out a node which has retired. It still waits for a node which has retired without learning of it, which the primary knows until the retirement is confirmed. No scenario changes. In an exploration of 20,000 changed scenarios, 398 runs ended without this confirmation because the nodes did not agree. With this change and the three which follow, 19 do. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A node keeps the nodes to which a primary sends in all_other_nodes. It added a node when a configuration named it, and removed one only once its retirement or its deletion was committed. A rollback removed a configuration, and not the nodes which that configuration had added. A node which is elected discards what follows its last signature. If that includes a configuration which adds a node, the new primary went on sending AppendEntries to a node which none of its configurations named, listed it in /node/consensus, and could nominate it as its successor. Quorums and votes count the configurations only, so what was committed stayed safe. A rollback now forgets the nodes which only the configurations rolled back named. A node whose retirement is not confirmed stays known: retirement is final, so no later configuration names it. The consensus model reads the peers of a node from its log, and cft_model_driver --explore found the difference in 152 of 20,000 changed scenarios. The scenario rollback_forgets_added_node and a unit test fail without the change. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A node which joins from a snapshot is given the configuration at that snapshot. Once elected, it sends neither entries nor a nomination to a node which retired before the snapshot and whose retirement is not confirmed. Every other node still knows that node, and so does the consensus model, which reads the peers of a node from its whole log. Replay now records the index at which each node starts, and treats what the model sends to such a node as lost, which the model allows. Only entries and nominations are excused, and only from a node to a node which a configuration before its snapshot had removed. cft_model_driver --explore found the difference in 3 of 20,000 changed scenarios. The scenario join_from_snapshot_after_retirement shows it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
CCF starts the election timer of a node when it appends or receives a signature while it is in its latest configuration, and never stops it. Until then the node ignores a nomination. The consensus model has no timer: a node campaigns if a signature covers an active configuration which names it, or if its retirement is committed and not confirmed. So a node which is added and removed between two signatures, or which retires before it first signs, ignores a nomination for which the model campaigns. Replay now treats a nomination which CCF ignores as lost if the model would campaign and the log of the node holds no signature which found it in its latest configuration. Otherwise it delivers the nomination, and the model must ignore it too. cft_model_driver --explore found the difference in 1 of 20,000 changed scenarios. The scenarios nominate_node_removed_before_signature and nominate_node_retired_before_signature show the two cases. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
cft_model_driver --explore changes the consensus scenarios at random, and reports as a finding a run in which the driver fails or the model rejects the trace. It exited with status 0 whatever it found, and cft_model_test explored 40 changed scenarios only to check that exploring works. The exploration now exits with status 1 if a run is a finding, and writes the program of each finding to its error output, so that the finding can be run again from the log of a test. cft_model_test fails on a finding as a result. With a rollback which no longer forgets the nodes of the configurations which it removes, 2,000 runs with the default seed include 9 of one finding. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The capture job of the Lean workflow builds cft_model_driver and runs every scenario with the consensus model. It now also explores 2,000 changed scenarios with a fixed seed, and fails if the implementation and the model disagree in one of them. It uploads the findings when it fails. The same seed gives the same changes, so the step tells whether a change breaks what those runs reach, and looks for nothing new. In a Debug build like that of the job, the exploration takes 87 seconds with 8 runs at once. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The replay job reports which outcomes of the model the traces of the scenarios show. A scenario still passes when a change makes it stop showing the outcome for which it was written, and 11 outcomes are shown by one scenario only. consensus-replay --outcomes takes --not-shown and the names of outcomes, and fails unless the outcomes which no trace shows are those named. The replay job names the one outcome which no scenario shows, so that losing another fails it, and so does showing that one without removing its name. 30 captures of the 82 scenarios in each of a Debug and a RelWithDebInfo build all show the same 78 outcomes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
On a runner, the 2,000 changed scenarios which the capture job explored take 33 seconds, in a job of 4.2 minutes. The longest job of the workflow takes 5.5 minutes. The job now explores 5,000. The seed is fixed, so these are the same runs each time, and more of them check more cases on every pull request which changes consensus. In a Debug build on a development machine they take 184 seconds with 8 runs at once, and none is a finding. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cft_model_driverlinks the consensus scenarios of #8773 with the Lean model of the protocol. It replays each scenario against the model in the process which runs it, and it explores changes to the scenarios, keeping the runs in which the model orcft.hdoes something which no scenario makes it do.It adds 23 scenarios: most came from that exploration or reproduce what it found, and the rest show what the driver could not do before. With them, the scenarios show 78 of the 79 outcomes which replay tells apart in the model, where those of #8773 show 66, and execute 87.8% of the lines and 65.7% of the branches of
cft.h, where those of #8773 execute 82.6% and 62.1%.An exploration of 20,000 changed scenarios now ends without a difference between
cft.hand the model. The Lean workflow explores 5,000 changed scenarios with a fixed seed, and fails on such a difference, or if the scenarios stop showing an outcome which they showed. The seed is fixed so that a pull request fails only for what it changes; exploring with new seeds each night is left to a later pull request.Stacked on #8773.
Changes to consensus behaviour
Five commits change
cft.hor the model, and not only tests and tools. Each resolves a disagreement between the two which the exploration found.6b7ff8449,cft.h). A transaction which removes the last trusted node of a service leaves a configuration in which nothing can be committed. The primary now throws before it appends that transaction or sends it to another node. The node still stops with a fatal error, as it did before, but the entry no longer reaches its ledger or the other nodes.12908658b, model only). When one commit confirms the retirement of a primary and of other nodes, CCF forgets the nodes with a lower identifier before the primary nominates a successor, and not those with a higher one. The model now nominates among the same nodes.cft.his unchanged.f77674c5a,cft.h). The follower threwversion must not move backwardsfrom its view history, logged it, and sent no response. It now acknowledges, as the model does.8dccd7741,cft.h). A follower which held transactions after its last signature recorded a later term as starting at the first of them. It then reported a wrong transaction ID for those transactions, or threw as in 3 and went on reporting the term of an entry which it had replaced. It now records the term where the AppendEntries starts, as it already did when one of the entries was a signature.d29a76ad2,cft.h). A node kept every node which a configuration had named until the retirement or the deletion of that node was committed, so a rollback removed a configuration and not the nodes which it had added. A primary went on sendingAppendEntriesto such a node, listed it inGET /node/consensus, and could nominate it as its successor. A rollback now forgets the nodes which no configuration that remains names.3 and 4 need a node which is elected while it holds a signature and has committed nothing, so a running service should not reach them. 5 needs the transaction which trusts a node to be rolled back on a node which is or becomes primary.
Replay accepts two more differences as they are, each as the loss of messages, which the model allows:
f951558e0).86b70d659).Alternatives
Committedfor a different one. Stepping down after the rollback avoids that, at the cost of an election and of a step which the model does not have.set_retired_committedcould forget every node which it is given before the primary nominates, which would make the order of the identifiers irrelevant. That changescft.h, which this leaves alone.Safety and compatibility
findSuccessorintobestSuccessorover the eligible peers, with the same meaning. The safety proofs need one new lemma, the proof of deadlock freedom another, and the proof inlean/mathlib-equivalenceunfolds one more name.all_other_nodes, which no quorum and no vote reads: those count the active configurations. A node which a remaining configuration names, or whose retirement is not confirmed, stays known. No end-to-end test covers it: a unit test and a scenario do.cft.hgained the trace eventrecv_append_entries_response_unknown_node, and the scenario driver starts the election timer of a node when a real node would.