Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
bbc8191
Report the model outcomes shown by consensus traces
achamayou Nov 13, 2026
486080d
Let consensus scenarios deliver any pending message
achamayou Nov 13, 2026
8614cda
Keep the entries of AppendEntries as sent in scenarios
achamayou Nov 13, 2026
2ef0050
Return the trace of consensus scenarios to Lean
achamayou Nov 13, 2026
be77db1
Replay consensus traces one driver command at a time
achamayou Nov 13, 2026
a4dc1c6
Run consensus scenarios with the Lean model in one process
achamayou Nov 13, 2026
655272b
Replay nominations without a successor and ignored responses
achamayou Nov 13, 2026
f3baec4
Let consensus scenarios hold a message back
achamayou Nov 14, 2026
52a0995
Explore changes to the consensus scenarios
achamayou Nov 14, 2026
c3ff11e
Add consensus scenarios found by exploration
achamayou Nov 14, 2026
c94ae7f
Start the election timer of scenario nodes as real nodes do
achamayou Nov 14, 2026
e7ed962
Add a scenario which rolls back a retirement signature
achamayou Nov 14, 2026
c94fad0
Shrink explored runs by the edges which they reach
achamayou Nov 14, 2026
4ee7d65
Add consensus scenarios for branches found by exploration
achamayou Nov 14, 2026
6b7ff84
Refuse a consensus configuration with no nodes on the primary
achamayou Nov 15, 2026
1290865
Order retirement confirmations in the consensus model as CCF does
achamayou Nov 15, 2026
f77674c
Acknowledge AppendEntries ending before entries of a later term
achamayou Nov 15, 2026
0707298
Do not require a scenario and its program to show the same outcomes
achamayou Nov 15, 2026
890ce59
Keep retired nodes out of later consensus scenario configurations
achamayou Nov 15, 2026
57b2a81
Add a consensus scenario command for joining from a snapshot
achamayou Nov 15, 2026
8dccd77
Record a new term at the first entry which a follower receives in it
achamayou Nov 15, 2026
1a954ac
Check what scenario nodes report against their ledgers
achamayou Nov 15, 2026
b4861b0
Confirm retirements in consensus scenarios as a primary does
achamayou Nov 15, 2026
1a3cbb1
Wait only for nodes which the primary knows in consensus scenarios
achamayou Nov 15, 2026
d29a76a
Forget nodes which only a rolled back configuration added
achamayou Nov 15, 2026
f951558
Replay a primary which does not know nodes retired before its snapshot
achamayou Nov 15, 2026
86b70d6
Replay a nomination ignored before the election timer starts
achamayou Nov 15, 2026
6616a62
Fail the exploration of consensus scenarios on a finding
achamayou Nov 15, 2026
acc35f9
Explore changes to the consensus scenarios in the Lean workflow
achamayou Nov 15, 2026
43715ba
Fail the Lean workflow when scenarios stop showing an outcome
achamayou Nov 15, 2026
7c38c70
Explore 5,000 changed consensus scenarios in the Lean workflow
achamayou Nov 15, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .changes/forget-rolled-back-nodes.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
A node which rolls back an uncommitted reconfiguration transaction now stops tracking the nodes which only that transaction added, where a primary previously continued to send them `AppendEntries` and to list them in `GET /node/consensus`.
See [Reconfiguration](doc/architecture/consensus/index.rst#reconfiguration).
2 changes: 2 additions & 0 deletions .changes/refuse-empty-configuration.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
A primary now refuses a transaction which would leave the service with no trusted node before it writes that transaction to its ledger or sends it to another node, where it previously recorded a configuration after which nothing could be committed.
The node still stops with a fatal error; see [Removing an Existing Node](doc/governance/common_member_operations.rst#removing-an-existing-node).
3 changes: 3 additions & 0 deletions .github/skills/testing/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@ Labels include `unit`, `e2e`, `partitions`, `perf`, `benchmark`, `cft_scenario`,

`cft_scenario_test` runs the [consensus scenarios](../../../tests/cft_scenarios/README.md), Lean programs linked into `cft_driver`. If CMake cannot find `lake` when configuring, it does not build `cft_driver` and disables the test; install Lean and reconfigure rather than reporting the disabled test as passing.

`cft_model_test` runs the same scenarios with the [consensus model driver](../../../tests/cft_model/README.md), which replays each trace against the Lean model of the protocol in the process which records it.
CMake registers it only when configured with `-DCFT_MODEL_DRIVER=ON`.

The release workflow sets `CCF_USE_RELEASED_PYTHON_PACKAGE=1` so `tests.sh`, installed-package tests, and sandboxes install the latest released `ccf` package from PyPI with `uv pip install --upgrade --reinstall-package ccf ccf`, replacing any previously installed editable SDK.
Without this flag, development and normal CI tests retain their editable local SDK installation.
Installed sandboxes still default to the SDK version matching the installed CCF release.
Expand Down
4 changes: 3 additions & 1 deletion .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,9 @@ Each matrix job caches its third-party dependency checkouts and build outputs, l
Caches are isolated by package and runner architecture and invalidated by changes to the Ubuntu version, Lean toolchain, package and common dependency manifests/configuration, or reusable workflow.
CCF package and common build outputs are not cached, and all verification steps still run.
CCF consensus, KV, and consistency trace captures run in parallel with package checks.
The CCF consensus capture job then builds the [consensus model driver](../../tests/cft_model/README.md) with `-DCFT_MODEL_DRIVER=ON`, and runs `cft_model_test`, which replays each scenario in the process which runs it.
Three independent trace-validation jobs each build their own replayer, download the corresponding capture artifacts, and replay the traces without depending on the package matrix.
The CCF consensus replay job replays each trace whole and [by driver command](../../lean/consensus/replay/README.md#replay-by-command), and adds to its summary the [outcomes](../../lean/consensus/replay/README.md#outcomes) of the model which the captured traces show, and those which none shows.
Consistency capture reuses the existing `consistency_trace_validation` e2e test, and its replay job also checks the consistency replay fixtures, including election and rollback traces.
A final summary job requires every package and trace job to succeed.

Expand Down Expand Up @@ -184,7 +186,7 @@ The dedicated replay job builds `kv-replay` separately to validate the captured

After the build, `scripts/lean-format-checks.sh` checks every tracked `.lean` file with the pinned leanfmt dependency.
The workflow runs on pull requests that change `lean/` (except `lean/mathlib-equivalence/`), any `.lean` file, the formatter script, or the workflow.
It also runs on changes to the C++ sources and trace runners checked against the CCF consensus and KV models: `src/consensus/aft/`, the CCF consensus scenario tests, `src/kv/`, the public KV and transaction headers, and the KV trace runner.
It also runs on changes to the C++ sources and trace runners checked against the CCF consensus and KV models: `src/consensus/aft/`, the CCF consensus scenario tests and model driver, `src/kv/`, the public KV and transaction headers, and the KV trace runner.
See the [local formatting commands](../../lean/disaster-recovery/README.md#formatting) to apply fixes.

File: `lean.yml`
Expand Down
13 changes: 9 additions & 4 deletions .github/workflows/lean-package.yml
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,7 @@ jobs:
for trace in lean/consensus/replay/fixtures/valid/*.ndjson; do
echo "::group::Validate $trace"
lean/consensus/.lake/build/bin/consensus-replay "$trace"
lean/consensus/.lake/build/bin/consensus-replay --by-command "$trace"
echo "::endgroup::"
done

Expand All @@ -120,14 +121,18 @@ jobs:
run: |
set -euo pipefail
shopt -s failglob
for trace in lean/consensus/replay/fixtures/invalid/*.ndjson; do
echo "::group::Reject $trace"
reject() {
status=0
lean/consensus/.lake/build/bin/consensus-replay "$trace" || status=$?
lean/consensus/.lake/build/bin/consensus-replay "$@" || status=$?
if [ "$status" -ne 1 ]; then
echo "::error::$trace: expected rejection (exit 1), got exit $status"
echo "::error::$*: expected rejection (exit 1), got exit $status"
exit 1
fi
}
for trace in lean/consensus/replay/fixtures/invalid/*.ndjson; do
echo "::group::Reject $trace"
reject "$trace"
reject --by-command "$trace"
echo "::endgroup::"
done

Expand Down
43 changes: 42 additions & 1 deletion .github/workflows/lean.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ on:
- "tests/cft_trace*.py"
- "tests/cft_scenarios*.py"
- "tests/cft_scenarios/**"
- "tests/cft_model/**"
- "src/consensus/aft/**"
- "src/kv/**"
- "include/ccf/kv/**"
Expand Down Expand Up @@ -68,7 +69,7 @@ jobs:
run: |
set -euo pipefail
git config --global --add safe.directory "$GITHUB_WORKSPACE"
cmake -S . -B "$RUNNER_TEMP/cft-build" -GNinja -DCMAKE_BUILD_TYPE=Debug -DVERBOSE_LOGGING=ON
cmake -S . -B "$RUNNER_TEMP/cft-build" -GNinja -DCMAKE_BUILD_TYPE=Debug -DVERBOSE_LOGGING=ON -DCFT_MODEL_DRIVER=ON
cmake --build "$RUNNER_TEMP/cft-build" --target cft_driver --parallel "$(( ($(nproc) + 1) >> 1 ))"
# RUNNER_TEMP can persist between jobs on reused self-hosted runners; drop files from earlier runs.
rm -rf "$RUNNER_TEMP/cft-build/consensus/raw"
Expand All @@ -84,6 +85,35 @@ jobs:
retention-days: 1
overwrite: true

# cft_model_driver also links the consensus model, which lake builds.
- name: Run scenarios with the consensus model
run: |
set -euo pipefail
cmake --build "$RUNNER_TEMP/cft-build" --target cft_model_test --parallel "$(( ($(nproc) + 1) >> 1 ))"
ctest --test-dir "$RUNNER_TEMP/cft-build" -VV -R "^cft_model_test$" --no-tests=error

# Fails if the implementation and the model disagree in a changed scenario.
- name: Explore changes to the scenarios
timeout-minutes: 15
run: |
set -euo pipefail
# RUNNER_TEMP can persist between jobs on reused self-hosted runners; drop files from earlier runs.
rm -rf "$RUNNER_TEMP/cft-explore"
"$RUNNER_TEMP/cft-build/cft_model_driver" --explore "$RUNNER_TEMP/cft-explore" \
--seed 1 --runs 5000 --jobs "$(( ($(nproc) + 1) >> 1 ))"

- name: Upload exploration findings
if: ${{ failure() }}
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: cft-exploration
path: |
${{ runner.temp }}/cft-explore/report.md
${{ runner.temp }}/cft-explore/findings
if-no-files-found: ignore
retention-days: 7
overwrite: true

kv_traces:
name: KV trace captures
runs-on: [gha-vmss-d16av7-ci-c]
Expand Down Expand Up @@ -269,9 +299,20 @@ jobs:
for trace in "$RUNNER_TEMP"/cft-scenario-traces/*.ndjson; do
echo "::group::Validate $trace"
lean/consensus/.lake/build/bin/consensus-replay "$trace"
lean/consensus/.lake/build/bin/consensus-replay --by-command "$trace"
echo "::endgroup::"
done

# An outcome which the scenarios showed and no longer show fails this step.
- name: Report model outcomes shown by CCF consensus traces
shell: bash
run: |
set -euo pipefail
shopt -s failglob
lean/consensus/.lake/build/bin/consensus-replay --outcomes \
--not-shown appendEntriesConflictIgnored \
"$RUNNER_TEMP"/cft-scenario-traces/*.ndjson >> "$GITHUB_STEP_SUMMARY"

models:
name: Lean models and proofs
needs:
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ CMakeFiles/
libuv/
cft_tests.md
tests/cft_scenarios/.lake/
tests/cft_model/.lake/
*.pyc
__pycache__/
sphinx/build/
Expand Down
95 changes: 95 additions & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -1312,6 +1312,101 @@ if(BUILD_TESTS)
)
endif()

# Consensus model driver: the scenario driver, linked also with the Lean
# model of the protocol, which replays the trace of a scenario in the
# process which runs it. Opt-in, because lake then builds the model, and
# fetches the formatter which the package of the model requires.
option(
CFT_MODEL_DRIVER
"Build cft_model_driver, which runs consensus scenarios with the Lean model"
OFF
)
if(CFT_MODEL_DRIVER)
if(NOT LAKE)
message(
FATAL_ERROR
"CFT_MODEL_DRIVER needs lake. Install Lean (https://lean-lang.org/install/)."
)
endif()

set(CFT_MODEL_DIR ${CMAKE_SOURCE_DIR}/tests/cft_model)
set(
CFT_MODEL_LIBRARIES
${CFT_MODEL_DIR}/.lake/build/lib/libcft__model_ModelDriver.a
${CMAKE_SOURCE_DIR}/lean/consensus/.lake/build/lib/libconsensus_Consensus.a
${CMAKE_SOURCE_DIR}/lean/common/.lake/build/lib/libccf__common_CCF.a
)
add_custom_target(
cft_model
COMMAND
${LAKE} build ModelDriver:static @consensus/Consensus:static
@ccf_common/CCF:static
WORKING_DIRECTORY ${CFT_MODEL_DIR}
BYPRODUCTS ${CFT_MODEL_LIBRARIES}
COMMENT "Building consensus model driver"
VERBATIM
)
# lake builds the scenarios for this package too, so never at once
add_dependencies(cft_model cft_scenarios)

add_executable(
cft_model_driver
${CMAKE_CURRENT_SOURCE_DIR}/src/consensus/aft/test/driver.cpp
)
add_dependencies(cft_model_driver cft_model)
add_san(cft_model_driver)
add_warning_checks(cft_model_driver)
enable_coverage(cft_model_driver)
# Count the edges of the consensus implementation which a run reaches,
# so that the model driver can tell the runs which reach new ones.
# Without inlining, they are the edges of that implementation only, and
# not also those of the library code which it calls.
if(CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
set(CFT_MODEL_EDGES ${CFT_MODEL_DIR}/edges.allowlist)
target_compile_options(
cft_model_driver
PRIVATE
-fsanitize-coverage=inline-8bit-counters
-fsanitize-coverage-allowlist=${CFT_MODEL_EDGES}
-fno-inline
)
set_property(
SOURCE ${CMAKE_CURRENT_SOURCE_DIR}/src/consensus/aft/test/driver.cpp
APPEND
PROPERTY OBJECT_DEPENDS ${CFT_MODEL_EDGES}
)
endif()
# As for cft_driver. The model driver library comes first, so that its
# main is linked, and not that of the scenario library.
target_link_libraries(
cft_model_driver
PRIVATE
${CFT_MODEL_LIBRARIES}
${CFT_SCENARIOS_LIBRARY}
ccfcrypto
ccf_tasks
gcc_s
${LEAN_SYSROOT}/lib/lean/libleanshared.so
)
target_include_directories(cft_model_driver PRIVATE src/consensus/aft)
target_include_directories(
cft_model_driver
SYSTEM
PRIVATE ${LEAN_SYSROOT}/include
)

add_test(NAME cft_model_test COMMAND cft_model_driver --all)
add_test_target(cft_model_test cft_model_driver)
if(COVERAGE)
# One process per scenario, as for cft_scenario_test
set_property(
TEST cft_model_test
APPEND
PROPERTY ENVIRONMENT "LLVM_PROFILE_FILE=cft_model-%p.profraw"
)
endif()
endif()

add_test(
NAME csr_test
COMMAND ${PYTHON} ${CMAKE_SOURCE_DIR}/tests/certs.py ./kp_cert_test
Expand Down
1 change: 1 addition & 0 deletions doc/architecture/consensus/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ For a live service, reconfiguration is materialised in two separate transactions

In contrast to normal transactions, reconfiguration transactions will only commit when the necessary quorum of acknowledgements is reached in **both** the previous and the new configuration it defines.
Transactions before an uncommitted reconfiguration transaction do not need acknowledgements from the new configuration it defines: they commit once their signature has a quorum in each active configuration defined at or before that signature.
A node which rolls back an uncommitted reconfiguration transaction stops tracking the nodes which only that transaction added: they are no longer sent ``AppendEntries`` and no longer appear in :http:GET:`/node/consensus`.

Disaster recovery is an exception to this retirement sequence.
Replaying the public ledger reconstructs the previous service's trusted-node configuration in the new consensus instance.
Expand Down
7 changes: 6 additions & 1 deletion doc/architecture/verification.rst
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,8 @@ Each replayer consumes traces produced by existing C++ tests:
* **Consensus**: the ``cft_scenario_test`` test runs :ccf_repo:`tests/cft_scenarios_runner.py`, which executes each scenario in :ccf_repo:`tests/cft_scenarios/Scenarios/` with the ``cft_driver`` binary.
The scenarios are Lean programs, which drive the implementation through the bindings in :ccf_repo:`tests/cft_scenarios/CftDriver.lean` and :ccf_repo:`src/consensus/aft/test/driver.cpp`; see :ccf_repo:`tests/cft_scenarios/README.md`.
The driver exports consensus events as Fluentd records with the ``ccf.cft`` tag, which the runner collects and writes as NDJSON traces to ``consensus/raw/`` in the build directory.
The ``cft_model_test`` test, built with ``-DCFT_MODEL_DRIVER=ON``, runs the same scenarios with ``cft_model_driver``, which links the driver with the replayer and replays each trace in the process which records it; see :ccf_repo:`tests/cft_model/README.md`.
``cft_model_driver --explore`` also changes the scenarios at random, and replays each changed run, to find runs in which the implementation and the model disagree, or which show an outcome that no scenario does.
* **KV**: building with ``-DCCF_KV_TRACING=ON`` enables test-only trace capture in ``kv_test``.
:ccf_repo:`tests/kv_trace_validation.py` captures the purpose-built ``KV trace *`` test cases and a seeded concurrent fuzzer, then checks every trace with ``kv-replay`` unless ``--capture-only`` is passed.
Tracing is disabled in normal builds.
Expand All @@ -80,7 +82,9 @@ The :ccf_repo:`Lean workflow <.github/workflows/lean.yml>` runs on pull requests

* A package job for each of ``common``, ``consensus``, ``consistency``, ``disaster-recovery`` and ``kv`` runs the kernel and axiom check (``lake check``), import coverage and architecture checks, formatting and linting, and, where applicable, the package's replay fixtures.
* Separate capture jobs build CCF and record consensus, KV and consistency traces from the tests described above.
The consensus capture job also runs ``cft_model_test``, and then ``cft_model_driver --explore`` with a fixed seed, which fails if the implementation and the model disagree in a changed scenario.
* Replay jobs build each replayer and validate every captured trace.
The consensus replay job also reports which outcomes of the model's handlers the captured traces show, and which none shows, and fails if the traces no longer show an outcome which they did; see :ccf_repo:`lean/consensus/replay/README.md`.
* A final ``Lean models and proofs`` job succeeds only if all of the above succeed.

The disaster recovery traces are replayed by the SNP Genoa job in the main CI workflow instead, since that is where the recovery scenarios run.
Expand All @@ -92,7 +96,8 @@ Running locally

Install Lean with `elan <https://lean-lang.org/install/>`__.
The Lean toolchain is pinned by :ccf_repo:`lean/lean-toolchain`, and the models are independent of normal CCF builds.
The only part of a CCF build which needs Lean is ``cft_driver``; without Lean, it is not built and ``cft_scenario_test`` is disabled.
The only part of a default CCF build which needs Lean is ``cft_driver``; without Lean, it is not built and ``cft_scenario_test`` is disabled.
``cft_model_driver`` also needs Lean, and is built only with ``-DCFT_MODEL_DRIVER=ON``.
From a package directory under ``lean/``, the common checks are:

.. code-block:: bash
Expand Down
4 changes: 4 additions & 0 deletions doc/governance/common_member_operations.rst
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,10 @@ The operator can establish if it is safe to remove a node by calling :http:GET:`

.. note:: If the now-retired node was the primary node, once the proposal successfully completes, an election will take place to elect a new primary.

.. warning::
Do not remove the last trusted node of a service.
Nothing could be committed once no trusted node is left, so the primary refuses the transaction which removes it, and stops with a fatal error before it writes that transaction to its ledger.


Updating Code Version
---------------------
Expand Down
5 changes: 4 additions & 1 deletion lean/consensus/Consensus.lean
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
module

public import Consensus.Properties
public import Consensus.Replay.Reduction
public import Consensus.Replay.Session

import Consensus.Tests.CanonicalTraces
import Consensus.Tests.ReplayDelivery
import Consensus.Tests.ReplayDifferences
import Consensus.Tests.ReplayOutcomes
import Consensus.Tests.ReplaySession
11 changes: 11 additions & 0 deletions lean/consensus/Consensus/Model/ClusterConfig.lean
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,17 @@ def retiredCommittedNodes (log : List Entry) : Finset NodeID :=
log.foldl
(fun retired entry => retired ∪ entry.content.metadata.retirementConfirmations) ∅

/--
Nodes whose retirement the log confirms before that of `node`: in an earlier
transaction, or in the same transaction with a smaller identifier. CCF confirms the
retirements which a commit covers in that order.
-/
def retirementsConfirmedBefore (log : List Entry) (node : NodeID) : Finset NodeID :=
let confirmations := log.map (·.content.metadata.retirementConfirmations)
let earlier := confirmations.takeWhile fun nodes => decide (node ∉ nodes)
let same := (confirmations.drop earlier.length).headD ∅
earlier.foldl (· ∪ ·) ∅ ∪ same.filter (· < node)

def hasConfigurationMajority (support : Finset NodeID) (configuration : Configuration)
: Prop :=
(support ∩ configuration.nodes).card * 2 > configuration.nodes.card
Expand Down
Loading