Date: March 09, 2026

Punchline: Testing distributed protocols with random simulation and stateful property-based testing (PBT) is not enough! Yes, running a simulator for days is better than doing manual testing or just running unit tests. But you will miss states, which may expose bugs. Even on very small systems. I have been saying exactly this to many software engineers. Many times. However, whiteboard arguments do not help. As humans, we have a great deal of trust in probabilities, and our intuitive understanding of randomness is often wrong. Hence, I am giving you concrete figures and plots in this blog post. I must admit that my own intuition was also wrong: I expected fewer random walks to be needed to achieve good coverage. For a quick glance, see Quick summary.

Achieving complete coverage with random walks is hard. This is especially important to know, if you are using them to produce test cases for your implementation. It is also crucial to know, in case you generate an implementation of a distributed protocol with AI tools and hope for random walks/PBT to work as an ultimate guardrail.

Don’t get me wrong. I like PBT and simulators (having written the Quint simulator). I believe that these tools are must-have tools for testing. See my recent blog post on Property-based testing, adversarial developers, and LLMs. However, they are not the only tools that we need to make sure that our systems work as expected. This is especially true now, when we do not have time to properly design and review the AI-generated code.

Why now? It has always been difficult to compare search procedures that were developed by different branches of computer science. Everyone wanted to promote their technique as the ultimate winner. Want to compare property-based testing and model checking? Bad luck. Different tools require different inputs. Some are libraries for programming languages (like QuickCheck), some are tools for specification languages (like TLC and Apalache). Now it is much faster to design frameworks, to experiment with multiple search procedures. It is also easier to do reproducible experiments with LLMs. Good times, if you know how to conduct experimental research.

1. Quick summary for the impatient readers

Look at two groups of figures below. They summarize the results of running random walks on specifications of three prominent distributed protocols: two-phase commit, readers-writers, and FPaxos (see Benchmarks). The figures show the coverage achieved by random walks, with 100% being the numbers of distinct states (reported by the model checker TLC). In addition, we plot the running times of the random walks, with the values plotted against the right y-axis. All running times are on a AMD Ryzen 9 5950X processor (16 physical, 32 logical cores), 128 GB memory.

1.1 Coverage for minimal instances

In this set of experiments, we do random walks for the minimal instances of the benchmarks. We start with the meaningful default of 100,000 random walks, with at most 100 steps per walk. As you can see from Figure 1, only in the case of two-phase commit and two resource managers, we achieve complete state coverage. This is not surprising, since this instance has only 56 states. It’s tiny! For two-phase commit with three resource managers and readers-writers with three actors, we achieve 85-90% coverage. This is also in the reasonable range. On FPaxos with two acceptors, we achieve the 77.5% coverage with 100k random walks. This is a bit worrying, since the state space is about 37k states.

The good news is that we can push all of the above benchmarks to achieve over 99% coverage. As you can see in the figures, it takes 10 million random walks to achieve 99% coverage. In addition to that, these runs require 1-2 hours.

All of the above benchmarks are quite small by the model checking standards. They have tens of thousands of states. It takes TLC only 1-3 seconds to explore the state space and check the invariants for each of these benchmarks.

1.2 Slightly larger instances

What happens if we take the instances that are still small, but have 1-2 participants more? Figure 2 shows the results of doing random walks on these instances.

As you can see, with the meaningful default of 100,000 random walks, we achieve extremely poor coverage, about 25-30% on the benchmarks up to 2 million states. On FPaxos with 4 acceptors, we achieve only 3% coverage after 100,000 random walks. Really bad!

To see how far we could push the coverage, we did the experiments with 10-100 million random walks. It is clear that in 1-2 hours of simulation we get to 60-80% coverage. It is good, but not great. When we push FPaxos with 3 acceptors to 100 million random walks, we get to 94.5% coverage. Nice, though it took us 7.5 hours to get there. However, on FPaxos with 4 acceptors, we get a poor coverage of 60.4% even with 100 million random walks, which took us 8.5 hours to run. This benchmark has about 11 million states. So it is reasonably large, but, again, not that large by the model checking standards.

Again, it takes the model checker TLC up to 10 minutes to enumerate all the states and check the invariants for these instances, whereas we have been running the simulations for hours! This is especially striking, given that we are running optimized simulators in Rust.

2. The benchmarks

As benchmarks, we use three specifications of distributed protocols. These are prominent examples from the repository of TLA+ Examples:

• Two-phase commit. This is the famous two-phase commit. The specification is explained in Consensus on Transaction Commit by Jim Gray and Leslie Lamport. You can check the TLA⁺ specification in TwoPhase.tla.

• Readers-writers. This is a solution to the Readers-Writers Problem. The TLA⁺ specification by Stephan Merz can be found in ReadersWriters.tla.

• FPaxos. This is Flexible Paxos by Heidi Howard, Dahlia Malkhi, and Alexander Spiegelman. The TLA⁺ specification can be found in FPaxos.tla.

All of the above specifications are parameterized in the number of participating processes. We consider several instances of each benchmark. To give you an idea of their state space size (the number of reachable states), we compute the figures with TLC. The reachable states are called distinct states in TLC, whereas produced states are the number of states that TLC generates during the search. Another important metric is the diameter of the state space, which is the length of the longest shortest path between any two reachable states (read it again!).

As you can see from Table 1, these transition systems are not tiny, but they are actually small by the model checking standards. Surprisingly, they are sophisticated enough to challenge random walks! Distributed protocols are hard.

In the experiments, I am using a custom framework to represent the above specifications-as-code that makes it easy to experiment with different search procedures. To make sure that these specifications faithfully represent the original TLA⁺ specifications, I do the following:

1. do a code review (obviously),

2. automatically translate the specifications to TLA⁺ and check them with TLC,

3. run a custom-tailored model checker to compute the number of distinct states and check the invariants.

3. What are random walks and state enumeration?

I have mentioned random walks and state enumeration multiple times so far. Let’s clarify what these terms mean. The concept of a random walk is intuitively simple, though the details matter. Instead of looking at a large specification, let’s look at a simple example of a system that models adding and removing workers from a pool. This example is inspired by the example in Parameterize Your Actions by Hillel Wayne. We add the variable count to have a meaningful invariant. The specification is shown below. Even if you do not know TLA⁺, it should be easy to understand. If you still have trouble understanding it, just ask an LLM, they are good at explaining TLA⁺ specifications.

EXTENDS Integers, FiniteSets

CONSTANTS
    (* The set of workers to choose from. *)
    (* @type: Set(Int);                   *)
    Worker

VARIABLES
    (* The set of active workers.         *)
    (* @type: Set(Int);                   *)
    active,
    (* The number of active workers.      *)
    (* @type: Int;                        *)
    count

(* Add a worker w to the set of active workers, if it is not already active. *)
(* @type: (Int) => Bool;                                                     *)
Add(w) ≜ w ∉ active ∧ active' = active ∪ {w} ∧ count' = count + 1

(* Remove a worker w from the set of active workers, if it is active.        *)
(* @type: (Int) => Bool;                                                     *)
Remove(w) ≜ w ∈ active ∧ active' = active \ {w} ∧ count' = count - 1

(* Initialize the system with no active workers and a count of zero.         *)
Init ≜ active = {} ∧ count = 0

(* In a next state, either add a worker or remove a worker.                  *)
Next ≜ ∃ w ∈ Worker:
          Add(w) ∨ Remove(w)

(* An invariant: `count` matches the cardinality of the active set.          *)
Inv ≜ (count = Cardinality(active))

If we fix the set of workers to be Worker = {1, 2}, we get a nice labelled transition system (LTS) of 4 states. The graphical representation of this LTS is shown below.

TLA⁺ does not have any built-in notion of randomness or probabilities. It is what is usually called a qualitative specification. When evaluating Next in a state, we can only evaluate whether a specific transition is possible under a specific choice of w and the action scheduling decision (whether to execute Add(w) or Remove(w)). This is the standard semantics under the definition of behaviors. We can enumerate all reachable states for the above system by breadth-first search or depth-first search. This is what the model checker TLC does (it uses breadth-first search). This is what I will call state enumeration in this blog post.

We could also interpret the choice of w and the action scheduling decision as a random choice. Since the above specification is small, we can visualize it as a Markov decision process (MDP). The states are the same as in the LTS, but we also attach probabilities to the transitions.

Notice that we assign probabilities for choosing the value of w and for choosing the action to execute: Add(w) or Remove(w). For example, in the initial state, we choose w=1 with probability 0.5, then the action Add(1) with probability 0.5, which gives us a transition to the state where active = {1} and count = 1 (with probability 0.25). However, if we choose w=1 and the action Remove(1), we have to backtrack to the initial state, since the precondition of Remove(1) is not satisfied.

A random walk is a path through the MDP. It is a sequence of states that we get by making random choices at each step. In the above figure, you can see one walk in blue and one walk in red. To avoid too many backward edges, we have a retry budget, typically, 3-10 retries per step. We take this simple approach in our custom framework. It is similar to what the randomized simulator in Quint is doing, though the Quint simulator is trying a bit more locally before backtracking. Probabilities are basically used to produce various random walks. There is no inherent statistical meaning to these probabilities in random walks. This is very much how stateful property-based testing works, too, though PBT frameworks usually use biased coins, instead of uniform ones.

4. Which states are missing?

Since we can measure state coverage now, the next question is: What are these states that we are missing? Maybe these states are not important at all. To check that, I ran the random walks for the two-phase commit benchmark with 2 resource managers for 10,000 instead of 100,000 runs. Conveniently, exactly one state was missing from the coverage. As our specifications are code, I just asked Claude to instrument the search to experimentally evaluate the visit frequencies per run for each reachable state. Figure 6 is quite detailed. Click on it to see the full-size version.

As we can see, the missing state (with the frequency of 0) is the state where the transasction manager aborts the transaction, one resource manager also aborts the transaction, and the other resource manager is in the “prepared” state. This is an interesting state in this protocol, as the other resource manager still has the potential to commit the transaction, though it should not do that.

Bottom line: We may miss important states with random walks.

5. More coverage plots

Figure 7 shows the coverage evolution for the large instances of the benchmarks. With this, we can see how increasing the number of random walks helps to increase the coverage. It also demonstrates the growing volume of covered and missing states.

I wanted to share these flame plots with you. I find them cool.

6. Conclusions

Random walks are not sufficient to achieve complete coverage except for very small state spaces. Moreover, random walks take significantly longer than the model checker. This is especially striking, given that we are running optimized simulators in Rust. Another issue with state coverage by random walks is that you would not even know that you achieved complete coverage. You can measure the speed of discovering new states, but understanding that the simulator has converged basically requires a model checker.

Interestingly, random walks behaved badly on FPaxos with four acceptors. This is a relatively benign benchmark, not having a state explosion like specifications of Byzantine consensus protocols (PBFT). In PBFT, the minimal configurations contain 4-6 replicas, depending on the protocol. Hence, we should expect a significantly worse coverage by random walks on PBFT.

Why do engineers keep running randomized experiments? Well, it is relatively easy to write a simulator. (It is not that easy to write one that actually works!) I have seen people playing with action distributions in the simulator, just to drive the search towards “interesting” states. Whenever I was asking, where the distributions were coming from, they could not explain this. Simulators are deceptive. You have to understand what you are doing, or, better, incorporate feedback. The most basic feedback is state coverage, though we can implement more sophisticated feedback mechanisms.

From our experiments it may look like state enumeration is all we need. I would argue that it is true as long as the set of reachable states fits into memory. We do not have to store the states directly in memory, practical model checkers store hashes of states. We can go as far as to store 2-3 bits per state, assuming that collisions are acceptable (still better than random walks!). Having a machine with 128 GB of memory, we can store roughly 50 billions of states. This is way more than the number of states in our benchmarks – dozens of billions vs. thousands and millions.

There are cases where randomness may find bugs, where state enumeration gets stuck:

1. Value domains are quite large. For example, if we choose values from the set of all 64-bit integers, it is not feasible to enumerate all successors even for a single state. A random walk can still do some progress without getting stuck. One can argue that choosing a value from the set $[0, 2^{64})$ uniformly at random is shooting in the dark, but sometimes it helps us to find bugs, especially if the large set has just a few large equivalence classes. Arguably, one should be able to apply data abstraction in this case. Also, this is usually the moment when you should consider using a model checker that supports symbolic representation of states, like Apalache.

2. Guided search. If we have an heuristic that guides the search towards interesting states, we can achieve better coverage with random walks faster. Maybe we use reinforcement learning to learn such a heuristic. Maybe we use an LLM to predict which actions are more likely to lead to interesting states. The main issue is that it is quite hard to find a direction for the search in the state space of distributed protocols.

Want to talk?

Date: February 12, 2026

tl;dr:

• AI coding tools now reduce development costs, but they also accelerate the creation of software that appears high-quality while hiding serious correctness and reliability risks.

• When both code and tests are autogenerated, traditional quality checks lose their signaling value, increasing the likelihood of costly failures, outages, and liability exposure in production systems.

• To responsibly capture productivity gains without undermining trust, organizations must pair AI-generated code with specification-driven development and automated validation techniques that verify real system behavior rather than surface-level compliance.

In 2025, we saw plenty of enthusiastic announcements about LLMs generating (or, more correctly, replicating) relatively complex projects, like web applications or video games. Apparently, many people got so tired of all this that you could hear the words “AI slop” quite often. So often that some very important people asked all of us not to call the output of their amazing tools “slop”.

Anyhow, by the end of 2025, the amazing AI tools became visibly more amazing. In the early 2025, I was only using ChatGPT and Copilot to produce small code snippets and scripts, as well as to search for design solutions. In the summer of 2025, I used Copilot & Sonnet to produce boilerplate code. Now, I am using Claude Code and Copilot (both with Opus and Sonnet) to generate code and tests as well as to fix linting errors (still the hardest task!). I still have to define the core data structures, write non-standard code and explain in detail what I want to achieve. It is still hit and miss (see the most notable examples below). However, it becomes economically feasible for me to use these tools, unless they get 10x more expensive. By the way, after finishing my experiment with Symbolic testing of TFTP, I was still not sure, whether I wanted to use agentic tools every day. The feedback loop was energy draining. It looks like the tools became better, and I’ve learnt how to give them more focused and smaller-scoped tasks.

It still remains to see an AI-generated product that generates revenue. Are there any examples, except the AI coding assistants themselves? In any case, this is not what I wanted to write about. I wanted to write about something that looks like a new phenomenon to me. We all have heard the saying: When everyone is digging for gold, sell shovels. Just over a couple of weeks, there was an unusual number of announcements about development tools that were generated with AI. This is what I call AI-generated shovels. These announcements bring so much joy to AI influencers that it’s hard to find anything else. Do these tools actually work though? At a closer look, some of the shovels break on a first try, some happen to work only under very specific conditions. Most likely, you have seen some announcements, and you know what I am talking about. It is also very likely that you have not seen all of the announcements that I have in mind. Since we are talking about development tools, libraries, or even languages that do not actually work, not web apps, it is not just slop, it is a second-order slop!

I am not going to call any names, or do any fingerpointing. This is not the point. What makes me seriously concerned about the second-order slop is that the software development industry was cutting corners everywhere even before the AI boom. “Move fast and break things!”, minimum-viable products (or are they solutions?), product-market-fit, etc. A couple of years ago, I was joking that I would rather not use an MVP compiler, operating system, or database. Well, AI tools generate compilers. Here we are.

Shovel ad! Since I have been working on pre-LLM shovels like Apalache and Quint myself, I am in the shovel business, too! (Do you know that SMT solvers were also considered AI?) Of course, I am developing new shovels, and they are also AI-generated and AI-compatible, and they are the best in town, by the way. So if you want to talk, drop me a message. To be fair, my time tracker shows that I’ve burnt six weeks of my time on the latest shovel, in addition to burning through and over my Copilot and Claude budgets, so it’s not entirely AI-generated. Perhaps, a bit artisanal.

Good shovel or slop? How do we distinguish a robust AI-generated shovel from a second-order slop? In the pre-LLM years, I could just look at the test suite and say, whether the team was serious or not. Those amazing days when blockchain engineers would nod their heads to the question: Do you have integration tests? They were proudly demonstrating a single integration test that was 3-5 KLOC long. Also, by looking at the code, you could sense whether it was written just yesterday, or someone had time to think about it.

In 2026, the code may look professionally written and follow all the best practices and still be completely broken. On top of that, LLMs generate well-looking tests, if you ask them. A lot of tests! The more tests you have, the more tokens you have to pay for. Win-win. Moreover, the generated tests may check that the code works, but this does not mean that the code does what you expect. This happened to me (see below).

So when we evaluate an AI-generated shovel, we want to answer two questions:

1. Does this shovel do what the authors claim it should do?

2. Does this shovel work beyond a few simple tests?

These are not new questions. The testing and verification communities have been trying to automate validation and verification for long time. Interestingly, these questions did not get much attention over the last two decades. It was expected that open source projects and products by respectable companies were “more or less” correct and complete. In my understanding, two factors contributed to that:

1. The code was written and reviewed by highly-skilled engineers, for fun or profit.

2. The projects were extensively tested with continuous integration tools.

Now, if an LLM generated the code just yesterday, and all tests pass, are we good? It is hard to tell. If we follow the brand new spec-driven development, we have a bunch of markdown files. Apparently, we should ask a few other LLMs to check whether the implemented code matches the markdown specs. Something like that.

Can we do better? I believe we can. For example, if you are developing a distributed system, do not generate it directly. First, write or AI-generate a sequential reference implementation (e.g., in Python) or, even better, a formal specification (e.g., in TLA⁺). Second, use this artifact to produce the code for the actual distributed system.

Why does this help? For two reasons:

1. It is easier to compare the reference implementation or specification against the markdown requirements than to compare the entire codebase.

2. The reference implementation/specification is an actionable artifact. Use it to produce tests for the distributed system. Instead of generating 10 KLOC tests once (and paying for loading them into the LLM context), automatically produce as many tests as you can. This is where property-based testing and model checking start to shine. See Symbolic testing of TFTP for an example.

Examples of LLMs hit and miss. If you follow me on LinkedIn, you could have seen some of the examples. Below are the most curious instances that I would regret missing in a code review (by Sonnet 4.5 and Opus 4.5):

• The set minimum. When I asked an LLM to implement the search for the minimal element of a set by using its string representation (called repr in Python), it collected all set elements in a list, sorted them by repr and picked the first one. It looks like my requirement was slightly non-standard.

• Sets with duplicates. An LLM has produced a unit test that constructed the data structure called “Set” from the list [ V(1), V(2), V(3), V(1) ] and asserted that the set cardinality was 4. The test passed, since V did not have equality defined, and two different instances of V(1) had different references. So it was doing the things right, but it was not doing the right things!

• Performance bottleneck. An LLM translated my Python function into a Rust function. Perfectly looking code. However, instead of adding a big integer x to the big integer y, it used an iterator that made y increments of x. Almost like a theorem prover! A logically correct solution, but my Rust code was slower than the Python code. I only spotted it after running the profiler. Again, a bit non-standard setup threw it off.

Want to talk?

Date: December 22, 2025

I present a simple example that illustrates how property-based testing (PBT) and model checking can help us catch unexpected behaviors of LLMs when they are used to generate code. The example is inspired by the talk on property-based testing by Scott Wlaschin. If you are looking for a light example that stresses the importance of writing good properties and having them checked, this post is for you.

1. Adversarial Developer

A few days ago, I watched the talk on property-based testing by Scott Wlaschin. He started the talk by introducing a persona that he called the Enterprise Developer from Hell. This is basically someone who implements a feature to satisfy the given requirements, but they do it in creatively evil (or just stupid) and unexpected ways. I will call such a persona an adversarial developer in the rest of this post.

Then, Scott¹ showed how an adversarial developer could ruin as simple task as adding up two numbers. For example, if we give them two tests $2+2=4$ and $10+33 = 43$, they will implement exactly those cases by case distinction. I am not going to repeat Scott’s talk. Watch it! It’s instructive and entertaining.

Back in 2020, of course, Scott added that we are often aversarial developers ourselves, and our peers are rarely that evil. It could be an enthusiastic junior developer, who has just started and now wants to rewrite the whole code base. Now, we are a few weeks away from 2026, and we definitely have such a peer! It is called an LLM, or just AI, as the corporate marketers prefer. LLMs are not necessarily evil, but they are definitely less predictable than experienced human software engineers. I am not talking about prompt injection here, which is another real issue with LLMs.

To be clear, in the rest of this text, I am talking about how an LLM could behave like an adversarial developer when it generates code. It does not mean that I ran one of the commercial LLMs and got those results.

2. Property-Based Testing

The point of Scott’s talk was to show that a few data points (typical unit tests) are insufficient to demonstrate correctness of the implementation. A totally valid point!

In addition to the standard unit tests, we should also write the expected properties of our implementation. The PBT frameworks test the code by producing input values at random. For example, have a look at Hypothesis. While this may seem to be a silly idea at first, property-based tests uncovers tricky bugs. Moreover the input value distribution does not have to be uniform. Keep reading to see how this helps us catch the adversarial developer.

Here are the three properties of addition that Scott used to defeat the adversarial developer:

• identity: for every number $x$, we have $x + 0 = x$,
• commutativity: for every numbers $x$ and $y$, we have $x + y = y + x$, and
• associativity: for every numbers $x$, $y$, and $z$, we have $(x + y) + z = x + (y + z)$.

At this point of the talk, I was like: Wait a minute! I could continue this game of the adversarial developer. Before doing this, let’s look at where we are with respect to the code and the properties. Here is the obvious implementation of integer addition in Python, since the language has built-in support for unbounded integers:

def add(a: int, b: int) -> int:
    return a + b

Here are the property-based tests in Hypothesis, generated by Claude Sonnet 4.5:

@given(st.integers())
def test_identity(a):
    """Test identity property: a + 0 = a."""
    assert add(a, 0) == a
    assert add(0, a) == a


@given(st.integers(), st.integers())
def test_commutativity(a, b):
    """Test commutativity property: a + b = b + a."""
    assert add(a, b) == add(b, a)


@given(st.integers(), st.integers(), st.integers())
def test_associativity(a, b, c):
    """Test associativity property: (a + b) + c = a + (b + c)."""
    assert add(add(a, b), c) == add(a, add(b, c))

You can find these and further examples in the example repository.

We run these tests with pytest to make sure that they all pass:

$ git clone https://github.com/konnov/pbt-example-summation.git
$ cd pbt-example-summation/python
$ poetry run pytest tests/test_add.py \
  -k "test_identity or test_commutativity or test_associativity" --verbose
...
tests/test_add.py::test_identity PASSED                                        [ 33%]
tests/test_add.py::test_commutativity PASSED                                   [ 66%]
tests/test_add.py::test_associativity PASSED                                   [100%]

3. Symbolic model checking with Apalache

I decided to go even further and write a TLA⁺ specification, to check the three properties with Apalache and Z3 (only showing the relevant parts):

----------------------------------- MODULE Add --------------------------------
(*
 * A simple TLA+ specification of different kinds of addition.
 *
 * Igor Konnov, 2025
 *)
EXTENDS Integers

VARIABLE
    \* @type: Int;
    x,
    \* @type: Int;
    y,
    \* @type: Int;
    z

AddMath(a, b) == a + b

InitMath ==
    /\ x \in Int
    /\ y \in Int
    /\ z \in Int

Next == UNCHANGED <<x, y, z>>

Identity(F(_, _)) ==
    F(x, 0) = x

Commutativity(F(_, _)) ==
    F(x, y) = F(y, x)

Associativity(F(_, _)) ==
    F(F(x, y), z) = F(x, F(y, z))

InvMath ==
    /\ Identity(AddMath)
    /\ Commutativity(AddMath)
    /\ Associativity(AddMath)

With the above specification, we define a very simple state machine that non-deterministically picks three integers x, y, and z with InitMath. These variables do not change their values in the state machine, as you can see from the definition of Next. We use x, y, and z to define three properties of addition: Identity, Commutativity, and Associativity. As you can see, these definitions are parameterized by the operator F, which is Add for now. Our invariant InvMath is simply the conjunction of the three properties.

This is how we run Apalache to check the invariant:

$ cd pbt-example-summation/tla-spec
$ docker pull ghcr.io/apalache-mc/apalache
$ docker run --rm -v `pwd`:/var/apalache ghcr.io/apalache-mc/apalache \
  check --init=InitMath --inv=InvMath --length=0 Add.tla

With the above command, we tell Apalache to check the invariant InvMath starting from the initial state InitMath. The --length=0 option tells Apalache to unroll Next zero times, which is sufficient in our case, since the state machine does not change the values of the state variables.

4. Playing adversarial

Ok, the code and the specification above seem to be correct. But what if our friendly AI produced something unexpected?

4.1. Hallucinating addition over 32-bit unsigned integers

Since we are dealing with an adversarial developer, they could simply use a different definition of addition. So far, we have been talking about unbounded mathematical integers, which Python conveniently implements for us.

Now, the adversarial developer gives us this implementation:

def add32(a: int, b: int) -> int:
    return (a + b) % (2**32)

This implementation is actually not wrong. An LLM could copy it from a code base that emulates a 32-bit CPU architecture in Python. This is a bit of a stretch, but possible.

Let’s add property-based tests for this implementation as well:

# Tests for add32 (32-bit natural numbers with wrapping)

@given(st.integers(min_value=0, max_value=2**32 - 1))
def test_add32_identity(a):
    """Test identity property for add32: a + 0 = a."""
    assert add32(a, 0) == a
    assert add32(0, a) == a


@given(st.integers(min_value=0, max_value=2**32 - 1), st.integers(min_value=0, max_value=2**32 - 1))
def test_add32_commutativity(a, b):
    """Test commutativity property for add32: a + b = b + a."""
    assert add32(a, b) == add32(b, a)


@given(
    st.integers(min_value=0, max_value=2**32 - 1),
    st.integers(min_value=0, max_value=2**32 - 1),
    st.integers(min_value=0, max_value=2**32 - 1)
)
def test_add32_associativity(a, b, c):
    """Test associativity property for add32: (a + b) + c = a + (b + c)."""
    assert add32(add32(a, b), c) == add32(a, add32(b, c))

These tests also pass:

$ poetry run pytest tests/test_add.py \
  -k "test_add32_identity or test_add32_commutativity or test_add32_associativity" \
  --verbose
...
tests/test_add.py::test_add32_identity PASSED                                [ 33%]
tests/test_add.py::test_add32_commutativity PASSED                           [ 66%]
tests/test_add.py::test_add32_associativity PASSED                           [100%]

What is going on? Well, identity, commutativity, and associativity also hold for 32-bit integers with overflow semantics. If we let AI generate not only the implementation but also the properties, we may end up with a correct implementation, but not the one we wanted! In this case, imagine an LLM has added the @given decorators for the inputs to be in the range in $[0, 2^{32})$, whereas we wanted unbounded integers! This is an example of the classical question in requirements engineering: do we get things right vs. do we get the right things.

Just to double check that it is not the random chance, I ran Apalache on the TLA⁺ specification above with Add32 instead of Add:

$ docker run --rm -v `pwd`:/var/apalache ghcr.io/apalache-mc/apalache \
  check --length=0 --init=Init32 --inv=Inv32 Add.tla
...
Checker reports no error up to computation length 0
Total time: 2.205 sec

Further, the SMT solver Z3 confirms that identity, commutativity, and associativity hold for 32-bit integers with overflow semantics. This is provided that we pick the integers from the range $[0, 2^{32})$, which we do with Init32.

However, this is not what we wanted initially. Let’s catch the adversarial developer with the PBT tests that pick unbounded non-negative integers:

@given(st.integers(0))
def test_add32_unbounded_inputs_identity(a):
    """Test identity property for add32: a + 0 = a."""
    assert add32(a, 0) == a
    assert add32(0, a) == a

@given(st.integers(0), st.integers(0))
def test_add32_unbounded_inputs_commutativity(a, b):
    """Test commutativity property for add32: a + b = b + a."""
    assert add32(a, b) == add32(b, a)


@given(
    st.integers(0),
    st.integers(0),
    st.integers(0)
)
def test_add32_unbounded_inputs_associativity(a, b, c):
    """Test associativity property for add32: (a + b) + c = a + (b + c)."""
    assert add32(add32(a, b), c) == add32(a, add32(b, c))

This time, Hypothesis catches the issue with identity:

$ poetry run pytest tests/test_add.py  --verbose \
  -k "test_add32_unbounded_inputs_identity or test_add32_unbounded_inputs_commutativity or test_add32_unbounded_inputs_associativity"
...
a = 4294967296

    @given(st.integers(0))
    def test_add32_unbounded_inputs_identity(a):
        """Test identity property for add32: a + 0 = a."""
>       assert add32(a, 0) == a
E       assert 0 == 4294967296
E        +  where 0 = add32(4294967296, 0)
E       Falsifying example: test_add32_unbounded_inputs_identity(
E           a=4_294_967_296,
E       )

4.2. Is property-based testing a magic tool?

Let’s stop and think about our example. How did Hypothesis catch the issue over an unbounded integer domain? Even it was picking the integers from the interval $[0, 2^{64})$, the chance of picking 4294967296 by uniform random sampling is pretty slim. Yet, Hypothesis keeps picking this number.

Well, the trick is that its input generator tries the well-known “magic numbers” such as 0, 1, -1, 2**32, 2**64, etc. In this sense, Hypothesis does not use uniform random sampling. See the discussion on domain and distribution in the Hypothesis documentation for more details.

What if our adversarial developer hallucinated an implementation that stays undetected by Hypothesis? This is what our next example is about.

4.3. Hallucinating addition over 256-bit unsigned integers

This time, the adversarial developer uses 256-bit unsigned integers with overflow semantics:

def add256(a: int, b: int) -> int:
    return (a + b) % (2**256)

If you think that using 256-bit integers is absurd, well, the Ethereum Virtual Machine (EVM) does exactly that. So an LLM could have adapted the above code from an EVM-related code base.

@given(st.integers(0))
@settings(max_examples=100000)
def test_add256_unbounded_inputs_identity(a):
    """Test identity property for add256: a + 0 = a."""
    assert add256(a, 0) == a
    assert add256(0, a) == a

@given(st.integers(0), st.integers(0))
@settings(max_examples=100000)
def test_add256_unbounded_inputs_commutativity(a, b):
    """Test commutativity property for add256: a + b = b + a."""
    assert add256(a, b) == add256(b, a)


@given(
    st.integers(0),
    st.integers(0),
    st.integers(0)
)
@settings(max_examples=100000)
def test_add256_unbounded_inputs_associativity(a, b, c):
    """Test associativity property for add256: (a + b) + c = a + (b + c)."""
    assert add256(add256(a, b), c) == add256(a, add256(b, c))

This time, the adversarial developer gets away, all tests pass:

$ poetry run pytest tests/test_add.py --verbose -k \
  "test_add256_unbounded_inputs_identity or test_add256_unbounded_inputs_commutativity or test_add256_unbounded_inputs_associativity" 
...
tests/test_add.py::test_add256_unbounded_inputs_identity PASSED                [ 33%]
tests/test_add.py::test_add256_unbounded_inputs_commutativity PASSED           [ 66%]
tests/test_add.py::test_add256_unbounded_inputs_associativity PASSED           [100%]

Why? Hypothesis does not try $2^{256}$ as a magic number. I gave it the budget of 100,000 examples, so it had a chance to try multiple powers of two, but it did not try anything above $2^{256} - 1$.

We can make sure that the identity test indeed fails when we pass $2^{256}$ as an example:

@given(st.integers(0))
@settings(max_examples=100000)
@example(2**256)  # This should fail!
def test_add256_unbounded_inputs_identity(a):
    """Test identity property for add256: a + 0 = a."""
    assert add256(a, 0) == a
    assert add256(0, a) == a

4.4. Catching the adversarial developer with Apalache

Here is how we modify the TLA⁺ specification to use Add256:

Add256(a, b) == (a + b) % (2^256)

InitNat ==
    /\ x \in Nat
    /\ y \in Nat
    /\ z \in Nat

Inv256 ==
    /\ Identity(Add256)
    /\ Commutativity(Add256)
    /\ Associativity(Add256)

Apalache immediately finds the issue with identity when we run it with Add256:

$ docker run --rm -v `pwd`:/var/apalache ghcr.io/apalache-mc/apalache \
  check --length=0 --init=InitNat --inv=Inv256 Add.tla
...
State 0: state invariant 0 violated.
Total time: 2.272 sec

If we check the counterexample, we see that the solver picks the value $2^{256}$ for x:

$ head -n 14 _apalache-out/Add.tla/2025-12-22T15-20-31_8166638658721555415/violation.tla
---------------------------- MODULE counterexample ----------------------------
EXTENDS Add

(* Constant initialization state *)
ConstInit == TRUE

(* Initial state [_transition(0)] *)
State0 ==
  x
      = 115792089237316195423570985008687907853269984665640564039457584007913129639936
    /\ y = 0
    /\ z = 0

This is not just luck and not an heuristic! Apalache delegates solving to the SMT solver Z3, which solves integer constraints. If you want to make sure that it’s not using magic numbers, go and change the modulo operator in Add256 to a large prime number, e.g., $2^{256} + 297$. Rerun the model checker, and it will still find the issue with identity.

4.5. For the curious: how Apalache and Z3 work together

Our example is so simple that we can even go over the actual SMT constraints that Apalache generates. Let’s run Apalache with the option --debug:

$ docker run --rm -v `pwd`:/var/apalache ghcr.io/apalache-mc/apalache \
  check --debug \
  --length=0 --init=InitNat --inv=Inv256 Add.tla

Open the file _apalache-out/Add.tla/2025-12-23T09-31-46_16436938462355564409/log0.smt (the timestamp will be different on your machine). The log is pretty verbose. Here are the crucial parts, which I’ve accompanied with explanations:

-- the initial value of `x`
(declare-const $C$6 Int)
-- `x` is a natural number
(assert (>= $C$6 0))
-- introduce a boolean variable for the identity property
(declare-const $C$9 Bool)
-- Encode the identity property for `Add256` and `x`.
-- The huge number is 2^256.
(assert (= $C$9
   (= (mod $C$6
           115792089237316195423570985008687907853269984665640564039457584007913129639936)
      $C$6)))
-- assert that the identity property is violated
(declare-const $C$10 Bool)
(assert (= (not $C$10) $C$9))
(assert $C$10)
-- check, whether the above constraints have a solution
(check-sat)

If you want to understand what is going on, read the comments above. At first, I was actually surprised that the SMT constraints did not contain addition at all. Then I recalled that Apalache has a bunch of rewriting rules that simplify the constraints. In this case, the symbolic model checker has applied the property a + 0 = a internally to get rid of the addition (yeah, it is the identity property!). It was an equivalent transformation, so we are still left with the modulo operator.

Essentially, we are asking Z3 to solve these inequalities over integers:

What is crucial here is that the SMT solver Z3 has a strict contract with the user. When we give it a set of constraints and ask it to check their satisfiability, it will apply sound algorithms to arrive at one of the three answers:

• sat: there is a solution to the above constraints, i.e., an assignment of values to the variables that makes the constraints true,

• unsat: there is no solution, and

• unknown: the constraints are too hard, or it took the solver too long to solve them.

In contrast to PBT, it is not just like “I tried a few random inputs and did not find a bug”. If Z3 answers sat, there is indeed a solution to the constraints, and the solver gives it to us as a model. If it returns unsat, there is no solution. Whenever you see unknown, it’s a bad day. Sometimes, it also indicates a bug in the solver itself, as I’ve reported to the Z3 developers a few times. However, Z3 is pretty reliable in my experience, and producing an unknown is an achievement, unless you set very tight timeouts, or use tricky non-linear arithmetic.

If you are further interested in how Z3 actually solved the above constraints, a simple answer is that it used something like the Simplex algorithm for integer linear programming. The constraints are linear in our case, so Z3 could apply this algorithm to find a solution. Most likely, Z3 used a more recent algorithm, but the idea is similar.

In case you really want to know how SMT solvers work under the hood, I recommend starting with the book on Decision Procedures by Kroening and Strichman.

5. Conclusions

We all have to learn how to write high-quality properties and understand the boundaries of the “magic” tools. Even if you don’t use LLMs, your peers will.

How to learn writing good properties? We can play with property-based testing. However, PBT tools are not reliable teachers. By their random nature, a PBT tool may miss a bug on one run and find it on another run. Don’t get me wrong. Property-based testing has its value, as many other testing and verification techniques. However, PBT is not a silver bullet. It may miss bugs, especially if the input generator does not cover the right input space well enough.

Shall we use interactive provers like Lean and Rocq? Learning how to prove code correct definitely helps! However, these tools only tell us that the proof does not go through. It would not give us a counterexample. State-of-the-art provers also recommend using PBT for bug finding.

In my opinion, model checking is the best way to learn how to write good properties. You can write as many properties as you like, and the model checker will produce you counterexamples, or not. Importantly, model checkers come with a guarantee of not having a bug in their search scope, if they terminate. See my blog post on the value of model checking on that.

Usually, I recommend people to start with TLC. It works by state enumeration and easy to understand. If your search scope is small, TLC is a good learning tool. In our example, the search scope is astronomical. In this case, Apalache is there to help.

By the way, our example was so simple, that we could encode it in Z3 directly via its python bindings. We could use other model checkers. If you do that, let me know!

6. Bonus: Hypothesis + Crosshair

Hypothesis offers an integration with Crosshair, which is a symbolic execution engine for Python using Z3. I did not explore this integration in depth. Claude told me that it is sufficient to just add this import to the test:

import hypothesis_crosshair_provider

Well, this did not help me to find the violation of identity for add256. If you know how to make Crosshair work with Hypothesis, please let me know!

When we run Crosshair directly on the add256 implementation, it finds the issue with identity right away:

$ poetry run crosshair check tests.test_add_crosshair.check_add256_identity
.../python/tests/test_add_crosshair.py:12: error: false when calling check_add256_identity(115792089237316195423570985008687907853269984665640564039457584007913129639936) (which returns False)

The Crosshair test looks like follows:

from crosshair.core_and_libs import standalone_statespace
from pbt_add import add256


def check_add256_identity(a: int) -> bool:
    """
    Check identity property for add256: a + 0 = a.
    
    pre: a >= 0
    post: _
    """
    return add256(a, 0) == a and add256(0, a) == a

Date: December 15, 2025

Note: I mostly stopped using LLMs for proof-reading my texts, so you know it is not all generated. Enjoy my typos and weird grammar!

Abstract. As promised in the blog post on small-scope hypothesis, I am continuing with the main body of the talk that I presented at the internal Nvidia FM Week 2025. This blog post is rather long. If you do not want to read the whole post, here are the most exciting new developments:

• A new JSON-RPC server API for Apalache, which allows external tools and scripts to drive the symbolic execution of TLA⁺ specifications and interact with the solver. Read the section on The new JSON-RPC API of Apalache.

• A new approach to conformance testing of TLA⁺ specifications and real implementations, called interactive symbolic testing. This approach is inspired by the work of McMillan and Zuck (2019) on testing of the QUIC protocol with IVy and SMT. Read the section on Interactive symbolic testing with SMT.

• A case study on testing multiple open-source implementations of TFTP, including unexpected (but not harmful) deviations from the protocol. This case study includes the experience report on using Claude to bootstrap the harness for testing TFTP implementations against the TLA⁺ specification. Read the section on Bootstrapping the testing harness with Claude and Testing against adversarial behavior. My point is not not to brainwash you into LLMs, but to show what works for me and what does not.

• The specification and the test harness are openly available. Check the Github repository.

In this blog post, I am using TLA⁺. The same tooling and results equally apply to Quint.

Contents:

1. Introduction
2. Model-based testing and trace validation
3. Interactive symbolic testing with SMT
4. The new JSON-RPC API of Apalache
5. Case study: TFTP protocol
6. Initial TLA⁺ specification of TFTP
7. Bootstrapping the testing harness with Claude
8. Debugging the TLA⁺ specification with the implementation
9. Testing against adversarial behavior
10. The specification as a differential testing oracle
11. Prior Work
12. Conclusions

1. Introduction

This work aims at demonstrating how to answer the following two questions with Apalache:

1. How to test the actual implementation against its TLA⁺ specification?

2. How to test the TLA⁺ specification against the actual implementation?

For long time, these questions have been mostly ignored by the TLA⁺ community. Over the last 4-5 years, researchers started to look into these two questions and found out that having a connection between the specification and the implementation is much more useful than it was initially thought. (The engineers were telling this to me all the time!) Check the prior work section for the papers and talks on this topic. Roughly speaking, the approaches follow the two ideas:

• Model-based testing (MBT). The TLA⁺ specification is used to generate test cases that are then executed against the implementation. This is an answer to question 1 above. The state exploration is driven by the specification. Hence, we are testing, whether the implementation matches the inputs and outputs, as produced by the specification.

• Trace validation (TV). The traces are collected from the implementation and checked against the TLA⁺ specification. This is an answer to question 2 above. State exploration is driven by the implementation, e.g., by executing the existing test suites, or just by running the system for some time. Hence, we are testing whether the specification matches the inputs and outputs of the implementation. Alternatively, we may check whether the implementation states may be lifted to the specification states, in order to produce a feasible trace in the specification.

If you re-read the description of MBT and TV above, you may notice that there are two more dimensions of how to do testing:

• State-based. In this case, we have to establish a relation between the implementation states and the specification states in each step of the trace. This usually done by defining mapping functions, either from the implementation states to the specification states, or vice versa. Notice that mapping an implementation state to a specification state is usually much easier, as it involves state abstraction (e.g., dropping some variables). Mapping a specification state to an implementation state is more difficult, as it involves state concretization, e.g., choosing a representative concrete value for each abstract value in the specification state. For example, if the specification says $x \in [10, 20]$, then we have to choose a concrete value for $x$ in this range, e.g., at random.

• Action-based. In this case, we have to establish a relation between the implementation actions and the specification actions. Again, we would need to define mappings. Interestingly, in my experience, defining action mappings is way easier than defining state mappings.

2. Model-based testing and trace validation

2.1. Model-based testing in one picture

Without going into too many details, the following picture illustrates the main idea of model-based testing. We generate an “interesting” trace with a model checker, e.g., with Apalache. This trace is fed to the test harness that: (1) does action concretization, (2) executes the actions against the implementation. The moment the implementation refuses to replay an action, we know that there is a divergence. Notice that we often do not even need to query the system for its current state, as we only care about the actions.

One downside of this approach is that the model checker can be quickly overwhelmed by the many possible action interleavings unless the search scope is further restricted. In my experience, the SMT solver Z3 slows down dramatically when it must solve two problems simultaneously:

1. Choose a sequence of actions (a schedule) to explore, and

2. Find variable assignments (states) that produce a feasible trace for the chosen schedule.

When a schedule is fixed, the SMT solver must solve far fewer constraints because it mainly propagates values through the actions. If the solver must also pick a schedule, it must backtrack along two axes: (1) schedules and (2) states. This increases solving times in practice.

To mitigate this, Apalache lets you randomly sample schedules and execute them symbolically. To enumerate different “interesting” schedules, the user can define a view operator, which usually projects state variables to more abstract values. The model checker will then produce traces projected onto those views. This works significantly better for test generation in practice. However, this exploration strategy is fixed and cannot be changed without modifying Apalache itself.

2.2. Trace validation in one picture

Trace validation is conceptually simpler than model-based testing. We simply execute the system under test (SUT) and collect traces. These traces are then mapped to the abstract states, if necessary, and checked against the specification.

This approach has been tried in multiple projects that use the exhaustive-state model checker TLC as the back-end solver. See the prior work section.

Trace validation also has its challenges:

1. We need a good test suite, in order to produce “interesting” traces. However, test cases are usually written for the happy-path scenarios. Hence, it is easy to miss handling of error cases and faults. Srinidhi Nagendra et al. (2025) address this issue by fuzzing the tests.

2. Someone has to instrument the SUT to trace the relevant events. In some cases, it easy to do, e.g., by tracing message exchanges, as presented by Markus Kuppe et. al. (2024). In other cases, it may be quite difficult to do, e.g., when we want to dump the internal states of the SUT. In a concurrent system this may require a global lock and traversing large data structures. In a distributed system, this may further require a distributed snapshot or using vector clocks.

3. We have to run the whole system to collect traces. It is hard to isolate one component, e.g., one network node.

3. Interactive symbolic testing with SMT

As we can see, both model-based testing and trace validation in their above formulation are non-interactive. They both require a complete trace to be produced first, and there is no feedback loop between the specification and the implementation.

There is a third way to do conformance testing that leverages SMT solvers, yet receives feedback from the implementation during the testing. I will call it interactive symbolic testing. I think the first time I heard about this approach was from Giuliano Losa, when he explained the paper by Ken McMillan and Leonore Zuck (2019) to me. If you have not read this paper yet, I highly recommend doing so. On the naming side, McMillan and Zuck call their approach “specification-based testing”. I find this name to be a bit non-descriptive, as MBT is also specification-based.

The idea is to generate an action with the SMT solver by following the specification, execute it against the implementation, and then feed the results back to the SMT solver to generate the next action. This way, we can systematically explore the protocol specification while getting feedback from the implementation.

The picture below illustrates this approach, by approximately following the internal transition executor of Apalache.

To implement this approach to testing with Apalache, we would have to find a way for Apalache and the test harness to communicate. My experience with development of Apalache shows that fixing exploration strategies inside the model checker is not a good idea. People always want to tweak them a bit for their purposes. Given this observation, Thomas Pani and I have decided to implement a simple server API for Apalache that would allow external tools to drive the symbolic execution of TLA⁺ specifications.

4. The new JSON-RPC API of Apalache

Thomas and I wanted to have a lightweight API that we could use from any programming language without writing too much boilerplate code. At this point, every engineer would whisper: hey, you need gRPC, I’ve got some. Well, we tried gRPC in the integration of Apalache with Quint. It is hard to call gRPC lightweight.

So we have decided to go with JSON-RPC this time, which is a very simple protocol that works over HTTP/HTTPS. Implementing a JSON-RPC server is quite straightforward. Since Apalache is written in Scala, which is JVM-compatible, we can use the well-known and battle-tested libraries. Perhaps, a bit unexpectedly for a Scala project, I’ve decided to implement this server with Jetty for serving the HTTP requests and Jackson for JSON serialization. (The reason is that we have already burnt ourselves with fancy but poorly supported libraries in Scala.) The resulting server is lightweight and fast. Moreover, it can be tested with command-line tools like curl.

The state-chart diagram of the Apalache JSON-RPC server for a single session is shown below.

To see a detailed description of this API, check Apalache JSON-RPC. Just to give you the taste of it, here is how you start the server without having anything installed but Docker:

$ docker pull ghcr.io/apalache-mc/apalache
$ docker run --rm /tmp:/var/apalache -p 8822:8822 \
    ghcr.io/apalache-mc/apalache:latest \
    server --server-type=explorer

Now, we create a new Apalache session with a TLA⁺ specification (in a separate tab):

$ SPEC=`cat <<EOF | base64
---- MODULE Inc ----
EXTENDS Integers
VARIABLE
  \* @type: Int;
  x
Init == I:: x = 0
Next == (A:: (x < 3 /\\ x' = x + 1)) \\/ (B:: (x > -3 /\\ x' = x - 1))
Inv3 == Inv:: x /= 0
\* @type: () => <<Bool, Bool, Bool>>;
View == <<x < 0, x = 0, x > 0>>
=====================
EOF`
$ curl -X POST http://localhost:8822/rpc \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"loadSpec","params":{"sources": [ "'${SPEC}'" ],
       "invariants": ["Inv3"], "exports": ["View"]},"id":1}'

Is not that amazing? No protobuf, no code generation, just pure shell and readable JSON.

Having the specification loaded, we load the predicate Init into the solver context, which is encoded as transition 0:

$ curl -X POST http://localhost:8822/rpc \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"assumeTransition","params":{"sessionId":"1",
       "transitionId":0,"checkEnabled":true},"id":2}'

Assuming that the previous call returned ENABLED, we switch to the next step, which applies the effect of Init to the current symbolic state:

$ curl -X POST http://localhost:8822/rpc \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"nextStep","params":{"sessionId":"1"},"id":3}'

Now, we can check the invariant Inv3 against all states that satisfy Init:

$ curl -X POST http://localhost:8822/rpc \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"checkInvariant",
       "params":{"sessionId":"1","invariantId":0},"id":3}'

Since invariant Inv3 is violated by the initial state, the server returns VIOLATED, along with a counter-example trace:

{
  "jsonrpc": "2.0",
  "id": 3,
  "result": {
    "sessionId": "1",
    "invariantStatus": "VIOLATED",
    "trace": {
      "#meta": {
        "format": "ITF",
        "varTypes": { "x": "Int" },
        "format-description": "https://apalache-mc.org/docs/adr/015adr-trace.html",
        "description": "Created by Apalache on Thu Dec 11 16:56:47 CET 2025"
      },
      "vars": [ "x" ],
      "states": [ {
          "#meta": { "index": 0 },
          "x": { "#bigint": "0" }
      } ]
    }
  }
}

The trace is encoded in the ITF format, which is a simple JSON-based format for TLA⁺ and Quint traces.

Had the invariant been violated on a deeper trace, we would have to assume more transitions by calling assumeTransition and nextStep multiple times.

If you want to access this API from Python right away, use two helper libraries:

• apalache-rpc-client for interacting with the JSON-RPC server of Apalache, and

• itf-py for serializing and deserializing ITF traces.

5. Case study: TFTP protocol

To experiment with interactive symbolic testing and the new JSON-RPC API, I wanted to choose a relatively simple network protocol that had multiple implementations. After several sessions with ChatGPT, I ended up with the Trivial File Transfer Protocol (TFTP) as a reasonable target for this small project.

The Wikipedia page on TFTP gives us a good overview of the protocol. In short, TFTP is a simple protocol to transfer files over UDP. It supports reading and writing files from a remote server. It is mostly used for booting from the network. The protocol is simple enough to be specified in TLA⁺ without too much effort, yet it has enough complexity to make the testing effort interesting. Actually, I’ve only specified reading requests (RRQ) and no writing requests (WRQ) to keep the scope manageable.

You can find more detailed specifications in the original RFC 1350, as well as in its extensions RFC 2347, RFC 2348, and RFC 2349. RFC 1350 defines a simple non-negotated version of the protocol. Below is an example of such an interaction between the client and the server. Notice that the client first sends a read request (RRQ) to the server on the control port 69, which responds with the first data block (DATA) on a newly allocated ephemeral port. The client acknowledges (ACK) the received data block on the same ephemeral port. This continues until the server sends the last data block, which is smaller than the maximum block size (512 bytes by default).

Further, RFC 2347 defines an option negotiation phase that happens right after the read request. The client and the server may negotiate options like block size, timeout, and transfer size. RFC 2348 defines the block size option, while RFC 2349 defines the transfer size option. Below is an example interaction with option negotiation:

The cool thing about TFTP is that it has multiple open-source implementations of TFTP clients and servers in different programming languages. Here are some of them:

• tftp-hpa is the canonical implementation of TFTP for Linux (and UNIX?) in C.

• atftpd is advanced TFTP, which is intended for fast boot in large clusters, also in C.

• dnsmasq is a lightweight DNS and DHCP server that also includes a TFTP server, in C.

• rs-tftpd (Rust) is an implementation of a TFTP server in Rust.

• gotfpd (Go) is an implementation of a TFTP server in Go.

• busybox also has its minimalistic implementation for file reads.

6. Initial TLA⁺ specification of TFTP

In the first stage of this experiment, I read the RFCs and wrote a TLA⁺ specification of the TFTP protocol. At that stage, I did not introduce packet loss, duplication, or reordering. I just wanted to have a simple working specification that I could use for testing the implementations. This stage took me just two days. Well, I have been writing plenty of TLA⁺ specifications in the past.

You can check this initial specification in the initial commit of the testing repo. The main body of the specification lives in tftp.tla, which imports several several auxiliary modules:

• typedefs.tla defines the types of the data structures and the basic constructors for these data structures. Since I am using Apalache, the specification needs type definitions. Luckily, these days, I just write the type definitions in comments and let Claude generate the auxilliary operators such as constructors and accessors. If you already have an untyped specification, Claude is good at figuring out the types in the agent mode. Just use this prompt.

• util.tla defines common utility definitions such as Min, Max, and option conversions.

Finally, MC2_tftp.tla defines a protocol instance of two clients and one server. If you stumble upon the definitions that end with View there, ignore them. They are not essential for this blog post. I used them to experiment with more advanced symbolic exploration scripts.

If you are not familiar with TLA⁺, or your TLA⁺ skills are rusty, I recommend giving one of the definitions and this prompt to ChatGPT. It actually explains TLA⁺ quite well:

Assume that I am a software engineer. I don't know TLA+ but know Golang or Rust.
Explain me this TLA+ snippet using my knowledge: ...

To see the kinds of actions this initial specification had, have a look at the definition of Next in tftp.tla:

Next ==
    \* the actions by the clients
    \/  \E srcIp \in CLIENT_IPS, srcPort \in PORTS:
            \E filename \in DOMAIN FILES, timeout \in 1..255:
                \* "man tftpd": 65464 is the theoretical maximum for block size
                \* https://linux.die.net/man/8/tftpd
                \E tsize \in 0..FILES[filename], blksize \in 0..65464:
                    \* choose a subset of the options to request
                    \E optionKeys \in SUBSET OPTIONS_RFC2349:
                        LET options ==
                            mk_options(optionKeys, blksize, tsize, timeout)
                        IN
                        ClientSendRRQ(srcIp, srcPort, filename, options)
    \/  \E udp \in packets:
            \/ ClientRecvDATA(udp)
            \/ ClientRecvOACK(udp)
            \/ ClientRecvErrorAndCloseConn(udp)
    \/  \E ipPort \in DOMAIN clientTransfers:
            ClientTimeout(ipPort)
    \* the server
    \/  \E udp \in packets:
            \/ ServerRecvRRQ(udp)
            \/ ServerSendDATA(udp)
            \/ ServerRecvAckAndCloseConn(udp)
            \/ ServerRecvErrorAndCloseConn(udp)
    \/  \E ipPort \in DOMAIN serverTransfers:
            ServerTimeout(ipPort)
    \* handle the clock and timeouts
    \/  \E delta \in 1..255:
            AdvanceClock(delta)

Falsy invariants. As I always do, I also specified “falsy invariants” to produce interesting examples. For example, using the invariant RecvThreeDataBlocksEx below, I can easily produce a trace where a client receives three data blocks from the server.

\* Check this falsy invariant to see an example of a client receiving 3 blocks.
RecvThreeDataBlocksEx ==
    ~(\E p \in DOMAIN clientTransfers:
        Len(clientTransfers[p].blocks) >= 3)

If you want to try it right way without installing anything, just do this with docker:

$ git clone git@github.com:konnov/tftp-symbolic-testing.git
$ git checkout 6fb00d1878b7e37a629868ac25b853d95b16cbdc
$ docker pull ghcr.io/apalache-mc/apalache
$ docker run --rm -v `pwd`:/var/apalache ghcr.io/apalache-mc/apalache \
  check --inv=RecvThreeDataBlocksEx MC2_tftp.tla

Trace visualization. Since Apalache emits traces in the ITF format, which has a very simple schema in JSON, it was easy for me to convince Claude to produce a Python script that would convert ITF traces to human-readable state charts in Mermaid. Here is just an example of such a trace produced by Apalache when checking the invariant RecvThreeDataBlocksEx in Mermaid:

sequenceDiagram
    participant ip10_0_0_3_port65000 as 10.0.0.3:65000
    participant ip10_0_0_1_port10000 as 10.0.0.1:10000
    participant ip10_0_0_1_port69 as 10.0.0.1:69

    ip10_0_0_3_port65000->>ip10_0_0_1_port69: RRQ(file1, blksize=0, timeout=4)
    ip10_0_0_1_port10000->>ip10_0_0_3_port65000: DATA(blk=1, 512B)
    ip10_0_0_3_port65000->>ip10_0_0_1_port10000: ACK(blk=1)
    ip10_0_0_1_port10000->>ip10_0_0_3_port65000: DATA(blk=2, 512B)
    ip10_0_0_3_port65000->>ip10_0_0_1_port10000: ACK(blk=2)
    ip10_0_0_1_port10000->>ip10_0_0_3_port65000: DATA(blk=3, 0B)
    ip10_0_0_3_port65000->>ip10_0_0_1_port10000: ACK(blk=3)

This is how it looks like when rendered by Mermaid:

Note on abstractions. Similar to the McMillan and Zuck, I tried to avoid unnecessary abstractions and approximations in the specification. If you look at the type definition of a TFTP packet in typedefs.tla, you will see that all fields except data are modeled as strings and integers:

  // TFTP Packet Types
  @typeAlias: tftpPacket =
    // Read Request (RFC 1350, Figure 5-1, RFC 2347).
    // See RFCs 2348-2349 for the options.
      RRQ({ opcode: Int, filename: Str, mode: Str, options: Str -> Int })
    // Write Request (RFC 1350, Figure 5-1, RFC 2347).
    // See RFCs 2348-2349 for the options.
    | WRQ({ opcode: Int, filename: Str, mode: Str, options: Str -> Int })
    // Acknowledgment (RFC 1350, Figure 5-3)
    | ACK({ opcode: Int, blockNum: Int })
    // Option Acknowledgment (RFC 2347)
    | OACK({ opcode: Int, options: Str -> Int })
    // Data packet (RFC 1350, Figure 5-2)
    // In our specification, we simply pass the length of data instead of the
    // data itself. The test harness should pass the actual data.
    | DATA({ opcode: Int, blockNum: Int, data: Int })
    // Error packet (RFC 1350, Figure 5-4)
    | ERROR({ opcode: Int, errorCode: Int, msg: Str })
  ;

Thinking about it now, I could even model data as a sequence of bytes, but it was obvious to me that only the length of data matters for the protocol logic.

7. Bootstrapping the testing harness with Claude

Now, we have the initial TLA⁺ specification of TFTP and the standard implementation tftp-hpa, which is the default tftpd server on Linux.

I wanted to avoid running the TFTP server on my laptop. What if I accidentally find a bug that corrupts my file system? So I have decided to run the server and the client harnesses in Docker containers. This way, I could easily reset the SUT and have an isolated network for the TFTP server and clients.

Below is the architecture of the test harness that I had in mind. It’s quite a bit overengineered for testing TFTP. I also wanted to experiment with Docker networking and managing multiple containers for potential future projects.

┌──────────────────────────────────────────────────────────────────┐
│                        Host Machine                              │
│                                                                  │
│  ┌────────────────────────────────────────────────────────────┐  │
│  │ harness.py                                                 │  │
│  │  - Coordinates symbolic execution                          │  │
│  │  - Manages Apalache server                                 │  │
│  │  - Controls Docker containers                              │  │
│  │  - Generates and saves test runs                           │  │
│  └────────┬────────────────────────┬──────────────────────────┘  │
│           │                        │                             │
│           ▼                        ▼                             │
│  ┌─────────────────┐     ┌──────────────────────────┐            │
│  │ Apalache Server │     │  Docker Manager          │            │
│  │  (port 8822)    │     │  - Network: 172.20.0.0/24│            │
│  └─────────────────┘     └──────────┬───────────────┘            │
│                                     │                            │
└─────────────────────────────────────┼────────────────────────────┘
                                      │
                         ┌────────────┴──────────────┐
                         │   Docker Network          │
                         │   (172.20.0.0/24)         │
                         │                           │
         ┌───────────────┼───────────────────────────┼─────────────┐
         │               │                           │             │
         ▼               ▼                           ▼             │
  ┌─────────────┐ ┌─────────────┐          ┌─────────────┐         │
  │ TFTP Server │ │  Client 1   │          │  Client 2   │         │
  │ 172.20.0.10 │ │ 172.20.0.11 │          │ 172.20.0.12 │         │
  │             │ │             │          │             │         │
  │ tftp-hpa    │ │ Python      │          │ Python      │         │
  │ Port: 69    │ │ TCP: 15001  │          │ TCP: 15002  │         │
  │ Data:1024-27│ │ (control)   │          │ (control)   │         │
  └─────────────┘ └─────────────┘          └─────────────┘         │
         ▲               │                           │             │
         │               │    UDP TFTP packets       │             │
         └───────────────┴───────────────────────────┘             │
                                                                   │
                         Docker Containers                         │
                                                                   │
                         tftp-test-harness:latest                  │
                                                                   │
└──────────────────────────────────────────────────────────────────┘

LLMs will do the work? As you could have guessed, I had no interest in writing the Docker files and the test harness from scratch. Having heard from so many people that LLMs are so amazing, I have decided to give Claude a try at generating the test harness.

Hence, I spent about four hours writing a very detailed prompt for Claude that explained how I want the test harness to look like (the above architecture diagram is actually generated by Claude from my prompt).

Pushing the button! So I’ve run Claude in the agent mode with my prompt and went for a coffee break. You can see the first generated version in this commit. The result looked so exciting and amazing until I looked at CHECKLIST.md:

## Notes

- The framework is complete and production-ready
- Remaining work is mostly about connecting components
- Each task is independent and can be tackled separately
- Estimated effort: 4-8 hours for core integration (tasks 1-4)
- Additional 2-4 hours for polish and testing (tasks 5-8)

What is going on? Claude left me homework? I was also baffled by the hourly estimates: Are these Claude hours or my hours? In the hindsight, the estimate was surprisingly accurate. It took me about 1.5 days to make this code do the first test run that made the harness exchange UDP packets with the TFTP server.

Then I looked at harness.py, which was supposed to be “complete and production-ready”. Guess what? The main loop was left as a TODO!

        # TODO: Implement actual TFTP operation execution
        # This would involve:
        # - Querying Apalache for the transition details
        # - Sending commands to the TFTP client in Docker
        # - Collecting UDP packet responses
        # - Parsing the responses

The overall structure was there, but the most important pieces were left as TODOs. Fine. It did the tedious part at least. So I started to chat with Claude again to implement the missing pieces. If you look at the commit history, you will see plenty of spaghetti code generated by Claude. In the end, it became a bit better after my guidance, but I had to rewrite it at some point.

Even though I am making jokes about LLMs here, I must say that Claude really helped me to debug the Docker setup and produce the python code for communicating over UDP in modern Python. I could easily lose a couple of days there.

Of course, the exploration logic was totally broken. After all, there is not much for LLMs to learn from. We are doing something new here!

1.5 days later. Something was working, but even the happy path was not there. So I had to do the baby steps with Claude. Here are just a few examples from my Copilot chat:

Let’s implement sending the RRQ packet over the wire.

…

Now I am receiving a response like below. This is good! What I want you to do next. Decode the response and construct the expected packet for the TLA+ specification. Save this as the expectation that we will use in the next step.

…

You should not construct a TLA+ expression. Rather, convert the packet to an ITF value using itf-py.

…

Can you implement this case for receiving OACK from the server and sending ACK by the client to the server.

2 more days later. I had the happy path working. At this point, I was tired of reading the harness logs. So I needed some form of visualization for each run. Obviously, I wanted to have the same kind of Mermaid diagrams as before.

So I asked Claude to generate a script that would reconstruct the sequence diagram from the harness logs. Well, it took me longer than expected. At some point, Claude was producing quite convoluted log parsers with regular expressions and python loops. Of course, it needs a human to define a simple log format instead.

Below is an example of such a test run, visualized from the log by the generated script in Mermaid:

If you look at the above diagram carefully, you will notice that server responses come in two flavors:

1. The dashed arrows indicate that the client has received the UDP packet from the UDP socket.

2. The solid arrows indicate that the UDP packet was successfully replayed with the TLA⁺ specification.

8. Debugging the TLA⁺ specification with the implementation

At that point, the tests started to produce actual interactions between the TLA⁺ specification (as solved by Apalache and Z3) and the real TFTP server. This brought a lot of surprises! I am going to present some of them below.

In this debugging session, I am keeping the scorecard of how many times the TLA⁺ specification was wrong versus how many times the implementation (tftp-hpa) was wrong. The scorecard at this point looks like this:

Actually, tftp-hpa is a quite mature implementation, so I was not expecting any bugs there. Keep reading to see what I found.

8.1. Sending errors on read request

The first surprise came from my misunderstanding of how exactly TFTP is supposed to reply to a malformed read request (RRQ). Since a client sends RRQ to the control port 69 of the server, I thought that the server would reply with an error packet (ERROR) from the port 69, instead of introducing a new ephemeral port.

This is what RFC 2347 says about option negotiation:

…the server should simply omit the option from the OACK, respond with an alternate value, or send an ERROR packet, with error code 8, to terminate the transfer.

No explanation about the port from which the ERROR packet is sent. Well, my understanding was wrong. The server always allocates a new ephemeral port for sending the ERROR packet. This kind of makes sense, as the implementation simply forks on a new request. One score to the implementation:

8.2. The server may send duplicate packets

Well, I knew that, but was lazy to write an action in the specification that would handle duplicate packets. This is a typical shortcut when writing a specification, since duplicate packets do not change the specification state and considered “stuttering” steps. The server implementation retransmitted a DATA packet, which produced a deviation in the TLA⁺ specification. Another score to the implementation:

Formally speaking, this action does not affect protocol safety, so it is tempting to simply skip duplicates. However, in conformance testing, we have to handle all possible actions of the implementation, even if they produce stuttering steps in the theory of TLA⁺.

8.3. Input-output conformance does not work with UDP!

The next issue was quite interesting. When I read the papers on input-output conformance testing from the 1990s, there was always an assumption that the system under test (SUT) is input-enabled. This means that the SUT can always accept any input at any time and respond to it, possibly, with an error message. This assumption makes sense for synchronous systems (such as vending machines?), where the tester can wait for the SUT to be ready to accept the input.

However, TFTP is not like that at all. The client may send an ERROR packet at any point in time, and the server does not have to reply to it! This is exactly a deviating test run I saw produced by the harness.

So instead of waiting for a reply from the server on each client action, the test harness has to optimistically send the next UDP packet and then retrieve the UDP packets from the server (remember that they live in Docker!).

This is where Claude was useful again. It helped me to collect the UDP packets on the Docker client. Before taking the next step, the harness would retrieve the buffered UDP packets from the Docker clients and replay these packets in the TLA⁺ specification, in arbitrary order.

This makes our testing approach a bit more sensitive to the timing of extracting the buffered UDP packets, but it worked for TFTP.

8.4. The server recycles an ephemeral port on ERROR

Another interesting deviation happened when the server recycled an ephemeral port. RFC 1350 explains how the server allocates ephemeral ports:

In order to create a connection, each end of the connection chooses a TID for itself, to be used for the duration of that connection. The TID’s chosen for a connection should be randomly chosen, so that the probability that the same number is chosen twice in immediate succession is very low.

Well, in our test run, the event of low probability happened (actually, I gave the TFTP server a small range of ports to use):

Actually, this theme of reusing the same ephemeral port happened multiple times in the following debugging iterations. It is probably the most problematic aspect of the protocol, as there is no notion of a session in TFTP. Another score to the implementation:

8.5. The server recycles an ephemeral port on success

Guess what? A very similar thing happened on a successful file transfer as well. Here is a pruned version of the trace that shows this behavior (the initial sequence of RRQ-OACK-DATA-ACK is omitted for brevity):

This behavior seems to be consistent with Section 6 of RFC 1350, though it seems to be ambiguous to me. Anyway, another score to the implementation:

8.6. Mixing the protocol versions

TFTP essentially has two versions: the original version defined in RFC 1350 and the extended version with option negotiation defined in RFC 2347. In combination with packet duplication, this produced a very interesting deviation. I’ve not saved the full trace, but here is what happened. The server processes an RRQ with options and sends an OACK, as per RFC 2347. After that, the TLA⁺ specification of the server receives an earlier RRQ without options and sends a DATA packet in response, as per RFC 1350. This corrupts the internal state of the server in the specification.

Obviously, this is caused by non-determinism in the TLA⁺ specification, which allows the protocol to behave according to both protocol versions at the same time. I had to fix the specification by disallowing the server to behave according to RFC 1350, when it receives an RRQ with options. One score to the implementation:

8.7. More deviations on the specification side

At some point, I got tired of collecting the precise deviations. They still can be recovered from the commit log though. Here are some of the further deviations on the specification side that I fixed:

• The client must send tsize = 0 in RRQ.

• The server should send default timeout if it’s not specified in the options.

• The server may send invalid (e.g., outdated) packets.

• My understanding of TFTP timeouts was wrong. I thought that a timeout was meant to close a transfer session. Instead, timeouts in TFTP are just triggering packet retransmissions. The number of retries is not specified in the RFCs. In practice, tftp-hpa seems to retry 5 times before giving up.

• The server specification should store transfers for triplets (clientIP, clientPort, serverPort) instead of pairs (clientIP, clientPort).

In the end, the implementation scored another 7 points, before tests started to work.

It looks like my TLA⁺ specification was a bit sloppy, in comparison to the mature implementation of tftp-hpa. I have not designed this protocol and did not give much thought to it. Obviously, the engineers have spent much more time thinking about its behavior. You can check the specification in the repository.

9. Testing against adversarial behavior

At some point I thought: My clients are too well-behaved! They never lose, duplicate, or reorder packets. What if they start to misbehave within the protocol boundaries? Would I be able to find bugs in the implementation? Yes, I did. Keep reading.

Hence, I have added one more action that simply lets a client retransmit a previously sent packet in Next:

    \/ \E udp \in packets:
        ClientSendDup(udp)

Below is the action ClientSendDup. It does not change the specification state at all. However, it produces an action that retransmits a packet in the harness:

\* A client resends a duplicate packet that it sent in the past.
\* This is to test for the Sourcerer's Apprentice syndrome.
\* @type: $udpPacket => Bool;
ClientSendDup(_udp) ==
    ClientSendDup::
    /\ _udp.destIp = SERVER_IP
    /\ lastAction' = ActionRecvSend(_udp)
    /\ UNCHANGED <<packets, serverTransfers, clientTransfers, clock>>

You can find the complete specification here.

Protocol deviation. It mostly worked as expected. However, a few traces were reporting deviations. Here is one of them. It’s pretty long. Look for an explanation below.

The last UDP packet is an acknowledgment for block 1 from the server. If you think about the protocol, the server should never send an ACK in the sessions associated with read requests (RRQ). ACK packets are only sent by the clients. Yet, this is what was happening. To double check this, I’ve asked Claude to capture the traffic in pcap files in the Docker containers. Indeed, Wireshark was showing the ACK packet from the server. Moreover, the packet was malformed. It looked like the option acknowledgment (OACK) packet, but had the first bytes of an ACK packet. Sounds like memory corruption!

Here is the core sequence of events that produced this behavior (a few details removed):

1. The client sends RRQ("file1", blksize=NN) to the server (172.20.0.10:69).

2. The server sends a few OACK packets to the client.

3. The client erroneously sends ACK(1) to the server, which is a duplicate packet from an earlier transfer. It could be simply a delayed packet though.

4. The server responds with ACK(1) of length 64, which is basically the OACK packet with the first 4 bytes coming from ACK(1).

Investigation. Luckily, the source code is readily available. I’ve looked into the function tftp_sendfile of tftp-hpa that handles read requests. Indeed, the option negotiation loop receives the option acknowledgment packet OACK and waits for an ACK from the client. There are two cases:

• When it receives an ACK for block 0, it breaks out of the loop and continues with sending data blocks. This is the happy path.

• When it receives an acknowledgment for a block other than 0, block, it simply continues the loop, retransmitting OACK. The issue is that the code uses the same buffer for sending OACK and receiving ACK packets via different pointers! Hence, it later sends an OACK packet that is corrupted with the contents of the ACK packet. I don’t think I would have found this by code review!

Just for fun, I checked it with Claude. It could not identify this issue. The trick is that the same buffer is pointed to by two different pointers, so Claude is not clever enough to track this aliasing. When I explained the issue to Claude, it was ecstatic: You have found a critical!

I’ve continued looking for the blast radius of this bug. Even though it somewhat of memory corruption, it cannot crash the server, as the code is still writing to the same buffer, allocated by the server itself. All it can do is to produce malformed packets. Hence, it could probably crash a sloppy client, but would not do much harm to a well-behaved client and itself. Moreover, if a client crashes in such a case, anybody else on the network could have sent the malformed ACK as well.

So this is a bug (from the specification p.o.v.), but it does not result in a vulnerability. In any case, it was a deviation from the protocol specification. Finally, one point to the specification!

Contacting the author. To be on the safe side, before writing this blog post, I’ve contacted the author of tftp-hpa. As I expected, he also replied that TFTP is an unencrypted unauthenticated protocol, so we should not expect much security there.

10. The specification as a differential testing oracle

After finding the above implementation bug, I have decided to test other TFTP implementations as well. This is where Claude was super useful again. I just asked it to generate Dockerfiles for other implementations, which it did quickly. It happened that a similar issue existed in another implementation. I could not figure out the root cause in the source code of that other implementation, as it is a bit harder to read than tftp-hpa. Hence, not giving the details here.

Except this second deviation, the other implementations worked fine. Overall, the specification scored another point:

What I find really interesting here. Whenever I talk to engineers about formal specifications, they tell me that they would like to do differential testing instead of writing specifications. Meaning that they would like to compare the behavior of one implementation against another implementation. However, differential testing is not magic. It requires test inputs to compare the implementations. Hence, if the test suite is missing adversarial test cases, both implementations may pass the tests, even though they are both wrong.

What we did here with the TLA⁺ specification is something more than just differential testing. First, we have debugged the specification against tftp-hpa, so we have extracted its expected behavior into a relatively small and precise formal specification. Second, we have used this specification to produce the tests for another implementation!

11. Prior Work

In this section, I’ve collected the previous work on model-based testing and trace validation with TLA⁺:

• Nagendra et. al. Model guided fuzzing of distributed systems (2025). Check the talk.

• Cirstea, Kuppe, Merz, Loillier. Validating Traces of Distributed Systems Against TLA+ Specifications (2024). Check the arxiv paper.

• Chamayou et. al. Validating System Executions with the TLA+ Tools (2024). See the talk.

• Halterman. Verifiability Gap: Why We Need More From Our Specs and How We Can Get It (2020). See the talk.

• Davis et al. eXtreme Modelling in Practice (2020). See the talk.

• Kupriyanov, Konnov. Model-based testing with TLA+ and Apalache (2020). See the talk.

• Pressler. Verifying Software Traces Against a Formal Specification with TLA⁺ and TLC (2018). Check the paper.

I am pretty sure that this list is incomplete, so please let me know if you are aware of any other relevant work.

12. Conclusions

This was a lot of text! Thank you for reading it till the end. It may look like this project took me eternity to complete. In reality, it took me about two weeks of part-time work to do it from the start to the end. On one hand, I could probably do some parts of it faster, if I did not rely too much on Claude for generating the test harness. On the other hand, Claude quickly generates the code to start and stop services, parse their logs, etc. All the things Docker were done by Claude, and I did not have to touch them. This is the work that I find annoying and LLMs just do. In this experiment, I’ve burned all of my monthly premium requests included in the Copilot plan. To be fair, I also had to add a few features to the new Apalache API, as I was still experimenting with it.

What I find interesting in the approach outlined here is that it presents a (relatively) lightweight way to testing real-world protocols. Thinking of fuzzing in this context, I don’t think a standard fuzzer would have found the above deviations in TFTP. Indeed, the implementation was not crashing. Nor it was accessing memory out of bounds. It was just producing malformed packets occasionally. To detect this, we needed a test oracle that would tell us, whether a deviation happened. Writing such an oracle manually would be tedious and error-prone. Instead, we have used a formal specification as a precise and unambiguous oracle. Unambiguous does not mean deterministic though. Our oracle is non-deterministic, but it precisely defines the allowed behaviors of the protocol.

In addition to that, tftp-hpa is not just a piece of code that was written by a startup over a weekend, or generated by an LLM. It is a very mature project that has been written by professionals in the times when people had time to think. They took care of the Sorcerer’s Apprentice Syndrome. This is why I was quite surprised to see an unexpected packet from the server.

On the Apalache side, we finally have a symbolic approach that scales much better than bounded model checking! In my experiments with TFTP, the new JSON RPC API was showing the signs of slowing down only after about 200 steps of symbolic execution. This is a huge improvement over the previous approach, where Apalache was slowing down after about 10-20 steps. It is easy to see why. We feed the concrete responses from the implementation into the SMT context, which immediately produce a lot of simplifications.

We can improve this even further to essentially unlimited number of steps. All what is needed is to keep the concrete trace on the harness side and initialize the SMT context with the last state of the trace. We can do it every step, or every N steps. The cool thing is that it all can be done outside of Apalache, on the harness side! This opens the door to quick experimentation with various strategies of mixing symbolic and concrete execution.

Date: December 9, 2025

1. Introduction

In August 2025, Aztec Labs engaged Thomas Pani and Igor Konnov to formally specify and verify the new Aztec Governance Protocol – the core on-chain system that governs Aztec Network.

Over the course of five weeks, we reviewed every line of code in scope and developed a precise formal specification, verified automatically with Apalache. The result: scalable, massively parallel automated verification that explored the entire protocol state space to formally confirm correctness and uncover subtle, cross-contract issues that conventional audits or fuzzing can easily miss.

The team at Aztec Labs reviewed our findings and addressed all of them.

At-a-Glance Metrics

For the impatient reader, here are some key figures:

These runtimes are comparable to large-scale fuzzing campaigns – but with a crucial difference: formal verification explores every possible transaction symbolically, offering exhaustive reasoning rather than probabilistic coverage.

Highlights

Some of the key highlights from this article include:

• Representative issue: Governance Insolvency with root-cause analysis and fix (§9.1)
• Choosing the right tools (§4)
• Bootstrapping the formal specification with AI (§5.2)
• Making verification scale with compositional reasoning and inductive invariants (§5.3, §7.3)
• Showing that the protocol can progress: witnesses of liveness (§7.4)

Our formal report can be accessed via this link and the specifications can be found via this link.

2. Overview of Aztec Governance

Aztec Network’s governance is implemented as a suite of on-chain Solidity contracts. We summarize its multi-contract architecture, which required compositional analysis to verify, in the diagram below. The current implementation extends and formalizes the concepts from the Aztec Governance RFC – see Aztec Governance for the canonical documentation. This post reflects the protocol as of the commit used in our engagement. Parts of the codebase have evolved since our engagement.

(We highlight the key contracts, with no special meaning attached to the colors.)

Rollups and Registry. Aztec Network manages a system of rollups recorded in the Registry, which directs inflationary rewards to a single, designated canonical rollup.

GovernanceProposer. GovernanceProposer (derived from EmpireBase) forms the foundational layer of the voting system, implementing a round-based signaling mechanism to determine which proposals advance to Governance.

Governance. The Governance contract manages the full proposal lifecycle, including submission, voting, and execution. Given their critical role in managing the Aztec Network, Governance incorporates access control, such as whitelisting beneficiaries that participate in voting. On top of that, it implements an emergency proposal mechanism which requires a substantial token lock.

Governance Staking Escrow (GSE). Governance stakes and corresponding voting rights are managed by the Governance Staking Escrow (GSE) contract. GSE enables seamless migration of staked assets to new canonical chains, addressing the “cold-start” problem by ensuring immediate operational support during network upgrades. Proposals made through GovernanceProposer tie back to GSE during execution, verifying that at least two-thirds of the total stake is allocated to the latest rollup.

SlashingProposer and Slasher. The SlashingProposer contract, also derived from EmpireBase, uses the same round-based signaling mechanism to determine which slashing proposals are forwarded to the Slasher contract.

Libraries. The main contracts are supported by a set of custom libraries that use storage-layout compression for gas optimization. These libraries enable the system to retrieve historical, checkpointed state for computing voting power, implement a custom checkpointed set data structure built on OpenZeppelin’s Checkpoints.Trace224 library, implement the vote-tallying algorithm, and provide helper functions that encode the proposal lifecycle state machine.

3. Attack Surface

The attack surface of the Governance Protocol is significant – with over 40 external state-mutating functions across multiple contracts in scope. Moreover, problematic scenarios typically:

• involve multiple contracts,
• exercise them over several transactions, and
• can even involve multiple instances of the same contract (e.g., several Rollups, several Governance contracts, etc.).

Reasoning about time. Governance and GovernanceProposer use the block timestamp to organize signaling and voting phases. GovernanceProposer slots are short (fractions of a minute), while Governance voting periods are much longer (minutes to days). We must therefore reason about long time horizons interrupted by short-lived events.

Malicious external inputs. To make things harder, we also considered scenarios in which a canonical rollup produces erroneous readings from time to time, e.g., due to a fault. For example, what if a canonical rollup starts to return slot numbers from the past (or far in the future)?

This poses both a challenge and an opportunity: standard techniques such as fuzzing, random simulation, or bounded model checking would not get us far – the state and action spaces are prohibitively large.

4. Choosing the Right Tools

With the attack surface in view, the next question was tooling: how to verify the protocol logic without drowning in bytecode.

4.1. Protocol-Level Specification

From an engineer’s perspective, the ideal solution would be to verify correctness directly at the implementation level – that is, to automatically reason about the Solidity code itself. Tools such as Certora Prover, Kontrol, Halmos, and HEVM aim to do exactly this by automating formal reasoning over smart contracts. These tools are remarkable engineering achievements, but their task is inherently complex: among other things, they must reason precisely about stack behavior, memory, storage, and external calls – all the way down to the EVM bytecode.

Before diving into such low-level reasoning, however, we believe it is essential to ensure that the protocol’s logic is sound. If high-level properties of the protocol are violated, then verifying bit-level correctness provides limited value. Once the protocol logic is verified, attention can shift to the implementation.

In this project, we focused on specifying the Aztec Governance Protocol at the logic level. Our main objectives were to:

• specify the high-level behavior of the protocol,
• identify its core invariants, and
• prove these invariants correct (or demonstrate violations through counterexamples).

4.2. Languages and Tools

Several specification languages could serve this purpose. For instance, we could have expressed the protocol directly in an interactive theorem prover like Lean or Rocq. However, both would offer little automation, which would make limited progress feasible within our one-month timeframe. (Recently, Lean has seen exciting developments such as the grind tactic and research by the VERSE group. We may explore these in a future engagement!)

TLA⁺ and its tooling. TLA⁺ is perhaps the most well-known practical specification language. It is supported by two model checkers (TLC and Apalache) and an interactive theorem prover TLAPS. We use the methodology of TLA⁺ to reason about the Governance Protocol as a collection of interacting state machines over large state spaces. Since many engineers find the syntax of TLA⁺ confusing, we use the surface syntax Quint to write the specifications. As co-authors of both Quint and the Apalache model checker – together with Gabriela Moreira, Shon Feder, Jure Kukovec, and others – we have a deep understanding of their internals and how to apply them to large-scale protocol verification. This expertise was essential for scaling our analysis to a system as complex as Aztec Governance.

5. Specification Decisions and Challenges

With the goals and tools defined, the next step was to translate the Governance Protocol into a precise, analyzable specification.

5.1. Writing the Specification

Modeling the states. Before specifying the contract behavior, we must decide how to model the contract states. We first define the shape of individual contract states. For example, below is the state of a GovernanceProposer.

// GovernanceProposer contract state
type GovernanceProposerState = {
  // the state of the parent EmpireBase contract
  empireBase: EmpireBaseState,
  // mapping(uint256 proposalId => address proposer)
  proposalProposer: Uint256 -> Address,
  // immutable config (set in constructor)
  REGISTRY: Address,
  GSE: Address
}

You can see that many concepts from Solidity (like mappings) are seamlessly expressed in Quint. The full protocol state – including all relevant contracts – is captured by EvmState. In our case, an EVM state is structured as follows:

type EvmState = {
  block_timestamp: Uint256,
  // all possible instances of ERC20 used as assets
  assets: Address -> ERC20State,
  // all possible instances of Governance
  governances: Address -> GovernanceState,
  // all possible instances of GovernanceProposer
  governanceProposers: Address -> GovernanceProposerState,
  // all possible instances of GSE
  gses: Address -> GSEState,
  // all possible instances of Registry
  registries: Address -> RegistryState,
  // all possible instances of RewardDistributor
  rewardDistributors: Address -> RewardDistributorState,
  // all instances of Slasher
  slashers: Address -> SlasherState,
  // all instances of SlashingProposer
  slashingProposers: Address -> SlashingProposerState,
  // IEmperor(...).getCurrentSlot() for each rollup
  rollupSlot: IHaveVersion -> int,
  // IEmperor(...).getCurrentProposer() for each rollup
  rollupProposer: IHaveVersion -> Address,
  // mapping rollup addresses to their versions
  // Corresponds to _rollup.getVersion() call in Registry.sol:53
  ROLLUP_VERSIONS: IHaveVersion -> Uint256,
  // mapping rollup address to the reward distributors that they create
  REGISTRY_REWARD_DISTRIBUTORS: IHaveVersion -> IRewardDistributor
}

As you can see, we do not have to focus on nitty-gritty low-level details – like how storage is laid out in EVM. This frees us to focus on protocol logic and high-level correctness, rather than low-level implementation concerns. It also makes the reasoning problem more tractable for automated verification.

Modeling the contract functions. The contract functions are simply pure functions over the EVM state. For instance, we define the function initiateWithdraw in Quint as:

// Governance.sol#L341
pure def Governance::initiateWithdraw(__evm_state: EvmState,
      __self: IGovernance, __msg_sender: Address,
      _to: Address, _amount: Uint256): Result[EvmState] = {
  val __state = __evm_state.governances.get(__self)
  val config = __state.configuration

  // ConfigurationLib.sol#L36:
  //   Timestamp.wrap(Timestamp.unwrap(_self.votingDelay) / 5) +
  //     _self.votingDuration + _self.executionDelay;
  val withdrawDelay = config.votingDelay / 5
      + config.votingDuration + config.executionDelay

  // L342: _initiateWithdraw(msg.sender, _to, _amount,
  //                         configuration.withdrawalDelay());
  Governance::_initiateWithdraw(__evm_state, __self, __msg_sender,
                                _to, _amount, withdrawDelay)
}

In Quint, we explicitly model all side-effects of the Solidity code, including exceptions and reverts. While it makes our specification more verbose, all branches and assignments become immediately visible at the code level – auditors do this in their heads all the time. For example, _initiateWithdraw computes and returns an updated EvmState, unless it reverts:

// Governance.sol#L694
pure def Governance::_initiateWithdraw(__evm_state: EvmState,
      __self: IGovernance, _from: Address, _to: Address,
      _amount: Uint256, _delay: Timestamp): Result[EvmState] = {
  val __state = __evm_state.governances.get(__self)
  // L695: users[_from].sub(_amount);
  val fromAmount = __state.users.getOrElse(_from, checkpoints::constructor)
  val userTraceOrError = checkpoints::sub(__evm_state, fromAmount, _amount)
  if (isErr(userTraceOrError)) {
    err(__evm_state, userTraceOrError.err)
  } else {
    // L696: total.sub(_amount);
    val totalTraceOrError = checkpoints::sub(__evm_state, __state.total, _amount)
    if (isErr(totalTraceOrError)) {
      err(__evm_state, totalTraceOrError.err)
    } else {
      // L698: uint256 withdrawalId = withdrawalCount++;
      // L700: withdrawals[withdrawalId] = Withdrawal({...});
      val withdrawal = {
          amount: _amount,
          unlocksAt: __evm_state.block_timestamp + _delay,
          recipient: _to, claimed: false
      }

      val __state1 = {
        ...__state,
        users: __state.users.put(_from, userTraceOrError.v),  // L695
        total: totalTraceOrError.v,                           // L696
        withdrawals: __state.withdrawals.append(withdrawal),  // L700
      }
      ok({...__evm_state,
        governances: __evm_state.governances.put(__self, __state1)
      })
    }
  }
}

Modeling transactions. We model transactions, e.g., initiated by externally-owned accounts (EOAs), via Quint actions:

action governance_initiate_withdraw = {
  nondet _g = evm.governances.keys().oneOf()
  nondet _sender = ALL_SENDERS.oneOf()
  nondet _to = ALL_ADDRESSES.oneOf()
  nondet _amount = 0.to(MAX_UINT256).oneOf()
  val result = Governance::initiateWithdraw(evm, _g, _sender, _to, _amount)
  all {
    is_valid_sender(_sender) and isOk(result),
    evm' = result.v,
    // ...
  }
}

This directly controls the domains from which input parameters are drawn. When we run the Quint randomized simulator, non-deterministic values are sampled uniformly at random. When we run the Apalache model checker, it uses logic constraints in the Z3 SMT solver to reason about all possible non-deterministic values at once.

5.2. Bootstrapping the Formal Specification with AI

The above specification looks a bit machine-generated. This is not far from the truth. We used an LLM to produce the initial specifications, given the source code in Solidity and the Quint data types.

Obviously, an LLM cannot make high-level modeling decisions, like how to structure the EVM state, or how best to turn Solidity into functional definitions – this requires years of practical experience. We developed a custom system prompt that gives the LLM clear instructions and examples for translating Solidity into Quint. (It’s an internal tool we refine and apply with clients when we bootstrap their specifications.)

Of course, as with all AI assistants, we had to carefully proofread the translation results. Also, we were fortunate to have the model checker Apalache on our side – it automatically pointed us to some inconsistencies in the translation. Compared to writing the specification by hand, this approach allowed us to bootstrap the project very quickly and to start evaluating the protocol early on.

5.3. Compositional Reasoning

Some security researchers believe that formal verification does not scale to more than 1–2 smart contracts, or to exploit scenarios longer than 1–2 external calls deep. We have organized our specification in such a way that the verification tools can deal with the behavior of 10–20 smart contracts, and arbitrarily long transaction sequences. This level of scalability requires not just formal verification expertise, but a deep understanding of how model checkers and provers work internally and interact with protocol architecture. It builds directly on our prior formal verification work – including zkSync Governance, ChonkyBFT, and Ethereum 3-slot-finality – where we pushed verification tools to reason compositionally across complex systems. More on this in Section 7.3. Inductive Invariants: Making Verification Scale.

6. Protocol Invariants

From Aztec’s documentation and source code, we extracted and formalized 125 key invariants of the Governance Protocol. To get a taste of the invariants, here are a few examples in English (more of them are in the report):

• GOV-16: If the proposal has not been active yet, then no votes have been cast.
• GOV-20: The timestamps in the users traces are ordered.
• GOV-26: For each timestamp t, total[t] equals the sum of the users’ voting power at t.
• GP-02-01: For each submitted proposal in proposalProposer, there is round accounting for a corresponding executed proposal (i.e., submitted to Governance).
• GP-08: A proposal cannot be executed without a quorum.
• GSE-17: for each proposal that the delegatee has powerUsed on, Governance contains that proposal.
• GSE-19: powerUsed cannot exceed the attester’s voting power at the time of the proposal’s pendingThrough.
• GSE-23: delegation.supply at each checkpoint is the sum of all delegation.ledgers[instance].supply at that time.
• SP-10: lastSignalSlot is in the valid range. This range is [round * ROUND_SIZE, (round + 1) * ROUND_SIZE).
• SP-11: The number of signals is correct. It does not exceed lastSignalSlot % ROUND_SIZE + 1.

Formalized invariants in Quint. We formalized all 125 invariants in Quint as well. For example, the Governance contract should uphold the Solvency Invariant (‘The Holy Grail’, as coined by FV researchers at Certora):

// GOV-28: Solvency: Governance holds enough balance to cover all future
// withdrawals.
pure def governance_solvency_inv(_evm: EvmState, ga: IGovernance): bool = {
  pure val g = _evm.governances.get(ga)
  and {
    // the withdrawals that happen in the future
    pure val payable = g.withdrawals.indices().fold(0, (sum, i) => {
      pure val withdrawal = g.withdrawals[i]
      sum + if (withdrawal.claimed) 0 else withdrawal.amount
    })
    // the total user's balance, add payable, is below the contract's balance
    pure val asset = _evm.assets.get(g.ASSET)
    g.total.latest() + payable <= asset.balances.getOrElse(ga, 0)
  }
}

Turns out, the solvency invariant is actually violated under certain conditions. We will get back to it in Section 9.1. Governance Insolvency.

With the key invariants defined, we started verifying them using Quint and Apalache.

7. Formal Verification Workflow

As soon as parts of the specification stabilized, we began verification – moving from randomized simulation to full symbolic and inductive reasoning.

7.1. Randomized Simulator: Stuck at Unproductive Inputs

The Quint randomized simulator operates similarly to property-based testing for implementation languages: it assigns concrete values to nondet declarations and resolves non-deterministic control choices by selecting one branch at random.

Limitations. We briefly experimented with this approach, but it proved ineffective for our purposes. The simulator’s uniform random sampling consistently failed to produce valid configurations that would even satisfy the protocol’s initial state:

$ quint run --max-samples=100000 --max-steps=10  --invariant=past_signals \
    spec/slashing_proposer_machine.qnt
An example execution:

[ok] No violation found (768ms at 130208 traces/second).
Trace length statistics: max=0, min=0, average=0.00

We believe the randomized simulator could be improved in future versions. If you’d like to explore in more detail why this happens – and how it could be mitigated – check out our workshop 25-Minute Solidity Fuzzer: Fuzzing Smarter, Not Harder.

In its current form, however, it did not help us uncover issues. This led us to use the symbolic analysis tools in Apalache, which can reason over all possible inputs symbolically rather than sampling concrete ones.

7.2. Symbolic Random Walks: Scaling Up

With symbolic random walks (part of Apalache), we quickly checked several invariants. The following run revealed an issue: the system could receive outdated (“past”) signals when the canonical rollup was faulty:

$ quint verify --random-transitions=true --max-steps=10 \
  --invariant=past_signals spec/slashing_proposer_machine.qnt
...
[violation] Found an issue (22181ms)

When an invariant is violated, Apalache produces a counterexample with all details needed to understand the issue. We omit it here because it is quite verbose.

Limitations. Even though this approach proved to be quite useful in bootstrapping and debugging our specification, it reached its limits when we began dealing with multiple contracts. This limitation stems from the protocol’s scale – with over 40 external functions, many of which can be invoked at nearly any point in time, the number of possible symbolic paths grows combinatorially with path length. We then moved to proving inductive invariants automatically.

7.3. Inductive Invariants: Making Verification Scale

To scale our formal verification efforts further, we specified 125 invariants that together capture any arbitrary state of the Governance Protocol. For example, below is the invariant gse_rollups_inv that groups the invariants GSE-28 to GSE-32:

pure def gse_rollups_inv(evm: EvmState, gsea: IGSE): bool = {
  val gse = evm.gses.get(gsea)
  val chkpts = gse.rollups._checkpoints
  and {
    // GSE-28: `rollups` is an ordered checkpointed trace with ascending timestamps
    _trace_is_ordered(gse.rollups),
    chkpts.indices().forall(i => and {
      // GSE-29: `rollups` values are rollup addresses
      chkpts[i]._value.in(ROLLUP_ADDRESSES),
      // GSE-30: the bonus instance does not appear in the `rollups` history
      chkpts[i]._value != BONUS_INSTANCE_ADDRESS,
      // GSE-31: `rollups` values are registered in `instances`
      chkpts[i]._value.in(gse.instances.keys()),
    })
  }
}

The following command checks that all protocol invariants (all_inv) – including gse_rollups_inv – hold whenever the protocol is in a state that satisfies all_inv and one of the contracts makes a single step:

$ ./scripts/quint-inductive.sh spec/invariant_model.qnt 31 32 5 100 all_inv

Beware that the above command runs over 900 verification runs in parallel (in the example above using at most 5 CPUs at once). This can easily overwhelm your laptop. If you want to reproduce our experiments, read the next section on our experimental setup.

Scalable verification. The core technique that enables this level of scalability is the use of inductive invariants. Instead of exploring all possible symbolic paths of the specification from an initial state (this approach, used by most code-level symbolic tools, is called symbolic execution), we start with a much richer set of states (captured by the inductive invariant all_inv) and simply enumerate all possible external functions and make them execute exactly once from any state in the inductive invariant. By assuming that all_inv holds in an arbitrary state and showing that it still holds after symbolically executing any single transaction, our check extends inductively to all possible executions.

Note for sticklers. We still have to show that the initial states satisfy the inductive invariant. In our case, this is easy. Essentially, the initial state of the protocol is an “empty” EVM state where none of the governance contracts are deployed yet.

7.4. Witnesses of Liveness: Proving the Protocol Can Progress

When verifying safety, there is always a risk that we introduce a bug in the specification that restricts the protocol behavior too much. This would still keep the protocol “safe” from the verification point of view, but, obviously, the protocol would not do as many useful things as it is meant to do. To avoid this pitfall, we introduce “falsy invariants” that instruct Apalache to generate a witness of the protocol reaching an “interesting” state. Below is an example to produce an execution to a state in which at least one governance proposal has been executed:

// Check this invariant to find an example of having at least one executed proposal:
// quint verify --max-steps=0 --invariant=gov_proposals_executed_ex \
//   spec/invariant_model.qnt
val gov_proposals_executed_ex = {
  not(evm.governances.keys().forall(ga => {
    val g = evm.governances.get(ga)
    g.proposals.indices().exists(proposalId => {
      val proposal = g.proposals[proposalId]
      proposal.cachedState == ProposalState_Executed
    })
  }))
}

This ability to automatically generate an execution trace to an ‘interesting’ state is a superpower of symbolic model checkers like Apalache – a functionality that would be far more difficult to automate with an interactive theorem prover such as Lean or Rocq. (Provers have property-based testing tools, but they are not tuned to bug finding in distributed protocols like Apalache.)

8. Experimental Setup and Verification Runs

Experimental setup. As mentioned above, checking the inductive invariant of our specification produces 992 verification tasks in total (for the combinations of a specific invariant and an external function call). Apalache decomposes invariant checking into smaller tasks, so we employ GNU parallel to massively parallelize the verification. We use two servers to run the experiments:

1. AMD Ryzen 9 5950X processor (16 physical, 32 logical cores), 128 GB memory
2. 2× Intel Xeon Platinum 8280 processor (56 physical, 112 logical cores total), pinned at 3.1 GHz, 240 GB memory

Verification Runs. Some of the verification tasks take a few minutes to check, and some of them take a few hours. This is caused by the nature of the SMT constraints. It is well-known that SMT solvers, including Z3, are challenged by non-linear integer arithmetic – in this project, they naturally appear, e.g., as part of Aztec’s vote tallying logic.

Instead of writing many words, we simply show you the plot below. It visualizes the running times of Apalache when checking the 992 verification tasks. The X-axis shows the number of verification conditions solved (roughly, individual constraints in the inductive invariant), sorted from fastest to slowest. Each point corresponds to one verification condition. The Y-axis represents the running time per verification condition, formatted in human-readable units (milliseconds to hours). Notice the logarithmic scale!

As we can see from the plot, over 85% of the verification conditions are checked in less than 10 minutes each, about 7% are checked in several hours, and about 8% of the verification conditions require plenty of running time.

Timeouts. As it happens with SMT solvers, 3% of our verification conditions time out. These are the runs at the end of the “hockey stick”. We capped the running time of Z3 at 12 hours. Since we are decomposing the inductive invariant into smaller pieces, these problematic conditions are well-localized. We have investigated these conditions. They all have to do with non-linear arithmetic.

Below is an example of such an invariant. Notice that the very last expression involves modulo over a non-constant value, since ROUND_SIZE is initialized in the contract constructor.

// GovernanceProposer invariant on last signals and total signals
pure def governance_proposer_signal_inv(evm: EvmState,
                                        ga: IGovernanceProposer): bool = {
  val gp = evm.governanceProposers.get(ga)
  gp.empireBase.rounds.keys().forall(rollup => {
    gp.empireBase.rounds.get(rollup).keys().forall(round => {
      val rollupRounds = gp.empireBase.rounds.get(rollup)
      val accounting = rollupRounds.get(round)
      and {
        // GP-12: ...
        // ...
        // GP-13: The number of signals is in the right range
        // It does not exceed `lastSignalSlot % ROUND_SIZE + 1`.
        // This property is very hard for Z3. It is not falsified.
        and {
          gp.empireBase.ROUND_SIZE <= MAX_ROUND_SIZE,
          totalSignalCount >= 0,
          totalSignalCount <= gp.empireBase.ROUND_SIZE,
          totalSignalCount <=
            (accounting.lastSignalSlot % gp.empireBase.ROUND_SIZE) + 1,
        }
      }
    })
  })
}

We classify the small number of the verification conditions that time out as not falsified rather than verified. Usually, we recommend verifying such conditions with a theorem prover such as Lean or Rocq. Another solution is to fix these non-constant values to known production configuration values to gain further confidence.

9. Findings

Our verification of the Aztec Governance Protocol uncovered five Medium, three Low, and six Informational findings. Most arose from subtle cross-contract interactions that are difficult to identify through conventional testing, fuzzing, or simulation alone. We reported all issues to Aztec Labs, who acknowledged and/or fixed them in subsequent pull requests.

Below we explain one representative issue: a violation of the solvency invariant.

9.1. Governance Insolvency

Recall the solvency invariant from Protocol Invariants. When we check it, Apalache produces a counterexample. Below is the root cause of this issue:

function deposit(address _beneficiary, uint256 _amount) external
        override(IGovernance) isDepositAllowed(_beneficiary) {
  ASSET.safeTransferFrom(msg.sender, address(this), _amount);
    // <--- if msg.sender == address(this), then the balances do not change
  users[_beneficiary].add(_amount);
    // <--- ...but the liabilities always get increased
  total.add(_amount);
  emit Deposit(msg.sender, _beneficiary, _amount);
}

In short, executing an approved governance proposal can invoke Governance.deposit(...). Inside deposit, this performs an ERC-20 self-transfer – leaving token balances unchanged – while crediting _beneficiary and increasing total. Governance’s liabilities go up, but its assets don’t – the contract becomes insolvent. The diagram below illustrates the problematic scenario.

On ERC-20 approvals. Most ERC20 token implementations would require Governance to execute an explicit token approval for the self-transfer before the call to deposit() (while executing the governance proposal). Calling ASSET is forbidden by the current Governance implementation. However, certain tokens like WETH do not require approvals for transferFrom() if from == msg.sender.

Resolution. We raised this finding with Aztec Labs who addressed it in PR #16917 by forbidding Governance from calling deposit() itself. In addition, the current AZTEC token implementation uses OpenZeppelin’s ERC-20 implementation, which does require explicit approval of self-transfers.

9.2. Other Findings

For details on all our findings, refer to our formal report.

10. Conclusion: Scalable Formal Verification in Practice

Our formal verification of the Aztec Governance Protocol went far beyond a traditional audit. It was a compositional, protocol-level analysis using state-of-the-art tools and techniques that we helped create. We formally proved 125 high-level invariants across a multi-contract system – reasoning over a search space beyond the reach of traditional testing and most formal verification tools. These invariants were automatically decomposed into 992 verification conditions, which let us further parallelize the verification task.

By combining inductive invariants, symbolic reasoning, and massive parallelization (321 CPU-days of compute), we showed that formal verification can scale to the complexity of modern, mission-critical smart contract systems. Our methodology enables exhaustive, automated reasoning about real-world governance mechanisms and other smart contract protocols.

For systems like Aztec Governance, where bugs are subtle but potentially catastrophic, deep understanding of the tools and underlying logic is essential. This project demonstrates that scalable, unbounded formal verification is not just theoretically possible – it’s practical today for mature, production-grade protocols.

Differential Testing: Spec / Implementation Conformance

A natural next step would be to connect the formal protocol specification with the actual Solidity implementation to close the verification loop (known as differential or conformance testing). With our methodology, it suffices to check that each external function call in Solidity conforms to its formal specification in Quint. Traditionally, this is done by writing and proving pre- and post-conditions in Hoare logic – e.g., using Certora Prover. We suggest that a more pragmatic approach is to fuzz external Solidity functions directly against the formal specification.

Enabling diff testing in Apalache: We have just implemented a new Apalache JSON-RPC API, which enables interactive differential testing between implementation and specification. This delivers fast, actionable, and reproducible results while still providing a high level of assurance grounded in rigorous formal modelling.

Tags: specification tlaplus tlc

A couple of weeks ago, I gave a talk at the internal Nvidia FM Week 2025. Many thanks to Markus Kuppe for the organization and invitation! I am going to write a longer blog post about interactive spec conformance testing with Apalache later. Today, I want to talk a bit about the question posed by Markus (to find the question, continue reading).

Let’s talk about the small scope hypothesis. As formulated by Jackson in the technical report (1997), this hypothesis reads as follows:

"...most errors can be demonstrated by counterexamples within a small scope."

As you will see below, my example fits into this hypothesis quite well. However, having spoken to many engineers over the years, I believe that there is a mismatch between what engineers understand by “small scope” and what verification engineers understand by “small scope”.

In this blog post, I’ve decided to try a new format. Since everyone is using LLMs nowadays, I will follow the protocol. I will present the example and the problem of finding a small scope. Then, it is your turn to decide how this blog post should continue. If someone gives me an interesting example or insight in a comment, I will update this blog post accordingly.

1. Example 1: Buggy circular buffer

1.1. The specification

I started the talk with a TLA⁺ specification of a buggy circular buffer. You can find the full specification, the model checking models, and the TLC configuration files here. The specification looks as follows:

----------------------------- MODULE BuggyCircularBuffer -----------------------------
(**
 * A very simple specification of a circular buffer with a bug.
 * Generated with ChatGPT and beautified by Igor Konnov, 2025.
 * ChatGPT learned abstraction so well that it omitted the actual buffer storage!
 *)
EXTENDS Integers

CONSTANTS
    \* Size of the circular buffer.
    \* @type: Int;
    BUFFER_SIZE,
    \* The set of possible buffer elements.
    \* @type: Set(Int);
    BUFFER_ELEMS

ASSUME BUFFER_SIZE > 0

VARIABLES
    \* The integer buffer of size BUFFER_SIZE.
    \* @type: Int -> Int;
    buffer,
    \* Index of the next element to POP.
    \* @type: Int;
    head,
    \* Index of the next free slot for PUSH.
    \* @type: Int;
    tail,
    \* Number of elements currently stored.
    \* @type: Int;
    count

\* Initial state
Init ==
  /\ buffer = [ i \in 0..(BUFFER_SIZE - 1) |-> 0 ]
  /\ head = 0
  /\ tail = 0
  /\ count = 0

\* Buggy PUT: Advance tail, increment count, but no fullness check!
Put(x) ==
  Put::
  LET nextTail == (tail + 1) % BUFFER_SIZE IN
  /\ buffer' = [buffer EXCEPT ![tail] = x]
  /\ head' = head
  /\ tail' = nextTail
  /\ count' = count + 1

\* GET: Only allowed when count > 0.
Get ==
  Get::
  LET nextHead == (head + 1) % BUFFER_SIZE IN
  /\ count > 0
  /\ UNCHANGED buffer
  /\ head' = nextHead
  /\ tail' = tail
  /\ count' = count - 1

\* Either Put or Get may happen in any step.
Next ==
    \/ \E x \in BUFFER_ELEMS:
        Put(x)
    \/ Get

vars == <<buffer, head, tail, count>>

\* Complete specification
Spec == Init /\ [][Next]_vars

\* Safety property we *intend* to hold, but it is violated:
\* count must never exceed the buffer capacity.
SafeInv == count <= BUFFER_SIZE

Since I wanted to experiment with different buffer sizes and potential buffer elements, I have introduced two parameters in the specification:

• BUFFER_SIZE is the size of the cyclic buffer, and
• BUFFER_ELEMS is the set of possible buffer elements.

Now, my previous experience with introducing TLA⁺ to engineers suggests that there are two ways to set these parameters:

1. The Engineer’s way: Set the parameters to relatively small yet reasonable values. For example, BUFFER_SIZE = 10 and BUFFER_ELEMS = 0..255. These are not the minimal possible values, but they kind of make sense: The buffer should hold up to 10 bytes. Obviously, BUFFER_ELEMS are set to the minimal possible type in their programming language of choice, e.g., char in C, or u8 in Rust.

2. The Verification Engineer’s way: Start with the smallest possible values of the parameters, e.g., BUFFER_SIZE = 2 and BUFFER_ELEMS = {0, 1}. The idea is to check the specification in the smallest possible scope first. If there are no bugs found, increase the parameters gradually until you reach the reasonable values.

1.2. Checking the specification Engineer’s way

To check the specification the Engineer’s way, I have created the TLA⁺ model MC10u8_BuggyCircularBuffer.tla with BUFFER_SIZE = 10 and BUFFER_ELEMS = 0..255. For technical reasons, we also need the TLC config file MC.cfg. Follow the links to see the details. Further, I’ve run TLC on this model to check the invariant SafeInv:

$ java -cp tla2tools.jar "-XX:+UseParallelGC" tlc2.TLC \
  -config MC.cfg MC10u8_BuggyCircularBuffer.tla

I wanted to see how far TLC could go, so I gave it a machine with 128 GB of RAM and 32 cores. TLC has explored around 3 billion states in about 40 minutes. After consuming 400 GB of disk space, it has run out of disk space and terminated. No bug was found. Is this surprising? Not really. In this configuration, TLC has to enumerate $(2^8)^{10} * 10 * 10 * 10 \approx 2^{90}$ states. (Thanks to Thomas Pani for correcting the initially wrong estimate.)

Obviously, anyone who used TLC for some time would have asked the same question as Markus did:

What about the small scope hypothesis? Can we use smaller parameters?

The answer to this question is basically the second approach, which I called the Verification Engineer’s way.

1.3. Checking the specification Verification Engineer’s way

This time, we use the instance MC2u1_BuggyCircularBuffer.tla that has BUFFER_SIZE = 2 and BUFFER_ELEMS = {0, 1}. Let’s run TLC on this instance to check the invariant SafeInv:

$ java -cp tla2tools.jar "-XX:+UseParallelGC" tlc2.TLC \
  -config MC.cfg MC2u1_BuggyCircularBuffer.tla
...
Error: Invariant SafeInv is violated.
...
10 states generated, 10 distinct states found, 5 states left on queue.

Yay! Just after visiting 10 states, TLC has found a violation of the invariant!

So if we pick the right small scope, exhaustive model checking with TLC finds the bug quite fast. In this example, it is hard to find a small scope that would not reveal the bug. Of course, when we know that the bug exists, it is easy to experiment with different values of the parameters and find the bug.

1.4. Checking the specification randomly

Surprisingly, if we forget about exhaustive enumeration, TLC finds an invariant violation for BUFFER_SIZE = 10 and BUFFER_ELEMS = 0..255 in less than a second. To do this, we run TLC with the option -simulate, which simply picks successor states at random:

$ java -cp tla2tools.jar "-XX:+UseParallelGC" tlc2.TLC \
  -simulate -config MC.cfg MC10u8_BuggyCircularBuffer.tla
...
Error: Invariant SafeInv is violated.
...

This effectiveness of randomized search is actually not a one-off thing. The Quint simulator finds the bug in less than a second. Similarly, the Rust property-based testing with proptest finds the bug almost immediately.

Interestingly, we did not have to tune the scope to be as tiny as possible, as we did for exhaustive model checking. Maybe this is why some engineers want to use property-based testing for every problem?

2. Thinking about the small scope hypothesis

In Example 1, we indeed found several assignments to BUFFER_SIZE and BUFFER_ELEMS that revealed an invariant violation. Actually, this bug is so simple that almost any assignment to the parameters would reveal it. We could even set BUFFER_SIZE = 1 and BUFFER_ELEMS = {0} to find the bug! If you want to push it further, think, whether BUFFER_ELEMS = {} would allow us to find an invariant violation.

In fact, if we go back to Alloy, the way Alloy restricts the scope is quite different from what we did in Example 1. Alloy limits the number of elements of each type in the specification. For example, if we had specified the circular buffer in Alloy, we could restrict the search scope as follows:

• All integers, including BUFFER_SIZE and buffer indices, have the bit width of 4.

• The number of unique buffer elements is $2^8$.

As a result, Alloy would consider all possible values of BUFFER_SIZE from 0 to 15, all possible values of buffer elements from 0 to 255, as well as all possible combinations of buffers of size up to 15. This is a much more flexible way to restrict the search space. In case of TLC, we did not have this flexibility: We had to give concrete values to BUFFER_SIZE and BUFFER_ELEMS.

Hence, we have to distinguish between small scopes and small parameter assignments in TLC. After thinking about this question a bit more, I’ve asked myself:

Are there examples of specifications that have a small scope for a specific invariant violation, but it is hard to find concrete parameter assignments within this scope?

Even though my intuition says “yes”, there must be plenty of such examples, I could not come up with with non-artificial examples immediately. On top of my head, I can think of the following directions to look for such examples:

• Examples from abstract interpretation. If we have non-trivial math with overflows and underflows, it might be hard to find concrete assignments that would trigger these overflows and underflows.

• Examples from graph theory. For instance, non-planar graphs must contain subgraphs that are subdivisions of $K_5$ or $K_{3,3}$ (see Kuratowski’s theorem on planar graphs). So if a bug shows up only in non-planar graphs, there must be a small scope that reveals the bug. However, our concrete graph would have to contain a subdivision of $K_5$ or $K_{3,3}$, which is far from an arbitrary graph. Unfortunately, I do not know any concurrent or distributed algorithm that would have something to do with planar or non-planar graphs.

3. Your turn

It is your turn to decide how this blog post should continue. If someone gives me an interesting example or insight in a comment, I will update this blog post accordingly.

Tags: specification tlaplus apalache tlc

This must be my shortest blog post ever. I just wanted to run TLC to check a specification that uses Apalache modules. For example, the typed version of two-phase commit MC3_TwoPhaseTyped.tla uses the module Variants. Sure, TLC can do that, but it requires a small trick.

Let’s do it step by step for MC3_TwoPhaseTyped.tla. Say, we want to see an example of all participants committing:

RMAllCommittedEx ==
    ~(\A rm \in RM: rmState[rm] = "committed")

Step 1. Checkout the Apalache repository, if you don’t have it already:

$ git clone git@github.com:apalache-mc/apalache.git
$ export APALACHE_HOME=$(pwd)/apalache

Step 2. Download TLA⁺ Tools:

$ wget https://github.com/tlaplus/tlaplus/releases/download/v1.7.4/tla2tools.jar
$ export TLC_HOME=$(pwd)

Step 3. Introduce a configuration file MC3_TwoPhaseTyped.cfg with the following content:

$ cd $APALACHE_HOME/test/tla
$ cat >MC3_TwoPhaseTyped.cfg <<EOF
INIT Init
NEXT Next
INVARIANT RMAllCommittedEx
EOF

Step 4. Run TLC with the option -cp, which extends the Java classpath. TLC will look for non-standard modules in the specified directory, that is, in the directory ${APALACHE_HOME}/src/tla. This is the trick!

$ java -cp ${TLC_HOME}/tla2tools.jar:${APALACHE_HOME}/src/tla \
  "-XX:+UseParallelGC" tlc2.TLC \
  -config MC3_TwoPhaseTyped.cfg MC3_TwoPhaseTyped.tla

As expected, TLC finds an example of RMAllCommittedEx:

Running breadth-first search Model-Checking...
...
State 11: <Next line 16, col 1 to line 16, col 22 of module MC3_TwoPhaseTyped>
/\ msgs = { [tag |-> "Commit", value |-> "0_OF_NIL"],
  [tag |-> "Prepared", value |-> "0_OF_RM"],
  [tag |-> "Prepared", value |-> "1_OF_RM"],
  [tag |-> "Prepared", value |-> "2_OF_RM"] }
/\ rmState = [0_OF_RM |-> "committed", 1_OF_RM |-> "committed", 2_OF_RM |-> "committed"]
/\ tmState = "committed"
/\ tmPrepared = {"0_OF_RM", "1_OF_RM", "2_OF_RM"}
...
1119 states generated, 287 distinct states found, 7 states left on queue.

Bottom line

This is it! If you have any questions, please feel free to reach out. I’m happy to help.

Tags: specification lean distributed proofs tlaplus

In the previous blog post, we looked at proving consistency of Two-phase commit in Lean 4. This proof followed the well-trodden path: We found an inductive invariant, quickly model-checked it with Apalache and proved its inductiveness in Lean. One of the immediate questions that I got on X/Twitter was: What about liveness?

Well, liveness of the two-phase commit after the TLA⁺ specification does not seem to be particularly interesting, as it mostly depends on fairness of the resource managers and the transaction manager. (A real implementation may be much trickier though.) I was looking for something a bit more challenging but, at the same time, something that would not take months to reason about. Since many Byzantine-fault tolerance algorithms work under partial synchrony, a natural thing to do was to find a protocol that required partial synchrony. Failure detectors seemed to be a good fit to me. These algorithms are relatively small, but require plenty of reasoning about time.

Hence, I opened the book DP2011 on Reliable and Secure Distributed Programming by Christian Cachin, Rachid Guerraoui, and Luís Rodrigues and found the pseudo-code of the eventually-perfect failure detector (EPFD). If you have never heard of failure detectors, there are a few introductory lectures on YouTube, e.g., this one. Writing a decent TLA⁺-like specification of EPFD and its temporal properties in Lean took me about eight hours. Since temporal properties require us to reason about infinite executions, this required a bit of experimentation with Lean. Figuring out how to capture partial synchrony and fairness was the most interesting part of the exercise. I believe that this approach can be reused in many other protocol specifications. You can find the protocol specification and the properties in Propositional.lean. See Section 2 for detailed explanations.

To prove correctness of EPFD, we have to show that it satisfies strong completeness and strong accuracy (see temporal properties). I chose to start with strong completeness. The proof in the book is just four (!) lines long. In contrast to that, my proof of strong completeness in Lean is about 1 KLOC. It consists of 13 lemmas and 2 theorems. As one professor once asked me: Do you want to convince a machine that distributed algorithms are correct? Apparently, it takes more effort to convince a machine than to convince a human. By a machine, I mean a proof assistant such as Lean, not an LLM, which would be easy to convince in pretty much anything. The real reason for that is that we take a lot of things about computations for granted, whereas Lean requires them to be explained. For instance, if a process $q$ crashes when the global clock value is $t$, it seems obvious that no process $p$ can receive a message from $q$ with the timestamp above $t$. Not so fast, this has to be proven! For the impatient, the complete proofs can be found in PropositionalProofs.lean. See Section 3 for detailed explanations.

It took me about 35 hours to finish the proof of strong completeness. I remember to have a working proof for a pair of processes $p$ and $q$ after the 25 hour mark already. However, the property in the book is formulated over all processes, not just a pair. Proving the property over all processes took me about additional 10 hours. This actually required more advanced Lean proof mechanics and solving a few curious proof challenges with crashing processes, e.g., how we define them. Also, bear in mind that this was literally my first proof of temporal properties in Lean. I believe that the next protocol would require less time.

Below is the nice diagram that illustrates the dependencies between the theorems (green) and lemmas (yellow) that I had to prove, culminating in the theorem strong_completeness_on_states. Notice forall_FG_implies_FG_forall is not a lemma. Actually, it is a general theorem about swapping universal quantifiers and eventually-always, which could be reused in other proofs. Once I realized that I had to apply this lemma twice in the proof of eventually_always_suspected_meet, I finished the proof quickly. This is one more instance of that temporal logic helps us with high-level reasoning.

Surprisingly, even though strong completeness is usually thought of as a safety counterpart of strong accuracy, the proof required quite a bit of reasoning about temporal properties, not just state invariants. It also helped me a lot to structure the proofs in terms of temporal formulas, rather than in terms of arbitrary properties of computations. Of course, it would be interesting to see how this proof compares to a proof in TLAPS, which is specifically designed to reason about temporal properties.

If you look at how the lemma statements are written in Section 3, you will see that they are all temporal formulas, just written directly using quantifiers $\forall$ and $\exists$ instead of modal operators like $\square$ and $\Diamond$. To emphasize this similarity, I wrote alternative statements in TLA⁺. If you know temporal logic or TLA⁺, just have a look at all these statements:

Although a time investment of about a week to prove strong completeness of EPFD may seem like a lot, this approach has certain benefits in comparison to using tools like the explicit-state model checker TLC or the symbolic model checker Apalache:

1. Re-checking the proofs takes seconds. It’s also trivial to integrate proof-checking in the GitHub continuous integration.

2. All tools require a spec to be massaged a bit. I always felt bad about not being able to formally show that these transformations are sound with a model checker. With Lean, it is usually easy.

3. If you manage to decompose your proof goals into smaller lemmas, there is a sense of progress. Even though I had to prove 4-5 unexpected lemmas in this experiment, I could definitely say whether I was making progress or not. In the end, I only proved one lemma that happened to be redundant. With model checkers, both explicit and symbolic, it is often frustrating to wait for hours or days without clear progress.

Obviously, the downside of using an interactive theorem prover is that someone has to write the proofs. For a customer, it may make a difference whether they pay for 2–4 weeks of contract work, or for 1 week of contract work and then wait 3 weeks for a model checker. However, if time is critical, it makes sense to invest in both approaches.

Table of contents

• 1. Eventually perfect failure detector in pseudo-code
• 2. Eventually perfect failure detector in Lean
  • 2.1. Basic type definitions
  • 2.2. Partial synchrony
  • 2.3. Specifying the actions
  • 2.4. Specifying the temporal properties
  • 2.5. Specifying fairness and fair runs
• 3. Proving strong completeness in Lean
  • 3.1. Shorthand temporal definitions
  • 3.2. Warming up with simple temporal lemmas
  • 3.3. Proving completeness for two processes
    • 3.3.1. Main lemma: Eventually q is always suspected by p
    • 3.3.2. Eventually q is never alive for p
    • 3.3.3. Non-crashing p resets alive infinitely often
    • 3.3.4. A crashed process q stops sending messages
    • 3.3.5. No message sent from the future
    • 3.3.6. Other lemmas
  • 3.4. From 2 to N processes
    • 3.4.1. Defining the crashed processes
    • 3.4.2. Where do the suspected sets meet?

1. Eventually perfect failure detector in pseudo-code

To avoid any potential copyright issues, I am not copying the pseudo-code from the book. If you want to see the original version, go check DP2011¹, Algorithm 2.7, p. 55. Below is an adapted version, which simplifies the events, as we do not have to reason about the interactions between different protocol layers in the proof. Every process $p \in \mathit{Procs}$ works as follows:

upon Init do
  alive := Procs
  suspected := ∅
  delay := InitDelay
  set_timeout(delay)

upon Timeout do
  if alive ∩ suspected ≠ ∅ then
    delay := delay + InitDelay
  suspected := Procs \ alive
  send HeartbeatRequest to all p ∈ Procs
  alive := ∅
  set_timeout(delay)

upon receive HeartbeatRequest from q do
  send HeartbeatReply to q

upon receive HeartbeatReply from p do
  alive := alive ∪ {p}

Intuitively, the operation of a failure detector is very simple. Initially, a process $p$ considers all the processes alive and suspects no other process of being crashed. Also, it sets a timer to $\mathit{InitDelay}$ time units. Basically, nothing interesting happens in the time interval $[0, \mathit{InitDelay})$, except that some processes may crash.

Once a timeout is triggered on a process $p$, it updates the set of the suspected processes to the set of processes that have not sent a heartbeat to it in the previous time interval (not alive), resets the set of the alive processes and sends heartbeat requests to every process, including itself. Additionally, if $p$ finds out that it prematurely suspected an alive process, it increases its timeout window by $\mathit{InitDelay}$. Importantly, $p$ also sets a new timeout for $delay$ time units.

Finally, whenever a process receives a heartbeat request, it sends a reply. Whenever, a process receives a heartbeat reply from a process $q$, it adds $q$ to the set of alive processes.

The algorithm looks deceivingly simple. However, the pseudo-code is missing another piece of information, namely, how the distributed system behaves as a whole. What does it mean for processes to crash? When messages are received, if at all? It’s not even clear how to properly write this in pseudo-code. Normally, academic papers leave this part to math definitions. Since we want to prove correctness, we cannot avoid reasoning about the whole system. Instead of appealing to intuition, we capture both the process behavior and the system behavior in Lean.

2. Eventually perfect failure detector in Lean

In contrast to two-phase commit, where we started with a functional specification, I decided to start with the propositional specification immediately. A functional specification would be closer to the implementation details. Perhaps, we will write one in another blog post.

2.1. Basic type definitions

Before specifying the behavior of the processes, we have to figure out the basic types. You can find them in Basic.lean. First, we declare the type Proc that we use for process identities:

variable (Proc : Type) [Fintype Proc] [DecidableEq Proc] [Hashable Proc] [Repr Proc]

If you compare it with the type RM in two-phase commit, this time, we require Proc to be of Fintype. By doing so, we avoid carrying around the set of all processes. With Fintype, we can simply use Fintype.univ for the set of all processes!

Next, we define the types of message tags and messages:

/-- A message tag: `HeartbeatRequest` or `HeartbeatReply`. -/
inductive MsgTag where
  | HeartbeatRequest
  | HeartbeatReply
  deriving DecidableEq, Repr

/-- A message that is sent by one process (src) to another process (dst).
    Every message is equipped with a timestamp, which is equal to the
    clock value at the time of sending the message.
 -/
@[ext]
structure Msg where
  kind: MsgTag
  src: Proc
  dst: Proc
  timestamp: ℕ
  deriving DecidableEq, Repr

Now, most of it should be obvious, except, perhaps, for the field timestamp. What is it? If we look at the original paper on failure detectors by Chandra and Toueg (CT96), we’ll see that they assume the existence of a global clock. The processes can’t read this clock, but the system definitions refer to it. Hence, a message’s timestamp refers to the value of the global clock at the moment the message is sent.

Finally, we define a protocol state with the following structure:

structure ProtocolState where
  alive: Std.HashMap Proc (Finset Proc)
  suspected: Std.HashMap Proc (Finset Proc)
  delay: Std.HashMap Proc Nat
  nextTimeout: Std.HashMap Proc Nat
  sent: Finset (Msg Proc)
  rcvd: Finset (Msg Proc)
  clock: Nat
  crashed: Finset Proc

The first group of fields should be clear. They map process identities to the corresponding values of the variables in the pseudo-code:

• alive[p]! stores the set of alive processes as observed by a process p,
• suspected[p]! stores the set of suspected processes as observed by a process p,
• delay[p]! stores the current value of delay by a process p,
• nextTimeout[p]! stores the timestamp of the next timeout by a process p. The timestamp refers to the global clock.

The second group of fields is less obvious. They do not represent the process states, but rather the rest of the global state of the distributed system:

• sent is the set of all messages sent by the processes,
• rcvd is the set of all messages received by the processes,
• clock is the value of the fictitious global clock,
• crashed is the set of all processes that have crashed.

While the second group of fields is needed to formally capture a state of the distributed system, we notice that the processes cannot have access to those fields. Otherwise, detecting failures would be trivial, we would just access the field crashed.

If you find this representation of the global state surprising, it’s actually quite common to reason about such a global snapshot of a distributed system in TLA⁺. Here, we’re simply following the TLA⁺ methodology, albeit reproduced in Lean.

2.2. Partial synchrony

The algorithm is designed to work under partial synchrony. Unfortunately, DP2011 does not give us a precise definition of what this means. So we go back to the paper by Dwork, Lynch, and Stockmeyer (DLS88) who introduced partial synchrony. There are several kinds of partial synchrony in the paper. We choose the one that is probably the most commonly used nowadays: There is a period of time called global stabilization time (GST), after which every correct process $p$ receives a message from a correct process $q$ no later than $\mathit{MsgDelay}$ time units after it was sent by $q$. Both $\mathit{GST}$ and $\mathit{MsgDelay}$ are unknown to the processes, and may change from run to run. It is also important to fix the guarantees about the messages that were sent before $\mathit{GST}$. We assume that they are received by $\mathit{GST} + \mathit{MsgDelay}$ the latest.

Now we can write a formal definition of what it means for a message to be received on time under partial synchrony:

/--
  Given a message that was sent at `timestamp`, can a process receive it at time `clock`.
  -/
def isMsgTimely (GST: Nat) (MsgDelay: Nat) (timestamp: Nat) (clock: Nat): Bool :=
  clock ≥ timestamp && clock ≤ (max GST timestamp) + MsgDelay

Both DLS88 and DP2011 mention that in practice partial synchrony means that the periods of asynchrony and synchrony alternate. DLS88 mention that for their consensus algorithms one should be able to compute the time of convergence after GST. I am not actually sure how it would work in case of failure detectors, as it is impossible to predict how long it takes a process to crash. Hence, in our model, there is no alternation of asynchrony and synchrony. After GST, communication becomes synchronous, in the sense that every message is delivered not later than $\mathit{MsgDelay}$ time units after it was sent.

2.3. Specifying the actions

Now we are ready to specify the actions of a distributed system that follows the algorithm. You can find all definitions in Propositional.lean.

We start with the protocol parameters and the variables s and s' that we use throughout the definitions:

-- The abstract type of processes
variable (Proc : Type) [Fintype Proc] [DecidableEq Proc] [Hashable Proc] [Repr Proc]

-- The initial delay Δ used by the processes
variable (InitDelay: ℕ)

-- The global stabilization time GST, unknown to the processes
variable (GST: ℕ)

-- The message delay after GST, unknown to the processes
variable (MsgDelay: ℕ)

-- The state `s` is a state of the protocol, explicitly added to all the functions.
variable (s: ProtocolState Proc)

-- The state `s'` is the "next" state of the protocol.
variable (s': ProtocolState Proc)

Below is the definition of receiving a heartbeat request:

/--
  A process `dst` receives a heartbeat request from `src`.
  -/
def rcv_heartbeat_request (src: Proc) (dst: Proc) (timestamp: ℕ) :=
  let req := { kind := MsgTag.HeartbeatRequest, src, dst, timestamp }
  dst ∉ s.crashed
  ∧ req ∈ s.sent
  ∧ isMsgTimely GST MsgDelay timestamp s.clock
  ∧ s'.rcvd = s.rcvd ∪ { req }
  ∧ let reply :=
      { kind := MsgTag.HeartbeatReply, src := dst, dst := src, timestamp := s.clock }
    s'.sent = s.sent ∪ { reply }
  ∧ s'.crashed = s.crashed
  ∧ s'.clock = s.clock
  ∧ s'.alive = s.alive
  ∧ s'.suspected = s.suspected
  ∧ s'.delay = s.delay
  ∧ s'.nextTimeout = s.nextTimeout

As you can see, the definition of rcv_heartbeat_request captures the behavior of the whole system, when dst handles a heartbeat request. In particular, dst cannot be in the crashed state when it is receiving the message, the message has to be timely, etc. Similar to TLA⁺, we specify that certain fields preserve their values. Actually, we could update the structure s' instead of writing down multiple equalities over the fields. However, it would make the proofs more cumbersome. I could not find a simple way to express something like TLA⁺’s UNCHANGED over multiple variables.

Interestingly, I accidentally swapped src and dst in the initial version of reply in rcv_heartbeat_request. I only found that when trying to prove one of the lemmas towards strong completeness.

Similar to rcv_heartbeat_request, we specify rcv_heartbeat_reply:

/--
  A process `dst` receives a heartbeat reply from `src`.
  -/
def rcv_heartbeat_reply (src: Proc) (dst: Proc) (timestamp: ℕ) :=
  let reply := { kind := MsgTag.HeartbeatReply, src, dst, timestamp }
  dst ∉ s.crashed
  ∧ reply ∈ s.sent
  ∧ isMsgTimely GST MsgDelay timestamp s.clock
  ∧ s'.rcvd = s.rcvd ∪ { reply }
  ∧ let nextAlive := s.alive[dst]! ∪ { src }
    s'.alive = s.alive.insert dst nextAlive
  ∧ s'.sent = s.sent
  ∧ s'.crashed = s.crashed
  ∧ s'.clock = s.clock
  ∧ s'.suspected = s.suspected
  ∧ s'.delay = s.delay
  ∧ s'.nextTimeout = s.nextTimeout

The definition of timeout is the longest one, as a lot of things happen on timeout:

/--
  A process `p` timeouts.
  -/
def timeout (p: Proc) :=
    p ∉ s.crashed
  ∧ s.clock = s.nextTimeout[p]!
  -- if `p` suspects an alive process, increase the delay
  ∧ let nextDelay :=
      if s.alive[p]! ∩ s.suspected[p]! ≠ ∅
      then s.delay[p]! + InitDelay
      else s.delay[p]!
    s'.delay = s.delay.insert p nextDelay
  -- recompute the set of suspected processes
  ∧ let nextSuspected := Finset.univ \ s.alive[p]!
      /- q ∉ s.alive[p]! is equivalent to the original code:
        on q ∉ s.alive[p]! ∧ q ∉ s.suspected[p]! trigger Suspect q
        on q ∈ s.alive[p]! ∧ q ∈ s.suspected[p]! trigger Restore q
        else keep q ∈ s.suspected[p]!
       -/
    s'.suspected = s.suspected.insert p nextSuspected
  -- send heartbeat requests to all processes, including `p` itself
  ∧ s'.sent = s.sent ∪ Finset.univ.image (fun q => {
      kind := MsgTag.HeartbeatRequest, src := p, dst := q, timestamp := s.clock
    })
  -- set alive to empty and reset the timer
  ∧ s'.alive = s.alive.insert p ∅
  ∧ s'.nextTimeout = s.nextTimeout.insert p (s.clock + s.delay[p]!)
  -- everything else remains unchanged
  ∧ s'.rcvd = s.rcvd
  ∧ s'.crashed = s.crashed
  ∧ s'.clock = s.clock

As you can see, the sequential logic from the pseudo-code is compressed into multiple equalities, very much in the spirit of TLA⁺. Our proofs are complex enough, so it’s good that we do not have to deal with sequential execution inside actions. If this is not convincing enough, we could write sequential code and prove that it refines the corresponding propositional definition.

We have defined the three actions, as in the pseudo-code (the definition of init comes later). Are we done? Not quite. Since we’re specifying the behavior of the entire distributed system, not just individual processes, we need two more actions.

The first additional action is crash:

/--
  A process `p` crashes. This action is not part of the protocol itself, but
  rather a part of the environment (or the adversary).
  -/
def crash (p: Proc) :=
    p ∉ s.crashed
  ∧ s'.crashed = s.crashed ∪ { p }
  ∧ s'.sent = s.sent
  ∧ s'.rcvd = s.rcvd
  ∧ s'.clock = s.clock
  ∧ s'.alive = s.alive
  ∧ s'.suspected = s.suspected
  ∧ s'.delay = s.delay
  ∧ s'.nextTimeout = s.nextTimeout

Yes, we have to specify what it means for a process to crash, as there is no built-in semantics of crashing in Lean.

What is left? Remember that we had the fictitious global clock? We have to advance it from time to time:

/--
  The global system clock advances. We advance the clock by exactly one unit.
  If we had a rational clock, we would have to advance it by `delta` units.
  -/
def advance_clock :=
    s'.clock = s.clock + 1
  ∧ s'.crashed = s.crashed
  ∧ s'.sent = s.sent
  ∧ s'.rcvd = s.rcvd
  ∧ s'.alive = s.alive
  ∧ s'.suspected = s.suspected
  ∧ s'.delay = s.delay
  ∧ s'.nextTimeout = s.nextTimeout

I have cut a corner in the definition of advance_clock by incrementing it, instead of advancing it by a positive delta. This works since we declared the clock to be a natural number rather than a rational or real. Incrementing the clock instead of advancing it by delta simplifies the proofs a bit.

Finally, we define the initialization and the transition relation as follows:

/--
  Initialize a map with the default value `v` for each process in `all`.
  -/
noncomputable def init_map {α: Type} (v: α) : Std.HashMap Proc α :=
  Finset.univ.toList.foldl (fun m p => m.insert p v) (Std.HashMap.emptyWithCapacity 0)

/--
  The initial state of the protocol.
  -/
def init: Prop :=
    s.crashed = ∅
  ∧ s.sent = ∅
  ∧ s.rcvd = ∅
  ∧ s.clock = 0
  ∧ s.alive = init_map Proc ∅
  ∧ s.suspected = init_map Proc ∅
  ∧ s.delay = init_map Proc InitDelay
  ∧ s.nextTimeout = init_map Proc InitDelay

/--
  The transition relation of the protocol.
  -/
def next: Prop :=
    advance_clock Proc s s'
  ∨ ∃ p: Proc,
        timeout Proc InitDelay s s' p
      ∨ crash Proc s s' p
      ∨ ∃ q: Proc, ∃ t: ℕ,
            rcv_heartbeat_request Proc GST MsgDelay s s' p q t
          ∨ rcv_heartbeat_reply Proc GST MsgDelay s s' p q t

Notice the noncomputable qualifier in front of init_map. Lean requires it, as we are converting Finset.univ to a list. If we want to write an executable specification, we have to work around this, perhaps, by passing the list of all process identities to the initializer.

So far, our definitions looked very much like a typical specification in TLA⁺, even though we had to use Lean’s data structures such as finite sets and hash maps, instead of TLA⁺’s sets and functions. I believe that there is an advantage in keeping this resemblance. First, if we choose to translate this specification to TLA⁺, e.g., to use the model checkers, it is not hard. (Actually, I did that; it was almost no-brainer with Copilot). Second, we can reuse the standard specification idioms from TLA⁺.

2.4. Specifying the temporal properties

In two-phase commit, we were only concerned with state invariants and, thus, only had to reason about lists of actions. In the case of failure detectors, we have to reason about temporal properties. In general, temporal properties require us to reason about infinite behaviors. Surprisingly, it is quite easy to specify properties of infinite behaviors in Lean. We just use a function seq of natural numbers to ProtocolState.

Here is how we specify strong completeness of the failure detector:

def is_strongly_complete
    (Crashed: Finset Proc)
    (seq: ℕ → ProtocolState Proc): Prop :=
  (∀ p: Proc, p ∈ Crashed ↔ ∃ i: ℕ, p ∈ (seq i).crashed)
    → ∃ k: ℕ,
        ∀ i: ℕ,
          ∀ p q: Proc,
            p ∉ Crashed ∧ q ∈ Crashed → q ∈ (seq (k + i)).suspected[p]!

The above definition may seem to be a bit loaded. The left part of → requires that the set Crashed indeed contains all the processes that crashed in the run. The set Crashed happened to be hard to define. More on that later. The right part of → says that there is a point k in seq, so that starting with $k$, every further state $seq (i + k)$ satisfies $q \notin (seq (i + k)).suspected[p]!$ for a correct $p$ and a crashed $q$.

If you know temporal logic, e.g., as defined in TLA⁺, the right-hand side of → could be written like (<> is usually called “eventually” and [] is called “always”):

<>[](∀ p q: Proc, p ∉ C ∧ q ∈ C → q ∈ suspected[p])

Similar to is_strongly_complete, this is how we specify strong accuracy:

def is_eventually_strongly_accurate
    (Crashed: Finset Proc)
    (seq: ℕ → (ProtocolState Proc)) : Prop :=
  (∀ p: Proc, p ∈ Crashed ↔ ∃ i: ℕ, p ∈ (seq i).crashed)
    → ∃ k: ℕ,
        ∀ i: ℕ,
          ∀ p q: Proc,
            p ∉ Crashed ∧ q ∉ Crashed → q ∉ (seq (i + k)).suspected[p]!

Again, in temporal logic it would look like:

<>[](∀ p q: Proc, p ∉ crashed ∧ q ∉ crashed → q ∉ suspected[p])

Don’t we need a framework for temporal logic? Well, actually not. Instead of [] and <>, we can simply use ∀ and ∃ over indices. There is even a deeper connection between linear temporal logic and first-order logic with ordering, shown by Hans Kamp. For example, see a recent Proof of Kamp’s theorem by Alexander Rabinovich. Temporal formulas are often easier to read. So I prefer to accompany properties in Lean with temporal properties in the documentation.

2.5. Specifying fairness and fair runs

Now that we have described the system behavior, can we proceed with the proofs? Not so fast. If you have ever tried to prove liveness, you know that we have to restrict our analysis to fair system executions.

For instance, our definition of next allows the scheduler to always choose advance_clock as the next action. So we would end up with a sequence that consists of states that only have the increasing clock values. Is it an interesting sequence? Not really. We have not even had a chance to try other actions. Usually, such executions are called unfair. We want to restrict our liveness analysis to fair executions.

To save you guess work, here are the three kinds of conditions we want from a fair execution in our failure detector:

1. For every message $m$ that is sent, the message destination receives it on time, unless it crashes before the message $m$ expires. This is the constraint right from partial synchrony.

2. If a process $p$ has a scheduled timeout, $p$ should process the timeout before the global clock advances too far, unless $p$ crashed before the timeout had to be handled. While this may sound obvious, this requirement is crucial for the failure detector.

3. The global clock must advance infinitely often. Indeed, it is possible to construct a sequence of states that have the global clock increased only finitely many times. This effect is usually called zenoness, after Zeno’s paradoxes. We want to avoid such executions. If you do not believe this is possible, look carefully at the definition rcv_heartbeat_request. It can receive the same message multiple times! Sure, we could eliminate this behavior by receiving every message at most once. It would be harder to do that in a more complex protocol. Just requiring non-zenoness is much simpler.

Lean has no idea about distributed algorithms and fair executions. We can get some inspiration from TLA⁺. Unfortunately, fairness in TLA⁺ is a bit too complicated. If we wanted to transfer this approach to our proofs, we would have to figure out how to write our fairness constraints with strong fairness, weak fairness, and ENABLED.

To avoid this complex ceremony, we recall that our actions have very simple structure. Essentially, every protocol state is constructed by executing one of the six actions:

inductive Action where
  | Init
  | AdvanceClock
  | Timeout(p: Proc)
  | Crash(p: Proc)
  | RcvHeartbeatRequest(src: Proc) (dst: Proc) (timestamp: ℕ)
  | RcvHeartbeatReply(src: Proc) (dst: Proc) (timestamp: ℕ)

Our key idea here is that we could explicitly force some of the actions to be taken in a fair execution. To this end, we refine our transition relation next with next_a:

def next_a (a: @Action Proc): Prop :=
match a with
| Action.Init =>
    s' = s -- dummy action
| Action.AdvanceClock =>
    advance_clock Proc s s'
| Action.Timeout p =>
    timeout Proc InitDelay s s' p
| Action.Crash p =>
    crash Proc s s' p
| Action.RcvHeartbeatRequest src dst timestamp =>
    rcv_heartbeat_request Proc GST MsgDelay s s' src dst timestamp
| Action.RcvHeartbeatReply src dst timestamp =>
    rcv_heartbeat_reply Proc GST MsgDelay s s' src dst timestamp

Of course, we would have to prove equivalence of next and next_a. Actually, we have to account for the case Init, where we just require $s’ = s$. This is easy to do in Lean. Below is the theorem statement. Check PropositionalProofs.lean for the actual proof:

theorem next_a_iff_next
    (s: ProtocolState Proc)
    (s': ProtocolState Proc):
      s' = s ∨ next Proc InitDelay GST MsgDelay s s'
        ↔ ∃ a: Action Proc, next_a Proc InitDelay GST MsgDelay s s' a := by

Now, instead of reasoning just about sequences of protocol states, we can reason about sequences of states and actions. Formally, we introduce the definition of a trace and related definitions:

structure StateAction where
  s: ProtocolState Proc
  a: @Action Proc

/--
  A trace is an infinite sequence of pairs:
   - a state and
   - the action that produced the state from the previous one.

  The initial state is produced by the dummy action `Init`.
  We do not enforce the states to be connected by the `next` relation.
  See `is_path` and `is_run` for stronger conditions.
  -/
abbrev Trace := ℕ → StateAction Proc

/--
  Interpret a trace as a sequence of protocol states.
  -/
def states_of_trace (tr: Trace Proc) :=
  fun i: ℕ => (tr i).s

Not every trace can be produced by the failure detector protocol. We define what it means for a trace to be a run of the protocol, not necessarily a fair one, and what it means for a trace to be a fair run of the protocol:

/--
  A trace is a path, if every pair of state-action pairs `((s_i, _), (s_{i+1},
  a_{i+1})` is a transition via `next_a`. A path does not have to start with an
  initial state.
  -/
def is_path (tr: Trace Proc) : Prop :=
  ∀ i: ℕ,
    next_a Proc InitDelay GST MsgDelay (tr i).s (tr (i + 1)).s (tr (i + 1)).a

/--
  A trace is a (protocol) run, if it starts with an initial state,
  and it is a path.
  -/
def is_run (tr: Trace Proc) : Prop :=
  let s0 := (tr 0).s
  init Proc InitDelay s0
    ∧ is_path Proc InitDelay GST MsgDelay tr

/--
  Does a trace constitute a fair run of the protocol?
  -/
def is_fair_run (tr: Trace Proc) : Prop :=
  is_run Proc InitDelay GST MsgDelay tr
    ∧ is_reliable_communication Proc GST MsgDelay tr
    ∧ is_fair_timeout Proc tr
    ∧ is_fair_clock Proc tr

Having all these definitions, we proceed with our fairness constraints. The simplest one is is_fair_clock:

def is_fair_clock (tr: Trace Proc) : Prop :=
  ∀ i: ℕ,
    ∃ j: ℕ,
      j > i ∧ (tr j).a = Action.AdvanceClock

Essentially, is_fair_clock says that we observe AdvanceClock in a trace infinitely often. In TLA⁺, it would be written like <>[]<advance_clock>_vars. If you don’t know what it means, just skip it.

Further, we define is_fair_timeout as follows:

def is_fair_timeout (tr: Trace Proc): Prop :=
  ∀ i: ℕ,
    ∀ p: Proc,
      ∃ k: ℕ,
        (p ∉ (tr (i + k - 1)).s.crashed → (tr (i + k)).a = Action.Timeout p)
          -- TODO: is this a bit too strong in the presence of is_fair_clock?
          ∧ (tr (i + k)).s.clock = (tr i).s.nextTimeout[p]!

Finally, is_reliable_communication has the longest definition:

def is_reliable_communication (tr: Trace Proc) : Prop :=
  ∀ k: ℕ,
    ∀ m ∈ (tr k).s.sent,
      ∃ i: ℕ,
        let { s := s_j, a := a_j } := tr (k + i)
        isMsgTimely GST MsgDelay m.timestamp s_j.clock
          ∧ m.dst ∈ s_j.crashed
            ∨ match m.kind with
            | MsgTag.HeartbeatReply =>
                a_j = Action.RcvHeartbeatReply m.src m.dst m.timestamp
            | MsgTag.HeartbeatRequest =>
                a_j = Action.RcvHeartbeatRequest m.src m.dst m.timestamp

Now we are ready for the proofs!

3. Proving strong completeness in Lean

The main theorem that we want to prove is strong_completeness_on_states, which basically delegates the work to strong_completeness over fair traces:

/--
  Show that the property `is_strongly_complete` holds on fair runs.
  -/
theorem strong_completeness_on_states
    (tr: Trace Proc)
    (h_is_fair_run: is_fair_run Proc InitDelay GST MsgDelay tr)
    (Crashed: Finset Proc):
      is_strongly_complete Proc Crashed (states_of_trace Proc tr) := by
  unfold is_strongly_complete states_of_trace
  intro h_crashed
  have h_is_crashing_set: is_crashing_set tr Crashed := by exact h_crashed
  exact strong_completeness InitDelay GST MsgDelay
    tr h_is_fair_run Crashed h_is_crashing_set

The below figure summarizes the lemmas (yellow) and theorems (green) that I had to prove, in order to show strong_completeness_on_states.

If you want to understand the proofs, you should inspect PropositionalProofs.lean with the Lean extension for VSCode. I will only give you human-readable summaries as well as my observations about how I wrote these proofs.

3.1. Shorthand temporal definitions

I found it convenient to define shorthands for the temporal properties that are used throughout the proofs. For instance, below is the definition of never_crashes:

/-- A process `p` never crashes, i.e., `[](p ∉ crashed)`.  -/
def never_crashes (tr: Trace Proc) (p: Proc): Prop :=
  ∀ i: ℕ,
    p ∉ (tr i).s.crashed

This property can be visualized as follows:

[](p ∉ crashed) holds

The negation of never_crashes is eventually_crashes:

/-- A process `p` eventually crashes, i.e., `<>(p ∈ crashed)`.  -/
def eventually_crashes (tr: Trace Proc) (p: Proc): Prop :=
  ∃ i: ℕ,
    p ∈ (tr i).s.crashed

<>(p ∈ crashed) holds

We also need eventually_never_alive:

/--
  Eventually, `p` never registers `q` as alive, i.e., `<>[](q ∉ alive[p])`.
  -/
def eventually_never_alive (tr: Trace Proc) (p q: Proc): Prop :=
  ∃ k: ℕ, ∀ i: ℕ,
    q ∉ (tr (k + i)).s.alive[p]!

<>[] (q ∉ alive[p]) holds

We also need q_is_always_suspected and eventually_q_is_always_suspected:

/--
  `p` suspects `q` permanently from some point `k`, i.e.,
  `tr[k,...] ⊧ [](q ∈ suspected[p])`.
  -/
def q_is_always_suspected (tr: Trace Proc) (p q: Proc) (k: ℕ): Prop :=
  ∀ i: ℕ,
    q ∈ (tr (k + i)).s.suspected[p]!

/--
  Eventually, `p` suspects `q` permanently, i.e., `<>[](q ∈ suspected[p])`.
  -/
def eventually_q_is_always_suspected (tr: Trace Proc) (p q: Proc): Prop :=
  ∃ k: ℕ,
    q_is_always_suspected tr p q k

<>[] (q ∈ suspected[p]) holds

Finally, we need the definition of the set of crashing processes:

/--
  A set of processes `C` is a crashing set if every process in `C`
  eventually crashes, and every process not in `C` never crashes.
 -/
def is_crashing_set (tr: Trace Proc) (C: Finset Proc): Prop :=
  ∀ p: Proc, p ∈ C ↔ eventually_crashes tr p

3.2. Warming up with simple temporal lemmas

Before discussing hard-to-prove lemmas, let’s have a look at a few very simple ones. To start with, we can easily show that the global clock never decreases in a single step. The proof is basically done by the simp tactic:

/--
  A single step does not decrease the clock value. In temporal logic,
  `[](clock' ≥ clock)`.
  -/
lemma clock_is_monotonic_in_one_step
    (s: ProtocolState Proc) (s': ProtocolState Proc) (a: Action Proc)
    (h_next: next_a Proc InitDelay GST MsgDelay s s' a):
      s'.clock ≥ s.clock := by
  unfold next_a crash rcv_heartbeat_reply advance_clock
         rcv_heartbeat_request timeout at h_next
  cases a with
  | Init => simp at h_next; rw [h_next]
  | AdvanceClock | RcvHeartbeatRequest _ _ _
  | RcvHeartbeatReply _ _ _ | Timeout _ | Crash _ =>
    simp [h_next]

Very much similar to clock_is_monotonic_in_one_step, we can prove that the set of the crashed processes can only grow in a single step:

/--
  A single step does not decrease the set of the crashed processes.
  In temporal logic, `[](crashed' ⊇ crashed)`.
  -/
lemma crashed_is_monotonic_in_one_step
    (s: ProtocolState Proc) (s': ProtocolState Proc) (a: Action Proc)
    (h_next: next_a Proc InitDelay GST MsgDelay s s' a):
      s'.crashed ⊇ s.crashed := by
  -- literally the same proof as above
  unfold next_a crash rcv_heartbeat_reply
         advance_clock rcv_heartbeat_request timeout at h_next
  cases a with
  | Init => simp at h_next; rw [h_next]
  | AdvanceClock | RcvHeartbeatRequest _ _ _
  | RcvHeartbeatReply _ _ _ | Timeout _ | Crash _ =>
    simp [h_next]

We use these simple lemmas to prove that $clock$ never decreases in a fair run, and once a process has crashed, it always remains crashed. In both cases, the proof is done by simple induction over the indices in a fair run. For example, here is the lemma crashed_is_monotonic_in_fair_run, together with its proof:

/--
  The set `crashed` grows monotonically in a fair run.

  In temporal logic, `∀p: Proc, [](p ∈ crashed) → [](p ∈ crashed))`.
  -/
lemma crashed_is_monotonic_in_fair_run
    (tr: Trace Proc)
    (h_is_fair_run: is_fair_run Proc InitDelay GST MsgDelay tr)
    (p: Proc) (k: ℕ) (h_p_crashed: p ∈ (tr k).s.crashed) (i: ℕ):
      p ∈ (tr (k + i)).s.crashed := by
  induction i with
  | zero => exact h_p_crashed
  | succ i ih =>
    unfold is_fair_run at h_is_fair_run
    rcases h_is_fair_run with ⟨ h_is_run, _ ⟩
    unfold is_run at h_is_run
    rcases h_is_run with ⟨ _, h_is_path ⟩
    unfold is_path at h_is_path
    specialize h_is_path (k + i)
    -- apply crashed_is_monotonic_in_one_step to the last step
    have h_last_step_mono :=
      crashed_is_monotonic_in_one_step InitDelay GST MsgDelay
        (tr (k + i)).s (tr (k + i + 1)).s (tr (k + i + 1)).a h_is_path
    exact h_last_step_mono ih

Further, we prove another useful lemma: Given a clock value $t$, eventually the global clock reaches the value $t$:

/--
  Every fair run covers every clock value `t`. Note that this requires fairness.
  Otherwise, the clock may not advance at all.

  In temporal logic, `∀t ∈ ℕ, <>(clock ≥ t)`.
  -/
lemma eventually_clock_is_t
    (tr: Trace Proc)
    (h_is_fair_run: is_fair_run Proc InitDelay GST MsgDelay tr)
    (t: ℕ):
      (∃ i: ℕ, (tr i).s.clock ≥ t) := by

You can check the full proof of eventually_clock_is_t. It is not a long one, but it requires a bit of linear arithmetic to reason about indices and clock constraints.

3.3. Proving completeness for two processes

Before we dive into the results for all processes, we focus on just two processes in a fair run tr:

• a process p that never crashes in tr,
• a process q that eventually crashes in tr.

Since the Lean proofs are quite detailed, I provide the lemmas with short human-readable proofs. Actually, I had to write proof schemas on paper, before developing detailed proofs. The math-like proofs below are summaries of the detailed Lean proofs, as my pen & paper proofs had several flaws.

3.3.1. Main lemma: Eventually q is always suspected by p

To show strong completeness for $p$ and $q$, we prove the following key lemma:

Lemma eventually_crashes_implies_always_suspected. If $q$ crashes at some time $j$ and $p$ never crashes, then there exists $k$ such that for all $i \ge k$, we have $q \in \text{suspected}[p]$ at $i$.

Using the TLA⁺ notation, we could write this lemma in temporal logic:

This is how the lemma is formulated in Lean:

lemma eventually_crashes_implies_always_suspected
    (tr: Trace Proc)
    (h_is_fair_run: is_fair_run Proc InitDelay GST MsgDelay tr)
    (p q: Proc)
    (h_p_never_crashes: never_crashes tr p)
    (h_crashes: eventually_crashes tr q):
      eventually_q_is_always_suspected tr p q := by

You can check the detailed proof in Lean. The proof is about 100 LOC long. I have asked ChatGPT to summarize the proof similar to a mathematician’s proof. It looked very convincing, but the AI has hallucinated a lot, by inventing additional lemmas and mixing process names. Instead, here is my proof summary, 100% organic:

Proof. Since $q$ eventually crashes, we apply Lemma eventually_crashes_implies_never_alive (see below) to show that there is an index $k$ such that for all $i \ge k$, we have $q \notin alive[p]$ at $i$. Now, we may still have $q \in suspected[p]$ at $k$. Hence, we apply the fairness constraint is_fair_timeout to show that there is an index $j > k$ such that $p$ timeouts at $j$. By the definition of timeout, we have $q \in suspected[p]$ at $j + 1$, as the action timeout updates suspected[p] with Finset.univ \ alive[p], and $q \notin alive[p]$ at $j$.

It remains to show that $q \in suspected[p]$ at an arbitrary $i > j$. We do this by induction on $i$. All actions except Timeout preserve the value of the field suspected, so we have $q \in suspected[p]$. In case of Timeout r, we consider two cases: (1) $r \ne p$, and (2) $r = p$. When $r \ne p$, the value of $suspected[p]$ does not change. When $r = p$, we again invoke the conclusion that $q \notin alive[p]$ at $i$. Similar to the above reasoning about the action timeout, we conclude $q \in suspected[p]$ at all $i > j$. $\blacksquare$

3.3.2. Eventually q is never alive for p

As you have noticed, we invoked Lemma eventually_crashes_implies_never_alive. This is how it looks like in a human-readable form:

Lemma eventually_crashes_implies_never_alive. If $q$ crashes at some time $j$ and $p$ never crashes, then there exists $k$ such that for all $i \ge k$, we have $q \notin \text{alive}[p]$ at $i$.

Using the TLA⁺ notation, we could write this lemma in temporal logic:

This is how the lemma is formulated in Lean, the detailed proof in Lean is 170 LOC:

lemma eventually_crashes_implies_never_alive
    (tr: Trace Proc)
    (h_is_fair_run: is_fair_run Proc InitDelay GST MsgDelay tr)
    (p q: Proc)
    (h_p_never_crashes: never_crashes tr p)
    (h_crashes: eventually_crashes tr q):
      eventually_never_alive tr p q := by

Proof. Since $q$ eventually crashes, there is an index $i_{crash}$, so $q \in crashed$ at $i_{crash}$. Let’s denote with $t_{crash}$ the clock value at $i_{crash}$. However, $p$ may still receive heartbeats from $q$ that were sent in the past. Hence, we choose the time point $t_{magic}$:

We invoke Lemma eventually_clock_is_t to show that eventually the global clock reaches the value $t_{magic}$. Further, we invoke Lemma eventually_alive_is_empty (see below) to show that eventually $alive[p] = \emptyset$ after that point. Hence, we have an index $i_{empty}$ with the following constraints at:

1. $alive[p] = \emptyset$ at $i_{empty}$,

2. $q$ has crashed and no heartbeats from $q$ can longer arrive, as the global clock is past $\mathit{GST} + \mathit{MsgDelay}$,

The rest of the proof goes by induction over $i \ge i_{empty}$. We have already shown the inductive base. The inductive step is proven by contradiction: Assume that there is an index $i + 1$, where $alive[p] \ne \emptyset$. We do case analysis on the action that produces the state at $i + 1$. There are two interesting cases:

1. A process $r$ times out. If $r \ne p$, the $r$ keeps the value of $alive[p]$. If $p$ times out, it resets $alive[p]$ to $\emptyset$. In both cases, $alive[p] = \emptyset$.

2. A process $\mathit{dst}$ receives a heartbeat reply $m$ from a process $\mathit{src}$. The cases of $dst \ne p$ or $src \ne q$ are trivial, as the predicate $q \in alive[p]$ does not change in those cases. The case of $src = q$ and $dst = p$ is the hardest one. First, we apply Lemma crashed_process_does_not_send (see below) to show that the message timestamp $m.ts$ is not greater than $t_{crash}$. Second, we apply Lemma clock_is_monotonic_in_fair_run to show that $clock \ge t_{magic}$ at point $i$. Third, we apply the constraint isMsgTimely from the definition of rcv_heartbeat_reply. We arrive at the following combination of linear constraints that do not have a solution:

This inductive argument finishes the proof. $\blacksquare$

3.3.3. Non-crashing p resets alive infinitely often

We invoked Lemma eventually_alive_is_empty in the previous section. This is how this lemma looks like in a human-readable form:

Lemma eventually_alive_is_empty. If $p$ never crashes, then for every $k > 0$, there is $i \ge k$ such that we have $\text{alive}[p] = \emptyset$ at $i$.

Using the TLA⁺ notation, we could write this lemma in temporal logic:

This is how the lemma is formulated in Lean, the detailed proof in Lean is 30 LOC:

lemma eventually_alive_is_empty
    (tr: Trace Proc)
    (h_is_fair_run: is_fair_run Proc InitDelay GST MsgDelay tr)
    (p: Proc)
    (h_p_never_crashes: never_crashes tr p)
    (k: ℕ)
    (h_i_positive: k > 0):
      ∃ i: ℕ, (tr (k + i)).s.alive[p]! = ∅ := by

Proof. Since $p$ never crashes, we apply the fairness constraint is_fair_timeout to the index $k$. Hence, there exists an index $i$, so that $p$ times out at $k + i$. When processing the action timeout, process $p$ resets $alive[p]$ to the empty set. $\blacksquare$

3.3.4. A crashed process q stops sending messages

We invoked Lemma crashed_process_does_not_send in the proof of eventually_crashes_implies_never_alive. This is how this lemma looks like in a human-readable form:

Lemma crashed_process_does_not_send. If $q$ is crashed at $k$ and the clock value at $k$ equals to some $c$, then for every $i \ge 0$ and every message $m \in sent$ at $k + i$, if $m.src = q$, then $m.ts \le c$.

Using the TLA⁺ notation, we could write this lemma in temporal logic:

This is how the lemma is formulated in Lean, the detailed proof in Lean is 110 LOC:

lemma crashed_process_does_not_send
    (tr: Trace Proc)
    (h_is_fair_run: is_fair_run Proc InitDelay GST MsgDelay tr)
    (p: Proc) (k: ℕ) (h_p_crashed: p ∈ (tr k).s.crashed):
      ∀ i: ℕ,
        ∀ m ∈ (tr (k + i)).s.sent,
          m.src = p → m.timestamp ≤ (tr k).s.clock := by

Although this lemma seems to be obvious, the proof is relatively long. It is mostly technical.

Proof. The proof is done by induction on $i$. It invokes two other lemmas:

1. Lemma crashed_is_monotonic_in_fair_run that we discussed before, and

2. Lemma no_sent_from_the_future (see below), which states that no message can have a timestamp above the current value of the global clock.

The induction goes by case analysis on the action that is executed at point $i$. There are two interesting cases that extend the set of sent messages $sent$:

1. A timeout by process $r$. Since $q$ crashed at $k$, it remains crashed at $k+i$. Hence, $q$ cannot timeout, and, thus, $r \ne q$. Therefore, the new heartbeat requests in $sent$ do not have $q$ as their source.

2. Receiving a heartbeat request by process $r$. Again, $q$ is crashed at $k+i$, and $r \ne q$. Therefore, the new heartbeat replies in $sent$ do not have $q$ as their source. $\blacksquare$

3.3.5. No message sent from the future

Lemma no_sent_from_the_future plays an important role in the proof of crashed_process_does_not_send. This is how this lemma looks like in a human-readable form:

Lemma no_sent_from_the_future. For every point $i \ge 0$ and every message $m \in sent$ at $i$, it holds that $m.ts \le clock$.

Using the TLA⁺ notation, we could write this lemma in temporal logic:

This is how the lemma is formulated in Lean, the detailed proof in Lean is 110 LOC:

lemma no_sent_from_the_future
    (tr: Trace Proc)
    (h_is_fair_run: is_fair_run Proc InitDelay GST MsgDelay tr):
      ∀ i: ℕ,
        ∀ m ∈ (tr i).s.sent,
          m.timestamp ≤ (tr i).s.clock := by

Proof. The proof is done by induction on $i$ and case analysis on the action executed at $i$. The proof is quite mechanical. We basically show that either the set $sent$ does not change, whereas the value of $clock$ does not decrease, or the new messages have their timestamp set to $clock$. Such messages are sent in timeout and rcv_heartbeat_request. $\blacksquare$

Most likely, a human reader would immediately infer this lemma without much thought. However, the lemma’s proof works only because we’re using the global clock value $clock$ when sending messages. If we used local clocks for timestamps, the lemma would no longer hold.

3.3.6. Other lemmas

The proof of no_sent_from_the_future invokes another lemma called inductive_inv. It provides us with a general proof of inductive invariants in the context of our protocol. I expected more proofs to use inductive_inv, but it happened that the other proofs required additional temporal reasoning beyond simple inductive invariants.

Finally, we have another lemma called when_clock_is_positive_step_is_non_init. It is just a technical lemma to work around a corner case that could not be solved by the tactic omega. You can check this lemma. There is really nothing interesting in it.

Overall, when we go down the dependency tree of our lemmas, the proofs in the top require quite a bit of creative thinking. The proofs in the bottom like crashed_process_does_not_send and no_sent_from_the_future are quite mechanical. They are long but they do not require much thinking. It would be great if those proofs could be derived automatically.

3.4. From 2 to N processes

With main lemma, we have proven strong completeness for a pair of processes. Actually, we could just stop there. However, I decided to go the last mile and prove strong completeness for arbitrary sets of processes, exactly as the properties are written in DP2011. The last mile happened to be harder than I anticipated. Nevertheless, the findings and the proof technique are quite interesting.

3.4.1. Defining the crashed processes

When I was writing a proof on paper, I was writing something along these lines:

Given a fair run, let us define the set Crashed that contains exactly those processes that crash in the run.

Hence, I tried to write a definition like this in Lean:

def crashed_set (tr: Trace Proc) :=
  { p: Proc | ∃ i ∈ Nat, p ∈ (tr i).s.crashed }

Lean produced an a bit obscure error: “failed to synthesize Membership ?m.10217 Type”.

So I thought, OK, it seems to be hard to define a potentially infinite set that uses a proposition over an infinite sequence. Let’s try finite sets:

def crashed_set (tr: Trace Proc) :=
  Finset.univ.filter (fun p => ∃ i ∈ ℕ, p ∈ (tr i).s.crashed)

The same error. What is going on? If we rewrite the definition like this, it works (obviously, it does not do what we want though):

def crashed_set (tr: Trace Proc) :=
  Finset.univ.filter (fun p => p ∈ (tr 0).s.crashed)

Ugh. It looks like Lean does not like that we have to prove existence of a member of an infinite set to filter a finite set. It would be fine to use that in a proposition, but not in a definition. Well, this kind of makes sense. We cannot just compute crashed_set, as we cannot predict when processes crash! Lean is a bit strict about random mathy stuff.

Interestingly though, given a fair run, we should be able to define the set of the crashed processes. This set is bounded from above with the finite set Finset.univ of type Finset Proc. Also, as we showed in crashed_is_monotonic_in_fair_run, the set of crashed processes can only grow, not shrink. Hence, in theory, we should be able to define crash_set as the fixpoint of the operator that transforms $s_i$ into $s_{i+1}$ in our run. We should be able to apply Knaster-Tarski theorem. Conversations with ChatGPT about Knaster-Tarski in Lean opened a new rabbit hole.

This was getting too hard, all of a sudden. So I have decided that it was not worth the effort. If I had a conversation like that with a customer, they would tell me to stop right there. Hence, I decided that the properties should simply have two parameters:

• The set (Crashed: Finset Proc), and
• a proof that it’s exactly the set of the crashing processes.

This is why our temporal properties have these two parameters.

3.4.2. Where do the suspected sets meet?

Recall that we have proven the main lemma for two processes, and it looks like this:

lemma eventually_crashes_implies_always_suspected
    (tr: Trace Proc)
    (h_is_fair_run: is_fair_run Proc InitDelay GST MsgDelay tr)
    (p q: Proc)
    (h_p_never_crashes: never_crashes tr p)
    (h_crashes: eventually_crashes tr q):
      eventually_q_is_always_suspected tr p q := by

The yet-to-prove theorem strong_completeness looks like this:

theorem strong_completeness
    (tr: Trace Proc)
    (h_is_fair_run: is_fair_run Proc InitDelay GST MsgDelay tr)
    (Crashed: Finset Proc)
    (h_is_crashing_set: is_crashing_set tr Crashed):
      ∃ k: ℕ, ∀ i: ℕ, ∀ p q: Proc,
        (p ∉ Crashed ∧ q ∈ Crashed) → q ∈ (tr (k + i)).s.suspected[p]! := by

The challenge here is that the theorem claims existence of a single index $k$ for all processes, whereas we get different indices when applying the lemma. Intuitively, we should be just able to pick the maximum index among them. This starts to smell like the above problem with the crashing sets. On the other hand, choosing the maximum among the values of a finite set should be possible. This made me think about Well-founded induction. Intuitively, we should be able to start with the empty set, add elements one-by-one and pick the maximum of two numbers at each inductive step: the maximum of the smaller set, and the value for the new element. This is what Finset.induction can do for us!

Another way to see the issue is by comparing these two temporal formulas in TLA⁺:

Hence, to go from Equation (1) to Equation (2), we have to move two quantifiers $\forall p$ and $\forall q$ inside $\Diamond \square (\dots)$. This observation together with Finset.induction gave me this nice theorem:

/--
  For a finite set of values `vs` and a state proposition `P`, show that we can
  swap a universal quantifier and eventually-always. In temporal logic `∀ v ∈
  vs, <>[](P v) → <>[](∀ v ∈ vs, P v)`.
  -/
theorem forall_FG_implies_FG_forall
    (P: TraceProp)
    (vs: Finset Val):
    (∀ v ∈ vs, ∃ k: ℕ, ∀ i: ℕ, P (k + i) v) →
      (∃ k: ℕ, ∀ i: ℕ, ∀ v ∈ vs, P (k + i) v) := by

The proof of the theorem is not hard, but it is 60 LOC. So you can check it online.

By using this theorem twice, we finally arrive at the final lemma:

/--
  For a set of crashing processes `C` and a trace `tr`, show that if for every
  crashing process `q` and every correct process `p`, it holds that `p`
  eventually suspects `q` forever, then there is a common time point `k` such
  that all correct processes suspect all crashed processes forever.

  In temporal logic, `∀ q ∈ Crashed, ∀ p ∈ Correct, <>[](q ∈ suspected[p]!)`
  implies `<>[] ∀ q ∈ Crashed, ∀ p ∈ Correct, q ∈ suspected[p]!`.
  -/
lemma eventually_always_suspected_meet
    (tr: Trace Proc)
    (Crashed: Finset Proc)
    (h_suspected:
      ∀ q ∈ Crashed,
        ∀ p ∈ Finset.univ \ Crashed,
          eventually_q_is_always_suspected tr p q):
      ∃ k: ℕ,
        ∀ i: ℕ,
          ∀ q ∈ Crashed,
            ∀ p ∈ Finset.univ \ Crashed,
              q ∈ (tr (k + i)).s.suspected[p]! := by
  -- fix the set of correct processes
  let Correct := Finset.univ \ Crashed
  -- we have to bubble up `∃ k: ℕ` two times
  -- bubble up `∃ k: ℕ` the first time
  have bubble_once: (q: Proc) → (h_q_crashed: q ∈ Crashed) →
      ∃ k: ℕ, ∀ i: ℕ, ∀ p ∈ Correct, q ∈ (tr (k + i)).s.suspected[p]! := by
    intro q h_q_crashed
    specialize h_suspected q h_q_crashed
    let P (i: ℕ) (p: Proc) := q ∈ (tr i).s.suspected[p]!
    exact forall_FG_implies_FG_forall P Correct h_suspected
  -- the predicate `P` to use in the next instance of `forall_FG_implies_FG_forall`
  let P (i: ℕ) (q: Proc) :=
    ∀ p ∈ Correct, q ∈ (tr i).s.suspected[p]!
  -- bubble up `∃ k: ℕ` the second time
  exact forall_FG_implies_FG_forall P Crashed bubble_once

With this lemma, we finally prove strong_completeness:

theorem strong_completeness
    (tr: Trace Proc)
    (h_is_fair_run: is_fair_run Proc InitDelay GST MsgDelay tr)
    (Crashed: Finset Proc)
    (h_is_crashing_set: is_crashing_set tr Crashed):
      ∃ k: ℕ, ∀ i: ℕ, ∀ p q: Proc,
        (p ∉ Crashed ∧ q ∈ Crashed) → q ∈ (tr (k + i)).s.suspected[p]! := by

The proof is just a technical application of eventually_crashes_implies_always_suspected and eventually_always_suspected_meet. It is 40 LOC of unfolding definitions and repacking them into the right format.

Conclusions

This was probably the longest blog post I have ever written. It almost feels like an academic paper. I don’t expect many people to read all of it. If you have read the whole blog post and reached the conclusions, leave me a comment! I really want to know, whether anyone manages to read the whole writeup.

Footnotes

Tags: specification lean distributed proofs tlaplus

In the previous blog post, we discussed specification, randomized simulation, and property-based testing of Two-phase commit in Lean 4. The obvious question is whether we can use Lean for what it was designed for, namely, proving correctness of the protocol. Yes, we can! Here is our proof plan:

In short, I have managed to write full proofs of consistency in Lean 4, starting with a functional specification. Except for a few moments, it was clear how to proceed, though interactive proofs are tedious. In total, it took me 29 hours to write the proofs, excluding the time that was needed to read the Lean manuals. Together with specification and simulation from the previous blog post, the whole effort required 45 hours.

I believe the proofs went quickly because the inductive invariant was already correct, since we have found it with the model checker Apalache. In fact, I could probably reduce the proof times even further if I focused on minimizing the inductive invariant. If the invariant had not been correct, though, the process likely would not have gone as smoothly.

Let us have a look at the statistics in the table below.

The ratio of proofs (propositional and inductive) to the system code (propositional) is about 15. This fits into the empirical ratio of software verification, where the proofs are 10-20 longer than the source code.

In this blog post, we have explored a “traditional” path of interactive theorem proving, though we have cut corners by finding the inductive invariant with the model checker.

Another route to explore is to prove equivalence between our specification in Propositional.lean and the Veil specification. The Veil examples already contain a version of two-phase commit, though it is slightly different from the two-phase commit in TLA⁺ and our specification in Lean. Perhaps, this is a good topic for another exercise.

Certainly, this is not the first exercise in using interactive theorem provers to verify safety of a distributed algorithms. To name a few examples, there were larger-scale efforts such as IronFleet, Verdi, Disel, and Bythos.

Table of contents

1. What to prove?
2. Connecting functional and propositional specs
3. Finding an inductive invariant
4. Proving the inductive step in Lean 4
5. Proving consistency with the inductive invariant
6. Proving the inductive base

1. What to prove?

The task does not seem simple, though. How do we approach it? Our goal is to prove the consistency of the protocol. Fortunately, we started with the specification in TLA+, so we can stand on the shoulders of giants and reuse the TLA⁺ methodology. Here is how consistency is specified in TLA⁺:

TCConsistent ==  
  (*************************************************************************)
  (* A state predicate asserting that two RMs have not arrived at          *)
  (* conflicting decisions.                                                *)
  (*************************************************************************)
  \A rm1, rm2 \in RM : ~ /\ rmState[rm1] = "aborted"
                         /\ rmState[rm2] = "committed"

This invariant looks quite similar in Lean:

def consistency (s: ProtocolState RM) : Prop :=
  ∀ rm₁ rm₂: RM,
    s.rmState[rm₁]? ≠ some RMState.Committed ∨ s.rmState[rm₂]? ≠ some RMState.Aborted

By following the TLA⁺ methodology, to show that $TCConsistent$ is an invariant of Two-phase commit, it suffices to find a state predicate $IndInv$ and prove three properties:

1. The initial states satisfy the invariant $IndInv$, that is, $Init \Rightarrow IndInv$.

2. The transition relation preserves the invariant $IndInv$, that is, $Next \land IndInv \Rightarrow IndInv’$.

3. The invariant $IndInv$ implies the state invariant $TCConsistent$, that is, $IndInv \Rightarrow TCConsistent$.

The invariant $\mathit{IndInv}$ is called an inductive invariant, since it allows us to reason about all states that are reachable from $\mathit{Init}$ via $\mathit{Next}$ by induction. The interesting thing is that it is sufficient to prove this principle only once and reuse it for all specifications. This is why we simply use this approach without re-proving it every time. The cool thing about Lean is that we still can re-prove this inductive principle, if we want to.

2. Connecting functional and propositional specs

Now we have to understand what to use as the initial predicate $Init$ and the transition relation $Next$. If you have read the previous blog post, you remember that we had two kinds of specifications:

• A functional specification in Functional.lean and System.lean, and

• A propositional specification in Propositional.lean.

Luckily, Ilya Sergey warned me that writing proofs at the functional level is hard, so I did it at the propositional level. To connect the functional spec and the propositional spec, we prove two theorems:

theorem tp_init_correct (all: List RM) (s: ProtocolState RM):
    tp_init all s ↔ init all = s := by

theorem tp_next_correct (s: ProtocolState RM) (s': ProtocolState RM):
    tp_next s s' ↔ ∃ a: Action, next s a = some s' := by

You can find complete proofs in PropositionalProofs.lean. What’s interesting, it took me just 2.5 hours to write the equivalence proofs for all seven actions. That was fast, because once I wrote three proofs, the remaining four were completely generated by Copilot! As I expected, these proofs were relatively easy to write.

For instance, here is the function tm_commit in the functional specification:

/--
  The transaction manager commits the transaction. Enabled iff the TM is its initial
  state and every RM has sent a `Prepared` message.
 -/
def tmCommit :=
    if s.tmState = TMState.Init && s.tmPrepared = s.all then some {
        s with
        tmState := TMState.Committed,
        msgs := s.msgs ∪ { Message.Commit }
    } else none

And here is the propositional version tm_commit, which looks very much like an action in TLA⁺:

/-- The proposition version of `tmCommit`. -/
def tm_commit: Prop :=
    s.tmState = TMState.Init
  ∧ s.tmPrepared = s.all
  ∧ s'.tmState = TMState.Committed
  ∧ s'.msgs = s.msgs ∪ { Message.Commit }
  ∧ s'.tmPrepared = s.tmPrepared
  ∧ s'.rmState = s.rmState
  ∧ s'.all = s.all

The theorem tm_commit_correct is connecting both of them.

theorem tm_commit_correct (s: ProtocolState RM) (s': ProtocolState RM):
    tm_commit s s' ↔ tmCommit RM s = some s' := by
  apply Iff.intro
  case mp =>
    intro hrel
    simp [tm_commit] at hrel
    rcases hrel with ⟨ h_tmState, h_tmPrepared, h_tmState', h_msgs',
      h_tmPrepared', h_rmState', h_all' ⟩
    simp [tmCommit, h_tmState, h_tmPrepared]
    apply ProtocolState.ext
    repeat simp [*]

  case mpr =>
    intro heq
    simp [tmCommit] at heq
    rcases heq with ⟨ ⟨ h_tmState, h_tmPrepared ⟩, h_seq ⟩
    unfold tm_commit
    simp [h_tmState, h_tmPrepared]
    cases h_seq
    repeat simp [*]

If you are like me, it is hard to make sense of this proof by just starring at it, in contrast to pen & paper proofs. If you want to understand the proof, download the spec and go over the proof line by line with the Lean plugin. Of course, you would have to understand how the proofs are organized in Lean. The book on Theorem Proving in Lean 4 explains this.

It took me one more hour to prove the theorem tp_next_correct. However, when I turned to tp_init_correct, I got carried away trying to prove a statement that was too difficult. The proof involved several inductive arguments about hash maps, and I ended up spending four hours wrestling with a challenging fact. Once I clarified that, it only took 30 minutes to write a simpler and more effective proof.

Basically, the entire set of refinement proofs could be completed in a single day! The fact that Copilot was able to fill in four out of seven cases suggests that these proofs could be generalized into a broader lemma. This is evident from the structure of the proofs. I decided to leave them as they are for now, but we should consider making them more compact for easier maintenance.

For the remainder of this post, we will use only the propositional specification.

3. Finding an inductive invariant

To follow the proof methodology, we have to find $IndInv$. We could try to use our goal invariant consistency as a candidate for the invariant. However, safety properties rarely work as inductive invariants. Intuitively, an inductive invariant should generalize all reachable states, and consistency is too weak for that role.

How do we find $IndInv$? One approach would be to start with True, then try to prove the three properties. Once we understood why True is not good enough, add constraints. Repeat. In theory, this approach could work. In practice, it is too hard, as we have to write proofs by hand. It may happen that we finish 90% of a proof just to find that our candidate for $IndInv$ was not good enough.

We do not have to go the hard way. Instead, we can just use a model checker for TLA⁺ to quickly iterate on a candidate for an inductive invariant for a small set of resource managers. This is exactly what we did for our paper at OOPSLA19 at some point in 2018. I guess, it should be available in the artifact. In the modern version of Apalache, the inductive invariant looks like this:

IndInv ==
    /\ TPTypeOK
    /\ TCConsistent 
    /\ (\E rm \in RM: rmState[rm] = "committed") => tmState = "committed"
    /\ tmState = "committed" => /\ tmPrepared = RM
                                /\ \A rm \in RM: rmState[rm] \notin {"working", "aborted"}
                                /\ MkCommit \in msgs
    /\ tmState = "aborted" => MkAbort \in msgs
    /\ \A rm \in RM:
      /\ rm \in tmPrepared =>
        /\ rmState[rm] /= "working"
        /\ MkPrepared(rm) \in msgs
      /\ rmState[rm] = "working" => MkPrepared(rm) \notin msgs
      /\ MkPrepared(rm) \in msgs => rmState[rm] /= "working" 
      /\ rmState[rm] = "aborted" =>
        \/ MkAbort \in msgs
        \/ MkPrepared(rm) \notin msgs
    /\ MkAbort \in msgs =>
        \* it is either the TM or an RM who was in the "working" state
        \/ tmState = "aborted"
        \/ \E rm \in RM:
          /\ rmState[rm] = "aborted"
          /\ rm \notin tmPrepared
          /\ MkPrepared(rm) \notin msgs                 
    /\ MkCommit \in msgs =>
        /\ tmPrepared = RM
        /\ \/ tmState = "committed"
           \/ \E rm \in RM: rmState[rm] = "committed" 

By running Apalache, we can make sure that our invariant is inductive for three resource managers:

$ apalache-mc check --length=0  --init=Init --inv=IndInv MC3_TwoPhaseTypedInv.tla
...
Total time: 1.282 sec
$ apalache-mc check --length=1 --init=IndInv --inv=IndInv MC3_TwoPhaseTypedInv.tla
...
The outcome is: NoError
Total time: 1.621 sec
$ apalache-mc check --length=0  --init=IndInv --inv=TCConsistent MC3_TwoPhaseTypedInv.tla
...
The outcome is: NoError
Total time: 1.342 sec

We can do the same for 7, 20, and even 50 resource managers. However, as we increase the number of resource managers, the model checker takes longer to verify the properties. For example, checking inductiveness for 20 resource managers takes about 8 seconds, compared to just 2 seconds for 3 resource managers.

To make sure that the model checker is not just printing "NoError" but also doing something useful, we replace "committed" with "aborted" in the last line of IndInv. In this case, the model checker immediately gives us a counterexample to induction.

$ apalache-mc check --length=1 --inv=IndInv --init=IndInv MC3_TwoPhaseTypedInv.tla
...
Check the trace in: [...]/violation1.tla
State 1: state invariant 5 violated.
Total time: 1.760 sec

Basically, this is how we speed up the guess-work of finding an inductive invariant with the model checker.

Interestingly, we can remove the line /\ TCConsistent, and Apalache does not complain. This constraint is redundant. It is actually great that we have removed this redundant constraint, as it would take us much longer to write interactive proofs with this constraint in place.

We IndInv as a conjunction of six lemmas in Lean, as it is easier to reason about the invariant this way:

-- Our inductive invariant that we use to prove the consistency property.
def invariant (s: ProtocolState RM) : Prop :=
  lemma1 s ∧ lemma2 s ∧ lemma3 s ∧ lemma4 s ∧ lemma5 s ∧ lemma6 s

You can find all six lemmas in InductiveProofs.lean. For example, here is how we write lemma1:

def lemma1 (s: ProtocolState RM): Prop :=
  (∃ rm: RM, s.rmState[rm]? = some RMState.Committed) → s.tmState = TMState.Committed

If you look carefully at lemma5 and lemma6 in InductiveProofs.lean, you will notice that they also have redundancies. I missed that and this resulted in a lot of extra work when writing proofs. It would only take several seconds to check with Apalache, whether we could remove these subformulas. More on that later.

4. Proving the inductive step in Lean 4

Now that we know our invariant is inductive for small sets of resource managers, we have good chances of proving the inductive step of invariant. The transition relation tp_next is a disjunction of seven smaller subformulas such as tm_commit and tm_abort. Further, invariant is a conjunction of six lemmas.

Thus, our plan is very simple: Prove inductiveness for each of the seven actions and each of the six lemmas. This gives us 42 facts to prove. While this sounds like a lot of work, the good news is that proving 42 smaller lemmas is easier than proving one huge theorem.

Actually, the above decomposition is not hand-waiving. Our theorem invariant_is_inductive is proven exactly by this decomposition. Below you can see the first six cases:

/--
 Showing that `invariant` is inductive, that is, it is preserved by the transition relation.
-/
theorem invariant_is_inductive (s: ProtocolState RM) (s': ProtocolState RM)
  (h_all: ∀ rm: RM, rm ∈ s.all) (h_inv: invariant s) (h_next: tp_next s s'):
    invariant s' := by
  unfold tp_next at h_next
  cases h_next
  case inl h_tm_commit =>
    -- action tm_commit
    unfold invariant
    -- prove the lemmas one by one
    apply And.intro
    . exact invariant_is_inductive_tm_commit_lemma1 s s' h_tm_commit
    . apply And.intro
      . exact invariant_is_inductive_tm_commit_lemma2 s s' h_all h_inv h_tm_commit
      . apply And.intro
        . apply invariant_is_inductive_tm_commit_lemma3 s s' h_tm_commit
        . apply And.intro
          . exact invariant_is_inductive_tm_commit_lemma4 s s' h_inv h_tm_commit
          . apply And.intro
            . exact invariant_is_inductive_tm_commit_lemma5 s s' h_inv h_tm_commit
            . exact invariant_is_inductive_tm_commit_lemma6 s s' h_tm_commit

  case inr h_rest =>

(If you know how to write the above decomposition nicer in Lean, please leave a comment below.)

Now we have to prove 42 lemmas by writing the proofs. For example, here is the proof for the action tm_commit and lemma1:

-- Effort: 10m
lemma invariant_is_inductive_tm_commit_lemma1 (s: ProtocolState RM) (s': ProtocolState RM)
  (h_tm_commit: tm_commit s s'):
    lemma1 s' := by
    unfold lemma1
    intro h_committed
    exact show s'.tmState = TMState.Committed by
      unfold tm_commit at h_tm_commit
      simp [h_tm_commit]

As you can see, this was an easy one. It took me just 10 minutes to write it. Actually, closer to the end of the proof, I was writing similar proofs in 1-2 minutes. You can find the remaining 41 proofs in InductiveProofs.lean. The plot below shows the total proof efforts:

There are several things to notice:

1. Over a half of the lemmas took me less than 15 minutes each.
2. Ten lemmas took me from 20 to 30 minutes.
3. Six lemmas took me about 1 hour.
4. There is one outlier that required over three hours.

I am not exactly sure what happened with the lemma that took me so long. It was the second one I had to prove, so I might have spent a lot of time just figuring out the right tactics to use. If you look at the proof of invariant_is_inductive_tm_commit_lemma2, it involves reasoning with universal quantifiers and some small proofs by contradiction. If I were to write it again now, it probably would not take nearly as much time.

Something interesting happened when I proved about 80% of the lemmas. I got stuck. This is not really surprising, as I was at the core argument that required reasoning about one resource manager being in the Aborted state and another resource manager receiving a Commit message. Basically, I was struggling with an argument for the constraints in the commented out section below:

def lemma5 (s: ProtocolState RM) : Prop :=
  Message.Abort ∈ s.msgs → s.tmState = TMState.Aborted
  /- Added when discovering the invariant with the model checker.
      It was redundant and complicated the proof.
  (s.tmState = TMState.Aborted
    ∨ ∃ rm: RM,
        s.rmState[rm]? = some RMState.Aborted
      ∧ rm ∉ s.tmPrepared
      ∧ Message.Prepared rm ∉ s.msgs)
  -/

The commented out disjunction is not exactly wrong. The second disjunct of it implies the first disjunct. However, the second disjunct is much harder to reason about. The solution? Just remove it.

Once I have removed the redundant part, the proof was quick and easy:

lemma invariant_is_inductive_rm_rcv_commit_msg_lemma5 (s: ProtocolState RM) (s': ProtocolState RM)
  (rm: RM) (h_inv: invariant s) (h_rm_rcv_commit_msg: rm_rcv_commit_msg s s' rm): lemma5 s' := by
    unfold lemma5
    unfold rm_rcv_commit_msg at h_rm_rcv_commit_msg
    have h_unchanged_msgs: s'.msgs = s.msgs := by simp [h_rm_rcv_commit_msg]
    have h_unchanged_tm_state: s'.tmState = s.tmState := by simp [h_rm_rcv_commit_msg]
    simp [h_unchanged_tm_state, h_unchanged_msgs]
    rcases h_inv with ⟨_, _, _, _, h_lemma5_s, _⟩
    unfold lemma5 at h_lemma5_s
    exact h_lemma5_s

A similar redundancy was present in lemma6. So I removed it, too. By removing these two redundancies, I have cut the proofs in half! Small differences in proof goals may have big implications on the proof complexity.

def lemma6 (s: ProtocolState RM) : Prop :=
  Message.Commit ∈ s.msgs →
    s.tmPrepared = s.all ∧ s.tmState = TMState.Committed
    /- Added when discovering the invariant with the model checker.
        It was redundant and complicated the proof.
        (s.tmState = TMState.Committed
      ∨ ∃ rm: RM, s.rmState[rm]? = some RMState.Committed)
      -/

5. Proving consistency with the inductive invariant

Once we are sure that invariant is inductive, it is easy to prove that it implies consistency. It took me just 15 minutes to write the proof:

-- Proving that the inductive invariant implies the consistency property.
-- Effort: 15m
theorem invariant_implies_consistency (s: ProtocolState RM) (h_inv: invariant s):
    consistency s := by
  unfold consistency
  intro rm₁ rm₂
  by_contra h_committed_and_aborted -- assume the opposite
  simp at h_committed_and_aborted
  rcases h_committed_and_aborted with ⟨h_rm1_committed, h_rm2_aborted⟩
  have h_ex_committed: ∃ rm: RM, s.rmState[rm]? = some RMState.Committed := by use rm₁
  unfold invariant at h_inv
  rcases h_inv with ⟨h_lemma1_s, h_lemma2_s, _⟩
  unfold lemma1 at h_lemma1_s
  unfold lemma2 at h_lemma2_s
  have h_tm_committed: s.tmState = TMState.Committed := by exact h_lemma1_s h_ex_committed
  simp [h_tm_committed] at h_lemma2_s
  rcases h_lemma2_s with ⟨_, _, h_no_working_or_aborted⟩
  specialize h_no_working_or_aborted rm₂
  rcases h_no_working_or_aborted with ⟨_, h_rm2_not_aborted⟩
  rw [h_rm2_aborted] at h_rm2_not_aborted
  exact h_rm2_not_aborted rfl

6. Proving the inductive base

Finally, we have to show that the initial states also satisfy invariant. Here is just the header of the theorem:

theorem init_implies_invariant (all: List RM) (s: ProtocolState RM)
    (h_all: ∀ rm: RM, rm ∈ s.all) (h_init: tp_init all s): invariant s := by

Initially, I thought that the proof would not be harder than the proof of the inductive step, since there are fewer conditions to prove. Essentially, we have to prove six lemmas for the initial states. In total, I think it took me about three hours to finish the proofs.

This proof had an unexpected complication. I had to show that the initialization of the hash map rmState sets all resource managers to the state Working. Here is the header of this lemma:

-- show that the initialization predicate sets all the resource managers to `Working`
lemma init_rm_state_post (all: List RM) (s: ProtocolState RM)
    (h_init: tp_init all s):
    ∀ rm ∈ s.all, s.rmState.get? rm = RMState.Working := by

I observed similar issues when going through Isabelle tutorials many years ago, so I remember that sometimes one had to generalize the proof goal. This is exactly what happened here. In the end, the proof required this additional lemma:

-- An additional lemma to reason about map initialization.
-- Figuring out that we need this lemma was probably the hardest part of the proof.
lemma init_rm_keys (rm: RM):
    ∀ all: List RM,
      ∀ hashmap: Std.HashMap RM RMState,
        (all.foldl (fun m rm' => m.insert rm' RMState.Working) hashmap)[rm]? =
          if rm ∈ all then some RMState.Working else hashmap[rm]? := by

Interestingly, this is the only proof that explicitly uses the induction tactic. Even though we have been reasoning about an inductive invariant, we did not need to go into induction. Coming from TLA⁺, this is interesting. I had to use foldl to initialize the hash map, since Lean does not seem to have convenient primitives such as the function constructor. In TLA⁺, we would just use:

  [ rm ∈ RM |-> "working"]

The function constructor does not introduce explicit iteration and actually has the semantics of what I had to prove with the lemma init_rm_state_post. (To be precise, it also specifies the function domain.) Perhaps, we could introduce higher-level primitives in Lean 4 to deal with this. If you know of a better alternative to using HashMap, please let me know in the comments.

Reviewers: Thomas Pani

Tags: specification lean distributed simulation pbt tlaplus

More and more people mention the Lean theorem prover in my bubble. Just the last week Ilya Sergey and co announced Veil, an IVy-like verification framework on top of Lean. Luckily, I heard a long tutorial on Lean by Sebastian Ullrich and Joachim Breitner at VSTTE24. The interesting thing about Lean is that it’s not only a theorem prover, but also a decent programming language.

So, I have decided to take on a case study of specifying a relatively simple yet interesting distributed protocol in Lean. For this reason, I did not choose a more complex consensus algorithm and instead settled on two-phase commit, for the following reasons:

• It is an interesting distributed protocol with applications in databases.

• It has been specified in TLA⁺ by Leslie Lamport, so we do not have to think about choosing the right level of abstraction. We simply follow the same level of abstraction as in the TLA⁺ specification.

It took me about three hours to translate the TLA⁺ specification into Lean spec, occasionally debating syntax and best practices with ChatGPT and Copilot along the way. See the Functional specification and System-level specification. I had to invent several patterns on the way, as specifying distributed algorithms in Lean looks like terra incognita :dragon: (when compared with TLA⁺). Importantly, I tried to stay close to the original TLA⁺ specification in spirit, but not in the dogmas. My goal was to specify the protocol in a way that looks natural in Lean, instead of literally replicating the TLA⁺ idioms. In addition to that, I wanted the specification to be executable, since I was not sure about writing complete proofs of correctness. When I was writing this blog post, I realized that it is also possible to write a Propositional specification. This specification looks even closer to the original TLA⁺ specification, and also much more “natural”, if you are really into TLA⁺, even though this version is not executable.

What is intriguing is that the resulting specification had the necessary ingredients to be executable, since the Lean tools can compile a large subset of the language into C. Obviously, it does not automatically generate a distributed implementation of our distributed protocol. As we know from Quint, it is quite useful to have a randomized simulator, especially, if there are no other automatic analysis tools around.

As the next step, I wrote a very simple randomized simulator to check the properties against the specification — see Randomized simulation. It turned out to be even easier to implement in Lean than I expected. After about four hours of work, I had the simulator running and producing counterexamples to the properties that are expected to be violated. Maybe this is because I knew exactly what was needed, having written the Quint simulator a couple of years ago. Or maybe it is because Lean does not shy away from the occasional need for imperative code. Of course, this is all done through “monads”, but they are relatively easy to use in Lean — even if you are not quite ready to buy into the FP propaganda. As a bonus point, this simulator is really fast.

Of course, if you have been reading the orange website, you should tell me that writing a simulator by hand is not the way to go. Instead, we should use property-based testing (PBT). Well, after the experiments with the simulator, this is exactly what I did with Plausible. See Property-based testing. To my disappointment, PBT took took me about six hours of work, delivering less impressive results in comparison to the simulator. I wasted so much time figuring out the generators and trying to define my own instances of Sampleable and Shrinkable, that I needed something Drinkeable after that. This was a bit unexpected for me. True, it was my first time using PBT in Lean — though not my first time with PBT in general, since I had prior experience using ScalaCheck in Scala for Apalache. While PBT might be the way to go in the long run, at the moment, I find Plausible a bit too hard to use. To be fair, ScalaCheck was not trivial to figure out, too. For some reason, the PBT frameworks assume that everybody knows Haskell and QuickCheck.

After having played with the simulator and the Plausible tests, I was quite satisfied with my spec. (My experiments with more advanced language features did not lead to serious improvements in the spec.) Obviously, the next question is whether we want to prove that the protocol satisfies its state invariants. At the high-level, it is clear how to proceed: Discover an inductive invariant, prove its inductiveness and show that the inductive invariant implies the state invariants of the protocol. I must have an inductive invariant for the TLA⁺ spec of two-phase commit lying somewhere. Even if the inductive invariant is lost, I am pretty sure that it is relatively easy to find it again by iteratively running Apalache. We have seen that this is doable with Ben-Or’s Byzantine consensus. Even TLC should work for the two-phase commit. At the lower levels, it is hard to predict how much effort it would take to write complete proofs with Lean tactics. This is definitely something to do in another sprint.

To sum it up, I quite liked the experience with Lean as a programming and specification language. I will definitely add it to my set of available tools. If someone wants to write executable protocol specs in Lean, say, instead of TLA⁺, Quint, or Python, I would be happy to help.

The rest of the blogpost goes into details. If you are interested, keep reading. If not, you are free to stop here.

Table of Contents

1. A brief intro to Two-phase commit and the TLA⁺ specification
2. Specification in Lean
   1. Data structures
   2. Functional specification in Lean
   3. System-level specification in Lean
3. Randomised simulator in Lean
   1. Why?
   2. What?
   3. How?
   4. Our simulator is really fast!
4. Property-based testing in Lean
5. Propositional specification in Lean

1. A brief intro to Two-phase commit and the TLA⁺ specification

I am giving only a very brief introduction in two-phase commit. It is quite a famous protocol. Gray & Lamport’04 (Sec. 3) explain the protocol nicely and by using the same vocabulary, as in the TLA⁺ specification. Moreover, Leslie Lamport explains the protocol idea and the specification in his Lecture 6 of the TLA⁺ Course on YouTube.

Here is the elevator pitch of two-phase commit: One transaction manager and $N$ resource managers have to agree on whether to commit or abort a transaction, e.g., a database transaction. If they all decide to commit the transaction, including the transaction manager, they all commit it. If at least one of them decides to abort, all of them should abort the transaction.

The sequence diagram below shows the happy path: Everyone decides to commit the transaction, and everybody does so:

The following diagram shows an unhappy scenario, when the transaction manager declines the transaction, even though all resources managers were ready to commit:

The TLA⁺ specification. Let’s have a quick look at the TLA⁺ specification of the two-phase commit. If you don’t know TLA⁺, you can skip this part. The whole specification is about 140 lines long. As usual, it starts with the declaration of the constants and state variables:

CONSTANT RM \* The set of resource managers

VARIABLES
  rmState,       \* $rmState[rm]$ is the state of resource manager RM.
  tmState,       \* The state of the transaction manager.
  tmPrepared,    \* The set of RMs from which the TM has received $"Prepared"$
                 \* messages.
  msgs           
    (***********************************************************************)
    (* In the protocol, processes communicate with one another by sending  *)
    (* messages.  Since we are specifying only safety, a process is not    *)
    (* required to receive a message, so there is no need to model message *)
    (* loss.  (There's no difference between a process not being able to   *)
    (* receive a message because the message was lost and a process simply *)
    (* ignoring the message.)  We therefore represent message passing with *)
    (* a variable $msgs$ whose value is the set of all messages that have  *)
    (* been sent.  Messages are never removed from $msgs$.  An action      *)
    (* that, in an implementation, would be enabled by the receipt of a    *)
    (* certain message is here enabled by the existence of that message in *)
    (* $msgs$.  (Receipt of the same message twice is therefore allowed;   *)
    (* but in this particular protocol, receiving a message for the second *)
    (* time has no effect.)                                                *)

We have one specification parameter RM that fixes the set of the resource managers. Further, we have four state variables: rmState, tmState, tmPrepared, and msgs.

Since the classic TLA⁺ is untyped, the specification comes with a special predicate that captures the possible values that the state variables could have:

Message ==
  (*************************************************************************)
  (* The set of all possible messages.  Messages of type $"Prepared"$ are  *)
  (* sent from the RM indicated by the message's $rm$ field to the TM\@.   *)
  (* Messages of type $"Commit"$ and $"Abort"$ are broadcast by the TM, to *)
  (* be received by all RMs.  The set $msgs$ contains just a single copy   *)
  (* of such a message.                                                    *)
  (*************************************************************************)
  [type : {"Prepared"}, rm : RM]  \cup  [type : {"Commit", "Abort"}]
   
TPTypeOK ==  
  (*************************************************************************)
  (* The type-correctness invariant                                        *)
  (*************************************************************************)
  /\ rmState \in [RM -> {"working", "prepared", "committed", "aborted"}]
  /\ tmState \in {"init", "committed", "aborted"}
  /\ tmPrepared \subseteq RM
  /\ msgs \subseteq Message

There is also a typed version for the Apalache model checker, as it needs types. If you are interested, go and check it.

As is typical, the specification contains a series of actions, which prescribe the behavior of the resource managers and the behavior of the transaction manager. For example, RMPrepare(rm) prescribe how a resource manager rm transitions from its state "working" to the state "prepared", while sending the message [ type |-> "Prepared", rm |-> rm ]:

RMPrepare(rm) == 
  (*************************************************************************)
  (* Resource manager $rm$ prepares.                                       *)
  (*************************************************************************)
  /\ rmState[rm] = "working"
  /\ rmState' = [rmState EXCEPT ![rm] = "prepared"]
  /\ msgs' = msgs \cup {[type |-> "Prepared", rm |-> rm]}
  /\ UNCHANGED <<tmState, tmPrepared>>

The transaction manager receives that message from rm in the action TMRcvPrepared(rm):

TMRcvPrepared(rm) ==
  (*************************************************************************)
  (* The TM receives a $"Prepared"$ message from resource manager $rm$.    *)
  (*************************************************************************)
  /\ tmState = "init"
  /\ [type |-> "Prepared", rm |-> rm] \in msgs
  /\ tmPrepared' = tmPrepared \cup {rm}
  /\ UNCHANGED <<rmState, tmState, msgs>>

The individual actions such as RMPrepare(rm) and TMRcvPrepared(rm) are put together with the next-state predicate:

TPNext ==
  \/ TMCommit \/ TMAbort
  \/ \E rm \in RM : 
       TMRcvPrepared(rm) \/ RMPrepare(rm) \/ RMChooseToAbort(rm)
         \/ RMRcvCommitMsg(rm) \/ RMRcvAbortMsg(rm)

2. Specification in Lean

You can find the whole specification in Lean on GitHub. I am presenting it in small pieces, to demonstrate the decisions that had to be made.

2.1. Data structures

Let’s start with the data structures. Since Lean is typed, we have to understand how to represent the parameter RM and the state variables rmState, tmState, tmPrepared, and msgs.

When it comes to the parameter RM, which was declared with CONSTANT RM in TLA⁺, we simply declare RM to be a type variable:

-- The abstract type of resource managers.
variable (RM : Type) [DecidableEq RM] [Hashable RM] [Repr RM]

Since Lean does not make default assumptions about types, we also specify what is required from the type RM:

• RM must have decidable equality, that is, we should be able to check rm1 = rm2 for two instances rm1: RM and rm2: RM. We have [DecidableEq RM].
• RM must be usable as a key in a hash table, that is, we have [Hashable RM].
• RM must be convertible to a string, that is, we have [Repr RM].

Now we have to understand how to deal with the types of the resource manager state and the transaction manager state, which are simply written in TLA⁺ as "init", "working", "prepared", etc. To this end, we declare two types RMState and TMState, which are similar to enum types, e.g., in Rust:

/-- A state of a resource manager. -/
inductive RMState where
    | Working
    | Prepared
    | Committed
    | Aborted
    deriving DecidableEq, Repr

/-- A state of the transaction manager. -/
inductive TMState where
    | Init
    | Committed
    | Aborted
    deriving DecidableEq, Repr

Further, we should understand the type of messages, which are written in TLA⁺ like [ type |-> "Prepared", rm |-> rm ]. Again, Lean’s inductive types fit in very nicely:

/-- A message that sent by either the transaction manager or a resource manager. -/
inductive Message where
    | Commit
    | Abort
    | Prepared(rm: RM)
    deriving DecidableEq, Repr

Finally, how shall we represent the state variables rmState, tmState, tmPrepared, and msgs? To this end, we simply declare the structure ProtocolState:

/-- A state of the Two-phase commit protocol. -/
structure ProtocolState where
    -- The set of the resource managers.
    -- It may differ from run to run, but remains constant during a run.
    all: Finset RM
    rmState: Std.HashMap RM RMState
    tmState: TMState
    tmPrepared: Finset RM
    msgs: Finset (Message RM)

As you can see, ProtocolState is the place where we had to make a number of decisions:

• The state carries the set all of all resource managers. The transaction manager needs this set to decided whether it had received the messages from all resource managers, see below. Perhaps, there is some Lean magic that would do that automatically.

• rmState is a hash map from the resource managers to RMState. Recall that it was simply defined as a function in TLA⁺.

• tmPrepared is a finite set of resource managers.

• msgs is a finite set of messages.

2.2. Functional specification in Lean

Now that we have chosen our data structures, we can specify the behavior of the two-phase commit. We do this in the module Functional.lean. For example, this is how we specify the behavior of the transaction manager on receiving the message Prepared rm, for a resource manager rm:

/-- The transaction manager receives a `Prepared` message from a resource manager `rm`. -/
def tmRcvPrepared (rm: RM) :=
    if s.tmState = TMState.Init && Message.Prepared rm ∈ s.msgs then some {
        s with tmPrepared := s.tmPrepared ∪ { rm },
    } else none

The above definition is very simple, but let’s go over it, just in case:

• We check, whether the input state s (more on that below) has the field tmState set to TMState.Init.

• Further, we check, whether the set of messages contains the message Message.Prepared rm. The operator $x ∈ S$ is set membership, which returns True, if and only if the set $S$ contains $x$ as its element.

• If the both of the above conditions hold true, then we produce a new state that is like the state s but its field tmPrepared is set to s.tmPrepared ∪ { rm }, that is, the set s.tmPrepared with the value rm added to it. Note that we return “some” value in this case, using the constructor some of the option type Option ProtocolState.

• If at least one of the conditions does not hold true, we return the value of type none, to indicate that the function parameter rm is not applicable in this case.

Where does the parameter s come from? It is not declared in tmRcvPrepared at all. This is an interesting feature of Lean. The parameter s is implicitly added as a parameter of tmRcvPrepared. To achieve this, we wrap the definitions in the section defs and declare s as a section variable there:

section defs
-- The state `s` is a state of the protocol, explicitly added to all the functions.
variable (s: ProtocolState RM)

def tmRcvPrepared (rm: RM) :=
  ...

def tmCommit :=
  ...

...
end defs

Here is another example of rmPrepare:

/-- Resource manager `rm` prepares. -/
def rmPrepare (rm: RM) :=
    if s.rmState.get? rm = RMState.Working then some {
        s with
        rmState := s.rmState.insert rm RMState.Prepared,
        msgs := s.msgs ∪ { Message.Prepared rm }
    } else none

Check the remaining definitions in Functional.lean.

If you’ve written TLA⁺ in the past, you probably expected the definition of tmRcvPrepared to look like this:

/-- The proposition version of `tmRcvPrepared`. -/
def tm_rcv_prepared (rm: RM): Prop :=
    s.tmState = TMState.Init
  ∧ Message.Prepared rm ∈ s.msgs
  ∧ s'.tmPrepared = s.tmPrepared ∪ { rm }

See the discussion about these two definitions in the Propositional specification.

2.3. System-level specification in Lean

Now we have the functional definitions of the resource managers and the transaction manager. How do we put these things together, to capture the behavior of the distributed system? We do this in the module System.lean.

Recall that the TLA⁺ specification does this via the predicates TPInit and TPNext:

TPInit ==   
  (*************************************************************************)
  (* The initial predicate.                                                *)
  (*************************************************************************)
  /\ rmState = [rm \in RM |-> "working"]
  /\ tmState = "init"
  /\ tmPrepared   = {}
  /\ msgs = {}

TPNext ==
  \/ TMCommit \/ TMAbort
  \/ \E rm \in RM : 
       TMRcvPrepared(rm) \/ RMPrepare(rm) \/ RMChooseToAbort(rm)
         \/ RMRcvCommitMsg(rm) \/ RMRcvAbortMsg(rm)

Initializing the system does not look hard. We do it like this:

/-- initialize the state of all resource managers to `Working` -/
def init_rm_state (all: List RM) :=
  all.foldl
    (fun m rm => m.insert rm RMState.Working)
    (Std.HashMap.emptyWithCapacity 0)

/-- The initial state of the protocol -/
def init (all: List RM): ProtocolState RM := {
    all := all.toFinset,
    rmState := init_rm_state all,
    tmState := TMState.Init,
    tmPrepared := ∅,
    msgs := ∅
}

The definition of init_rm_state definitely looks less elegant than the function constructor in TLA⁺, but it does its job: We iterate over the resource managers rm and add pairs (rm, RMState.Working) to the hash map.

What can we do about TPNext? This looks tricky: In every state, it should be possible to select one out of seven actions, as well as the parameter rm. This corresponds to control and data non-determinism, which poses a challenge to a functional definition, which is deterministic.

Luckily, the literature on distributed computing can help us with this problem. It is not unusual to define a system execution as a function of a schedule, or as a function of an adversary. Basically, we imagine that there is an external entity that tells us which steps to take.

Once we understand this trick, it becomes easy to use. We simply declare the type Action like this:

/--
Since Lean4 is not TLA+, it does not have a built-in syntax for actions.
Hence, we introduce the type that essentially turns control and data
non-deterministm into inputs. This trick is usually called a "schedule"
or "adversary" in the literature.
-/
inductive Action where
  | TMCommit
  | TMAbort
  | TMRcvPrepared(rm: RM)
  | RMPrepare(rm: RM)
  | RMChooseToAbort(rm: RM)
  | RMRcvCommitMsg(rm: RM)
  | RMRcvAbortMsg(rm: RM)
  deriving DecidableEq, Repr

Having defined the Action type, we define the function next of a protocol state s and an action a:

/-- The transition function (!) of the protocol. Since we provide `next` with
the action as an argument, `next` is a function, not a relation. -/
def next s a :=
  match a with
  | Action.TMCommit => tmCommit RM s
  | Action.TMAbort => tmAbort _ s
  | Action.TMRcvPrepared rm => tmRcvPrepared _ s rm
  | Action.RMPrepare rm => rmPrepare _ s rm
  | Action.RMChooseToAbort rm => rmChooseToAbort _ s rm
  | Action.RMRcvCommitMsg rm => rmRcvCommitMsg _ s rm
  | Action.RMRcvAbortMsg rm => rmRcvAbortMsg _ s rm

It looks simple and clean. I would argue that this definition of next is more elegant than the definition of TPNext in TLA⁺.

This is all we have in System.lean.

3. Randomised simulator in Lean

3.1. Why?

We have a formal specification in Lean. What is next? Obviously, our ultimate goal is to verify protocol correctness. In particular, we would like to verify consistency across the resource managers, for every intermediate state of the protocol:

-- the main invariant of the protocol, namely, that resource managers cannot disagree
def consistentInv (s: ProtocolState RM): Bool :=
  let existsAborted :=
    ∅ ≠ (Finset.filter (fun rm => s.rmState.get? rm = RMState.Aborted) s.all)
  let existsCommitted :=
    ∅ ≠ (Finset.filter (fun rm => s.rmState.get? rm = RMState.Committed) s.all)
  ¬existsAborted ∨ ¬existsCommitted

Basically, we want to show that it is impossible to find a protocol state, which has a resource manager in the Committed state and a resource manager in the Aborted state. Well, Lean offers us a lot of machinery for proving such properties. However, this machinery requires someone to write the proofs. Even though there is hope for large language models generating repetitive proofs, there is little hope for automatically proving properties of completely arbitrary algorithms.

Before going into a long-term effort of proving protocol properties, it is usually a good idea to try to falsify the protocol properties. This is what randomized simulations and model checkers can help us with. Even though such tools would not be able to give us a complete proof of correctness, they are quite useful in producing counterexamples. After all, if we want to prove a property $p$ and an automatic tool gives us a proof of $\neg p$, that is, a counterexample to $p$, we immediately know that the goal of proving $p$ is hopeless. Sometimes, model checkers can give us slightly better guarantees, see my recent blog post on value of model checking.

In contrast to TLA⁺, which has two model checkers TLC and Apalache, there is no model checker for Lean. Hence, the easiest approach to falsify a property in Lean is by using randomized techniques. In this section, we discuss randomized simulation. In the next section on Property-based testing, we discuss Plausible — a PBT framework for Lean.

3.2. What?

Our goal in this section is to quickly turn our Lean specification into a simulator that runs as follows:

$ lake exec RunTwophase4 100000 20 consistentInv 123

The above command runs the simulator RunTwophase4 that executes 100000 random runs, each having up to 20 steps. In each run, the simulator checks the state invariant consistentInv for each intermediate state. Since our simulator is randomized, we have to supply the seed. In our example, we give 123 as the seed. We could also supply the current time in seconds since Unix epoch by replacing 123 with $(date +%s).

Since the property consistentInv is not supposed to break, we also introduce “falsy invariants” that should actually break. By checking these invariants, we may convince ourselves that our specification is doing something useful. Here are the interesting invariants that we define right in the simulator:

-- This is a false invariant to demonstrate that TM can abort.
def noAbortEx (s: ProtocolState RM): Bool :=
  s.tmState ≠ TMState.Aborted

-- This is a false invariant to demonstrate that TM can commit.
def noCommitEx (s: ProtocolState RM): Bool :=
  s.tmState ≠ TMState.Committed

-- This is a false invariant for a trickier property:
-- Even though all resource managers are prepared, the TM can still abort.
def noAbortOnAllPreparedEx (s: ProtocolState RM): Bool :=
  s.tmState = TMState.Aborted → s.tmPrepared ≠ s.all

For example, this is how we find a schedule that makes the transaction manager commit:

$ /usr/bin/time -h lake exec RunTwophase4 1000000 20 noCommitEx 1745505753
❌ Counterexample found after 403736 trials
#0: Action.RMPrepare (RM.RM3)
#1: Action.RMPrepare (RM.RM4)
#2: Action.TMRcvPrepared (RM.RM3)
#3: Action.RMPrepare (RM.RM2)
#4: Action.TMRcvPrepared (RM.RM3)
#5: Action.TMRcvPrepared (RM.RM2)
#6: Action.TMRcvPrepared (RM.RM3)
#7: Action.RMPrepare (RM.RM1)
#8: Action.TMRcvPrepared (RM.RM4)
#9: Action.TMRcvPrepared (RM.RM1)
#10: Action.TMCommit
	2.06s real		1.88s user		0.53s sys

The simulator also finds an example, in which all resource managers were ready to commit a transaction, but the transaction manager decided to abort it:

$ /usr/bin/time -h lake exec RunTwophase4 1000000 20 noAbortOnAllPreparedEx 1745505754
❌ Counterexample found after 97860 trials
#0: Action.RMPrepare (RM.RM3)
#1: Action.RMPrepare (RM.RM4)
#2: Action.RMPrepare (RM.RM2)
#3: Action.TMRcvPrepared (RM.RM2)
#4: Action.TMRcvPrepared (RM.RM4)
#5: Action.TMRcvPrepared (RM.RM3)
#6: Action.RMPrepare (RM.RM1)
#7: Action.TMRcvPrepared (RM.RM1)
#8: Action.TMAbort
	1.21s real		0.67s user		0.32s sys

Actually, the diagrams in the Brief introduction are generated from the examples that are found by our simulator. As you can see the counterexamples are found in a few seconds, even though the simulator has to enumerate hundreds of thousands of schedules.

3.3. How?

You can find the whole simulator in Twophase4_run.lean. The core of our simulator is the following loop, written right in the entry point:

def main (args: List String): IO UInt32 := do
  -- parse the arguments
  -- ...
  -- run a loop of `maxSamples`
  for trial in [0:maxSamples] do
    let mut state := /- initial_state (...) -/ 
    -- run a loop of `maxSteps` steps
    for _ in [0:maxSteps] do
      -- ...
      let action := mkAction -- (random parameters)
      match next state action with
      | some new_state =>
        --- add to trace
      | none => -- do nothing
      -- check our invariant
      if ¬inv state then
        -- print the trace
        return 1

Before going into the details of the simulator loop, we have to introduce a bit of boilerplate code. First, we have to define the type of the resource managers RM. We do it by simply enumerating four constructors RM1, …, RM4:

-- An instance of four resource managers.
inductive RM
  | RM1
  | RM2
  | RM3
  | RM4
  deriving Repr, DecidableEq, Hashable, Inhabited

Second, since our simulator is randomized, we have to understand how to randomly produce schedules. To this end, we write the definition mkAction:

def mkAction (action_no: Nat) (rm_no: Nat): @Action RM :=
  let rm := match rm_no with
    | 0 => RM.RM1
    | 1 => RM.RM2
    | 2 => RM.RM3
    | _ => RM.RM4
  match action_no with
  | 0 => Action.TMCommit
  | 1 => Action.TMAbort
  | 2 => Action.TMRcvPrepared rm
  | 3 => Action.RMPrepare rm
  | 4 => Action.RMChooseToAbort rm
  | 5 => Action.RMRcvCommitMsg rm
  | _ => Action.RMRcvAbortMsg rm

Having defined mkAction, we simply generate two random numbers via Lean’s randNat that we supply to mkAction. Now we are ready to see the full implementation of the simulator loop:

  -- run a loop of `maxSamples`
  let mut rng := mkStdGen seed
  for trial in [0:maxSamples] do
    let mut state := init [ RM.RM1, RM.RM2, RM.RM3, RM.RM4 ]
    -- run a loop of `maxSteps` steps
    let mut trace: List (@Action RM) := []
    for _ in [0:maxSteps] do
      let (action_no, next_rng) := randNat rng 0 6
      let (rm_no, next_rng) := randNat next_rng 0 3
      rng := next_rng
      let action := mkAction action_no rm_no
      match next state action with
      | some new_state =>
        state := new_state
        trace := action::trace
      | none => pure ()

      -- check our invariant
      if ¬inv state then
        IO.println s!"❌ Counterexample found after {trial} trials"
        for (a, i) in trace.reverse.zipIdx 0 do
          IO.println s!"#{i}: {repr a}"
        return 1

As you can see, the simulator is extremely simple! There is really not much to explain here. The large body of work is done by Lean itself. Our job was to simply produce schedules, that is, lists of Action values.

3.4. Our simulator is really fast!

We have seen that in the cases when the invariant was violated — e.g., when checking noAbortEx, noCommitEx, or noAbortOnAllPreparedEx — our simulator was finding examples in a few seconds. However, we do not know the number of samples that is sufficiently large to convince ourselves that the invariant is not violated. Hence, our simulator should be fast enough to crunch plenty of runs.

Let us run the simulator on 1 million runs up to 30 steps each:

$ /usr/bin/time -h -l lake exec RunTwophase4 1000000 30 consistentInv 1745505754
	10.93s real		10.34s user		0.39s sys
           422985728  maximum resident set size
...

Nice! The simulator came back in 11 seconds and it consumed up to 434 MB of memory. We can also run it for a larger number of samples. As we can see, the simulator is quite robust. As expected, the running times are growing linearly, and there are no visible memory leaks:

Is it actually fast? I think it is really damn fast! What can we use as the baseline? We can compare it with the Quint simulator. All we have to do is to get the Quint spec of two-phase commit, which is also a translation of the same TLA⁺ spec that we used as the original source. To make the comparison faithful, we have to add the instance of four resource managers like this:

module two_phase_commit_4 {
  import two_phase_commit(resourceManagers = Set("rm1", "rm2", "rm3", "rm4")).*
}

Now we can run the Quint simulator like this:

$ /usr/bin/time -l -h quint run --max-samples 1000000 --max-steps=30 \
  --invariant=consistencyInv \
  examples/classic/distributed/TwoPhaseCommit/two_phase_commit.qnt --main two_phase_commit_4
...
	27m57.62s real		28m20.04s user		21.03s sys
           217235456  maximum resident set size

This is quite interesting. Although the Quint simulator uses half as much memory, it is 168 times slower when running one million samples. I believe there are several reasons for this. First, Quint is essentially a JavaScript interpreter, while our Lean simulator is transpiled to C and compiled with full optimization for the target architecture. Second, the Quint simulator makes a slightly greater effort to find satisfying assignments randomly. Nevertheless, the performance gap is substantial.

Quint also has a new backend in Rust. Let’s try it, too:

/usr/bin/time -l -h quint run --max-samples 1000000 --max-steps=30 \
  --invariant=consistencyInv examples/classic/distributed/TwoPhaseCommit/two_phase_commit.qnt \
  --main two_phase_commit_4 --backend=rust
...
	9m27.57s real		9m24.46s user		1.53s sys
           120913920  maximum resident set size

Nice! The rust backend is three times faster and uses half as much memory as the JavaScript backend. Still, 10 minutes is nowhere close to the 10 seconds by our simulator in Lean. Apparently, it’s really hard to compete with a transpiler to C, even if the interpreter is written in :crab:

4. Property-based testing in Lean

Instead of writing our own simulator — even though it happened to be easy in the case of two-phase commit — we employ Plausible in this section. Plausible also applies randomization to check properties, but it does it in a slightly different way. Basically, it generates data structures of a given type at random up to some predefined bound and checks the property. What is interesting about property-based testing is that, once the framework finds a property violation, it tries to minimize the failing input.

You can find the whole PBT setup in Twophase4_pbt.lean and Twophase4_pbt_failing.lean. If you are using only the standard types, using Plausible looks pretty simple: Just write a quantified property, and Plausible will check it. In our case, we have two user-defined data types, namely, RM and Action. This is not unusual. In my experience, writing generators is the hardest and the most time-consuming part of using PBT. Here is how we are defining the custom type for RM and its generator:

-- An instance of three resource managers.
inductive RM
  | RM1
  | RM2
  | RM3
  | RM4
  deriving Repr, DecidableEq, Hashable, Inhabited

-- define a generator for RM
instance RM.shrinkable: Shrinkable RM where
  shrink := fun (_: RM) => []

def genRm: Gen RM := (Gen.elements [ RM.RM1, RM.RM2, RM.RM3, RM.RM4 ] (by decide))

instance : SampleableExt RM :=
  SampleableExt.mkSelfContained genRm

Basically, we define the generator genRm that randomly selects one of the four values RM1, …, RM4. Since we do not know how to further minimize a value of type RM, we simply define the instance of Shrinkable as returning the empty list.

Similar to that, we define a generator for Action RM. This time, we delegate the choice of RM values to genRm:

-- define a generator for Action RM
instance Action.shrinkable: Shrinkable (@Action RM) where
  shrink := fun (_: @Action RM) => []

def genAction: Gen (@Action RM) :=
  Gen.oneOf #[
    Gen.elements [ Action.TMCommit, Action.TMAbort] (by decide),
    (do return (Action.TMRcvPrepared (← genRm))),
    (do return (Action.RMPrepare (← genRm))),
    (do return (Action.RMChooseToAbort (← genRm))),
    (do return (Action.RMRcvCommitMsg (← genRm))),
    (do return (Action.RMRcvAbortMsg (← genRm)))
  ]
  (by decide)

instance : SampleableExt (@Action RM) :=
  SampleableExt.mkSelfContained genAction

Once we have defined the generator for actions, it is very easy to define a generator of schedules, by simply applying a standard generator in Plausible:

def genSchedule: Gen (List (@Action RM)) :=
  Gen.listOf genAction

The good news is that defining the generators was the hardest part. At least, it was the hardest part for me. Now, since Lean does not have any idea about how our specification should be executed, we define two simple functions applySchedule and checkInvariant:

-- given a concrete schedule, inductively apply the schedule and check the invariant
def applySchedule (s: ProtocolState RM) (schedule: List (@Action RM))
    (inv: ProtocolState RM → Bool): ProtocolState RM :=
  schedule.foldl (fun s a => if inv s then (next s a).getD s else s) s

-- apply a schedule to the initial state
def checkInvariant (schedule: List (@Action RM)) (inv: ProtocolState RM → Bool): Bool :=
  let init_s := init [ RM.RM1, RM.RM2, RM.RM3, RM.RM4 ]
  let last_s := applySchedule init_s schedule inv
  inv last_s

Now we are ready to write our first property to be checked by Plausible. This is how we check, whether it’s possible to reach a state where the transaction manager goes into TMState.Aborted:

-- noAbortEx
example schedule:
    let inv := fun (s: ProtocolState RM) =>
      s.tmState ≠ TMState.Aborted
    checkInvariant schedule inv
  := by plausible (config := { numInst := 3000, maxSize := 100 })

In the above definition we say that whatever the schedule we generate, checkInvariant does not return false. We configure Plausible to produce 3000 instances with the data structures bounded to 100. Comparing it to our simulator, this would mean 3000 random schedules of up to 100 steps.

In this case, Plausible finds a counterexample and minimizes it:

===================
Found a counter-example!
schedule := [Action.TMAbort]
issue: false does not hold
(1 shrinks)
-------------------

Similar to noAbortEx, we define two other properties:

-- noCommitEx
example schedule:
    let inv := fun (s: ProtocolState RM) =>
      s.tmState ≠ TMState.Committed
    checkInvariant schedule inv
  := by plausible (config := { numInst := 3000, maxSize := 100 })

-- noAbortOnAllPreparedEx
example schedule:
    let inv := fun (s: ProtocolState RM) =>
      s.tmState = TMState.Aborted → s.tmPrepared ≠ { RM.RM1, RM.RM2, RM.RM3, RM.RM4 }
    checkInvariant schedule inv
  := by plausible (config := { numInst := 3000, maxSize := 100 })

If we are lucky, both properties are violated, because they do not hold true. In my case, Plausible does not find a counterexample to noCommitEx, and it finds a counterexample to noAbortOnAllPreparedEx:

===================
Found a counter-example!
schedule := [Action.RMPrepare (RM.RM3),
 Action.RMPrepare (RM.RM4),
 Action.TMRcvPrepared (RM.RM3),
 Action.RMPrepare (RM.RM1),
 Action.RMPrepare (RM.RM2),
 Action.TMRcvPrepared (RM.RM1),
 Action.TMRcvPrepared (RM.RM4),
 Action.TMRcvPrepared (RM.RM2),
 Action.TMAbort]
issue: false does not hold
(23 shrinks)
-------------------

Finally, we define the property consistentInv, which should not be violated:

-- consistentInv
#eval Testable.check <| ∀ (schedule: List (@Action RM)),
    let inv := fun (s: ProtocolState RM) =>
      let existsAborted :=
        ∅ ≠ (Finset.filter (fun rm => s.rmState.get? rm = RMState.Aborted) s.all)
      let existsCommitted :=
        ∅ ≠ (Finset.filter (fun rm => s.rmState.get? rm = RMState.Committed) s.all)
      ¬existsAborted ∨ ¬existsCommitted
    checkInvariant schedule inv

So PBT seems to work and looks fine! The only thing that I could not figure out is how to increase the number of samples to large values, e.g., millions of samples, as we did in the simulator. Unfortunately, Plausible produces a stack overflow even on 10K examples, even when run via lake test. This needs more debugging.

5. Propositional specification in Lean

Finally, let’s discuss, whether our specification style is the most natural one when translating TLA⁺ specifications. We used the functional approach to define individual behavior of resource managers and transition managers in Functional.lean and System.lean. When we did a similar trick in Quint, some people were telling me that it’s not TLA⁺. I think it is easier to see what they meant by this in Lean!

Instead of defining functions, we can write the specification as propositions. As before, we define two variables: s for the current state, and s' for the next state.

-- The state `s` is the "current" state of the protocol.
variable (s: ProtocolState RM)
-- The state `s'` is the "next" state of the protocol.
variable (s': ProtocolState RM)

Now we can translate TLA⁺’s action TMRcvPrepared as a proposition:

/-- The proposition version of `tmRcvPrepared`. -/
def tm_rcv_prepared (rm: RM): Prop :=
    s.tmState = TMState.Init
  ∧ Message.Prepared rm ∈ s.msgs
  ∧ s'.tmPrepared = s.tmPrepared ∪ { rm }

This looks very similar to TLA⁺! Moreover, we should be able to use some definitions that do not have an executable implementation. For instance, we can use quantifiers over sets, instead of filtering sets. I also suspect that it would be easier to write proofs over such propositions rather than over functional code.

Can we find a connection between the functional definitions and the propositions? Well, in Lean we can just write simple theorems like tm_rcv_prepared_correct:

theorem tm_rcv_prepared_correct (rm: RM):
    tm_rcv_prepared s s' rm ↔
      tmRcvPrepared RM s rm = some s' := by sorry

I do not know how to write a Lean proof of the above theorem yet. In principle, it should be quite easy to write for someone proficient with Lean proofs. Indeed, the functional definition and the proposition are very close in their syntactic structure.

I find this connection especially appealing. This would let us stop arguing about which level is the right one. We could simply write both functional and propositional definitions and connect them via (hopefully!) simple proofs.