Handling reconfigurations

[kim]

I started working with reconfigurations, and this makes me scratch my head:

Calling reconfigure() will propose a StopSign entry to be appended. If it gets decided, the log is sealed and prevented from being further appended.

As the example shows, this implies creating a new OmniPaxos instance with an empty storage.
Indeed, re-using the old (persistent) storage won't work, as it still contains the stop sign.

I'm wondering, however, how this is supposed to work, as all replicas will restart with an empty storage at the same index.

Am I supposed to initialize the new storage from the old one, such that the decided state is carried over?

[haraldng]

We assume the application state machine to be decoupled from the omnipaxos log. So after the reconfiguration, users should just use the new instance to decide new entries and update the same state machine.

But I guess you'd like to keep appending to the log but using different nodes after the reconfiguration? That is not supported right now, but I'd be happy to help with some guidance if you'd be interested in contributing?

Otherwise, using the current API, one would have to build a log abstraction on top of the different omnipaxos instances. E.g., if the stopsign is decided at entry 100, then the first entry of the new configuration corresponds to entry 101.

[kim]

Yes, I'd like to keep appending to the application log, and the application can ensure that a quorum of nodes of the old configuration remains in the new configuration.

Our application log's sequence number is not the same as the omnipaxos index (an entry can contain multiple transactions), and it is conceivable that we just apply an offset after a stopsign and the reset of the omnipaxos index that follows.

For example, the stopsign is decided at 100, and the next post-reconfiguration index is 1, so we continue counting 100 + idx at the application level.

However, this requires that we store the stopsign at least until entry 101 is decided.

This seems to suggest that an abstraction would essentially entail a sequence of omnipaxos logs, of which we retain at least the two most recent entries. Is this what you had in mind?

[haraldng]

Yes, a sequence of omnipaxos logs is what the application/user needs to maintain. The important part for safety is that you only read the entries of the new configuration when the old configuration has really decided the stopsign.

For performance reasons (avoid down-time), it is possible to even start a new configuration before the old one has stopped,, but just make sure to not expose it to the application yet.

[kim]

Circling back to this, what makes me stop is that it's not obvious how to solve this without joint consensus as described in the Raft paper: before we can stop C_i, we need proof that C_i+1 is a continuation of the log, but to provide this proof we need the stop sign.

If we need to literally stop C_i before C_i+1 can commence, then that's operationally downtime (for which, if we accept that, the stop sign mechanism is not needed).

What might I be missing?