Hot-failover with load-balanced spare

[jevolk]

We have an undocumented configuration option rocksdb_secondary which allows for multiple instances of Tuwunel in a single-writer multiple-reader topology. The synchronization is undeveloped, so the readers will be stuck at a snapshot from when they were started. This filesystem-level clustering, though primitive, is a highly economical path to provide:

• Failover for uninterrupted service on server failure or upgrade.
• Load-shedding to exceed the vertical limitation of a provider (i.e. when 64 or 128 cores is not enough). Requests to Matrix protocol GET/read-only endpoints can be balanced over the cluster of secondaries.

[alythemonk]

I am sure that rocksdb was chosen for a good reason. Would it be a consideration to leverage a proper DB like PGSQL and allow the code to potentially become “stateless” and you could simply rack up VMs or containers against a proper RDBMS?

Or would this be a big lift? Or is it just not a good idea overallv

[jevolk]

It's been exhaustively considered, but it's just not a good fit for the existing codebase and likely inferior for Matrix as an application. Synapse is some of the best evidence we have against it, and dendrite only confirmed that in another language.

[iwalkalone69]

I am really insterested in this feature. Can you post an example on how rocksdb_secondary works? Thanks.

[jevolk]

This is just a quick example of what the option does, although this is all it does right now 😉

The top screen has a normally running primary server as usual. In the bottom screen, the first command shows what happens when running another instance of the same server; it's prevented by a filesystem lock as expected. The second command then uses rocksdb secondary mode. There is no error anymore, though it's technically running in a read-only mode. The next step would be to implement the synchronization signals that keep the secondary up to date with the primary.

[meovary150]

Hi @jevolk and contributors,

Thanks for the detailed explanation on rocksdb_secondary mode — it's very insightful.

We’re currently working on a commercial-grade Matrix-based service using Tuwunel, and we’re exploring options to achieve high availability and load-balancing without fully rewriting the storage layer.

Do you have any recommended architecture patterns or best practices for using Tuwunel + RocksDB in production where we want:

Read-scalable or standby nodes that don't conflict with the primary

Fault-tolerant failover mechanisms that avoid data corruption

Possibly proxying or gateway-level balancing across multiple read replicas?

Any suggestions or examples you could share would be extremely helpful. Also, is there any ongoing or planned effort toward a more robust synchronization model between secondaries and the primary?

Thanks again for your work — this project is impressive!

[jevolk]

It's at least a few months out on the roadmap. The mechanism to synchronize the secondaries with the primary is crude, but workable. I have just added an admin debug command !admin debug resync-database which should be linked to this issue soon and make it into the next release. In theory one could set the config variable admin_signal_execute = ["debug resync-database"] and then send the server SIGUSR2 to update from the primary. This is just a small step and not close to being a production solution. Facebook also has a slightly less crude alternative called Follower rather than Secondary to do this automatically but it's still experimental. I've been considering a custom inotify(7) solution as well.

Nevertheless the real challenge here is ensuring the Tuwunel API surface is truly capable of read-only service, otherwise there will be panics. That's just the first major step. The transition from Secondary to Primary requires closing and reopening the database, which is basically no different from restarting the server right now.

This is several months out. And as an aside, we're slightly more inclined to address performance issues for vertically scaling a single-server rather than throw in with this mode of horizontal scaling. We haven't come close to reaching the limit of a single machine or found it cost-prohibitive to continue trying.

[meovary150]

Hi @jevolk , thanks again for the detailed response.

We’re evaluating long-term paths for scaling, and I’d like to ask two follow-up questions:

How feasible would it be to replace RocksDB with a distributed KV engine like TiKV?
We understand this would be a major lift, but given the modular nature of Tuwunel, is this something you’ve ever considered or prototyped internally?

Has the Tuwunel team ever benchmarked the maximum throughput achievable on a single instance?
We'd be very interested in any figures you might have regarding sustained room traffic, concurrent clients, or messages per second — especially under optimized conditions (e.g. NVMe SSD, tuned compaction settings, etc.).

This information would help us better assess our scaling roadmap and infrastructure planning. Appreciate your time and incredible work on the project!

[jevolk]

How feasible would it be to replace RocksDB with a distributed KV engine like TiKV?

The loads of the Matrix application require a high volume of low-latency random-access point-queries. It also requires fast conditional iterations. Many iterations also launch additional random-access queries from each result. TiKV doesn't provide this well. TiKV requires foreknowledge of a range of keys which it then sends from its server to its client in a batch; the client (our server) cannot start interpreting the first result until the last result has been received from the TiKV server. If the iteration only requires the first result to satisfy a condition, the remainder become "tail latency" when they're discarded.

One suggestion to improve TiKV would be refactoring their current implementations for asynchronous streaming results. For example, rather than presenting the result of a range as Vec<(Key, Val)> it would instead provide impl Stream<Item = (Key, Val)>, where the first item is available immediately, and the last item is subject to backpressure on the stream and may never even be read from the server's storage if the stream is dropped mid-way.

Overall the disadvantages end up outweighing the advantages and it becomes a bad fit for the application. The executive summary is that the modern cloud hosting environment can scale monolithic storage rather generously, and failover can occur at the infrastructure level to satisfy the needs of very large deployments (50-100k registered users). It's more effective for us right now to optimize the wasted storage space in the current db schema first, afterward considering how to shard the server itself at the application-logic level to eliminate the obvious point of failure. Matrix is mostly compressible text, so we're not talking about serious amounts of data here when managed properly.

Has the Tuwunel team ever benchmarked the maximum throughput achievable on a single instance? We'd be very interested in any figures you might have regarding sustained room traffic, concurrent clients, or messages per second — especially under optimized conditions (e.g. NVMe SSD, tuned compaction settings, etc.).

A benchmarking suite will eventually be part of the CI infrastructure here but it hasn't been a priority in the near-term. We're still working out bugs and scalability hazards from the original fork that don't require empirical analysis to address, but that point will be reached.