Risk of data loss due to Patroni configuration defaults

[nsakkos]

During a recent outage, we observed data loss when a former primary with divergent history rejoined the cluster. I believe this is linked to the default Patroni configuration, which can remove and recreate a data directory when timelines have diverged.

Charm defaults for patroni configuration:

use_pg_rewind: true
remove_data_directory_on_rewind_failure: true
remove_data_directory_on_diverged_timelines: true

From the patroni official documentation:

remove_data_directory_on_diverged_timelines: Patroni will remove the PostgreSQL data directory and recreate the replica if it notices that timelines are diverging and the former primary can not start streaming from the new primary.

Steps to reproduce

1. Deploy a 3-unit Postgres cluster using the charm
2. Manipulate the quorum so that two sub-clusters can both elect a leader
3. Write data to the isolated primary
4. Force a leadership change on the other primary (the one on the 2-node cluster) to increase the timeline
5. Restore connectivity and/or restart patroni on the isolated primary, so that it rejoins the cluster
6. Observe whether the isolated primary wipes its data directory when joining the cluster

Expected behavior

• A diverged primary should not automatically discard its data without human intervention.
• Documentation should clearly highlight the tradeoffs of the current defaults (risk of data loss vs automatic cluster recovery)

Actual behavior

In the observed incident, a node broke away from the cluster, but continued to receive updates from the consuming application. When it was manually restarted and attempted to rejoin, its timeline was behind the cluster's leader. As a result, it joined as a replica and dropped its local data to sync to the leader's timeline, leading to data loss.

Versions

The data-loss incident occurred on the following juju/charm versions, but the patroni default configuration is the same on the newer versions of the charm.

Operating system:

Juju CLI: 3.6.8-ubuntu-amd64

Juju agent: 3.1.8

Charm revision: 331

LXD:

Log output

Juju debug log:

I apologize for the internal-only links

https://pastebin.canonical.com/p/Cr92FPB8wZ/
https://pastebin.canonical.com/p/TNS6BYK4dd/

Additional context

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/DPE-8337.

This message was autogenerated

[delgod]

Agreed — let’s disable these three options to minimise the risk of data loss.
Apologies for the oversight; I enabled them in error.

[7annaba3l]

Hi @ALL,
Let me start by asking by clarifying a few points:

1. "we observed data loss when a former primary with divergent history rejoined the cluster." If we talk about former primary and divergent history, it means automatically that there is a probability of data loss. There are 2 cases:

• All replicas are synchronous so there can not be a committed transaction before acknowledged by all replicas => There can not be a data loss following a switch of primary.
• Some replicas are not synchronous. There can be a data loss by design even when the previous primary was correctly isolated

2. In the reproduction steps, I see
   step 3 "Write data to the isolated primary" => This should not be possible unless there is a bug. This is what we need to investigate.

Therefore I think that we are not investigating/addressing the root cause here.
In my opinion, we should have:
use_pg_rewind: true
remove_data_directory_on_rewind_failure: false
remove_data_directory_on_diverged_timelines: false

This is specially true for big databases where a full copy can stress the system further probably leading to more complications during recovery. I think it makes sense to manually supervise the deletion of the data directory when rewind did fail as it might hide bigger issues.

[taurus-forever]

@7annaba3l your understanding is absolutely correct.
This is exactly the path I am exploring: #1173

Note: the 14/stable used in bugreport has old defaults: primary/0 + sync_standby/1 + replica/2.
All sync_standby are in 14/edge as for now only.

Anyway, in case of primary/0 outage, replica/2 should sync with new primary/1 and became a new sync_standby/2...
while into the earthquake environment we might have not enough time for replica/2 became sync_standby/2 => might have no data from primary/0 on election as primary/2.

Searching STR here... re: Manipulate the quorum so that two sub-clusters can both elect a leader .
@nsakkos can you please share some steps-to-reproduce here? Ideas?
We are trying to use #712 , not really reproducible. Tnx in advance!