Designing PostgreSQL High Availability with Patroni, etcd, and a Stable Endpoint
Date Published
Separate consensus from database replication
PostgreSQL streaming replication moves database changes. It does not decide which node is allowed to accept writes. Patroni manages PostgreSQL roles and failover, while etcd stores the distributed consensus state. Keep this boundary clear: a healthy replica is not automatically a safe leader, and an available etcd cluster does not prove that database replication is current.
Use three voting members
A three-member etcd cluster tolerates one member failure while preserving quorum. Place members on independent failure domains when possible and monitor leader changes, peer latency, disk latency, and free space. Never treat etcd as disposable configuration; back it up and secure both client and peer communication. Loss of quorum should pause automated leadership changes rather than create two writable primaries.
Give applications one stable route
Applications should not contain a hand-maintained list of database nodes. A floating VIP can follow the Patroni leader and provides a simple read-write endpoint. HAProxy can add separate read-write and read-only ports by checking Patroni REST endpoints. Use a VIP alone for the simplest reliable path; add HAProxy only when read scaling or endpoint policy justifies another component.
Plan failover behavior, not just failover mechanics
Choose maximum replication lag for promotion, synchronous replication policy, failure timeouts, and fencing behavior deliberately. Short timeouts can turn a network pause into an unnecessary failover. Strict synchronous replication can protect data but may stop writes when no synchronous replica is available. The correct choice follows the business recovery objective.
Test the ugly paths
Exercise leader shutdown, network partition, etcd member loss, replica lag, full disks, expired certificates, and the return of a former leader. Confirm client reconnection behavior and measure write unavailability. After every test, verify that exactly one primary exists, replicas are streaming on the same timeline, and the stable endpoint resolves to the intended leader.