300 GB deleted in seconds: what GitLab's database incident still teaches us

A command ran in the wrong terminal window. Within seconds, approximately 300 GB had disappeared from GitLab.com's primary PostgreSQL server.

That sentence makes the incident sound like one engineer's mistake. It was not. The command started the disaster, but broken backups, missing monitoring, fragile replication procedures, and an untested recovery plan turned it into an 18-hour outage.

The incident happened on January 31, 2017. GitLab documented the failure publicly, published a detailed postmortem, and livestreamed the recovery.[1][2] A recent Instagram reel brought the story back into circulation.[6]

The lesson is still current: a backup job that has never completed a tested restore is only an assumption.

What happened

GitLab.com was running a PostgreSQL primary and a secondary replica. A spike in database load caused the replica to fall behind. The primary removed WAL segments before the secondary could consume them, and GitLab had no WAL archive from which to retrieve the missing history.[2]

The team began rebuilding the secondary with `pg_basebackup`. The tool appeared to wait without useful output, and the recovery procedure was neither automated nor documented well enough. During repeated troubleshooting, an engineer intended to clear the secondary server's PostgreSQL data directory but ran the destructive command on the primary instead.[2]

The engineer stopped the command after one or two seconds. It was already too late. GitLab's initial incident report said that only about 4.5 GB remained from roughly 300 GB of production data.[1]

Why the safety net failed

The database deletion was only the first failure. Recovery exposed several more:

1. The scheduled `pg_dump` process used PostgreSQL 9.2 tools against a PostgreSQL 9.6 database, so the backup failed.

2. Failure notifications were sent by email, but the receiver rejected them because the messages were not DMARC-signed.

3. The S3 backup bucket was empty.

4. Azure disk snapshots were enabled for other servers, but not for the database servers.

5. Nobody owned regular restore testing, so the team discovered the broken recovery path during the outage.[2]

GitLab eventually restored a staging copy created about six hours earlier. Moving that data from slow storage back to production took around 18 hours. The company estimated that it lost approximately 5,000 projects, 5,000 comments, and 700 user accounts created during the missing window. Git repositories and wikis were stored separately and were not deleted.[2]

What should have been done

A safer design would have reduced both data loss and recovery time:

1. Continuously archive WAL outside the database hosts. This provides a recovery path between full backups and allows point-in-time recovery.

2. Keep backups in a separate failure domain. The database administrator should not be able to destroy production and its backup repository with the same credentials or terminal session.

3. Test restores automatically. A scheduled job should restore a backup into an isolated environment, start PostgreSQL, and confirm that the recovered database is usable.

4. Monitor recovery readiness, not job execution. Alert on the age of the latest verified backup, WAL gaps, failed integrity checks, and missed restore drills.

5. Put guardrails around destructive production work. Use unmistakable production prompts, separate access profiles, peer approval for destructive commands, and tested runbooks that remove guesswork during an incident.

Replication remains useful for availability, but it is not an independent backup. A replica can be unavailable, corrupted, or deliberately erased during maintenance, exactly as happened here.

Where CYBERTEC pg_hardstorage fits

CYBERTEC's `pg_hardstorage` is an open-source PostgreSQL backup and recovery tool for PostgreSQL 15 through 18. It runs as a static binary on a backup host, connects through PostgreSQL's replication protocol, takes base backups, and continuously streams WAL. Nothing needs to be installed on the database host.[3][4]

That architecture addresses several weaknesses exposed by the GitLab incident:

• Continuous WAL streaming supports point-in-time recovery to a timestamp, LSN, or named restore point.

• Backups can live on a separate host or supported remote storage backend.

• Each content-addressed backup is self-contained, so removing an older backup does not break a newer one.

• `pg_hardstorage verify` checks backup integrity, while an actual test restore can prove that PostgreSQL starts from the recovered data.

• Preflight checks inspect replication settings before protection begins, and metrics can feed Prometheus and Grafana monitoring.[3][4]

`pg_hardstorage` would not stop an engineer from deleting the wrong directory. No backup product can make every destructive command impossible. If a comparable incident happened today on a supported PostgreSQL version, a correctly isolated and continuously monitored `pg_hardstorage` repository could preserve the WAL and base backup required to restore to a point immediately before the deletion.[4]

The word "correctly" matters. The repository still needs separate credentials, restricted deletion rights, retention controls, monitoring, and repeated restore drills. Installing a tool is not the same as operating a recovery system.

How PGEE strengthens the wider PostgreSQL platform

`pg_hardstorage` addresses backup and recovery. CYBERTEC PostgreSQL Enterprise Edition, or PGEE, strengthens the database platform around it. Its security features include Transparent Data Encryption, data masking and obfuscation, encrypted PL/pgSQL code, and time-related permissions. PGEE also adds tuned defaults and Intel hardware acceleration, plus operational tools such as PGWatch, active execution-plan tracking, automatic incident inspection, `pg_qualstats` index advice, built-in scheduling, and automatic storage reorganisation.[7]

These features solve different parts of the risk. TDE protects database files at rest, masking reduces exposure outside production, and monitoring helps teams detect trouble sooner. Patroni high availability can keep service running through a node failure. None of them replaces an independent backup with tested point-in-time recovery.

PGEE support plans

Recovery tooling matters, but a serious incident also needs experienced people who can respond within a defined time. Worlber offers three PGEE support levels, with software licensing, updates, and support included according to the selected tier. Infrastructure costs remain separate.[7][8]

• Standard: PGEE: Optional; Availability: 8x5; SLA response: 1 hour; Main support features: PostgreSQL bug fixes, support for seven historical versions, and PostgreSQL in Docker

• Professional: PGEE: Optional; Availability: 8x5; SLA response: 45 minutes; Main support features: Standard coverage plus ticketing and emergency restore support

• Enterprise: PGEE: Included; Availability: 24x7; SLA response: 30 minutes; Main support features: Unlimited version support and tickets, emergency restore support, remote login, slow-query tuning, and a dedicated account manager

For a business where database downtime stops revenue or public services, the Enterprise plan is the closest match to this incident: 24x7 availability, a 30-minute response target, emergency restore assistance, and direct remote troubleshooting.[7][8]

A practical recovery-readiness standard

A production PostgreSQL service should be able to answer five questions at any time:

1. When was the latest successful base backup?

2. Is WAL protection continuous, and are there any gaps?

3. When did the last automated restore test succeed?

4. How much data can the business lose, and how quickly must service return?

5. Can one compromised production account also delete the backups?

If any answer is unknown, the recovery plan is incomplete.

Build for recovery, not for perfect operators

GitLab's strongest conclusion was not that engineers should never make mistakes. Its postmortem argued for an environment where mistakes can happen without causing long and irreversible damage.[2]

Worlber provides PostgreSQL infrastructure based on PostgreSQL Community and CYBERTEC PGEE, with Patroni high availability, Transparent Data Encryption, and automated backup capabilities.[5] For organizations in Saudi Arabia, the next useful step is a recovery-readiness review built around measurable recovery point and recovery time targets.

Sources

• [[1] GitLab.com database incident](https://about.gitlab.com/blog/2017/02/01/gitlab-dot-com-database-incident)

• [[2] Postmortem of database outage of January 31](https://about.gitlab.com/blog/2017/02/10/postmortem-of-database-outage-of-january-31)

• [[3] pg_hardstorage product page](https://www.cybertec-postgresql.com/en/products/pg-hardstorage)

• [[4] pg_hardstorage repository and documentation](https://github.com/cybertec-postgresql/pg_hardstorage)

• [[5] Worlber PostgreSQL services](https://worlber.sa/en)

• [[6] Instagram reel that inspired this article](https://www.instagram.com/reel/DeKKf55Dekh)

• [[7] Worlber CYBERTEC PGEE features and support packages](https://worlber.sa/en/services/cybertec-postgresql)

• [[8] Worlber PGEE pricing and support FAQ](https://worlber.sa/en/pricing)

Make your PostgreSQL recovery plan testable

Worlber helps Saudi organizations deploy PostgreSQL Community and CYBERTEC PGEE with Patroni high availability, encryption, monitoring, automated backups, and support plans matched to recovery targets.

Talk to Worlber about PostgreSQL recovery