A Backup You Haven’t Restored Is a Hypothesis: GitLab’s Database Outage
On January 31, 2017, one of GitLab’s engineers was trying to restore PostgreSQL replication between the production primary and secondary. The replica had fallen so far behind that it could no longer simply catch up, so the secondary had to be rebuilt: clear the PostgreSQL data directory and copy the database from the primary again using pg_basebackup.
The procedure was not going well. pg_basebackup would start, but for a long time it showed no clear progress. Several attempts changed nothing, and strace only showed that the process was sitting in poll and waiting. After another attempt, the engineer thought some files might have been left in the data directory and could be blocking a clean restart, so he decided to clear it again.
About a second or two later, he noticed the problem: the command was not running on the secondary, db2. It was running on the production primary, db1. The process was stopped immediately, but out of roughly 300 GB of PostgreSQL data on the primary, only about 4.5 GB remained.
The production database was effectively gone. And this would have been just a very bad story about using the wrong terminal if the replica, backups, and other recovery mechanisms had then done what they existed to do.
They didn’t.
To Delete the Primary, They First Had to Delete the Replica
At the time, GitLab.com used a fairly ordinary PostgreSQL architecture: one primary and one secondary running in hot-standby mode. The secondary existed mainly for failover, while almost all database load went through the primary.
That evening, the database was already under heavy load. GitLab was dealing with spam activity, and it was later discovered that a background job was also trying to delete a GitLab employee and data connected to their account after the account had accidentally entered an abuse workflow. In its final postmortem, GitLab identified these two events as the reason for the load spike that caused replication to start falling behind.
For this story, you only need to know one thing about PostgreSQL replication. The primary writes changes to the Write-Ahead Log, or WAL, while the secondary receives those records and replays them locally. If the secondary falls a little behind, that is normal: it reads the missing WAL and gradually catches up with the primary.
The problem starts when the lag becomes too large. GitLab was not using WAL archiving at the time, and the primary had already removed old WAL segments that the secondary had not received yet. PostgreSQL replication can fail when the WAL it needs has already been recycled, and that is exactly what happened at GitLab: the replica could no longer continue from its previous position and needed a new base backup.
Around 23:00, the secondary’s data directory was deleted and pg_basebackup was started. First, the team discovered that the primary did not have enough replication connections, so max_wal_senders was increased from 3 to 32. PostgreSQL then refused to restart because this required too many semaphores, which led to another interesting discovery: max_connections had been set to 8000 for almost a year. They reduced it to 2000, PostgreSQL started again, but pg_basebackup still looked like it was doing nothing.
Later, another production engineer explained that this behavior could have been normal: pg_basebackup sometimes simply waited for the primary to start sending replication data. The person who knew this was not around at the time, and the behavior was not described clearly enough in GitLab’s runbooks.
So the next attempt started with the idea of getting a clean data directory again. The operation itself was not the problem — rebuilding the secondary really did require removing its old state. The only problem was that the destructive command was run on db1, not db2.
By this point, the secondary was already empty as part of the normal rebuild procedure. Now the primary was empty too because of a mistake, and the two PostgreSQL hosts that had looked like redundancy only a few hours earlier suddenly could not provide even one recoverable production copy.
This is where the story stops being about rm.
Now you need a backup.
There Was a Backup. In Theory
GitLab had what looked like a perfectly reasonable recovery plan. Every 24 hours, pg_dump was supposed to create a logical backup and upload it to Amazon S3. There were also LVM snapshots of the production database, while Azure disk snapshots were used for some other parts of the infrastructure.
After the primary was deleted, the team went looking for the pg_dump backups and discovered that the S3 bucket was empty. It wasn’t that the latest backup was too old or corrupted. There simply was no current dump.
The reason was almost perfect for this series. The production database was running PostgreSQL 9.6, while the backup process was running pg_dump from PostgreSQL 9.2. For PostgreSQL 9.x, these were different major versions, and the older pg_dump failed when it tried to back up the newer server.
But where did 9.2 even come from if production was already running 9.6? GitLab Omnibus supported both versions and selected the correct binaries based on the local $PGDIR/PG_VERSION. On a database host, that worked logically enough: there is a PostgreSQL cluster, it says 9.6, so use 9.6.
But pg_dump was not running on the database server.
It was running on a regular application server. There was no local PostgreSQL data directory there, so there was no PG_VERSION either. Omnibus therefore fell back to PostgreSQL 9.2. The backup job ended up running pg_dump 9.2 against production PostgreSQL 9.6, got an error, and finished without creating a usable backup.
This was not a problem that appeared during the outage. The backup had already been broken for some time, which raises the obvious question: why had nobody noticed?
Technically, they had. Email notifications were configured for cron job errors, so the failed backup process really did generate messages. The problem was that those cron emails were rejected by the receiving mail server because of the DMARC setup, so the team never got the signal that pg_dump had stopped doing its job.
So the backup process was broken, monitoring saw the failure, a notification was created, but the people who needed to know never received it. On an architecture diagram, you could still draw pg_dump → S3 → alert. The real result was much simpler: there was no backup, and nobody knew.
Azure disk snapshots could have been the next option. GitLab did use them for other machines, including NFS servers, but snapshots were not enabled for the database servers. This was not simply an accidental omission. Restoring within the same Azure storage account could be fast, while moving data between different accounts could take hours or even days, so the team did not want to depend too heavily on this mechanism. In the postmortem, GitLab explicitly said database snapshots were not enabled because the other backup procedures were considered sufficient.
At this point, the situation was becoming absurd. The primary was deleted. The secondary had already been cleared for its rebuild. pg_dump backups were not being created. Emails reporting those failures never arrived. And there were no Azure snapshots of the database.
One option remained.
The LVM snapshot.
Production Was Saved by a Snapshot That Wasn’t Made for Recovery
GitLab’s LVM snapshots had a different main purpose. The production database was copied into staging so changes could be tested against realistic data, and this process ran automatically every 24 hours. GitLab specifically pointed out that these snapshots worked as intended, but they were not designed as the main disaster recovery mechanism.
If only the regular daily snapshot had been available, the data loss would have been much larger. But earlier that same day, at around 17:20, an engineer working on database load balancing in staging manually created an extra LVM snapshot so he could test against a fresher production copy.
And here is the detail that makes this whole story worth telling. In the original incident report, GitLab said this was the same team-member-1 who accidentally deleted the production database about six hours later. The person whose command finally killed the primary had personally created the freshest copy that later made recovery possible.
GitLab used the staging database created from this snapshot as the basis for recovery. That saved them from almost a full day of data loss, but the process was not fast. Staging ran on cheaper Azure disks without Premium Storage, and copying the data back into production took roughly 18 hours.
In the end, the database was restored to its state at 17:20 UTC. Changes made after that point were lost. GitLab estimated the loss at at least roughly 5,000 projects, 5,000 comments, and 700 users. Git repositories and wikis were stored separately, so the Git data itself was not lost with the database.
GitLab.com remained unavailable for about 18 hours. And for almost all of that time, the team was restoring production not from the recovery mechanism they had expected to depend on before the accident, but from a staging copy that existed because someone had manually created a snapshot for completely different work.
The Problem Wasn’t the Wrong rm
With hindsight, it is very easy to reduce this entire story to one engineer who failed to check the hostname before running a destructive command. It is a comfortable conclusion because it makes the solution seem simple: be more careful, add a red shell prompt, or ban rm, and the problem goes away.
After the outage, GitLab did start working on making hosts and environments harder to confuse. But the team deliberately went further in its postmortem. Follow-up work included Prometheus monitoring for backups, automated restore testing, better replication runbooks, hourly LVM snapshots, and a dedicated owner for data durability. Point-in-time recovery and continuous archiving were still being investigated at that point, not presented as solutions that had already been implemented.
What interests me much more is everything else. Before the outage, GitLab had several layers of protection: a replica, a procedure for creating pg_dump backups in S3, cron notifications for failures, LVM snapshots, and Azure snapshots for other parts of the infrastructure. But when recovery was actually needed, the replica had already been cleared for a rebuild, pg_dump backups were not being created, alerts were not arriving, and Azure snapshots were not enabled for the database servers at all. Out of that entire system, the only thing that was actually useful for recovery was a staging copy that was never meant to be the main disaster recovery path.
In its final root-cause analysis, GitLab asked a very simple question: why wasn’t the backup procedure tested regularly? The answer was even simpler — there was no ownership, so nobody was specifically responsible for regularly verifying the recovery procedure.
And that, in my view, is the main idea behind this incident. It does not matter how many backup jobs exist in cron, how many objects sit in storage, or how many green indicators your monitoring shows. Until someone regularly takes that backup, restores a system from it, and verifies that recovery actually works, all you really know is that the backup process is doing something.
You still don’t know whether you can recover.
A backup you haven’t restored is a hypothesis.
GitLab tested that hypothesis at the worst possible moment: after the production database was already gone. The answer cost roughly six hours of database data and about 18 hours of downtime.