How I lost and recovered messages

TL;DR: From April 12, 1 PM (UTC) until April 13, 9 AM the integration RSS feeds missed some messages.

On Friday, April 12 around 1 PM (UTC) I accidentally ran an update script in the wrong directory, which resulted in deleting some files with messages.

Fortunately, all files are backed up every night, so I downloaded the latest backup and started restoring the deleted files. I also temporarily disabled all integration RSS feeds because without the missing files they couldn't be rendered correctly. Since I didn't know which files exactly have been deleted, I decided to restore all files from the backup. It took about 4 hours, so by 5:30 PM all files, except the ones created after the latest backup has been made, have been restored. At this point, I also re-activated previously disabled RSS feeds.

After checking which files are still missing it turned out that they're all email messages. So I started restoring them semi-manually from Mailgun, which I use for sending and receving emails. By 9 AM, Saturday (April 13) I was able to restore them all.

What I did to prevent such data losses in the future:

  1. Change the server config, so it's harder to delete files accidentally.
  2. Optimized the restore process, so if I have to restore files in the future, it takes less time.

I'm sorry for the inconvenience! Let me know if you have got any questions.

Published on April 15
All posts