How I lost and recovered messages

TL;DR: From April 12, 1 PM (UTC) until April 13, 9 AM the integration RSS feeds missed some messages.

On Friday, April 12 around 1 PM (UTC) I accidentally ran an update script in the wrong directory, which resulted in deleting some files with messages.

Fortunately, all files are backed up every night, so I downloaded the latest backup and started restoring the deleted files. I also temporarily disabled all integration RSS feeds because without the missing files they couldn't be rendered correctly. Since I didn't know which files exactly have been deleted, I decided to restore all files from the backup. It took about 4 hours, so by 5:30 PM all files, except the ones created after the latest backup has been made, have been restored. At this point, I also re-activated previously disabled RSS feeds.

After checking which files are still missing it turned out that they're all email messages. So I started restoring them semi-manually from Mailgun, which I use for sending and receving emails. By 9 AM, Saturday (April 13) I was able to restore them all.

What I did to prevent such data losses in the future:

  1. Change the server config, so it's harder to delete files accidentally.
  2. Optimized the restore process, so if I have to restore files in the future, it takes less time.

I'm sorry for the inconvenience! Let me know if you have got any questions.

Published on April 15
ā† All posts