That bad feeling you get in your stomach. Can I restore my files?

The first thing I check when talking with a potential clients is if I can restore a file from their backup. Notice I did not say I check to see if they do backups because most do. Or they think they do. We go to test and.. that tape failed (74% eventually do). Panic!

This morning started with a phone call at 6:45. "Our system is down!" Where does your mind go? Immediately mine goes to "Did the backup run last night?" Can I trust it? Do I know for a fact that I can restore from it?

That is a sick feeling, even though just a week ago I did indeed test it to make sure I could restore. Worst case scenario I just lost a week of data. Not catastrophic but its not going to be a good day if that's the case.I still was not quite sure. No need for caffeine this morning. I am awake.

First step is to VPN in to the client's system to see what I can see. Can I ping it? (A ping is like throwing a ball as something you can not see in the dark. If it returns you know its there. If it doesn't.. YA)  YES!!  I can ping it so that means at least it is breathing. I can not log into it though.. (I guess I can breath a little..very little). 

I get on the phone with person who discovered our barely breathing system. The client is 45 minutes away. They open in about 6 minutes. HMMMMM....  Quickly I verify that the backup ran the night before. YES, but with errors. I can see when it lost communication. I am now in a state of elation and horror.. and it's Friday of course! Seems to always happen on a Friday...

I tell the client to PUSH and hold that power button and till it dies... I can hear the fans shut down.  Let it rest for a little bit. He hits the power button again and the server comes back to life.

At this point I know I there will not be an answer for about 8 -10 minutes (it is an old server). I jump in the shower thinking of all of the good, bad and ugly this day may turn into. When I get out I check my phone.. Nothing. I get dressed and go into my office. VPN, connect... AND!! Bingo.. All systems go and running like a champ. Test functions.. All look happy. Client says "THANK YOU"

I have had enough cases in my career similar to this where I wasn't quite sure if I needed a full restore I could do it. Major projects were never a problem because we did full backups and multiple test restores so confidence was there. I did have one case of a client get hit hard with a RansomWare virus (Encrypted half the network). The restore started OK but would die after 15 minutes. It was brutal but I did get it back because there was an additional layer of protection where it was backed up to a cloud server. It was a much longer time frame but it came back.

So the moral of this story is how much faith do you have in your backups AND your restores? My typical disaster recovery plan for clients is an on site storage system coupled with an upload to a cloud repository each night. I will probably never be 100% sure I can get everything back. That queasy little feeling will be there no matter how much I test. When it comes to security nothing is 100% but at least I know I am doing everything I can to give it a fighting chance.