Disaster Recovery & Business Continuity Planning

Disaster Recovery & Business Continuity Planning is discussed in our most recent entry into the Cyber Anxiety podcast series. Our hosts determine what a disaster is and where to start with Disaster Recovery planning and Business Continuity.

To read Simon's blog, click the link below:
http://blog.sembee.co.uk/post/podcast-disaster-recovery-and-business-continuity-planning

Luke Betteridge from Inbay hosts the Cyber Anxiety Podcast, with regular speakers including Daniel Welling from Welling MSP and Simon Butler from Sembee. We created this podcast to help support MSPs through the ever-evolving field of the digital world. The goal is to give tangible tips and strategies that MSPs and others in the tech industry can use while releasing the built-up anxiety around the sector. 

Transcript

00:00:03:03 - 00:00:22:07
Luke B
Hello. Hello, and welcome back to another Inbay podcast and another episode in our Cyber Anxiety series. My name's Luke Betteridge and I'm joined by our two series regulars and my favourite guests. Simon Butler and Daniel Welling. So, thank you guys for joining us again. So today we're having a bit of a focus on disaster recovery and business continuity planning.

00:00:22:20 - 00:00:41:17
Luke B
So, there is a bit of a misconception at the moment that actually, business continuity planning isn't really a priority now that we're in this new cloud world. So actually we want to address that and have a look at why really MSPs and your customers should be having a real focus on disaster recovery and business continuity. So Simon, just to come to you first.

00:00:41:24 - 00:00:53:26
Luke B
I don't know if you want to maybe give an overview of what the DR planning look like in the traditional sense. And in the traditional world, shall we say, and then kind of talk about how we need to be applying that again to this modern cloud world.

00:00:54:24 - 00:01:23:12
Simon B
Well, traditionally disaster recovery pretty much could be counted as the loss of something. So, loss of the building, loss of access to the building loss of data, loss of Internet, loss of electricity supply. But it was affecting everything you had in your office. So, where you know, in the old days where we had, you know, a room full of servers and we had racks and switches and all the other nice stuff that those of us, a certain generation, still get misty eyed over.

00:01:24:11 - 00:01:47:07
Simon B
You know, that was we lost power or anything like that. It was effectively a disaster for the business. Obviously now, a lot of that has moved to the cloud, but we're still sort of thinking, right? Well, you still have some elements on premise in the office. You know, you lose your internet in the office. How are you going to get to your cloud resources for a start?

00:01:47:08 - 00:02:14:00
Simon B
So, some of the elements of a traditional disaster recovery plan still apply. You know, you still got to get access to your cloud somehow. So how do you get access to it and things like this, So I think, yes, what's happened is a traditional DR plan has probably lost some of its weight. You now need to change your considerations on what you need to do in the event of a problem.

00:02:14:06 - 00:02:22:24
Simon B
In some respects, it could be easier to deal with a problem. In other respects, it can also be more difficult, particularly with regards to security.

00:02:23:09 - 00:02:33:23
Luke B
So, have you got some examples of some of the things that might be more challenging in this kind of the modern cloud world, what's an example of something that MSPs may have not considered up until this point?

00:02:35:14 - 00:02:57:16
Simon B
I think, well the main thing probably is going to be accessing the data if, as long as the cloud services up and we're not talking about that being the disaster, you know, if everything is set up for everyone to access everything in the office, you know, we talked about conditional access a little while ago. I was talking about how, you know, some clients have got their conditional access set up.

00:02:57:16 - 00:03:18:11
Simon B
So, you know, you can only access their cloud data in the office. Suddenly you've lost the office. So how are the end users now going to get access to their data? So are you just literally saying to them to send them home and then hitting it with, you know, home computers, which can have all sorts of weird and wonderful stuff on it

00:03:18:11 - 00:03:48:25
Simon B
Or are you then trying to spin up a remote desktop servers or Azure virtual desktop or something else to access that data? So that's the first thing is, you know, you know, what's going to be the easiest way for the clients to at least get access to that data, even if it's so, they can literally send out an email to say, “Oh dear, we've had a problem, we're dealing with it” to keep their customers happy and maintain that level of communication.

00:03:49:03 - 00:04:03:11
Simon B
I think that's the first thing is that the MSP should have is the how are they going to communicate what is gone wrong and what the end users need to do.

00:04:03:11 - 00:04:31:28
Luke B
So, I think that's the key thing really, isn't it, like we spoke about before the technology plays one part of it when it comes to having a backup in place and having a disaster recovery plan in place. But the key thing is actually the process you've got for the end users, for your staff, for everyone involved, really, how are they going to continue working again, in simplistic terms, the disaster recovery plan is to ensure that you can continue working as smoothly as possible so that your customers aren't really even aware that there's been a disaster or there’s been an issue.

00:04:32:12 - 00:04:46:21
Luke B
So, one of the key things is really having those processes in place, but then also having the right technologies behind you to back that up to make sure that you can carry on working as smoothly and as efficiently as possible.

00:04:46:21 - 00:05:07:12
Simon B
Quite right. That's effectively it you know, a disaster while it could impact the business. You're trying to limit that impact as much as possible. Yeah, tech is fine. You know, you've lost everything. You know, you can phone up a friendly VAR and whip out a credit card and all this sort of stuff, but, you know, that's only part of it.

00:05:07:12 - 00:05:14:18
Simon B
And to be honest, probably from an I.T. point of view, it’s probably very small parts of it, the tech’s the easy bit.

00:05:14:18 - 00:05:40:11
Luke B
Yeah. So, one of the things I want to jump into and again, we were talking earlier about some of the misconceptions around cloud technology and the need for a business continuity plan. So, one of the things is obviously majority of companies now using Microsoft 365 or sort of a cloud based email solution. What happens when you get, you know, kind of cryptolockered and all of your emails get locked up, you can't access them.

00:05:40:21 - 00:05:50:19
Luke B
There is an assumption out there, with a lot of MSPs and a lot of people that everything's backed up it’s in the cloud. Don't worry, I'll be able to access it in one way or another. What should people be doing in this type of scenario?

00:05:51:20 - 00:06:21:07
Simon B
That’s quite easy they've got to do backups Microsoft specifically state there is no backups for Office 365. Yes, there are data recovery capabilities but that's mainly for deleted data, but otherwise, yes, you need to back up all you're doing is a different kind of backup. So, you may not necessarily be using tape or anything like that because, you know, realistically bringing a lot of that data back to your office where your backup system is, is not going to be practical, particularly with the way that, you know, Internet connections are here in the UK.

00:06:21:29 - 00:06:48:10
Simon B
So, you could be looking at using another service to do that, backing up. But you've got to you've got to have backups. You know, the fact that you're in the cloud doesn't mean no backups, it just means different kinds of backups. You know, this is the underlying strategy around retaining your data hasn't changed. It's just that the methods being used are different.

00:06:48:10 - 00:06:57:07
Luke B
And just whilst we’re on the subject, are there any recommendations for a backup? Is there a particular product or is there something supplied by Microsoft in terms of backups for your emails?

00:06:57:07 - 00:07:19:22
Simon B
I don't think Microsoft actually have anything that they recommend. What I would normally advise my MSP clients is that you look at the underlying tech, where is the data being stored? Because the last thing you want to do is be backing up Microsoft to a customer to a service sorry, that is using Azure as their backup resource.

00:07:19:22 - 00:07:23:00
Luke B
defeating the point in a way if anything was to happen with Microsoft.

00:07:24:04 - 00:07:48:21
Simon B
If you're heavily into Microsoft 365 you probably then want to be looking at maybe something that goes to say, Amazon S3 or one of the S3 compatible services or someone is running their own infrastructure. Even the MSP could run their own infrastructure. You know, there are service software that you can buy that you can put on to effectively a, you know, a large NAS or something, you know, your own storage.

00:07:48:21 - 00:08:06:13
Simon B
If the MSP wants to go down that path, which would then, you know, poll the tenant and pull out the email, pull out the SharePoint data, pull out the OneDrive data, even pull out the teams conversation data and all this sort of stuff and retain it and all this sort of stuff for you. So that's what you've got to look at.

00:08:06:19 - 00:08:26:24
Simon B
But it's, you know, it's the same as the backups you were doing back when you were on prem. You didn't backup to the same server, you were backing up, you backed up to something else. And it's just the same. You know, you're looking if you're really paranoid, you back up into a different region. You know, if you're in the UK, you may be looking at, you know backing up into Central Europe or something like that.

00:08:26:24 - 00:08:39:25
Simon B
You know, so the you know, you're reducing the chances of the fact that your data might be in the same, even though it's on a different service, you know is in the same data centre in London, for example.

00:08:39:25 - 00:08:58:05
Luke B
no, it makes sense. And again, it's maybe sounds pedantic, but some people do want that level of security in a sense that, you know, making sure a plane is not going to suddenly fly into the data centre and destroy the whole thing or something as extreme as that. So, no, that makes complete sense. Daniel I just wanted to come to you because obviously there's two elements to having disaster recovery plans.

00:08:58:05 - 00:09:13:06
Luke B
Obviously, the MSP themselves need to have a DR plan in place, then it's also about providing that to your customers. Have you got kind of any insight on ways that you can approach providing as a service for your customers and almost commercialising the DR plan itself?

00:09:14:23 - 00:10:00:14
Daniel W
Yeah, and great, great phrase actually to the commercialisation of this. And really, I think it has to be interwoven into every piece of guidance and advice that the MSP is offering themselves and also to their to their customers. There's no point architecting a solution that doesn't have, let's say, business continuity as the modern phrase, sort of baked into it and yeah you can really go down a rabbit hole with the thinking on this as Simon was providing an example.

00:10:00:14 - 00:10:41:26
Daniel W
There for. I mean there's always a single point of failure and that is that we're on one planet currently. I'm sure in the future we might have backups to the moon, but yeah, currently there will always be a risk and it's important for the customer and the MSP to understand what that exaggerated risk is and therefore to make their own decision about how much they want to spend protecting against that risk and that follows through in terms of every aspect of their IT decision, their IT choices about what they buy and how they buy it.

00:10:42:05 - 00:11:17:18
Daniel W
So yeah, you were quite right earlier talking about a process that can't be stressed enough and the expectations that sits above that process has to be set at the very beginning of the relationship. It has to be woven into the account management process, which will undoubtedly have some sort of business review interface for the customer and then a roadmap to show the customer what the options are and how much they can spend.

00:11:17:27 - 00:12:09:28
Daniel W
But they need to be under no illusion that no matter what they spend, there will be outages, there will be data loss. And they need to protect themselves against that eventuality when it happens rather than if so, yeah and in terms of how far you commercialise it, 365 back up is a commodity purchase now and just as well because it's a nice and easy decision for the customer to make about spending the money, but a customer that has deeper concerns or an MSP that wants to offer a premium service should then be also offering rehearsal for the business continuity plans.

00:12:10:20 - 00:12:38:28
Daniel W
And you're only as good as your last recovery. that's the reality. You know you know if you back up sufficient if you if you're able to use it so yeah and again I would encourage MSPs to be thinking about offering an annual, a quarterly, a monthly restoration test and yeah sure very few customers are going to opt for a monthly one.

00:12:39:09 - 00:12:53:16
Daniel W
But there's probably a lot out there that would opt for an annual one at least and that's time it's revenue. So, it's an opportunity and it reduces the MSPs risk.

00:12:54:18 - 00:13:12:09
Luke B
And I guess with kind of doing restorations there’s different levels you could do of that isn’t there you can almost do it by department or by a specific kind of entity like we said maybe kind of going through what would happen if emails were to go down and then you could do one like you said, an annual one, which is kind of almost like everything's gone.

00:13:12:09 - 00:13:43:22
Luke B
What is the process? And it's like we said it before, again, that process is key. It's making sure that everyone knows what to do in that situation. Suddenly everything goes off, you know, every department and every individual worker knows exactly what they need to do in that moment to get back up online and running again. So, yeah, as I was saying I guess with restorations, you can do different levels of that and then you can offer that based on, you know, the frequency saying actually every month let's just, let's go department by department and then at the end of the year, that's a big one that involves the whole company or something along.

00:13:43:22 - 00:13:44:08
Luke B
those lines.

00:13:44:28 - 00:13:49:22
Daniel W
Yeah, exactly. Exactly right. And great, great suggestion.

00:13:49:22 - 00:14:07:04
Luke B
Just out of interest, is there any kind of like regular objections you see to putting like DR plans in place or business continuity elements in place? Is it normally to do with costs or is there another element that customers seem to be pushing back on? You guys can't see this, but there's a lot of nodding heads at the moment.

00:14:07:04 - 00:14:08:13
Luke B
As soon as I mentioned, costs.

00:14:08:13 - 00:14:40:05
Simon B
Cost is always, cost is always the main one, but the other the other one, particularly if we go back to traditional DR, it was always well it's not going to happen to us. And when I was doing traditional DR planning, the plan that I'd put in place for, say, a company that was two miles away from Heathrow compared to the company that was out in the middle of nowhere where, you know, the nearest neighbour is a sheep was completely different because, you know, the risk scenario was completely different.

00:14:40:05 - 00:15:06:11
Simon B
And the rural customer would say, well, you know What's gonna happen to us? You know, the chance of a plane hitting our building is quite small. The chances of the Heathrow, customer a chance of a plane hitting their building was much larger. And so, it was like, yeah, that's not going to happen to us. But now, with everything so much in the cloud and everything you can almost adopt the same DR plan for everybody because now the risk is almost the same for everybody.

00:15:06:22 - 00:15:24:24
Simon B
You know, we've taken away the almost the differentiating factors and you can say, well, it’s not going to happen to us. Well, I'm sorry, but if you're in office 365 it doesn't matter whether you're in Heathrow or outskirts of Cardiff or up a mountain, an office 365 outage is going to affect you in the same way.

00:15:25:04 - 00:15:45:22
Simon B
So, from MSPs point of view, they can always come up with a standard plan, which they can then implement and get away from this. It's not going to happen to us scenario, which is, you know, one of the major pushbacks, particularly with smaller businesses, then of course with smaller businesses and I'm talking we're talking, you know, less than say less than 25 or even smaller.

00:15:46:14 - 00:16:06:21
Simon B
Most of those, if they had a major disaster, they don't come back from So that can be a way that Daniel, when he goes in as a salesperson, can have on, you know, the method of, you know, if you go and had everything Cryptoed that's your business gone probably overnight. Yeah. How you know, then you can start to pull on the, you know, the heartstrings.

00:16:06:21 - 00:16:12:04
Simon B
How is, how are you going to put food on the table?

00:16:12:14 - 00:17:02:16
Daniel W
It is absolutely education. That's the only way to combat the budget objection and again the same underlying principles are just as true today as they were a decade ago of recovery point and recovery time objectives. So, if you are unable to access your data due to a ransomware attack, you, you've got to follow a process to investigate what the cause was, which means you can't immediately be recovering unless you've got a second environment ready, ready to go.

00:17:03:08 - 00:17:33:16
Daniel W
So just explaining that part of it actually helps a customer understand the fragility of their position. And guess what? they’re then going to be prepared to spend more on security in order to reduce the chance of that happening. But you know that the business continuity part of it will just be a natural follow on part of it.

00:17:34:14 - 00:18:13:13
Daniel W
And yeah, I guess the more we mature what’s happening out there in the marketplace with the correlation between business continuity security and of course insurance, we're starting to see the market firm up develop new products. You know, recently there's been issues where insurers are just refusing to insure or the premiums have got so expensive because it's caught up now with the insurers that they're supporting this ransom industry.

00:18:14:07 - 00:18:39:16
Daniel W
So, you know, we're going to we're going to see another change another transition, there'll be another level of education and understanding within the end customer marketplace as a result. And but then I'm sure the bad actors out there will come up with another naughty disruptive challenge for us to mitigate in time.

00:18:39:16 - 00:19:19:25
Daniel W
And so the cycle goes on. But yeah the customers have got to be educated they would of course need different levels of persuasion and time. And so you as the MSP, have got to be consistent in your messaging and not give up you know keep talking about the same topics and eventually the customers will have no choice but to come around and make decisions that reduce their’s and the MSPs risk.

00:19:21:06 - 00:19:40:02
Luke B
I think that's absolutely spot on. I think that's kind of a brilliant way to close off kind of you know our discussion around disaster recovery. Before we go, I thought we'd have a little bit of a fun one. Do you guys have any examples of where someone didn't have a DR in place and suddenly everything kind of went, shall we say, all went wrong.

00:19:40:02 - 00:19:58:23
Luke B
So I've got one story which I think is quite interesting is I had a customer once it’s going back a little while ago. Now obviously won't say any names, but they used to have two on prem servers on site and at the end of the day they used to just copy everything from that day onto the next server and then obviously carry on working and then the next day they would just copy it again.

00:19:58:24 - 00:20:17:11
Luke B
So every time they were overwriting the, you know, the day before. So essentially they had one 24 hours worth of backups and one day, you know, essentially we kept going up to them and saying to them you need to put a proper backup plan in place, proper, you know, DR solution. We need to do something. No, no, it's fine. We'll fix it when it breaks.

00:20:17:24 - 00:20:43:22
Luke B
So one night someone opened an email, got a Cryptolocker virus about 5:15 in the afternoon. Didn't realise when they opened email immediately everyone locked up at 5:30. went home. Obviously the whole system got cryptolockered and then it backed up to the server next to it with the Cryptolocker and then the next morning they had absolutely nothing. So again, this goes back to that example where sometimes until it happens to them, they don't think it's going to happen.

00:20:43:28 - 00:21:00:28
Luke B
There is that real kind of again, misconception that's like, well, we're never going to find ourselves in that situation until you do, and then it's almost too late. So, again, I don't know if you guys have got any kind of fun stories of people who haven't, put a DR plan in place.

00:21:01:18 - 00:21:23:13
Daniel W
It's a great example, actually Luke it normally comes down to a misconception rather than a deliberate not to have a back up, but misconceptions to how secure their back up was or whether in fact it was working. And that's what I would probably see is the majority of data loss instances in the past.

00:21:23:13 - 00:22:12:00
Daniel W
It's because a back up hadn't been working and maybe that then falls back on the notification monitoring process. But equally, that's why you layer this up with recovery testing to ensure that, you know, not just the process is rehearsed, but so you know, you've confidence in the technological part. But yeah, I think there's you know, I think everyone in in the MSP world has examples stories of data loss and that's why it's such a hot, hot topic for us because we know the pain of having that conversation with the customer and breaking the bad news because regard regardless of fault the MSP

00:22:12:13 - 00:22:40:19
Daniel W
you know someone has to be has to take the blame and there will be pain that comes from it regardless of whether you can pull out the I told you so card so yeah and you just have to have care for your customer and be thinking of that example in the future when you're having the frustrating conversation about how they should be doing more.

00:22:40:24 - 00:22:41:26
Luke B
Yeah definitely.

00:22:42:17 - 00:23:11:29
Simon B
I've got one quickly on the processes one I did a job for a client last year. Again no names where a crypto attack was launched during the middle of the night and the person who responded because they didn't have a process thought the best course of action was to cut the company off from the internet. But the problem was, of course, the encryption was still inside.

00:23:12:25 - 00:23:36:09
Simon B
He then alerted everyone else what was happening, but nobody could get in. So everyone basically had to then travel to the various offices to physically get inside because he cut the internet off. He just gone, right I've seen it happening. I've cut the Internet off. But of course it was already inside. So by the time people got into the office in 45 minutes to an hour, the damage was done.

00:23:36:10 - 00:23:37:04
Luke B
Yeah, it's too late.

00:23:37:04 - 00:24:00:19
Simon B
and because nobody could get in to the office to get into the systems, to actually do anything about it. And they were down for a very long time. I helped them to recover. But that was purely down to not having a process for dealing with an incident which made the problem a lot worse.

00:24:00:19 - 00:24:18:23
Simon B
You know, if they'd actually had a process, you know, just simply somebody remoting in and just shutting down their entire VMware platform probably would have saved a ton of headache, even if they literally, you know, got the UPS to yank the plug out the back or the virtual plug, if you like, by just cutting the power.

00:24:19:16 - 00:24:35:07
Simon B
You know, we could have recovered from that kind of disaster quite easily. You know, that's an easy scenario to deal with, you know, But everything was up and it was basically running riot for about an hour and a half until somebody could get in. And that's what they actually ended up doing, was pulling the plug out of everything.

00:24:35:18 - 00:24:55:29
Simon B
The damage was done. I've got another I've got just quickly, I've got a one of my own. This is one of my own. In my last full IT job, we were I just taken on and for some reason, the customer, the customer had two buildings and for some reason the core finance server was in the other building, everything else was in the primary building.

00:24:56:08 - 00:25:25:00
Simon B
And we went in on a Saturday to clean up the network switch and completely re cable the network switch. And during that I dropped something that broke the fibre connection between the two sites. And we ended up, the two of us going over to the other building, unplugging the server and physically carrying it across the car park in full view of the CCTV looks completely dodgy to get it on the other side to get it back up and running, you know.

00:25:25:00 - 00:25:41:27
Simon B
And then we had to get someone in the other office was basically unusable for about two days while we had the fibre repaired and found the person to repair the fibre and all this sort of stuff. And so, you know, that had an impact on the business. And that was purely, you know, my mistake. I think I dropped or

00:25:41:27 - 00:25:48:03
Simon B
I pulled it or something but fibre’s quite fragile where it was coming out and it broke and it was like oh dear.

00:25:48:13 - 00:26:09:25
Luke B
I think that's such a big thing, though, is how easily these kind of disasters can happen. Yeah, it can be a small thing, but it completely has a massive impact on your ability to be able to work. And again, one of the key takeaways, if you take away anything from today's podcast, I think the key for us is, process, you need to have a process in place when it comes to DR planning.

00:26:10:08 - 00:26:32:00
Luke B
So Daniel, Simon, thank you so much for joining me today and thank you for your insights on DR and business continuity planning. Looking forward to you joining us again in the next podcast and for everyone listening at home. Thanks a lot. If you have any questions or you want to reach out to us to discuss anything that was spoken about in today's podcast, then please feel free to reach out to us and have a great rest of your day.

00:26:32:00 - 00:26:32:21
Luke B
Thank you very much.

00:26:33:13 - 00:26:35:10
Daniel W
Thank you.