How Two MSPs Got Punched in The Mouth — Things You Don’t Consider Until You Get Hacked

Boxer Mike Tyson said, “Everyone has a plan ‘till they get punched in the mouth,” and that’s how Robert Cioffi of Progressive Computing Inc. felt after experiencing a cybersecurity incident firsthand.

During a recent panel discussion at ChannelCon 2022, “I’ve Been Compromised, Now What?” Cioffi and Jay Tipton of Technology Specialists got vulnerable about their experiences being hacked. Cioffi said when it comes to cybersecurity, preparedness only goes so far.

“You can’t cover every permutation; we don’t have infinite resources. You’ve got to do enough that’s prudent, you’ve got to keep improving and keep trying,” he said. “Experience is a really wicked teacher.”

A Tale of Two Hacks

“A headline-making July 4 cybersecurity attack hit about 1,500 companies in 2021,” said Wayne Selk, CompTIA’s vice president of cybersecurity programs, who hosted the panel discussion at CompTIA ChannelCon. While hackers demanded $70 million from their victims, Tipton and Cioffi found themselves spinning out instead of diving in.

“I’m not a touchy feeling kind of guy, and stress to me is something you deal with, but in that high stress situation we were figuratively coming apart at the seams in ways that I can’t describe,” Cioffi said. “Everybody in my company always looked to me as the guy with the answers and I was staring into an abyss.”

The stress caused him to freeze — a common traumatic stress response — and waste valuable time. Tipton experienced the same challenge. Even with an incident plan and a hundred priorities to address, his problem-solving skills failed him. “I couldn’t make a decision, I couldn’t process it all,” Tipton said. It took support from confidants in the cybersecurity community to get them refocused and ready to work on a solution.

Unexpected Challenges

Tipton had an incident response plan, but it didn’t do him much good. The only copy was tied up in the attack. “Everything was in [the platform],” said Tipton. “What I never thought to think of was, ‘What if that goes down? What if I can’t get to it?’” Eventually he remembered an uncorrupted backup where he could access a copy of the plan, but he didn’t think of that solution until a full day later.

Cioffi’s challenge was in communicating the right message. A breach coach walked him through some suggested language, but he couldn’t get behind it. “I said, ‘If I give them this legal speak, it’s going to destroy 30 years of relationships.” In the middle of a data breach, you’ve still got to maintain communication and manage your business relationships. 

Another thing the solution providers never accounted for: The actual speed it would take to bring all the data back online. “We had 16 terabytes for one client,” Tipton said. It took much longer than expected to recover after the attack.

What MSPs Can Learn from Getting Punched

It has been more than a year since the cybersecurity incident that sent Tipton and Cioffi spinning. Since then, they’ve had time to regroup and turn their experiences into valuable intel for other MSPs. Here are three things they recommend before you find yourself in a cyber attack.

Related Content: MSPs Double Down on Cybersecurity | Trend Watch

1. Make Multiple Plans

Develop both an overall response plan and a more specific disaster recovery plan, and make sure at least a few people have access to it. “It used to be the only one who knew the plan was me,” Tipton said. He’s taken the burden off himself and also made hardcopies of the incident recovery plan, which are kept in a secure location.

Plans aren’t always about data breaches either. Organizations need plans for business continuity, disaster recovery and incident responses — an ice storm can cause as much downtime as a cyber attack. 

“Focus on planning, but that’s not the end goal. You need to develop a process by which you can take these out and use them,” Cioffi said.

2. Practice the Process

When you’re managing an attack, you don’t have time to think about the process, you need to be able to do it in your sleep. As Tipton and Cioffi experienced, a freeze response is common in a cyber attack. “When the rubber meets the road, the stresses are going to kick in and your mind will go blank,” said Selk.

Practice your plans with your entire staff, everyone from development to HR and marketing.

“We squandered a good three hours in vacillating and calling the wrong people and doing the wrong things,” said Cioffi. “Thankfully we didn’t do any damage to ourselves, but we could have been on top of it a lot sooner.”

3. Communicate Carefully

In case you missed 10 Things You Missed at ChannelCon, “breach” is the new four-letter word. During a cybersecurity incident, communication overall should be handled carefully, and it’s especially important to use phrases like “reportable security incident” over the b-word.

Keep communication clear during a reportable cybersecurity incident, and that includes saying, “We can’t speculate on what is happening in the moment.” When developing your incident response plans, include a communication plan for both internal and customer-facing messaging.

It’s not a question of if, but when, you or one of your customers will be compromised by a cybersecurity incident or ransomware attack.