Turning our focus to the industry’s response.
It was the shot heard ’round the cyber world.
Flashback to Friday, December 10. Most of us were just waking up and looking forward to that first sip of coffee. Instead, we were served a huge cup of WTF reality. The CVE-2021-44228 vulnerability — a.k.a. Log4Shell — was disclosed, and the cyber world exploded. This was an entirely new exploit with a 10 CVSS severity rating — and the Apache Software Foundation probably would have given it an 11 if they could.
What makes this zero-day vulnerability so sobering is the expansiveness of its potential reach and the ubiquitousness of ways the triggering JNDI might access a system. Java itself boasts running on “billions of devices” and every one of them could potentially be vulnerable to this flaw. The nomenclature and syntax aspects make Log4Shell so wildly executable. The chat in Minecraft, the best-selling video game of all time, was a vehicle for logging it. People were even putting it on their license plates hoping a traffic camera would capture and log it.
So how DID the channel respond, and what can we learn from this experience?
LESSON 1: The Channel is an Amazing Ecosystem
A threat this wide-reaching could potentially cripple the software and cyber industry, but the channel’s response was the opposite of what the public might expect. Before many of us even had time to process what Log4Shell meant and what we were going to do, the channel ramped up and began working full-bore on solutions.
Three or four different GitHub repositories came out almost instantly looking for the Log4J vulnerabilities from the outside world and testing them. Proof of concept code was happening by 10:00 am MT. In less than an hour and a half, we saw at least five different videos from John Hammond and his team on ways to address the threat. Most impressive was Huntress releasing a free-to-all Log4Shell tester before the end of day one.
There’s a strong sense of comradery and shared responsibility that permeates the channel, and it was on full display starting Friday, December 10 when even competitors banded together. It’s been amazing to see all the ways different providers are working together to quickly create tools to address this threat. We’ve seen people working to educate each other, coming together to talk about the risk and what can be done. A palpable energy currently exists in the channel that we’ve never seen before.
LESSON 2: The Importance of Transparency
Another hugely impressive response to the Log4Shell vulnerability has been the transparency of communications coming from vendors and providers. Understandably, MSPs were frantically trying to figure out if they and their clients were affected, and what they needed to do. Overall, the channel displayed an impressive level of emotional intelligence in addressing the feelings of uncertainty, confusion, and even panic that this threat created.
One of the first responses that caught our attention was from ConnectWise, who quickly pushed out an initial notification that the vulnerability existed and that they were attacking the problem. What was essentially just a “We know it’s happening, but we don’t know much more right now,” message went a long way to easing the minds of those potentially affected. ConnectWise followed this up with well-cadenced updates throughout the day that kept their partners apprised of ongoing developments. We watched the maturity of the message grow with each communication as they started to learn more about the threat and provide actual details. As a security professional, I always want to know where my risk is, and ConnectWise did an excellent job of answering that.
At Pax8, we chose a different approach, releasing a one-time statement to our partners once our SecOps and engineering teams were able to rapidly assess our exposure, update our platform, implement security measures to mitigate the risk, and verify the absence of any malicious exploitation.
If you are a Pax8 partner and have questions or concerns about the CVE-2021-44228 (Log4Shell) vulnerability, please reach out to your Client Account Manager.
When it came to Log4Shell-related communications, Pax8 Manager of Cloud Solutions Dominic Kirby had his own amazing end user experience from Jefferson County Public Schools, which is one of the largest K–12 school districts in the state of Colorado. He received a phone notification from the district to check his email, in which they explained what was happening in plain language: that there was a vulnerability, so the district had decided to temporarily remove public access to their portals in order to safeguard student and other personal data.
“Pulling the entire system down was a tough, courageous decision to make, but Jeffco did it, and I’m really happy about that. As someone whose kids are in that system, I’m glad they made a decision to protect the information.”
– Dominic Kirby, Father and Pax8 Manager of Cloud Solutions
The takeaway here is that the worst thing we can do is not respond at all – people just want to know that we are aware and taking action, even if we don’t have all the answers yet. Whether it’s a cadenced response with ongoing updates, a single communication, or some other approach, we need to maintain strong lines of communication. Prompt, well-planned communication lets partners and their clients understand the threat, empowers them to take actions of their own, and builds trust.
LESSON 3: We Need to Build on this Momentum
In the response to the Log4Shell vulnerability, we’ve seen the channel utilizing muscles that we haven’t seen flexed at this speed before. While impressive, it should also be a wakeup call that, as an industry, we still have a lot of growing to do. We need to keep building the muscles required to respond to incidents like this, and it has to become part of our day-to-day priorities.
We need to make vulnerability management a core part of what we do, and we need to continue to develop defensible strategies. We also need to have these conversations with our clients, so we can teach and educate them. We need to see that our responses are crafted in a way that our partners can say, “Yes I was impacted,” or “No I wasn’t.”
The speed in which we’re increasing our capabilities in the MSP space blows my mind, and I love seeing it.
We are starting to see the signs of life of an operational capacity inside our industry that is more than just nascent working-side execution. This shouldn’t just be a crisis response that we develop. We need to carry this approach into our every-day work.
LESSON 4: Bringing MSPs into the Equation
While there isn’t a whole lot MSPs can do about Log4Shell beyond enumeration of their client environments and scanning for exploitation, this provides a chance for them to step in and be more actively involved in the cybersecurity community.
This is a perfect time for a tabletop exercise. Every MSP should look at each of the vulnerable and vital applications they run and see what the response was from each of those companies. How did they respond? Did they push out a response? Did their response help address your questions and your fears?
Another thing MSPs should do is start digging into the assessment capabilities of their own tool set. Find out if you have the ability to look back into your history and ask, “Did anybody inject something that matches this syntax?” If you can’t do that, take it as a sign that you need to increase your capabilities.
This is a terrific opportunity for MSPs to take a greater role in the channel, to pull a seat up to the table, to be part of the conversation. MSPs have just as much skin in the game and can be a perfect resource for helping shape the future of the channel.
Defining the Legacy of Log4Shell
In no way is this the first cyber risk we’ve faced this year. It’s actually been a fairly average year with just under 50,000 CVSS disclosed software security risks. Log4Shell just happens to be a security risk with the potential to touch almost every aspect of people’s business and personal lives.
While we won’t know the full effect of the Log4Shell exploit for quite a while (if ever), we can and should use this experience as an opportunity to examine our actions, both as individual providers and collectively as a channel. This could be a watershed moment where we embrace the wins, learn from the fails, and build a stronger, more sustainable channel of the future.
Dive deeper with the Pax8 team
Check out the recording of the December 13 livestream during which our own Dominic Kirby, Matt Lee and Ryan Cromar discuss some of the technical aspects of Log4Shell, what it means for the cyber world, and how the channel responded.
