Last week at the BlackHat convention in Las Vegas, Lidia Giuliano and Mike Spaulding gave a presentation entitled, “Lies, and Damn Lies: Getting Past the Hype of Endpoint Security Solutions.” Upon starting the presentation, Mike joked that they could have renamed it “Lawsuits, and Damn Lawsuits” for reasons that I’ll explain below.
With a combined 35 years in Information Security between them, Mike and Lidia have seen their share of infosec marketing hype. However, to effectively deal with rampant ransomware in their environment, they wanted to cut through all the hype.
Just like all security teams, they knew they needed to focus on protecting themselves, rather than buying into all the marketing buzzwords of the day. In their case, their business-centric goals were to reduce incidents, reduce people costs, keep the reputation of the firm, and keep the business running.
As they began talking to vendors about potential solutions, they quickly realized that they would need to create their own test framework to effectively test their own real-world scenarios. They initially selected eight vendors, but once Mike and Lidia explained that they would essentially be doing a bakeoff, three vendors quickly dropped out.
Vendors dropping out of their test framework project speaks volumes – many vendors must realize that their products perform in the subpar range during real-world testing.
Throughout the course of the project, vendors gave Lidia and Mike varying kinds of bad advice:
Lidia explained how she methodically worked from Oct 2016 through to May 2017 on business requirements, setting up a very large test framework and documenting her results. She also realized that it would be important to simulate never-seen-before malware variants, often referred to as zero-day attacks, so she setup her system to mutate malware to better represent a real-world attack scenario.
Attackers don’t typically just reuse malware that is known and easily blocked by traditional AV solutions – they want to get into the corporate network, so they often make minor changes to known malware to slip past the guards at the front door. The vendors in Lidia's bakeoff whose solutions rely heavily on hash-based signatures that have already been seen before tried to convince her to drop this approach, but Lidia knew better. And she persisted.
Ultimately, Lidia and Mike’s test framework grew to contain tens of thousands of samples. Lidia wrote scripts to automate much of the testing. In this way, they were easily able to test and retest on different OS’s, with and without network connectivity, with known samples and mutated samples – and easily replicate tests weeks or months later. Essentially, they created a much more realistic attack scenario than many of the existing testing procedures accounted for.
The difference in effectiveness and performance among the five solutions tested in various categories was stunning. The presentation walked though vendors named A through E. One vendor, vendor C, was clearly superior to the others in several categories of testing, both in effectiveness and performance. The failures of the other four vendors in many cases were simply striking.
Mike and Lidia had originally planned to share the detailed results, including all the vendor names. Yet, as word of their presentation spread, they were threatened with legal action from some of the vendors. As recently as Monday of last week, they had planned on at least announcing the name of the one vendor who ranked by far the highest in the overall testing.
However, after receiving a Cease and Desist notice last Monday, and speaking with their legal counsel, they decided to avoid any further legal hassle for their client and opted to anonymize all the vendor names in their bakeoff, at least for now.
I applaud Lidia and Mike’s near-heroic efforts to bring truth and honesty to this field, and their attempts to combat marketing hype. At Cylance, we fully support this kind of in-depth analysis of anti-malware efficacy that Mike and Lidia have researched, as well as the framework for testing. Their efforts present organizations with real results so that they can make the right decisions for their companies and cut through all the buzzwords.
Here at Cylance, the advice we give to customers has always been to test our products in your environment, so you can see for yourself how effective they can be.
Of course, there are no silver bullets in this industry. Yet, customers who take the time to test within their own environment will be well rewarded for their efforts, in that their environments will be significantly safer than if they just chose a vendor based on their marketing materials.
I encourage you to reach out to Mike (@fatherofmaddog) and Lidia (@pink_tangent) to check out their research. You may even want to join me in thanking them for their courageousness in the face of legal challenges for trying to do the right thing. Even better, they also have templates and a guide for testing available to the community free of charge. Kudos to Mike and Lidia for taking on this important topic.