Unveiling the Testing House Minority Reports

In my past blog, ‘Security Testing Houses: Know the Truth’ I exposed some unethical behaviors at security testing houses. But in that same exposé I also said, “Repeat after me: there is no such thing as a 100% efficacy rate in security. There is no single silver bullet that will provide total, unbreachable protection against every type of malware in every situation. Ever!”

And yet every week that goes by, I read testing reports that claim that Vendor X or Vendor Y has made it to the fabled efficacy rating of 100% with zero false positives.

With such a game-changing testing score, you’d think you’d soon see headlines stating that the world’s malware problems have finally been solved by these ‘bulletproof’ antivirus (AV) companies, and that every business and enterprise in the nation can now have total peace of mind because they will never have a malware issue or data breach ever again. Please note that I am holding up an extremely large sarcasm sign right now.

What Lies Behind the Door of the Oracle?

I can’t help but think of the movie Minority Report when it comes to testing security products. In Spielberg’s futuristic thriller, the pseudo-science of PreCrime is invented to create an Orwellian brave new world, using a supposedly infallible testing system to convict people before they commit a crime. The system is controlled by higher powers that have ulterior motives (profit) beyond simply preventing crime. This leads to the internal suppression of ‘false negatives’ to perpetuate a broken system that keeps those who run the system in a position of power.

In one key scene, our hero John Anderton takes a new recruit behind the scenes to a hidden room called The Temple to view the inner workings of the system. He remarks, “The power has always been with the priests, even if they had to invent the Oracle.”

I am here today to tell you that is exactly what some testing organizations want you to do: to trust an ‘infallible’ system run for profit whose inner workings are kept hidden from us. What, exactly, lies behind the Temple door? “There's nothing to see here folks, just believe us,” the priests reassure us. “Only we can talk to the Oracle. Don’t go in there. Don’t even try.”

Herein lies the problem. If everyone had the power to talk to the Oracle, the priests would be out of a job. By denying people access to the Oracle, the priests became the ones with the real power – including the power to interpret the “truth” they receive exactly as they (or their financial backers) wished.

Going back to the world of malware, we’re told time and again that it’s “impossible” for consumers and companies to test AV systems for themselves. After all, that’s why we have AV testing houses, right?

I recently attended a very informative meeting at the Anti-Malware Testing Standards Organization, where I was fortunate to be on a panel where I stated, to some controversy, that testing organizations must:

1.  Define a Curation Strategy for the Malware Repository – The process used in collecting the malware used in testing and how the malware is maintained over time.
2.   Provide Provenance of Malware – The evidence of the history and source origin of the malware.
3.   Conviction Process of Malware – The steps used in providing evidence on how the malware was convicted or charged as being malware.
4.   Creation of Malware – Allowance of any testing organization to create malware to be used in testing.
5.   Obfuscation and/or Modification of Malware – Allowance to change the malware from its original form for the purposes of changing the underlying properties of the malware, not necessarily the underlying actions of the malware.

None of this happened in the latest testing report. Once again the vendor, or vendors in this case, paid for perfection using questionable malware.

I will state this simply: using a malware list to which vendors and testers can contribute is simply hygiene testing. It’s equivalent to using malware from the 1970s. This is not prevalent malware being detected and defeated, because everyone already knows about the malware in question. Signatures that guard against it have already existed for some time. I would argue that this malware would never be seen in a modern environment. Its conviction is questionable and so is its provenance, so why curate it?

The simple answer is because testing houses know that they are in a position of power, and no one has ever questioned their authority… until now.

Why You Should Test For Yourself

Case in point: over the past two years, between two 'reputable' testing organizations, we have seen public tests produce a testing score of 100% accuracy (wait for it) 149 times.

How is it that we have seen 100% accuracy in 149 public security product tests, yet company after company continues to get breached because these products that produce 100% accuracy in so-called testing are failing miserably outside the rarified atmosphere of the testing environment?


Let me give you some real world examples. If the vendors in question have 100% efficacy, why are we still regularly finding malware on endpoints where these incumbents are installed?

The data below shows where Cylance’s proof of concepts detected malware when the incumbent with the supposed ‘100% efficacy’ rating did not. This anonymous data is representative from July to September 2016.  

Here is what the numbers say:


Figure 1: Real World POC Data From July to September 2016

So ask yourself this: are these vendors really getting 100% in your environment? When was the last time you had to re-image a system protected by these legacy endpoint protection products? While the vendor is paying for this 1970s-type malware hygiene testing that gives them ‘perfect’ scores, you as the user are paying ransom to the creators of the ransomware they missed.

AGAIN I say: Test For Yourself. Res Ipsa Loquitur! (The thing speaks for itself!)

Know the Truth!
Test For Yourself!


Chad Skipper,
VP Product Testing & Certifications, Cylance