What Constitutes a False Positive?

False positives are currently one of the more challenging problems faced by the antivirus (AV) industry today. Unfortunately, some AV vendors are choosing to capitalize on this and promote themselves by publicly shaming other vendors based upon their definition of 'false positives'.

Spending time and effort focusing on what <insert Product X> does not do rather than factually informing would-be customers what the product does do spreads fear, uncertainty, and doubt (FUD), and is more likely to damage the vendor’s reputation than instill confidence in buyers.

So what is a false positive? The SANS Institute, which provides IT industry information, security training, certification, and research, provides this simple definition:

"A false positive is any normal or expected behavior that is identified as anomalous or malicious."

So if a file is detected as 'malicious' by an AV product but has legitimate uses, this means it has been misclassified by the product’s detection system. This can have serious security implications for your organization.

The Failure of Blacklisting Technology

Personally speaking, I still carry the battle scars of the ‘false positive’ wars, having seen this tired spectacle play out so many times in a former life while managing one of the largest single-vendor AV implementations in the world. One such experience included the harmless generic host process Svchost.exe being misclassified as malicious twice in the same month, plus queue BSODs, .DAT rollbacks, genuine panic, and a process of definition file generalization by the vendor involved.

The limitations and failures of blacklisting technologies such as those used by signature-based legacy AV companies are well documented, therefore I don’t feel the need to throw more fuel on the FUD fire by going over them in this blog. Instead, I’d like to provide a brief overview of false positives and their relevance, given that there is a newer approach to thwarting malware-related incidents, using the power of artificial intelligence and machine learning to protect the endpoint.

Let’s start with a simple Q&A we might conduct to classify a file or program as either ‘good’ or ‘bad’:

Question 1: Is the Microsoft-developed tool PsExec a legitimate application?
Answer 1: Yes

Question 2: Because PsExec is a powerful tool used by system admins, can it be used for nefarious purposes?
Answer 2: Yes

Question 3: Do the bad guys know this and could they exploit PsExec to launch attacks?
Answer 3: Yes

Question 4: Is PsExec a tool that should always be classified as 'good'?
Answer 4: Absolutely not

We would therefore classify PsExec as a Potentially Unwanted Program (PUP) - Remote Access Tool due to the file having characteristics which are commonly found in tools developed for malicious purposes rather than good.

Many legacy AV vendors do not classify PsExec as a PUP due to the inflexibility inherent in signature-based systems: a definition means an absolute classification, therefore the file is always blocked completely. Completely blocking a file under all circumstances, as signature-based AV systems often do, may frustrate legitimate users and lead to them circumnavigating security to use the tool, which would lead to far more serious consequences.

Of course, PsExec is just one of the many examples I could have used. When Cylance’sThreatZero team works with our customers to define policies, the example above almost always comes up. Most of the time, IT power-users require access to PsExec while most other departments do not; these users are commonly the weak link in the security myriad.

The lesson to be learned is that regardless of how good the perimeter defenses are, non-IT personnel are far more likely than skilled IT users to click on a link, download a malicious attachment, fall for a phishing scam or unknowingly giving out their company user credentials, which opens up the entire organization to attack. That makes it imperative that an endpoint protection product should not block legitimate files and cause the chain reaction of the user disabling the security system to use a PUP, and by doing so, exposing themselves and their company to a slew of security risks.

Why You Should Always Test for Yourself

The bottom line is that if someone is online, researching endpoint security products and attempting to compare AV vendors, they almost certainly know they have an endpoint security problem. Customers both present and potential need to feel safe exploring their options with their trusted vendors, rather than feeling that those vendors are more focused on tearing down their competitors than they are on helping their customers.

If you’re out there looking for answers and no longer know who to trust, we have this to say: don’t trust anyone but your own good judgment. Don’t fall for clever marketing or the impossible ‘100% efficacy’ scores routinely produced by pay-to-play testing houses. Don’t even trust us!

Rather than focusing on what the market says an AV product can or can’t do, we always encourage you to ‘Test For Yourselff.’ We believe that the only security tests that matter are those done in the real world, in real environments. Only then will you know the truth.