imperva's anti-virus study is garbage

Enough is enough! I have had it with these motherf#$%ing flakes on this motherf#$%ing train of thought - (what i imagine samuel l. jackson might say if he were following this nonsense about imperva)

in case you are unfamiliar, imperva (a security vendor of some sort) commissioned a bunch of students from the technion-israel institute of technology to perform an evaluation of the efficacy of anti-virus (all anti-virus as a whole, apparently, rather than comparing them to each other) by uploading 82 found samples to virustotal. yes, you read that right, it's another virustotal-based test.

these days i have a number of alternative avenues to express myself that i didn't have when this blog was still young, and that can often sate my need to express my feelings on some topic. i can make snide comments on twitter, or even parody tweets from a satirical twitter account. in fact i can even make memes about it. unfortunately none of that has proven sufficient in this case because the hits just keep coming.

you see, imperva keeps shopping this quackery out to more and more media outlets where it gets gobbled up and regurgitated uncritically by writers/editors (who really ought to know better if reporting on this sort of topic is part of their actual job) and thus gets put in front of more and more eyeballs of those who realistically can't know better. along the way it can even collect somewhat supporting voices from venerated members of the security community like robert david graham

or wim remes

let me be clear, however - this is all wrong. as has been repeated over and over again, virustotal is for testing samples, not anti-malware. they say it themselves in their about page

The reason is that using VirusTotal for antivirus testing is a bad idea.

and

BAD IDEA: VirusTotal for antivirus/URL scanner testing

those statements alone should be enough but, because virustotal later talks specifically about comparative tests, imperva (and others) have tried to argue that imperva's test is OK because it doesn't compare products to each other. however...

VirusTotal's antivirus engines are commandline versions, so depending on the product, they will not behave exactly the same as the desktop versions: for instance, desktop solutions may use techniques based on behavioural analysis and count with personal firewalls that may decrease entry points and mitigate propagation, etc.

this makes it pretty clear that the product a customer installs is very much a different thing from the program that virustotal uses - they will in most cases behave very differently and so the results that virustotal spits out cannot be considered representative of what actual users of anti-malware products will experience.

(ironically,  a product that appears to fare best in a virustotal-based test may actually be the worst because a higher focus on the type of static (often signature-based) detection that virustotal best measures could be to cover for a weakness in (or absence of) more generic/dynamic detection capabilities.)

but don't just take my word for it, let's hear from a couple of people who actually work at virustotal

yes, that's right, imperva's study is a joke. this shouldn't be surprising to long time readers of this blog since when i first wrote about this problem four years ago the first reason i gave for why you might want to avoid performing virustotal-based tests was that those of us who know better will laugh at you. i'm sure a number of people are laughing at imperva's gross incompetence (hanlon's razor makes me choose this explanation over the more sinister alternatives) but i'm afraid i can't consider the mess they're making to be a laughing matter.

promulgating ignorance in a security context has the potential to do real harm, and that is where i draw the line. that's why i'm writing this, that's why the title gets straight to the point, and that's why i'm going to start naming some names of people/organizations who have helped make this mess and who really ought to have known better. imperva has behaved like a dung beetle, persistently rolling this turd around, but somehow it keeps getting bigger like some katamari damacy of bullshit, and i think it's important to see the scale and scope of it and hold the people responsible accountable. it's worth noting, however, that somewhere deep down someone at imperva must also have seen the potential for their message to do harm - that's why the caveat that they weren't advising eliminating AV was added (as an apparent afterthought).

a non-exhaustive list of people/orgs who really should have known better, tried harder, and ought to be held to account for this growing mess is as follows:

• the technion-israel institute of technology because they presumably performed the test so that the test could have an air of objectivity it would otherwise lack if imperva performed it themselves
• the students who actually did the work because apparently no one bothered to read virustotal's about page
• the students' supervisor(s) who also failed the reading test and whose job it is to look out both for their students and the reputation of the learning institution that employs them
• imperva for coming up with this cockamamie idea in the first place and for later trying to defend it
• rob rachwald who posted the defense of the test on imperva's blog
• tal be'ery who helped with the defense of imperva's test
• amichai shulman (imperva cto) who is quoted far and wide by uncritical reporters
• clinton karr who seems to have written the press release that many others have simply regurgitated as their own reporting, perhaps as a result of getting that press release onto reuters
• the register which took the bait imperva offered
• richard chirgwin who posted an uncritical account of imperva's research
• techworld which took the bait imperva offered
• john e dunn who posted an uncritical account of imperva's research
• cso online which unthinkingly reprinted the techworld article
• the new york times (yes, that's right) who once again took the bait imperva offered
• nicole perlroth who posted an uncritical account of imperva's research
• the wallstreet journal who apparently uncritically summarize the work of others
• michael hickens who ran a summary of nicole perlroth's article as if it was gold inside an aggregate of summarized stories
• dark reading which posted a regurgitation of imperva's press release - is that what dark reading is supposed to be for?
• ihotdesk which took the bait imperva offered
• steven gaskill who posted an uncritical account of imperva's research
• pc magazine which took the bait imperva offered
• damon poeter who posted an uncritical account of imperva's research
• business computing world which took the bait imperva offered
• christian harris who posted a regurgitation of imperva's press release
• security bistro which took the bait imperva offered
• anthony m freed who posted an uncritical account of imperva's research

(i'm aware there's a lot more than this that you can simply find by googling sentences from the press release, i wish i had the time to make this list exhaustive - that said: reuters, the new york times, and the wallstreet journal... that definitely caught a lot of eyeballs)

now, perhaps you're thinking i'm being too hard on the journalists involved here. after all, they aren't experts. frankly, however, they don't have to be experts to see what's wrong with this test. if you're the type of reporter who reports on this type of technology then you should already know about virustotal and about how it can and can't be used. this isn't rocket science, or even some obscure nuance that only matters every 5th wednesday - not in the context of reporting on this subject. this is something reporters covering security technology ought to know. it's table stakes. you need to be this tall to get on the ride.

perhaps you think i'm being too hard on the students and their supervisor(s)? but this is academia we're talking about. they're expected to do their research, and i don't just mean the experimental research, i mean looking up and reading about the issues involved in designing and performing tests on anti-malware products. and their supeverisor(s) should have made sure they were doing their due diligence in this regard. frankly, in my time i've seen lone rank amateurs perform better tests than this with fewer resources. this is not acceptable academic performance.

and as for imperva themselves, well... if you intend to occupy part of the security industry that hopes to steal some of the AV industry's market, then you better know this stuff like it's the back of your hand. the institutional incompetence going all the way up the chain of command to the chief technology officer is astonishing and i'm surprised they managed to find someone with too many dollars and too little sense to give them funding, but i guess p.t. barnum was right about there being one born every minute.

imperva - do yourselves a favour and put a stop to this mess before it gets any bigger. you can't defend this junk computer science, the truth will eventually come out (it seems to have already started). you can't sweep it under the rug either, you've let things get too out of hand. the kind of smear campaign you're currently running was already attempted by the whitelisting industry years before you, and while that industry itself is still around and may even still be pumping out this same kind of junk, it didn't stop them from drifting back into obscurity. the way i see it the only way you can move forward sustainably is

1. admit your error
2. publicly retract your study
3. reach out to the journalists whose reputations have been tarnished by listening to you and apologize
4. assist the students you dragged into this in learning the error of the experimental methodology they followed (you can probably find a lot of good info either on or linked to from the anti-malware testing blog)
5. start over with a more intelligent methodology and try to make your case again with valid data

and if you can't manage to follow these steps then i'll be glad to watch you fade away or get swallowed up in a few years time, because the kind of incompetence you've been proudly displaying so far is not the path to success.

it's not a detection rate

(this has been stewing for a little while now)

look, i realize that virustotal performs a series of detection tests in order to get it's results. i also know that it expresses those results as something that looks a lot like a rate. but as much as you may want to, as much intuitive sense as it may make, don't mistake those results for a detection rate.

first, let's deal with the elephant in the room. in the anti-malware world, the term "detection rate" has already been used for something else. traditionally a detection rate is arrived at by testing an anti-malware product against many malware samples in order to see how good the product is at detecting malware. this is what detection rate has meant for somewhere on the order of two decades, and it bares little relation to what virustotal does.

the inverse of that method, to test a single sample against many anti-malware products in order to see how bad anti-malware technology is, is in theory similar to what virustotal does but it differs in two very important ways:

1. the purpose of virustotal's test is to give the user an indication of whether the submitted sample is likely to be malware rather than to determine how bad anti-malware technology is
2. the way virustotal uses anti-malware products in it's testing does not lend itself to an accurate determination of whether a particular product can detect a particular sample (as i've discussed over and over again, and as hispasec themselves mention)

i mulled over the idea that even though it's not a detection rate, and it's not even an inverse detection rate, maybe it could at least represent the lower bound of an inverse detection rate since the most obvious methodological problems (if we were trying to interpret virustotal results the way some people seem to want) would lead to detection capabilities being under-reported. even if it could be called that, however, an inverse detection rate lower bound is so abstract that there's little benefit in using the term with the general population.

i'm tempted to suggest people just call the results a score, but when you compare "virustotal results" with "virustotal score" you realize you're not really saving much more than 2 keystrokes by using that term. there's no intuitive meaning to be had in either of them. it seems the kind of intuitive meaning that people hope to convey by calling it a detection rate simply can't be had.

as such, whether you're an recognized and well regarded expert like dancho danchev who explicitly calls it a detection rate, or brian krebs who tries to infer meaning about anti-malware technology by looking at the results, or even if you just someone who relies on such experts for their accurate analyses and informed opinions - remember that virustotal is for testing samples not anti-malware products. don't try to infer meaning from virustotal test results about something virustotal isn't meant to test. you will most likely fail.

anti-virus usage fail

how do you top a virustotal usage fail? you attempt to commit virustotal usage fail but use samples that aren't even malware in the first place...

that's what john strand did in a video embedded in daniel miessler's post "Metasploit 3.2 Makes AV Look Silly | DiD is the Only Answer"...

the premise may sound ok - you can create executables using metasploit that won't be detected by the (perhaps severely) cut-down versions of anti-virus products that is used by virustotal... supposedly this points to a problem with anti-virus technology in general, but ask yourself this - is it really a problem that you can create executables that virustotal can't detect? and if so, why?

is the output from metasploit malware? if it is then hdmoore is a bad man and should be stopped (and it's not like we can't find him)... i don't see a lot of people calling him a bad man, though, or suggesting he needs to be stopped - that says to me that metasploit and it's output are not malware or at least occupy that gray area between malware and benign software... as such, if these things aren't malware then why are we expecting anti-malware programs to detect them? the av world knows that if it ain't bad then you shouldn't be catching it and if metasploit output is bad then why aren't we doing more about it?...

if it's not malware then stop expecting anti-malware apps to do anything about it... if it is malware then go after the root cause (isn't that bejtlich is always talking about being the more effective strategy dealing with the malware problem?)...

ultimately, the video and the post it's in make a good point - that you shouldn't be relying on av as your sole form of protection - but the argument would be better served by using a legitimate av failure as an example instead... better still would be to take an approach that doesn't seek to tear down a largely successful anti-malware control in the first place - you can promote defense in depth without erroneously trying to make av look like it's useless... tearing av down does not actually promote defense in depth, it promotes the search for the next great anti-malware hope that we can replace av with - and that's not going to help anybody because all preventative measures (even whatever people replace av with) fail...

virustotal usage FAIL

from rich mogull's post There Are No Trusted Sites: Paris Hilton Edition:

The best part? Only 12 of 37 tested AV vendors catch the trojan. All of who that give me crap for hammering on AV can go away now.

yes, boys and girls - in spite of my prior warning on the matter, in spite of didier stevens' thoughtful post on the matter, and in spite of hispasec's own post on the matter, people still don't get that virustotal is for testing suspected malware not anti-malware...

it doesn't matter if your sample size is 1 or 1000, using bad virustotal results to bolster the argument that av sucks (when it's well known that virustotal's results don't/won't match av user experience) is a big fat FAIL...

rich isn't the only one failing here, though, he's just the most recent example... 'incident handlers' at the internet storm center do this on a regular basis, as do quite a few others...

the devil's in the details folks, start paying attention... since the detective capabilities displayed in the context of virustotal do not represent the real detective capabilities of the products used by virustotal, what point can there really be to posting the detection rates (as dancho danchev likes to call them)? that's right, basically none - not only do they bear no relationship to what is conventionally thought of as detection rates, but also they are NOT accurate...

now repeat after me: virustotal is for testing suspected malware, not anti-malware...

why perform virustotal-based av tests?

probably most people with any familiarity with the anti-malware field has heard of virustotal.com - for those that haven't, it's an online service that runs the commandline version of a collection of av scanners against submitted samples in order to perform static analysis on them and determine if they're known malware (or perhaps close enough to known malware to be picked up by static heuristics)...

as has been well stated by others - virustotal is for testing samples not for testing anti-malware software... unfortunately that doesn't seem to stop everyone and their grandmother (apparently) from performing comparative and/or effectiveness testing on anti-virus products using the virustotal service...

there are a number of reasons why you shouldn't perform av tests using virustotal, including:

• those of us who know better will laugh at you - no, seriously, we will
• virustotal doesn't (can't) include the full detective capabilities of the av products they're using and therefore tests based on their service misrepresent the effectiveness of those products
• even the people who run virustotal say such testing methodologies are bogus right on their own site
• retrospective testing already provides results on the effectiveness of av products against new/unknown malware (and it already makes av look pretty bad)

those seem like pretty compelling reasons not to do this kind of testing and yet the practice persists... here are a couple reasons why people might still do it regardless of the reasons not to:

• it costs too much to do things the right way (proper testing takes a lot of work, time, and resources)
• people are lazy and virustotal can appear to be a convenient short-cut to getting things done, even though it's really just a short-cut to irrelevance
• some people seem to be genuinely ignorant of the irrevocable problems with test designs that use virustotal to compare scanners or gauge anti-virus technology
• related to ignorance but on a grander scale, some people may simply not be capable of designing a scanner test that even flirts with validity, nevermind one that is actually somewhat valid
• there are some pervasive misconceptions about anti-virus products/technology/vendors/industry that some people have an irrational need to affirm

of course that's just for individual people, when a security company (or worse, an anti-malware company) uses virustotal for quick and dirty av testing then it raises serious questions about the competency of that company's staff... although i have hinted before at the connection between innovation and not being constrained by the 'this is the way we've always done things' mentality, that isn't a license for the security industry to throw scientific rigor out the door...

no more non-distribution for virustotal

the hispasec folks announced thursday that they would be getting rid of the virustotal feature that allowed people to scan samples without submitting those samples to anti-malware companies when appropriate...

i think that's great news, because it makes it that much harder for the bad guys to use the feature maliciously... now the hispasec folks may want to believe that wasn't really an issue but lets face it, if there are still script kiddies asking for malware on usenet (and it does happen from time to time) then it stands to reason that some malware writers would misuse virustotal to test their malware... they're people, not perfectly rational automatons, they do things that don't always make sense to us - sometimes because what they know and what we know doesn't always completely overlap, sometimes just because they're cheap and lazy (and aren't we all to some extent?)...

the article also points out that there's a technical reason why using virustotal to test malware with isn't useful for the malware writers - the results don't tell the whole story with regards to the detectability of the malware sample by the various anti-malware products used because virustotal only uses the scanner portion of those products... now, while non-scanner based anti-malware components in anti-malware suites are nothing new, they did fall out of favour for quite a while... thankfully, as the hispasec folks point out, they're coming back and that renders virustotal results meaningless to a malware writer... what they don't point out, however, is that it also renders virustotal results meaningless for the complete tools anti-virus critics who use virustotal results against new malware as proof that anti-malware companies aren't able to stay ahead of the threat...

there are 2 things in the article that baffle me though... first is how anyone could think they're protecting confidential documents by using the non-distribute feature (hello, you're submitting the file to a 3rd party, confidentiality is lost at that moment, not when that 3rd party shares copies with other 3rd parties)... the second is why legitimate anti-malware vendors would need to use virustotal with the non-distribute feature when the bad guys have shown us it's easy enough to setup your own multi-product scanning system...

but all in all i always think it's good when we find ways to be less helpful to the bad guys...