virustotal usage FAIL

from rich mogull's post There Are No Trusted Sites: Paris Hilton Edition:

The best part? Only 12 of 37 tested AV vendors catch the trojan. All of who that give me crap for hammering on AV can go away now.

yes, boys and girls - in spite of my prior warning on the matter, in spite of didier stevens' thoughtful post on the matter, and in spite of hispasec's own post on the matter, people still don't get that virustotal is for testing suspected malware not anti-malware...

it doesn't matter if your sample size is 1 or 1000, using bad virustotal results to bolster the argument that av sucks (when it's well known that virustotal's results don't/won't match av user experience) is a big fat FAIL...

rich isn't the only one failing here, though, he's just the most recent example... 'incident handlers' at the internet storm center do this on a regular basis, as do quite a few others...

the devil's in the details folks, start paying attention... since the detective capabilities displayed in the context of virustotal do not represent the real detective capabilities of the products used by virustotal, what point can there really be to posting the detection rates (as dancho danchev likes to call them)? that's right, basically none - not only do they bear no relationship to what is conventionally thought of as detection rates, but also they are NOT accurate...

now repeat after me: virustotal is for testing suspected malware, not anti-malware...

the ongoing n.runs saga

you my recall my previous post about n.runs... well, it seems i wasn't the only one who saw FUD as ryan permeh wrote on mcafee's blog about what n.runs was saying specifically about mcafee... now it seems that thierry zoller of n.runs has responded to the mcafee post, or at least he tried to...

he didn't do a particularly good job of it, however, as despite explaining that the graphs come from data gleaned from publicly available 3rd party vulnerability catalogs (something that was clear from their original press release and not in need of additional explanation), he didn't address the issue that ryan raised about not being able to verify n.runs' figures when looking at the raw data and instead mistakenly or intentionally mislead the reader into thinking that ryan was looking to verify the 800 figure (which was n.runs' own) when it was clear from his post that he was only trying to verify the figures that applied directly to mcafee and that were supposed to come have come from 3rd party databases...

thierry also denied making the claim and/or believing that running av makes you less secure in spite of the fact that an n.runs slide deck i posted about last november makes exactly that claim...

additionally, where ryan claimed there was no evidence of these vulnerabilities in mcafee's product being exploited in the wild thierry responds by saying that it's because of the way the vulnerabilities are reported - apparently ignoring the fact that being used in the wild means there should be malware samples implementing the exploit(s) and that mcafee should have seen some of these by now...

one thing that ryan didn't really bring up and so wasn't addressed by thierry is the absurdity of aggregating the vulnerability count across an entire industry (where the 800 vulnerabilities figure is supposed to come from)... it's not an actionable metric, it doesn't say anything about any particular product or vendor within that industry, and only serves to scare people... this is the kind of marketing that john mcafee (long absent from the company bearing his name) used back in the days of the michelangelo virus (have i just invoked the anti-malware industry's version of godwin's law?)... even if there technically are that many vulnerabilities across the product lines of the entire set of vendors in the av industry, it's an entirely pointless measurement...

and while we're on the subject of marketing, am i the only one whose noticed that dancho danchev has put rather a lot of effort into providing a platform for n.runs to spread their marketing message from? one might wonder if he were still as 'independent' as he claims to be, though a more reasonable explanation might be that his rather obvious anti-av leanings (he's frequently made disparaging insinuations on his blog in the past) have been kicked up a notch so that, given the obvious ammo this 800 vulnerabilities claim could provide to an anti-av agenda, he either doesn't care or isn't aware that it's a marketing message he's helping to spread... given a more recent post where he misleads by misusing terms that someone in his position has no legitimate excuse to mix up (samples != variants != families != signatures, so counts of one can't be compared to counts of another), this latter explanation seems all the more plausible...

no such thing as trusted sites anymore

darn, rich mogull beat me to the publish button - that will teach me to put work and home repair first... i think i'm going to post what i had anyways, though, because mine isn't exactly the same and frankly when it comes to principles that i anticipate needing to hammer home repeatedly it's nice when i have them all in one place... also, actual incidents like yahoo mail serving malware and not cleaning up promptly sort of drive this point home better than xss vulnerabilities found on security vendors sites; i'm a security blogger and even i rarely visit security vendor sites so i don't imagine the average person does much either - yahoo mail, on the other hand gets LOTS of traffic from average folks... so here's what i had in my draft folder with the addition of some links i was waiting to find time to look up...

once upon a time there used to be this piece of advice about online security that said don't go to dodgy sites and you should be just fine... the principle behind it is that if you won't get compromised by malicious web content if you only ever run trustworthy web content and if you only ever got to trustworthy sites then trustworthy web content should be all your browser is exposed to...

internet explorer's security zone model has this very principle in mind, some sites are trustworthy and some aren't and those that aren't don't get to take advantage of as much rich web-based functionality as those that are...

even the mighty noscript firefox plugin depends on this basic premise to protect those firefox users who use noscript (in fact, adding a site to noscript's whitelist is in many ways the same as adding a site to ie's trusted sites zone - only easier, more convenient, and more intuitive)...

the principle makes sense and the advice (even in absence of the technologies that try to make it automatic) has been one of the more successful bits of security know-how at gaining widespread adoption... unfortunately the principle is falling apart because malicious web content is increasingly finding it's way on to what would otherwise be considered trustworthy sites... dancho danchev often informs his readers of instances of web sites being directly compromised to serve malware, and sandi hardmeier regularly informs her readers of instances of sites serving malware indirectly by virtue of malvertizements (malicious advertisements) infiltrating the 3rd party ad networks the site owners use...

when (as this zdnet article suggests) such compromises are up 400% over last year, and when affected sites include such well known internet properties as yahoo mail, cnn (among others), or the superbowl then it begs the question "is there any such thing as trusted sites anymore?" and the answer i think has to be either "no" or "not for much longer"...

now, of course the efficacy of the tools like noscript aren't quite as affected as a basic careful internet user would be since the tools can look at where the content is coming from rather than just what the current site is, but it really makes you wonder about those oh-so-clever users who go around not using any anti-malware software and thinking they're fine because they don't go to dodgy sites...

prudence alone isn't really enough anymore, you need good tools to help control what web content is allowed to run (i.e. some kind of whitelist like noscript), in what environment it runs (i.e. some kind of web sandbox whether it's multiple browsers/browser profiles, or sandboxing software like sandboxie, or even a full virtual machine like vmware), and to detect when something slips through the cracks (i.e. a scanner, preferably one that implements an lsp) to help prevent it from stealing data you enter and/or using your browsing session as a staging point for an attack on other things on your network (like your router) or the internet at large...