you my recall my previous post about n.runs... well, it seems i wasn't the only one who saw FUD as ryan permeh wrote on mcafee's blog about what n.runs was saying specifically about mcafee... now it seems that thierry zoller of n.runs has responded to the mcafee post, or at least he tried to...
he didn't do a particularly good job of it, however, as despite explaining that the graphs come from data gleaned from publicly available 3rd party vulnerability catalogs (something that was clear from their original press release and not in need of additional explanation), he didn't address the issue that ryan raised about not being able to verify n.runs' figures when looking at the raw data and instead mistakenly or intentionally mislead the reader into thinking that ryan was looking to verify the 800 figure (which was n.runs' own) when it was clear from his post that he was only trying to verify the figures that applied directly to mcafee and that were supposed to come have come from 3rd party databases...
thierry also denied making the claim and/or believing that running av makes you less secure in spite of the fact that an n.runs slide deck i posted about last november makes exactly that claim...
additionally, where ryan claimed there was no evidence of these vulnerabilities in mcafee's product being exploited in the wild thierry responds by saying that it's because of the way the vulnerabilities are reported - apparently ignoring the fact that being used in the wild means there should be malware samples implementing the exploit(s) and that mcafee should have seen some of these by now...
one thing that ryan didn't really bring up and so wasn't addressed by thierry is the absurdity of aggregating the vulnerability count across an entire industry (where the 800 vulnerabilities figure is supposed to come from)... it's not an actionable metric, it doesn't say anything about any particular product or vendor within that industry, and only serves to scare people... this is the kind of marketing that john mcafee (long absent from the company bearing his name) used back in the days of the michelangelo virus (have i just invoked the anti-malware industry's version of godwin's law?)... even if there technically are that many vulnerabilities across the product lines of the entire set of vendors in the av industry, it's an entirely pointless measurement...
and while we're on the subject of marketing, am i the only one whose noticed that dancho danchev has put rather a lot of effort into providing a platform for n.runs to spread their marketing message from? one might wonder if he were still as 'independent' as he claims to be, though a more reasonable explanation might be that his rather obvious anti-av leanings (he's frequently made disparaging insinuations on his blog in the past) have been kicked up a notch so that, given the obvious ammo this 800 vulnerabilities claim could provide to an anti-av agenda, he either doesn't care or isn't aware that it's a marketing message he's helping to spread... given a more recent post where he misleads by misusing terms that someone in his position has no legitimate excuse to mix up (samples != variants != families != signatures, so counts of one can't be compared to counts of another), this latter explanation seems all the more plausible...
Showing posts with label nruns. Show all posts
Showing posts with label nruns. Show all posts
Sunday, July 27, 2008
Wednesday, July 09, 2008
lies, damn lies, and statistics
nruns (who, by the way, have a product designed to fix the problem they're playing chicken little over) are reporting that there are 800 vulnerabilities in anti-virus products...
that's a pretty scary number, isn't it? 800... wow...
but wait, that's not 800 vulnerabilities in your av product, or in my av product, that's the aggregate number across more or less all av products... when was the last time anyone reported an undifferentiated vulnerability metric for an entire class of software? when has that even been interesting the the past?
imagine how many vulnerabilities there are in operating systems - not just one operating system, but all of them combined... and while we're imagining this metric lets make sure that we count each distribution of linux independently, just so we can see how high we can make the metric go...
i wonder how many vulnerabilities there are in browsers or media players or word processors... not just individual products but the entire classes... again i'm sure the numbers are big - see how using browsers or media players or word processors become part of the problem? because that is one of the arguments nruns is making - security software increases the attack surface and therefore makes you less secure... something which obviously completely ignores the positive contributions they make to security - you need to examine both positive and negative contributions, folks... you have balanced a checkbook or budget at some point, right?
while we're examining things, lets examine the independent figures that nruns themselves point out... 50 security advisories for the period from 2002 to 2005, and 170 for the period from 2005 to 2007... first of all, this looks like growth but could just as easily be the result of increased focus on finding vulnerabilities in this class of software... second, 220 (170 + 50) is a far cry from 800... i'd be interested in knowing EXACTLY how that number was arrived at (since such knowledge is how one avoids being mislead by bad statistics)... is each vulnerability distinctly different? if 50 applications have the same vulnerability does it get counted 50 times? if the vulnerabilities were found by forensic analysis of binaries, were any vulnerabilities counted multiple times in the same product due to appearing in multiple places in the code? etc...
at a time when treating the av industry as security's whipping boy is at an all time high, such sensationalistic numbers probably make for good marketing for nrun's product, so long as people don't recognize it for the opportunistic FUD that it is... nruns has what sounds a bit like a scanner sandboxing product which on the face of it sounds like it might be a pretty good idea, but they clearly have a vested interest in making the av industry look bad (because that drives demand for their product) so even if you don't believe the 800 vulnerabilities figure is intentionally poorly defined or an example of opportunistic FUD, you should at least recognize that it's a figure that should be taken with more than a few grains of salt...
that's a pretty scary number, isn't it? 800... wow...
but wait, that's not 800 vulnerabilities in your av product, or in my av product, that's the aggregate number across more or less all av products... when was the last time anyone reported an undifferentiated vulnerability metric for an entire class of software? when has that even been interesting the the past?
imagine how many vulnerabilities there are in operating systems - not just one operating system, but all of them combined... and while we're imagining this metric lets make sure that we count each distribution of linux independently, just so we can see how high we can make the metric go...
i wonder how many vulnerabilities there are in browsers or media players or word processors... not just individual products but the entire classes... again i'm sure the numbers are big - see how using browsers or media players or word processors become part of the problem? because that is one of the arguments nruns is making - security software increases the attack surface and therefore makes you less secure... something which obviously completely ignores the positive contributions they make to security - you need to examine both positive and negative contributions, folks... you have balanced a checkbook or budget at some point, right?
while we're examining things, lets examine the independent figures that nruns themselves point out... 50 security advisories for the period from 2002 to 2005, and 170 for the period from 2005 to 2007... first of all, this looks like growth but could just as easily be the result of increased focus on finding vulnerabilities in this class of software... second, 220 (170 + 50) is a far cry from 800... i'd be interested in knowing EXACTLY how that number was arrived at (since such knowledge is how one avoids being mislead by bad statistics)... is each vulnerability distinctly different? if 50 applications have the same vulnerability does it get counted 50 times? if the vulnerabilities were found by forensic analysis of binaries, were any vulnerabilities counted multiple times in the same product due to appearing in multiple places in the code? etc...
at a time when treating the av industry as security's whipping boy is at an all time high, such sensationalistic numbers probably make for good marketing for nrun's product, so long as people don't recognize it for the opportunistic FUD that it is... nruns has what sounds a bit like a scanner sandboxing product which on the face of it sounds like it might be a pretty good idea, but they clearly have a vested interest in making the av industry look bad (because that drives demand for their product) so even if you don't believe the 800 vulnerabilities figure is intentionally poorly defined or an example of opportunistic FUD, you should at least recognize that it's a figure that should be taken with more than a few grains of salt...
Monday, November 19, 2007
defense in depth revisited
so as a result of my previous post on the use of multiple scanners as a supposed form of defense in depth i was pointed towards this set of slides for a presentation by sergio alvarez and thierry zoller at n.runs:
http://www.nruns.com/ps/The_Death_of_AV_Defense_in_Depth-Revisiting_Anti-Virus_Software.pdf
the expectation was that i'd probably agree with it's contents, and some of them i do (ex. some of those vulnerabilities are taking far too long to get fixed) my blog wouldn't be very interesting if all i did was agree with people so thankfully for the reader there's a number of things in the slides i didn't agree with...
the first thing is actually in the filename itself - did anyone catch that death of av reference in there? obviously this is qualified to be specific to a more narrow concept but the frame of reference is still fairly clear...
i'm not going to harp on that too much because that's just a taste of what's in store, actually... the main thrust of the presentation these slides are for seems to be that defense in depth as it's often implemented (ie. with multiple scanners at multiple points in the network) is bad because of all the vulnerabilities in scanners that malware could exploit to do anything from simply bypassing the scanner to actually getting the scanner to execute arbitrary code...
this says to me that instead of recognizing that you can't build defense in depth using multiple instances of essentially the same control, the presenters would rather call the construct a defense in depth failure and blame the failure on the scanners and the people who make them (and make no mistake, there certainly is some room for blame there)... the fact is that it was never defense in depth in the first place and if you want to assign blame, start with the people who think it is defense in depth because they clearly don't understand the concept... in a physical security context, if my only defensive layer is a wall and i decide to add a second wall (and maybe even a third) i add no depth to the defense... an attack that can be stopped by a wall will be stopped by the first one and one that can't be stopped by the first wall probably won't be stopped by subsequent walls...
the slides also have some rather revealing bullet points, such as the one that lists "makes our networks and systems more secure" as a myth... this goes back to the large surface area of potential vulnerabilities; the argument can be made that using such software increases the total number of software vulnerabilities present on a system or in a network - however this is true for each and every piece of software one adds to a system... i've heard this argument used in support of the idea that one shouldn't use any security software and instead rely entirely on system hardening, least privileged usage, etc. but it's no more convincing now than it has been in the past... yes the total number of vulnerabilities increase but there's more to security than raw vulnerability counts... the fact is that although the raw vulnerability count may be higher, the real world risk of something getting through is much lower because of the use of scanners... there aren't legions of malware instances exploiting these scanner vulnerabilities, otherwise we'd have ourselves an optimal malware situation...
another point, and one they repeat (so it must be important) is the paradox that the more you protect yourself the less protected you are... this follows rather directly from the previous point, using multiple scanners is bad because of all the vulnerabilities... the implication, however, is that if using more scanners makes you less secure then using fewer scanners should make you more secure and thus using no scanners would make you most secure... i don't know if that was the intended implication, i'm tempted to give the benefit of the doubt and suggest it wasn't, but the implication remains... again, there's more to security than what they're measuring - they're looking at one part of the change in overall security rather than the net change...
yet another point (well set of points, really) had to do with how av vendors handle vulnerability reports... as i said earlier, some of the vendors are taking far too long but some of the other things they complain about are actually quite reasonable in my eyes and i find myself in agreement with the av vendors... things like not divulging vulnerability details when there is neither a fix nor a work around ready yet (no point in giving out details that only the bad guys can actually act on), condensing bugs when rewriting an area of code (i develop software myself, it makes perfect sense to me that a bunch of related bugs with a single fix would be condensed into one report), fixing bugs silently (above all else, don't help the bad guys), and spamming vulnerability info in order to give credit to researchers (if you're in it for credit or other rewards you'll get no sympathy from me)...
finally, and perhaps the most novice error in the whole presentation was the complaint that scanners shouldn't flag archive files as clean if they're unable to parse them... there is a rather large difference between "no viruses found" and "no viruses present"... scanners do not flag things as clean (at least not unless the vendor is being intellectually dishonest) because a scanner cannot know something is clean - all a scanner can do is flag things that aren't clean and the absence of such flags cannot (and should not, if you know what you're talking about) be interpreted to mean a thing is clean... scanners tell you when they know for sure something is bad, not when they don't know for sure that something isn't bad... if you want the latter behaviour then you want something other than a scanner...
all this being said, though, i did take away one important point from the slides: not only is using multiple scanners not defense in depth, using multiple scanners in the faux defense in depth setup that many people swear by comes with security risks that most would never have considered...
http://www.nruns.com/ps/The_Death_of_AV_Defense_in_Depth-Revisiting_Anti-Virus_Software.pdf
the expectation was that i'd probably agree with it's contents, and some of them i do (ex. some of those vulnerabilities are taking far too long to get fixed) my blog wouldn't be very interesting if all i did was agree with people so thankfully for the reader there's a number of things in the slides i didn't agree with...
the first thing is actually in the filename itself - did anyone catch that death of av reference in there? obviously this is qualified to be specific to a more narrow concept but the frame of reference is still fairly clear...
i'm not going to harp on that too much because that's just a taste of what's in store, actually... the main thrust of the presentation these slides are for seems to be that defense in depth as it's often implemented (ie. with multiple scanners at multiple points in the network) is bad because of all the vulnerabilities in scanners that malware could exploit to do anything from simply bypassing the scanner to actually getting the scanner to execute arbitrary code...
this says to me that instead of recognizing that you can't build defense in depth using multiple instances of essentially the same control, the presenters would rather call the construct a defense in depth failure and blame the failure on the scanners and the people who make them (and make no mistake, there certainly is some room for blame there)... the fact is that it was never defense in depth in the first place and if you want to assign blame, start with the people who think it is defense in depth because they clearly don't understand the concept... in a physical security context, if my only defensive layer is a wall and i decide to add a second wall (and maybe even a third) i add no depth to the defense... an attack that can be stopped by a wall will be stopped by the first one and one that can't be stopped by the first wall probably won't be stopped by subsequent walls...
the slides also have some rather revealing bullet points, such as the one that lists "makes our networks and systems more secure" as a myth... this goes back to the large surface area of potential vulnerabilities; the argument can be made that using such software increases the total number of software vulnerabilities present on a system or in a network - however this is true for each and every piece of software one adds to a system... i've heard this argument used in support of the idea that one shouldn't use any security software and instead rely entirely on system hardening, least privileged usage, etc. but it's no more convincing now than it has been in the past... yes the total number of vulnerabilities increase but there's more to security than raw vulnerability counts... the fact is that although the raw vulnerability count may be higher, the real world risk of something getting through is much lower because of the use of scanners... there aren't legions of malware instances exploiting these scanner vulnerabilities, otherwise we'd have ourselves an optimal malware situation...
another point, and one they repeat (so it must be important) is the paradox that the more you protect yourself the less protected you are... this follows rather directly from the previous point, using multiple scanners is bad because of all the vulnerabilities... the implication, however, is that if using more scanners makes you less secure then using fewer scanners should make you more secure and thus using no scanners would make you most secure... i don't know if that was the intended implication, i'm tempted to give the benefit of the doubt and suggest it wasn't, but the implication remains... again, there's more to security than what they're measuring - they're looking at one part of the change in overall security rather than the net change...
yet another point (well set of points, really) had to do with how av vendors handle vulnerability reports... as i said earlier, some of the vendors are taking far too long but some of the other things they complain about are actually quite reasonable in my eyes and i find myself in agreement with the av vendors... things like not divulging vulnerability details when there is neither a fix nor a work around ready yet (no point in giving out details that only the bad guys can actually act on), condensing bugs when rewriting an area of code (i develop software myself, it makes perfect sense to me that a bunch of related bugs with a single fix would be condensed into one report), fixing bugs silently (above all else, don't help the bad guys), and spamming vulnerability info in order to give credit to researchers (if you're in it for credit or other rewards you'll get no sympathy from me)...
finally, and perhaps the most novice error in the whole presentation was the complaint that scanners shouldn't flag archive files as clean if they're unable to parse them... there is a rather large difference between "no viruses found" and "no viruses present"... scanners do not flag things as clean (at least not unless the vendor is being intellectually dishonest) because a scanner cannot know something is clean - all a scanner can do is flag things that aren't clean and the absence of such flags cannot (and should not, if you know what you're talking about) be interpreted to mean a thing is clean... scanners tell you when they know for sure something is bad, not when they don't know for sure that something isn't bad... if you want the latter behaviour then you want something other than a scanner...
all this being said, though, i did take away one important point from the slides: not only is using multiple scanners not defense in depth, using multiple scanners in the faux defense in depth setup that many people swear by comes with security risks that most would never have considered...
Subscribe to:
Posts (Atom)