Wednesday, July 09, 2008

lies, damn lies, and statistics

nruns (who, by the way, have a product designed to fix the problem they're playing chicken little over) are reporting that there are 800 vulnerabilities in anti-virus products...

that's a pretty scary number, isn't it? 800... wow...

but wait, that's not 800 vulnerabilities in your av product, or in my av product, that's the aggregate number across more or less all av products... when was the last time anyone reported an undifferentiated vulnerability metric for an entire class of software? when has that even been interesting the the past?

imagine how many vulnerabilities there are in operating systems - not just one operating system, but all of them combined... and while we're imagining this metric lets make sure that we count each distribution of linux independently, just so we can see how high we can make the metric go...

i wonder how many vulnerabilities there are in browsers or media players or word processors... not just individual products but the entire classes... again i'm sure the numbers are big - see how using browsers or media players or word processors become part of the problem? because that is one of the arguments nruns is making - security software increases the attack surface and therefore makes you less secure... something which obviously completely ignores the positive contributions they make to security - you need to examine both positive and negative contributions, folks... you have balanced a checkbook or budget at some point, right?

while we're examining things, lets examine the independent figures that nruns themselves point out... 50 security advisories for the period from 2002 to 2005, and 170 for the period from 2005 to 2007... first of all, this looks like growth but could just as easily be the result of increased focus on finding vulnerabilities in this class of software... second, 220 (170 + 50) is a far cry from 800... i'd be interested in knowing EXACTLY how that number was arrived at (since such knowledge is how one avoids being mislead by bad statistics)... is each vulnerability distinctly different? if 50 applications have the same vulnerability does it get counted 50 times? if the vulnerabilities were found by forensic analysis of binaries, were any vulnerabilities counted multiple times in the same product due to appearing in multiple places in the code? etc...

at a time when treating the av industry as security's whipping boy is at an all time high, such sensationalistic numbers probably make for good marketing for nrun's product, so long as people don't recognize it for the opportunistic FUD that it is... nruns has what sounds a bit like a scanner sandboxing product which on the face of it sounds like it might be a pretty good idea, but they clearly have a vested interest in making the av industry look bad (because that drives demand for their product) so even if you don't believe the 800 vulnerabilities figure is intentionally poorly defined or an example of opportunistic FUD, you should at least recognize that it's a figure that should be taken with more than a few grains of salt...

2 comments:

Anonymous said...

http://www.nruns.com/_downloads/PR-08-02_Reaction_to_McAfee_statement.pdf

kurt wismer said...

yet more marketing spin... at the end of the day i see no reason to trust a company selling a solution to problem X when they tell me how bad problem X is... that goes for n.runs and scanner vulnerabilities, it goes for av companies and the virus problem, and it goes for water filter companies and the water quality problem...

whenever a company tells you how badly you need their product, you need to take that with more than a few grains of salt...