whole product testing / whole attack testing: two sides of the same coin

(this has actually been sitting in the drafts pile for a while)

several weeks ago brian krebs asked me for my thoughts about a new NSS Labs test which i was happy to provide. aside from the fairly predictable spike in traffic that resulted from brian's subsequent article, i also found an unexpected treat in my inbox - NSS' rick moy reached out to me so that we could discuss a few things. now, this post isn't intended to bring up anything anyone did or said in private email, but i do want to thank rick because if he hadn't prompted me to engage on this topic further than i had already done with brian i might not have gotten to this point in understanding the duality of whole product testing vs. whole attack testing.

the idea of whole attack testing came to me as i was contemplating what little information i could find freely available about NSS' most recent test of how well anti-malware products prevented drive-by downloads. the name was a play on the the term "who product testing" that has become so popular in anti-malware testing circles, and which NSS themselves are big proponents of (after all, they try to use their own attempts at whole product testing as a differentiating factor to set themselves apart from other testing organizations). i thought it was a natural extension of the line of reasoning that brought us whole product testing, maybe even the logical conclusion of that line of reasoning. after all, if you're only testing against part of a multi-stage attack it seems like you encounter similar biases that you get when you only test part of a multi-layer product.

but now that i've thought about it some more i realize what that really means. you literally can't have whole product testing without whole attack testing. if you only test one part of a multi-stage attack then you're only testing the parts of a product that are designed to deal with that particular stage of attack. if you're testing exclusively with neutered or otherwise benign exploits, for example, then it doesn't matter if you're testing  entire products against those exploits, only the parts of the products designed to deal with exploits will be capable of raising an alert. as a result, the biases you encounter aren't just similar to the ones you encounter in testing individual parts of a product, they're identical - because you will effectively still only be testing individual parts of the product.

in order to get a true measure of how well a product prevents compromise in the face of real attacks it is necessary to test the whole product against real whole attacks. as difficult, expensive, and painful as that may be, if we really want to produce tests that tell the laymen what they are expecting tests to tell them, this is what has to be done.

numbers, context, and background

one of the things i've come across while reading various sources is an attempt to pin down an intended target nation for the stuxnet worm based on prevalence data. the theory goes something like 'since nation X is where most instances of stuxnet are found, therefore nation X was the intended target (because obviously more work was put into spreading it there)'.

this theory has some problems, however. first and foremost is that not all the numbers agree. while we have symantec saying that ~60% is in Iran, we also have eset saying that ~60 are in the US just 3 days earlier. they can't both be right - or can they? and if they are, what are the implications for the targeted nation theory?

as is always the case when there are contradictory numbers, we have to look at how those numbers were arrived at. in fact, even when there aren't contradictory numbers, we should still be paying close attention to how those numbers were arrived at.

close examination of vikram thakur's post on the symantec site suggests that there number represents actual infected machines trying to connect to their C&C server (on top of everything else, stuxnet is also a botnet) during a 3 day period between july 19 and july 22. they were able to gather this data because they redirected the domains hosting the C&C servers to themselves so it seems like it would be a pretty accurate snapshot of the pool of infected machines at a particular point in time.

eset's numbers in david harley's post came from their installed clients throughout the world. their cloud-based technology reported the instances - however, since stuxnet employs stealth it's more likely that rather than reporting infected machines (where it would be active and hidden) it's actually reporting infected USB drives. it could also be reporting both if eset's products can see through the stealth, but the key point is that eset's numbers almost certainly include infected USB drives while symantec's do not. the USB numbers are important because that's how this worm spreads and if one were going to work on targeting a particular nation, spreading infected USB drives in that nation would be the way to do it.

furthermore eset's numbers appear to be from the time detection for the worm was added until the time the statistic was reported, rather than just the 3 day period covered by symatec's figures. this means that eset's figures represent a measurement of how many instances of the worm there were over it's detected lifetime to that point, while symantec's figures represent a measurement of how many infected machines remained at that point. this is important because by the time symatec started collecting it's data, negative population controls had already been in effect for some time.

controls which, like worms they intend to control, are not necessarily uniformly effective across the entire globe. some products have greater market share in some regions that they do in others, and the dominant product in certain regions might be poor at controlling particular worms and thus allow those worms greater reproductive advantages in those regions than they might find in others. the presence of population controls like anti-malware software affects both the death and birth rates of worm instances and as anyone who's heard johnny long discuss hackers for charity knows, such controls are not uniformly present or effective across all regions.

there are a actually a variety of other factors, in addition to such controls, that contribute to how well and in what way a worm or virus spreads, as discussed in some detail by jeff kephart, david chess, and steve white in "Computers and Epidemiology". some of these factors, like the degree of connectedness of susceptible hosts (and how often adequate contact between such hosts happens), can be influenced by computing culture, which in turn can be influenced by culture in the more general sense, geopolitical climate, and even socioeconomic considerations. hypothetically speaking, a nation that is cut off from US technologies due to trade sanctions (as metioned by by brian krebs) could well exhibit a higher rate of software sharing as part of 'alternative' procurement techniques and in so doing raise the region above the epidemic threshold for some unspecified worm.


ultimately both symantec and eset could be right since they were measuring different things over different periods of time. what that means for the targeted nation theory is that things aren't as clear-cut as either set of numbers would suggest on their own. what we do know is that stuxnet appeared to enjoy more reproductive success in Iran than elsewhere. whether that's down to purely epidemiological factors or intentional injection of the worm into the local computing population by a malicious actor is unclear, but eset's data would support an argument against the latter option as the effort seems to have been expended elsewhere. on the other hand, if we were to entertain the notion that the US was the target based on the amount of infected materials floating around the computing population, then we are left once again with the conclusion that stuxnet was a failure since in spite of all that effort the prevalence of actual infected machines in the US was minuscule.

i don't think much can be read into the fact that there were more infected machines in Iran than elsewhere since such pockets of infection are actually normal - especially for self-replicating malware that must be spread by physical media. some region had to draw the short straw and this time it was Iran.

NSS Labs vs. AMTSO?

further to my previous post about NSS Labs report as blogged about by brian krebs, while reading brian's post i found myself wondering how he got such a negative impression of the AMTSO.

the AMTSO is a bunch of people from the anti-malware community trying to hammer out some better approaches to testing anti-malware products because frankly there are a lot of bad tests out there. the people in question come from independent testing organizations like av-comparatives and av-test.org (as well as others i'm less familiar with), and also from anti-malware vendors. many of them are employed by vendors precisely because that's one of the primary places where one with expertise in this field would find employment. often-times those people will even represent their employers but that doesn't make it a "vendor-driven consortium" as claimed by rick moy here, nor any kind of "cartel" as claimed by vik phatak in this video (at approximately 84:15).

in fact, NSS folks themselves were members of AMTSO at one point but pulled out for some reason. surely if they were members they thought there was merit to the project so why pull out? one possible reason might be sour grapes (certainly in keeping with the verbiage) over a review of one of their tests which you can read here. while the review was requested by particular vendors (which one might interpret as being vendor driven - though reviews are ancillary to the formation of the testing standards) it was carried out by people from other organizations, two of which were testing organizations themselves.

there are always many sides to any story, but painting AMTSO as a vendor driven consortium (or worse, a cartel) in the current climate, where vendors are a favoured whipping boy of the general security community, effectively demonizes AMTSO and everything they're working towards. that rubs off on people and shows through when brian used words like "cantankerous" and "cobbled" (not to mention the #fail hashtag he used when promoting the post on twitter).

the story brian got for the split between NSS and AMTSO was that AMTSO favoured fairness to vendors over representing reality. one of AMTSO's goals is to (as much as possible) eliminate bias from tests. eliminating bias is fair to vendors and results in a more accurate representation of reality at the same time. do NSS and AMTSO have a fundamental difference of opinion over what constitutes bias? did NSS decide to take their ball and go home when they couldn't get their way? after reading AMTSO's review of the NSS report as well as listening to vik phatak on the subject (he makes an oblique reference at approximately 72:35 in the video) and reading rick moy's blog post, i'm left to conclude that this is the case.

if you're setting out to test a product's ability to protect users from "socially engineered malware" (NSS' wording, not mine) then it stands to reason that you should include in that test the technologies that help to block the social engineering itself (e.i. the spam filter) in addition to the technology that blocks the malware. NSS could easily change their wording to allow themselves a more narrowly defined scope, but that would be moving away from the sort of whole product testing that NSS evangelizes. alternatively NSS could accept that spam filters, though often taken for granted and even dismissed as a non sequitur in malware testing, do have protective value and by ignoring anti-spam technologies NSS introduced bias into their report. NSS has done neither of these things but instead parted ways with AMTSO and now try to discredit the organization (with some apparent success).

despite AMTSO's efforts there is never going to be a perfect test. there is never going to be a complete absence of bias or a complete absence of measurement error. there will always be some grounds upon which any test can be criticized and testing organizations who can't take criticism would do well and would serve the public's interests if they got over themselves and learned to take that criticism as an opportunity to improve - because they should always be improving.

of lines an graphs

i made a comment on brian krebs' recent blog post "Anti-virus is a Poor Substitute for Common Sense" that seems to have gotten a number of negative reactions from other readers. i thought perhaps i should expand on my comment here so as to demonstrate why i wasn't just nitpicking.

first we need an example of a line graph like the one i brian's post:

unlike the graph from the NSS Labs report that brian posted about, i've made it clear where my actual data points are so that you can see clearly what interpolation can do to a graph. while i have absolutely no data points above 70 the line still goes up above 70 and then comes back down, just like the graph in the report in question.

now, if you couldn't see where the data points actually were you might easily think that the data actually showed the value went above 70 and then came back down again. in fact one could easily make the mistake that every point on the graph represented what really happened, even the points for which there is no actual data, because a line graph without the data points clearly denoted implies that there is continuous data along the entire graph. but the reality is our data is not continuous, it's discrete. we take a finite number of measurements at fixed points in time.

even with the data points clearly marked on my own example graph above there is the implication that had i actually measured at some point i would have gotten a value on the line, even though that isn't necessarily true, and in fact there are many many points where that is most certainly false. let's say for example that my graph shows how the detection rate changes over time on a fixed set of 10 items. the detection rate can never be 15%, it's simply not numerically possible even though the line implies it is.

these are some of the problems you encounter using a continuous data visualization for discrete data. i'm not trying to suggest these are huge problems but they are problems because they mislead the reader. these subtle sorts of things are exactly what it means to lie with numbers/statistics. it makes no sense for there to have been periods of time during which the detection rate on a fixed set of malware went down instead of up (go on, pull my other leg) and that was almost certainly the result of interpolation rather than something reflected by actual data.

the misapplication of continuous data visualizations for discrete data is a hallmark of junk science. i don't know if that is representative of the work NSS Labs actually does (i've yet to successfully penetrate their reg-wall in order to see for myself) or if it was just a simple mistake - greater transparency on their part (as apparently discussed by david harley) would allow better peer review and help eliminate uncertainty about such things. as it stands, however, i have an admittedly small reason to be skeptical of them. their own marketing (PDF) bills them as being scientific and expert but a scientist ought to understand his/her data better than this.