whole product testing / whole attack testing: two sides of the same coin

(this has actually been sitting in the drafts pile for a while)

several weeks ago brian krebs asked me for my thoughts about a new NSS Labs test which i was happy to provide. aside from the fairly predictable spike in traffic that resulted from brian's subsequent article, i also found an unexpected treat in my inbox - NSS' rick moy reached out to me so that we could discuss a few things. now, this post isn't intended to bring up anything anyone did or said in private email, but i do want to thank rick because if he hadn't prompted me to engage on this topic further than i had already done with brian i might not have gotten to this point in understanding the duality of whole product testing vs. whole attack testing.

the idea of whole attack testing came to me as i was contemplating what little information i could find freely available about NSS' most recent test of how well anti-malware products prevented drive-by downloads. the name was a play on the the term "who product testing" that has become so popular in anti-malware testing circles, and which NSS themselves are big proponents of (after all, they try to use their own attempts at whole product testing as a differentiating factor to set themselves apart from other testing organizations). i thought it was a natural extension of the line of reasoning that brought us whole product testing, maybe even the logical conclusion of that line of reasoning. after all, if you're only testing against part of a multi-stage attack it seems like you encounter similar biases that you get when you only test part of a multi-layer product.

but now that i've thought about it some more i realize what that really means. you literally can't have whole product testing without whole attack testing. if you only test one part of a multi-stage attack then you're only testing the parts of a product that are designed to deal with that particular stage of attack. if you're testing exclusively with neutered or otherwise benign exploits, for example, then it doesn't matter if you're testing  entire products against those exploits, only the parts of the products designed to deal with exploits will be capable of raising an alert. as a result, the biases you encounter aren't just similar to the ones you encounter in testing individual parts of a product, they're identical - because you will effectively still only be testing individual parts of the product.

in order to get a true measure of how well a product prevents compromise in the face of real attacks it is necessary to test the whole product against real whole attacks. as difficult, expensive, and painful as that may be, if we really want to produce tests that tell the laymen what they are expecting tests to tell them, this is what has to be done.