whole product testing / whole attack testing: two sides of the same coin

(this has actually been sitting in the drafts pile for a while)

several weeks ago brian krebs asked me for my thoughts about a new NSS Labs test which i was happy to provide. aside from the fairly predictable spike in traffic that resulted from brian's subsequent article, i also found an unexpected treat in my inbox - NSS' rick moy reached out to me so that we could discuss a few things. now, this post isn't intended to bring up anything anyone did or said in private email, but i do want to thank rick because if he hadn't prompted me to engage on this topic further than i had already done with brian i might not have gotten to this point in understanding the duality of whole product testing vs. whole attack testing.

the idea of whole attack testing came to me as i was contemplating what little information i could find freely available about NSS' most recent test of how well anti-malware products prevented drive-by downloads. the name was a play on the the term "who product testing" that has become so popular in anti-malware testing circles, and which NSS themselves are big proponents of (after all, they try to use their own attempts at whole product testing as a differentiating factor to set themselves apart from other testing organizations). i thought it was a natural extension of the line of reasoning that brought us whole product testing, maybe even the logical conclusion of that line of reasoning. after all, if you're only testing against part of a multi-stage attack it seems like you encounter similar biases that you get when you only test part of a multi-layer product.

but now that i've thought about it some more i realize what that really means. you literally can't have whole product testing without whole attack testing. if you only test one part of a multi-stage attack then you're only testing the parts of a product that are designed to deal with that particular stage of attack. if you're testing exclusively with neutered or otherwise benign exploits, for example, then it doesn't matter if you're testing  entire products against those exploits, only the parts of the products designed to deal with exploits will be capable of raising an alert. as a result, the biases you encounter aren't just similar to the ones you encounter in testing individual parts of a product, they're identical - because you will effectively still only be testing individual parts of the product.

in order to get a true measure of how well a product prevents compromise in the face of real attacks it is necessary to test the whole product against real whole attacks. as difficult, expensive, and painful as that may be, if we really want to produce tests that tell the laymen what they are expecting tests to tell them, this is what has to be done.

what is whole product testing?

whole product testing is a form of anti-malware testing that aims to measure the effectiveness of entire anti-malware products rather than just testing the known malware scanner or the heuristic engine within the product.

whole product testing came about in answer to the problem where testing individual parts of an anti-malware product in isolation didn't give an accurate view of how well the product as a whole could perform (for example a threat might slip by the known malware scanner but be picked up by a behavioural technique that wouldn't show on a scanner test) and there was no way to combine the results of tests of the various parts to represent the effectiveness of the whole product. only by giving every part of a product the opportunity to stop a threat can we have an idea of whether that threat would have been stopped on a end user's machine.

because of the wide array of passive and active defenses anti-malware products provide, whole product testing requires each malware sample in the test set to be launched and then the system checked for indications of how well or poorly the anti-malware product stopped the malware sample from compromising the system. after this the system has to be returned to a known-clean state (generally by restoring an image of the drive). this is quite a bit more time and labour intensive than simply running a scanner against a directory full of malware and as a result often requires the size of the test bed to be more modest due to practical considerations (not enough hardware, manpower, etc). while a smaller test bed size may potentially raise questions about statistical significance (depending on how small it is) the ability of the results to map more directly to what an end user can expect makes this type of testing more ideal than earlier testing of a product's individual parts.

back to index