what is whole product testing?

whole product testing is a form of anti-malware testing that aims to measure the effectiveness of entire anti-malware products rather than just testing the known malware scanner or the heuristic engine within the product.

whole product testing came about in answer to the problem where testing individual parts of an anti-malware product in isolation didn't give an accurate view of how well the product as a whole could perform (for example a threat might slip by the known malware scanner but be picked up by a behavioural technique that wouldn't show on a scanner test) and there was no way to combine the results of tests of the various parts to represent the effectiveness of the whole product. only by giving every part of a product the opportunity to stop a threat can we have an idea of whether that threat would have been stopped on a end user's machine.

because of the wide array of passive and active defenses anti-malware products provide, whole product testing requires each malware sample in the test set to be launched and then the system checked for indications of how well or poorly the anti-malware product stopped the malware sample from compromising the system. after this the system has to be returned to a known-clean state (generally by restoring an image of the drive). this is quite a bit more time and labour intensive than simply running a scanner against a directory full of malware and as a result often requires the size of the test bed to be more modest due to practical considerations (not enough hardware, manpower, etc). while a smaller test bed size may potentially raise questions about statistical significance (depending on how small it is) the ability of the results to map more directly to what an end user can expect makes this type of testing more ideal than earlier testing of a product's individual parts.

back to index