why perform virustotal-based av tests?

probably most people with any familiarity with the anti-malware field has heard of virustotal.com - for those that haven't, it's an online service that runs the commandline version of a collection of av scanners against submitted samples in order to perform static analysis on them and determine if they're known malware (or perhaps close enough to known malware to be picked up by static heuristics)...

as has been well stated by others - virustotal is for testing samples not for testing anti-malware software... unfortunately that doesn't seem to stop everyone and their grandmother (apparently) from performing comparative and/or effectiveness testing on anti-virus products using the virustotal service...

there are a number of reasons why you shouldn't perform av tests using virustotal, including:

• those of us who know better will laugh at you - no, seriously, we will
• virustotal doesn't (can't) include the full detective capabilities of the av products they're using and therefore tests based on their service misrepresent the effectiveness of those products
• even the people who run virustotal say such testing methodologies are bogus right on their own site
• retrospective testing already provides results on the effectiveness of av products against new/unknown malware (and it already makes av look pretty bad)

those seem like pretty compelling reasons not to do this kind of testing and yet the practice persists... here are a couple reasons why people might still do it regardless of the reasons not to:

• it costs too much to do things the right way (proper testing takes a lot of work, time, and resources)
• people are lazy and virustotal can appear to be a convenient short-cut to getting things done, even though it's really just a short-cut to irrelevance
• some people seem to be genuinely ignorant of the irrevocable problems with test designs that use virustotal to compare scanners or gauge anti-virus technology
• related to ignorance but on a grander scale, some people may simply not be capable of designing a scanner test that even flirts with validity, nevermind one that is actually somewhat valid
• there are some pervasive misconceptions about anti-virus products/technology/vendors/industry that some people have an irrational need to affirm

of course that's just for individual people, when a security company (or worse, an anti-malware company) uses virustotal for quick and dirty av testing then it raises serious questions about the competency of that company's staff... although i have hinted before at the connection between innovation and not being constrained by the 'this is the way we've always done things' mentality, that isn't a license for the security industry to throw scientific rigor out the door...