as has been well stated by others - virustotal is for testing samples not for testing anti-malware software... unfortunately that doesn't seem to stop everyone and their grandmother (apparently) from performing comparative and/or effectiveness testing on anti-virus products using the virustotal service...
there are a number of reasons why you shouldn't perform av tests using virustotal, including:
- those of us who know better will laugh at you - no, seriously, we will
- virustotal doesn't (can't) include the full detective capabilities of the av products they're using and therefore tests based on their service misrepresent the effectiveness of those products
- even the people who run virustotal say such testing methodologies are bogus right on their own site
- retrospective testing already provides results on the effectiveness of av products against new/unknown malware (and it already makes av look pretty bad)
those seem like pretty compelling reasons not to do this kind of testing and yet the practice persists... here are a couple reasons why people might still do it regardless of the reasons not to:
- it costs too much to do things the right way (proper testing takes a lot of work, time, and resources)
- people are lazy and virustotal can appear to be a convenient short-cut to getting things done, even though it's really just a short-cut to irrelevance
- some people seem to be genuinely ignorant of the irrevocable problems with test designs that use virustotal to compare scanners or gauge anti-virus technology
- related to ignorance but on a grander scale, some people may simply not be capable of designing a scanner test that even flirts with validity, nevermind one that is actually somewhat valid
- there are some pervasive misconceptions about anti-virus products/technology/vendors/industry that some people have an irrational need to affirm
of course that's just for individual people, when a security company (or worse, an anti-malware company) uses virustotal for quick and dirty av testing then it raises serious questions about the competency of that company's staff... although i have hinted before at the connection between innovation and not being constrained by the 'this is the way we've always done things' mentality, that isn't a license for the security industry to throw scientific rigor out the door...