certainty of bias

with all the talk of anti-malware testing recently, one of the subjects that has come up is the appearance of bias. more specifically, when vendors are involved in any way with the execution of the test, the development of the testing methodology, or even if they just funded the test, the suspicion is that those vendors have somehow influenced the test in subtle or not so subtle ways so that they'll come out better in the end.

this is why testing organizations often strive to maintain independence from vendors - so that they can avoid the appearance that their tests have been unduly biased by an association with a vendor.

so there seems to be a certain amount of irony at play here because for all NSS Labs' claims of independence, in fact of being one of the only truly independent testing organizations out there, vikram phatak (either CTO or CEO of NSS, depending on whether you go by how he's referenced in the media which says the former or by his linkedin profile which says the latter) sure seems cavalier about throwing all the bias minimizing benefits of independence away by openly declaring favourites, in public, on camera.

in the source boston anti-malware panel video that i've referenced a few times already, at approximately 55:30 minutes in, andrew jaquith asks what he characterizes as a "naughty question" - he asked the panelists to list their 2 most favourite and their 2 least favourite products. the fact that the panelists were told from the start that it was a "naughty question" should have been a great big neon sign of a clue that answering the question would cause trouble.

to his credit, av-comparatives' peter stelzhammer refused, without hesitation, to answer the question in the spirit it had been asked. in fact, he refused twice. it was a textbook example of how an independent tester should respond to that sort of question. mario vuksan of reversing labs didn't do too bad a job either - he beat around the bush a bit but the gist of it was that he couldn't give a real answer because he didn't have enough recent data about the full capabilities of all the products. vikram phatak, in contrast with the other 2 panelists, wasted no time nor minced any words in his answer - his favourites are trend and mcafee, and his least favourites are panda and avg.

it's hard to imagine that a testing organization lead by someone with such clear and unambiguous favourites, not to mention an apparent disregard for the consequences that picking favourites has, would manage to develop a testing methodology that doesn't express that favouritism, that bias in some subtle way. you might then expect that trend and mcafee do well in NSS tests (trend does, apparently). you might also expect avg and panda to do poorly - and given both avg and panda lent their support to sophos in requesting a review of an NSS report (PDF) that seems like a safe bet too.

at this point you could be thinking that vikram was just expressing the ceiling and floor of the results of recent testing and poorly wording it as 'favourites'. unfortunately, that interpretation doesn't quite explain why he later compares avg and panda to cheapskate american football owners (see the same video starting at approximately minute 87:00). there's no question in my mind that his bias against avg and panda goes beyond simple test performance explanations.

so the question i put to you the reader is this: how can party A be expected to judge party B in a fair, unbiased, and impartial way when party A has such clear animosity towards party B?