certainty of bias

with all the talk of anti-malware testing recently, one of the subjects that has come up is the appearance of bias. more specifically, when vendors are involved in any way with the execution of the test, the development of the testing methodology, or even if they just funded the test, the suspicion is that those vendors have somehow influenced the test in subtle or not so subtle ways so that they'll come out better in the end.

this is why testing organizations often strive to maintain independence from vendors - so that they can avoid the appearance that their tests have been unduly biased by an association with a vendor.

so there seems to be a certain amount of irony at play here because for all NSS Labs' claims of independence, in fact of being one of the only truly independent testing organizations out there, vikram phatak (either CTO or CEO of NSS, depending on whether you go by how he's referenced in the media which says the former or by his linkedin profile which says the latter) sure seems cavalier about throwing all the bias minimizing benefits of independence away by openly declaring favourites, in public, on camera.

in the source boston anti-malware panel video that i've referenced a few times already, at approximately 55:30 minutes in, andrew jaquith asks what he characterizes as a "naughty question" - he asked the panelists to list their 2 most favourite and their 2 least favourite products. the fact that the panelists were told from the start that it was a "naughty question" should have been a great big neon sign of a clue that answering the question would cause trouble.

to his credit, av-comparatives' peter stelzhammer refused, without hesitation, to answer the question in the spirit it had been asked. in fact, he refused twice. it was a textbook example of how an independent tester should respond to that sort of question. mario vuksan of reversing labs didn't do too bad a job either - he beat around the bush a bit but the gist of it was that he couldn't give a real answer because he didn't have enough recent data about the full capabilities of all the products. vikram phatak, in contrast with the other 2 panelists, wasted no time nor minced any words in his answer - his favourites are trend and mcafee, and his least favourites are panda and avg.

it's hard to imagine that a testing organization lead by someone with such clear and unambiguous favourites, not to mention an apparent disregard for the consequences that picking favourites has, would manage to develop a testing methodology that doesn't express that favouritism, that bias in some subtle way. you might then expect that trend and mcafee do well in NSS tests (trend does, apparently). you might also expect avg and panda to do poorly - and given both avg and panda lent their support to sophos in requesting a review of an NSS report (PDF) that seems like a safe bet too.

at this point you could be thinking that vikram was just expressing the ceiling and floor of the results of recent testing and poorly wording it as 'favourites'. unfortunately, that interpretation doesn't quite explain why he later compares avg and panda to cheapskate american football owners (see the same video starting at approximately minute 87:00). there's no question in my mind that his bias against avg and panda goes beyond simple test performance explanations.

so the question i put to you the reader is this: how can party A be expected to judge party B in a fair, unbiased, and impartial way when party A has such clear animosity towards party B?

AMTSO revisited

in keeping with my habit of subtracting 1 from infinity, i've found that kevin townsend's recent post about AMTSO is just calling out for correction. the challenge, it seems, is where to start.

there are two primary questions he tries to answer, the first of which being whether or not AMTSO is serious about improving anti-malware testing. he concludes that the answer is no and holds up VB100 as an example to support this conclusion because he thinks if they were serious about improving anti-malware testing they'd ban the VB100 test on the basis that it misleads the public. of course, in reality AMTSO hasn't done that because they can't. they don't have the power to do so. AMTSO is trying to create improved standards but they don't have the authority to enforce those standards. all they can do is use indirect means to exert pressure on testing organizations to improve their methods. anyone reading the AMTSO FAQs, especially the one about their charter, can plainly see that enforcement is neither explicitly mentioned nor implicitly referred to.

additionally, VB100 isn't actually a test in it's own right. it's a certification/award based on a subset of the results of a larger comparative review. kevin should have known this had he bothered to read the first sentence of the VB100 test procedures. furthermore, the VB100 award itself is not misleading. this is one instance when we really ought to be shooting the messenger because the misleading is being done by vendor marketing departments which happen to use the VB100 award in an incredibly superficial and manipulative way (and frankly there's little that testers could do to stop that). there actually isn't all that much wrong with the VB100 award except that, due to it's being based on the WildList, it has lost most of it's relevance. that said, certifications in general have limited relevance as all they really do is help to establish a lower bound on quality. a lot of people don't understand this or even what VB100 really is, but that lack of understanding is hardly the fault of virus bulletin, especially when most people don't even go to the virus bulletin site to learn what the results mean.

before i move on to the second of kevin's main questions, i'd like to take an aside and look at something he wrote about the WildList itself:

this latency means that, almost by definition, the Wild List includes little, if any, of the biggest threat to end-users: zero-day malware

what kevin and many before him have failed to realize is that the reason zero-day malware is as big a threat as it is today is because it's competition has been largely eliminated thanks to a focus on the WildList. without that we'd still be getting compromised by the exact same malware year after year because the stuff that was demonstrably in the wild wouldn't be getting higher priority treatment.

the second of kevin's main questions had to do with whose interests AMTSO was really serving. he concludes that they serve the vendors interests rather than the end user's based on his assumptions about the reason behind their adherence to the rule about not creating new malware, but also based on his decision to buy into the spin being put forth by NSS Labs CEO rick moy.

for starters i can't believe that after all these years people are still getting bent out of shape or trying to read ulterior motives into the 'no malware creation' rule. it's one of the oldest and most fundamental ethical principles in the anti-malware community. if people found out that the CDC was creating new diseases they'd be up in arms - worse still if one of those new diseases got out (something which has happened in the malware world) - but in the case of the anti-malware community outsiders assume it's because everyone in the anti-malware community has vendor ties and the vendors don't want to look bad in tests. we're not talking about the 'we mostly frown on malware except when it's useful to us' community, it's the ANTI-malware community. you can't really call yourself anti-X if you go around making X's. that would just make you a hypocrite.

furthermore, and speaking directly to the following rather uninformed rhetorical question kevin puts forward about the 'no malware creation' rule:

Why not? How can you test the true heuristic behavioral capabilities of an AV product without testing it against a brand new sample that you absolutely know it has never experienced before?

it is already possible to test anti-malware products against malware they've never seen before without creating new malware. it's been possible for a long, long time. it's called retrospective testing, and anyone with familiarity with tests (not even testing issues, just the tests themselves) knows that retrospective tests make vendors look terrible. detection rates around 40% used to be the norm but in more recent times they've edged up closer to 50%. there are still some below the 40% mark, though, and even some below the 20% mark.

as for believing the NSS spin and using the 2 test reviews available (yes, what an incredibly small sample size) on the AMTSO site to try and support that view i offer the following support to the counter-argument: retrospective tests have made far more vendors look far worse than NSS's test did and no one is challenging the results. no one is using the AMTSO review process to dismiss those tests, as kevin phrased it. how can the conspiracy theory about protectionism in AMTSO be true if nobody is trying to discredit tests that are even more damning and damaging than NSS'? if you think it's because the tests are too obscure, think again - they're produced by some of the top names in independent anti-malware testing (even NSS' own vikram phatak recognized one of the organizations as being independent in that video i've referenced twice before), who also happen to be a part of AMTSO.

kevin believed the spin, i suspect, because he was predisposed to. previous posts on his blog show an existing bias against AMTSO, apparently due in part to the involvement of vendors. there is a very sad tendency in the general security community to not be able to see past a person's vendor affiliations. apparently people think that if you work for a vendor you're nothing more than a mouthpiece for your employer and that the entire company is one big unified collective entity. no attempts are made to distinguish between divisions within the company and recognize the huge difference between the technical people and the business people in those companies (you never know what you're going to get when the two overlap, though - just compare frisk with eugene kaspersky). it's not the business people, the marketroids (so called in order to distinguish them from actual human beings), or the HR departments participating in AMTSO, it's the researchers.

one final idea that kevin put forward in his post is the importance of the user - going so far as to suggest that users should be part of AMTSO, that users determine whether tests are any good, etc. i don't know what on earth he was thinking, but the layman hasn't the tools to divine good science from bad. most users (and i say this as a user myself) haven't got the first clue about what makes a good test or a biased test. in fact most users don't read or interact with tests at all. the only thing they know about tests is what they read in vendor marketing material (usually on the cover of the box), which, as previously mentioned, neither testers nor AMTSO have any control over. i really don't see what users could bring to AMTSO, but i do see something that AMTSO could bring to users - that being tools for to help them understand the tests, to put them in the proper perspective, and yes to also be able to pick out the good ones from the bad.

to be perfectly honest, i understand some of the indignation kevin is directing towards testers and by extension AMTSO, but i think it's misdirected. for most people, marketing is the first and sometimes only voice they hear with respect to security. it's marketing's job to distort and/or omit facts in order to make the company and it's product/service look good. of course marketing does this at the behest of management, of CEOs and shareholders, and people whose concerns are business and profit rather than the good of the user. none of that has anything to do with testing or AMTSO, however.

sample this

i seem to be getting a lot of mileage out of the NSS folks lately. one might think it's because i have some sort of grudge or maybe that i'm one of the av industry's 'pets' and just defending my master (like i did back when i accused a big chuck of the av industry of being snake oil peddlars, i'm sure). the truth is, however, that NSS Labs have simply given me a wealth of material to work with. please rick or vikram, talk some more.

in a previous post i linked to this video taken at the source boston conference this year. it's a panel discussion with vikram phatak, peter stelzhammer, and mario vuksan, with andrew jaquith acting as moderator. i'm going to be referring to it again in this post - specifically at about 25:00 minutes into the video where vikram talks about exploits.

now i'm going to be generous and overlook the fact that he confuses exploits with vulnerabilities at one point. it's vulnerabilities that are the flaws, exploits are what takes advantage of the flaws. the truth is public speaking isn't necessarily easy and things can come out wrong (or not come out at all), as i found out not too long ago, so it seems reasonable enough to me that he actually meant vulnerability when he said "it's an exploit, it's a flaw in IE or in Firefox or..."

what doesn't seem reasonable to me, however, is the idea he was actually talking about vulnerabilities rather than exploits for that entire part of the discussion. i'm fairly certain he was talking about exploits for the most part, so when he trotted out this idea that there are no samples when it comes to exploits my immediate reaction was "that's bullshit".

we exist in a universe of cause and effect. well, ok, determinism breaks down at the quantum level but at the macroscopic level determinism is the overriding trend - and even more so when we're talking about digital computers because they are deterministic finite state machines. the exploitation of a vulnerability doesn't just happen by magic, this isn't some sort of cyber-voodoo, there is a causal agent, an actor, some chunk of data that the computer receives which, when the computer operates on it, results in the vulnerability being exploited.

and guess what that chunk of data is - that's right, it's the supposedly fictional exploit sample. it doesn't matter that it's not an *.exe or that it may not even be saved to disk, that just makes collecting it more challenging. when asked for a sample, that chunk of data (along with a way to replay it's delivery to the vulnerable system/subsystem) is what is being requested - an example of the agent which causes the vulnerability to be exploited.

at this point some of you might be thinking about the fact that an exploit for a particular vulnerability can take many different forms and might even be thinking that that would make the previously described sample worthless. 'many forms' is not a new concept in the anti-malware field. the fact that an exploit can come in many forms is just another aspect of polymorphism. that's probably not a word that gets associated with exploits very much, in part because of a rather arbitrary convention of differentiating between 'code' and data. code is data, however, and any data the computer bases a decision on is for all intents and purposes the same as an instruction (the data dictates what the computer will do next) and can be considered a kind of code. so while exploits may not superficially resemble other polymorphic things from the past like polymorphic viruses or server-side polymorphic trojans, exploits are programs of a sort and multiple different exploits for the same vulnerability are functionally equivalent programs - which, when considered in that light, actually does bear some conceptual similarity to server-side polymorphic trojans. as such an exploit sample doesn't become worthless just because there are other forms it could take, it just means that detecting that one form represented by the sample is only the beginning.

NSS Labs vs. AMTSO?

further to my previous post about NSS Labs report as blogged about by brian krebs, while reading brian's post i found myself wondering how he got such a negative impression of the AMTSO.

the AMTSO is a bunch of people from the anti-malware community trying to hammer out some better approaches to testing anti-malware products because frankly there are a lot of bad tests out there. the people in question come from independent testing organizations like av-comparatives and av-test.org (as well as others i'm less familiar with), and also from anti-malware vendors. many of them are employed by vendors precisely because that's one of the primary places where one with expertise in this field would find employment. often-times those people will even represent their employers but that doesn't make it a "vendor-driven consortium" as claimed by rick moy here, nor any kind of "cartel" as claimed by vik phatak in this video (at approximately 84:15).

in fact, NSS folks themselves were members of AMTSO at one point but pulled out for some reason. surely if they were members they thought there was merit to the project so why pull out? one possible reason might be sour grapes (certainly in keeping with the verbiage) over a review of one of their tests which you can read here. while the review was requested by particular vendors (which one might interpret as being vendor driven - though reviews are ancillary to the formation of the testing standards) it was carried out by people from other organizations, two of which were testing organizations themselves.

there are always many sides to any story, but painting AMTSO as a vendor driven consortium (or worse, a cartel) in the current climate, where vendors are a favoured whipping boy of the general security community, effectively demonizes AMTSO and everything they're working towards. that rubs off on people and shows through when brian used words like "cantankerous" and "cobbled" (not to mention the #fail hashtag he used when promoting the post on twitter).

the story brian got for the split between NSS and AMTSO was that AMTSO favoured fairness to vendors over representing reality. one of AMTSO's goals is to (as much as possible) eliminate bias from tests. eliminating bias is fair to vendors and results in a more accurate representation of reality at the same time. do NSS and AMTSO have a fundamental difference of opinion over what constitutes bias? did NSS decide to take their ball and go home when they couldn't get their way? after reading AMTSO's review of the NSS report as well as listening to vik phatak on the subject (he makes an oblique reference at approximately 72:35 in the video) and reading rick moy's blog post, i'm left to conclude that this is the case.

if you're setting out to test a product's ability to protect users from "socially engineered malware" (NSS' wording, not mine) then it stands to reason that you should include in that test the technologies that help to block the social engineering itself (e.i. the spam filter) in addition to the technology that blocks the malware. NSS could easily change their wording to allow themselves a more narrowly defined scope, but that would be moving away from the sort of whole product testing that NSS evangelizes. alternatively NSS could accept that spam filters, though often taken for granted and even dismissed as a non sequitur in malware testing, do have protective value and by ignoring anti-spam technologies NSS introduced bias into their report. NSS has done neither of these things but instead parted ways with AMTSO and now try to discredit the organization (with some apparent success).

despite AMTSO's efforts there is never going to be a perfect test. there is never going to be a complete absence of bias or a complete absence of measurement error. there will always be some grounds upon which any test can be criticized and testing organizations who can't take criticism would do well and would serve the public's interests if they got over themselves and learned to take that criticism as an opportunity to improve - because they should always be improving.