whose yer members

so apparently i've been defending AMTSO a bunch lately. well, i think it's time to restore a little bit of balance to the force and show that i can be an equal opportunity critic.

you see, there's something i just don't understand about AMTSO. something that just doesn't make sense. i know it's members are people - it's members (sorry if the assumption here is wrong, i just can't tell which individuals are in it and which aren't for the most part) even admit that AMTSO is made of people - so why does the members page list companies instead of people almost exclusively?

way to present yourselves as sell-outs, guys and gals. it's no wonder people worry about AMTSO being a tool of the 'av industrial complex' with a membership that's so impersonal and corporate.

it's not as though you can't list the person along with their organizational affiliation - that's how the WildList reporters have been listed for as long as i can remember.

recently a joint post by several members was published that spoke about (among other things) transparency - so where's the transparency in AMTSO's membership? who exactly are you and why are most of you hiding behind vendor names when AMTSO is supposedly not a tool of the vendors?

i mentioned in a previous post that i mostly agreed that AMTSO wasn't responsible for the misconceptions people have about them, but on this one point in particular i think they actually are responsible. membership is basically secret - although i can find out names of who's on what committee, the union of those committees does not appear to represent the entire membership - and with a secret membership i'm left wondering if AMTSO qualifies as a not-so-secret secret society (least secret secret society evar!).

i really think AMTSO is sending the wrong message by listing companies instead of people as members of the organization. i think the members should stand and be counted, take ownership of their participation and the views they bring to the table, even if those views happen to not be in their employers' best interests. also if the membership page showed people's names it would be harder for people to fraudulently claim membership (it may not be a problem yet, but as AMTSO gains more mindshare it will be).

i see a standards organization

ed moyle prefaced his recent post about how AMTSO is perceived by the industry by saying that he really didn't want to continue talking about this subject (he has, after all, penned a number of posts about AMTSO recently). having seen this blog go more or less dark in the past, i have no qualms about following whatever path my interests and creativity take. if the subject doesn't bore me, i see no reason not to write about it.

and the subject of how AMTSO is perceived has a few interesting bits to it, i think. first and foremost, while david harley may bend to the notion that the use of the word "standard" in AMTSO's name might mislead people, i think the use of the word "standard" is entirely appropriate. if people are mislead into thinking AMTSO is anything like ISO, it is actually ISO and organizations like it that have mislead people into thinking enforcement has anything to do with standards. a few pertinent definitions for standard:

noun:  a basis for comparison; a reference point against which other things can be evaluated

noun:  the ideal in terms of which something can be judged ("They live by the standards of their community")

developing a basis upon which anti-malware tests can be evaluated or an ideal which testers should strive for is precisely what AMTSO is about. it is not about enforcement - following the standards is entirely voluntary. if enforcement were on the table at all then testers wouldn't participate for 2 reasons:

1. many testing organizations were (and perhaps still are) too far away from the ideal. signing up for obligations at a time when one cannot meet them makes little or no sense. with voluntary standards the obligation, instead, is to keep improving and moving closer to the ideal.
2. enforcement would mean that the standards were actually rules, and nobody thinks vendors should be involved in making rules for testers.

but apparently, as ed moyle has pointed out, the security industry perceives AMTSO as something different from what it actually is. now when you get right down to it, when people's perceptions don't match reality it's because they lack knowledge of that reality. if AMTSO were being purposefully deceptive or secretive (basically acting to deprive/deny people of that knowledge) then one might legitimately be able to blame them for the false perception problem. ed doubts that AMTSO is to blame, and so do i (mostly), but then who is to blame? let's look at an example from ed's own post. near the end he constructs some hypothetical situations where a vendor might challenge a test and then have the AMTSO review board composed entirely of employees of that vendor. he asks where the line is - i can tell him where it is, it's in the 'fine' manual. the document describing the analysis of review process states on the second page that review committee members that are employed by the challenging party must recuse themselves from participating in the analysis. this document is freely available, easy to find, well labeled and not hidden in any way. anyone who wants to learn the answer of where the line is drawn can easily do so by downloading the document and reading it - so when they don't, when they make assumptions or treat it like a questions that needs answering instead of one that's already been answered, that's really on the person themselves.

page three of that same document has something for ed as well. he asks the following:

So if it’s not the role of AMTSO to standardize, it’s also clearly not their role to accredit.  But aren’t they doing just that?

the answer is no, they are not. AMTSO makes no judgments or endorsements of reviewers or products. for an AMTSO member to suggest otherwise is considered misrepresentation. the analysis of reviews is just that, analysis. the output serves as an interpretive aid for individuals wishing to know how close to ideal a particular review was. the review analysis that ed looked at as an example (the analysis of NSS' review) was actually quite close to the ideal (though apparently not close enough for NSS' liking). only 2 real problems were found, and they've been described by members as 'minor'. in the analysis itself, the explanation for the first one even goes so far as to say that NSS' test is still better than most out there in spite of the problem. ed moyle interprets this as a pass/fail sort of judgment and i suppose in the strictest sense the NSS test did fail to reach the ideal but it's hard to say the analysis of their test is calling it a failure when it's clearly stating it's better than most out there.

of course as an interpretive aid, the reader is free to pick and choose the ideals that are important to him/her - as ed does when he discusses testing features individually. the ideal that the testing industry is trying to move towards is whole product testing. the reason is because it's understood that different products have very different technologies and thus have different ways of stopping different threats. it's especially difficult for testers to devise testing methodologies that aren't biased in favour of certain technologies. if i test feature X and product A blocks 5 things but product B only blocks 3 things, how can i possibly show that a test of feature Y shows that product B blocks everything it missed in the feature X test and product A blocks nothing because it doesn't even have feature Y? and if i can't show that then is what i'm presenting really relevant? isn't the important thing that B blocks the threats in one way or another? does it really matter if it uses feature X or Y to do it? current opinion in the anti-malware community is no, it shouldn't matter, which is why whole product testing is becoming the standard. NSS themselves bang the drum of whole product testing pretty loudly, so it seems ironic to me that they failed to test the whole product (seemingly testing everything but the spam filter).

of course, as interpretive aids go, even AMTSO's analysis isn't necessarily perfect. i say this because point 7 of the NSS review analysis is interpreted by ed one way, and a different way by myself. i don't know if ed's interpretation is correct or if the analysis is implicitly assuming domain knowledge of NSS' practices. ed quotes the following from the analysis:

Does the conclusion reflect the stated purpose? No. The report’s Executive Summary states that test’s purpose was to determine the protection of the products tested against socially-engineered malware only. Later in the report (Section 4 -product assessments) it says: “Products that earn a caution rating from NSS Labs should not be short-listed or renewed.” This is clearly a conclusion that you can’t make out of the detection for socially‐engineered malware only, as the products have other layers of protection that the test did not evaluate.

ed's interpretation is that the conclusion supposedly didn't reflect the stated purpose simply because NSS failed to include spam filters in their test. my interpretation differs in part because i know that NSS breaks malware down into 2 categories and "socially engineered malware" is only one of those categories - so making purchasing recommendations on the basis of the results of the socially engineered malware test alone seems like a premature conclusion to me. i suspect that the spam filters were only 1 of many features that weren't tested since the other malware category NSS recognizes involves drive-by downloads and other sorts of malware that don't involve user intervention. but clearly, someone who doesn't know what i know may interpret the meaning of the analysis in an entirely different way than i did.

i understand why ed feels that perception is important, but the key to making perception match reality is knowledge and understanding and there's only so much anyone can do to impart those things on others. people have to be willing to look past their preconceptions and actually acquire new knowledge and understanding.

AMTSO revisited

in keeping with my habit of subtracting 1 from infinity, i've found that kevin townsend's recent post about AMTSO is just calling out for correction. the challenge, it seems, is where to start.

there are two primary questions he tries to answer, the first of which being whether or not AMTSO is serious about improving anti-malware testing. he concludes that the answer is no and holds up VB100 as an example to support this conclusion because he thinks if they were serious about improving anti-malware testing they'd ban the VB100 test on the basis that it misleads the public. of course, in reality AMTSO hasn't done that because they can't. they don't have the power to do so. AMTSO is trying to create improved standards but they don't have the authority to enforce those standards. all they can do is use indirect means to exert pressure on testing organizations to improve their methods. anyone reading the AMTSO FAQs, especially the one about their charter, can plainly see that enforcement is neither explicitly mentioned nor implicitly referred to.

additionally, VB100 isn't actually a test in it's own right. it's a certification/award based on a subset of the results of a larger comparative review. kevin should have known this had he bothered to read the first sentence of the VB100 test procedures. furthermore, the VB100 award itself is not misleading. this is one instance when we really ought to be shooting the messenger because the misleading is being done by vendor marketing departments which happen to use the VB100 award in an incredibly superficial and manipulative way (and frankly there's little that testers could do to stop that). there actually isn't all that much wrong with the VB100 award except that, due to it's being based on the WildList, it has lost most of it's relevance. that said, certifications in general have limited relevance as all they really do is help to establish a lower bound on quality. a lot of people don't understand this or even what VB100 really is, but that lack of understanding is hardly the fault of virus bulletin, especially when most people don't even go to the virus bulletin site to learn what the results mean.

before i move on to the second of kevin's main questions, i'd like to take an aside and look at something he wrote about the WildList itself:

this latency means that, almost by definition, the Wild List includes little, if any, of the biggest threat to end-users: zero-day malware

what kevin and many before him have failed to realize is that the reason zero-day malware is as big a threat as it is today is because it's competition has been largely eliminated thanks to a focus on the WildList. without that we'd still be getting compromised by the exact same malware year after year because the stuff that was demonstrably in the wild wouldn't be getting higher priority treatment.

the second of kevin's main questions had to do with whose interests AMTSO was really serving. he concludes that they serve the vendors interests rather than the end user's based on his assumptions about the reason behind their adherence to the rule about not creating new malware, but also based on his decision to buy into the spin being put forth by NSS Labs CEO rick moy.

for starters i can't believe that after all these years people are still getting bent out of shape or trying to read ulterior motives into the 'no malware creation' rule. it's one of the oldest and most fundamental ethical principles in the anti-malware community. if people found out that the CDC was creating new diseases they'd be up in arms - worse still if one of those new diseases got out (something which has happened in the malware world) - but in the case of the anti-malware community outsiders assume it's because everyone in the anti-malware community has vendor ties and the vendors don't want to look bad in tests. we're not talking about the 'we mostly frown on malware except when it's useful to us' community, it's the ANTI-malware community. you can't really call yourself anti-X if you go around making X's. that would just make you a hypocrite.

furthermore, and speaking directly to the following rather uninformed rhetorical question kevin puts forward about the 'no malware creation' rule:

Why not? How can you test the true heuristic behavioral capabilities of an AV product without testing it against a brand new sample that you absolutely know it has never experienced before?

it is already possible to test anti-malware products against malware they've never seen before without creating new malware. it's been possible for a long, long time. it's called retrospective testing, and anyone with familiarity with tests (not even testing issues, just the tests themselves) knows that retrospective tests make vendors look terrible. detection rates around 40% used to be the norm but in more recent times they've edged up closer to 50%. there are still some below the 40% mark, though, and even some below the 20% mark.

as for believing the NSS spin and using the 2 test reviews available (yes, what an incredibly small sample size) on the AMTSO site to try and support that view i offer the following support to the counter-argument: retrospective tests have made far more vendors look far worse than NSS's test did and no one is challenging the results. no one is using the AMTSO review process to dismiss those tests, as kevin phrased it. how can the conspiracy theory about protectionism in AMTSO be true if nobody is trying to discredit tests that are even more damning and damaging than NSS'? if you think it's because the tests are too obscure, think again - they're produced by some of the top names in independent anti-malware testing (even NSS' own vikram phatak recognized one of the organizations as being independent in that video i've referenced twice before), who also happen to be a part of AMTSO.

kevin believed the spin, i suspect, because he was predisposed to. previous posts on his blog show an existing bias against AMTSO, apparently due in part to the involvement of vendors. there is a very sad tendency in the general security community to not be able to see past a person's vendor affiliations. apparently people think that if you work for a vendor you're nothing more than a mouthpiece for your employer and that the entire company is one big unified collective entity. no attempts are made to distinguish between divisions within the company and recognize the huge difference between the technical people and the business people in those companies (you never know what you're going to get when the two overlap, though - just compare frisk with eugene kaspersky). it's not the business people, the marketroids (so called in order to distinguish them from actual human beings), or the HR departments participating in AMTSO, it's the researchers.

one final idea that kevin put forward in his post is the importance of the user - going so far as to suggest that users should be part of AMTSO, that users determine whether tests are any good, etc. i don't know what on earth he was thinking, but the layman hasn't the tools to divine good science from bad. most users (and i say this as a user myself) haven't got the first clue about what makes a good test or a biased test. in fact most users don't read or interact with tests at all. the only thing they know about tests is what they read in vendor marketing material (usually on the cover of the box), which, as previously mentioned, neither testers nor AMTSO have any control over. i really don't see what users could bring to AMTSO, but i do see something that AMTSO could bring to users - that being tools for to help them understand the tests, to put them in the proper perspective, and yes to also be able to pick out the good ones from the bad.

to be perfectly honest, i understand some of the indignation kevin is directing towards testers and by extension AMTSO, but i think it's misdirected. for most people, marketing is the first and sometimes only voice they hear with respect to security. it's marketing's job to distort and/or omit facts in order to make the company and it's product/service look good. of course marketing does this at the behest of management, of CEOs and shareholders, and people whose concerns are business and profit rather than the good of the user. none of that has anything to do with testing or AMTSO, however.

NSS Labs vs. AMTSO?

further to my previous post about NSS Labs report as blogged about by brian krebs, while reading brian's post i found myself wondering how he got such a negative impression of the AMTSO.

the AMTSO is a bunch of people from the anti-malware community trying to hammer out some better approaches to testing anti-malware products because frankly there are a lot of bad tests out there. the people in question come from independent testing organizations like av-comparatives and av-test.org (as well as others i'm less familiar with), and also from anti-malware vendors. many of them are employed by vendors precisely because that's one of the primary places where one with expertise in this field would find employment. often-times those people will even represent their employers but that doesn't make it a "vendor-driven consortium" as claimed by rick moy here, nor any kind of "cartel" as claimed by vik phatak in this video (at approximately 84:15).

in fact, NSS folks themselves were members of AMTSO at one point but pulled out for some reason. surely if they were members they thought there was merit to the project so why pull out? one possible reason might be sour grapes (certainly in keeping with the verbiage) over a review of one of their tests which you can read here. while the review was requested by particular vendors (which one might interpret as being vendor driven - though reviews are ancillary to the formation of the testing standards) it was carried out by people from other organizations, two of which were testing organizations themselves.

there are always many sides to any story, but painting AMTSO as a vendor driven consortium (or worse, a cartel) in the current climate, where vendors are a favoured whipping boy of the general security community, effectively demonizes AMTSO and everything they're working towards. that rubs off on people and shows through when brian used words like "cantankerous" and "cobbled" (not to mention the #fail hashtag he used when promoting the post on twitter).

the story brian got for the split between NSS and AMTSO was that AMTSO favoured fairness to vendors over representing reality. one of AMTSO's goals is to (as much as possible) eliminate bias from tests. eliminating bias is fair to vendors and results in a more accurate representation of reality at the same time. do NSS and AMTSO have a fundamental difference of opinion over what constitutes bias? did NSS decide to take their ball and go home when they couldn't get their way? after reading AMTSO's review of the NSS report as well as listening to vik phatak on the subject (he makes an oblique reference at approximately 72:35 in the video) and reading rick moy's blog post, i'm left to conclude that this is the case.

if you're setting out to test a product's ability to protect users from "socially engineered malware" (NSS' wording, not mine) then it stands to reason that you should include in that test the technologies that help to block the social engineering itself (e.i. the spam filter) in addition to the technology that blocks the malware. NSS could easily change their wording to allow themselves a more narrowly defined scope, but that would be moving away from the sort of whole product testing that NSS evangelizes. alternatively NSS could accept that spam filters, though often taken for granted and even dismissed as a non sequitur in malware testing, do have protective value and by ignoring anti-spam technologies NSS introduced bias into their report. NSS has done neither of these things but instead parted ways with AMTSO and now try to discredit the organization (with some apparent success).

despite AMTSO's efforts there is never going to be a perfect test. there is never going to be a complete absence of bias or a complete absence of measurement error. there will always be some grounds upon which any test can be criticized and testing organizations who can't take criticism would do well and would serve the public's interests if they got over themselves and learned to take that criticism as an opportunity to improve - because they should always be improving.