further to my previous post about NSS Labs report as blogged about by brian krebs, while reading brian's post i found myself wondering how he got such a negative impression of the AMTSO.
the AMTSO is a bunch of people from the anti-malware community trying to hammer out some better approaches to testing anti-malware products because frankly there are a lot of bad tests out there. the people in question come from independent testing organizations like av-comparatives and av-test.org (as well as others i'm less familiar with), and also from anti-malware vendors. many of them are employed by vendors precisely because that's one of the primary places where one with expertise in this field would find employment. often-times those people will even represent their employers but that doesn't make it a "vendor-driven consortium" as claimed by rick moy here, nor any kind of "cartel" as claimed by vik phatak in this video (at approximately 84:15).
in fact, NSS folks themselves were members of AMTSO at one point but pulled out for some reason. surely if they were members they thought there was merit to the project so why pull out? one possible reason might be sour grapes (certainly in keeping with the verbiage) over a review of one of their tests which you can read here. while the review was requested by particular vendors (which one might interpret as being vendor driven - though reviews are ancillary to the formation of the testing standards) it was carried out by people from other organizations, two of which were testing organizations themselves.
there are always many sides to any story, but painting AMTSO as a vendor driven consortium (or worse, a cartel) in the current climate, where vendors are a favoured whipping boy of the general security community, effectively demonizes AMTSO and everything they're working towards. that rubs off on people and shows through when brian used words like "cantankerous" and "cobbled" (not to mention the #fail hashtag he used when promoting the post on twitter).
the story brian got for the split between NSS and AMTSO was that AMTSO favoured fairness to vendors over representing reality. one of AMTSO's goals is to (as much as possible) eliminate bias from tests. eliminating bias is fair to vendors and results in a more accurate representation of reality at the same time. do NSS and AMTSO have a fundamental difference of opinion over what constitutes bias? did NSS decide to take their ball and go home when they couldn't get their way? after reading AMTSO's review of the NSS report as well as listening to vik phatak on the subject (he makes an oblique reference at approximately 72:35 in the video) and reading rick moy's blog post, i'm left to conclude that this is the case.
if you're setting out to test a product's ability to protect users from "socially engineered malware" (NSS' wording, not mine) then it stands to reason that you should include in that test the technologies that help to block the social engineering itself (e.i. the spam filter) in addition to the technology that blocks the malware. NSS could easily change their wording to allow themselves a more narrowly defined scope, but that would be moving away from the sort of whole product testing that NSS evangelizes. alternatively NSS could accept that spam filters, though often taken for granted and even dismissed as a non sequitur in malware testing, do have protective value and by ignoring anti-spam technologies NSS introduced bias into their report. NSS has done neither of these things but instead parted ways with AMTSO and now try to discredit the organization (with some apparent success).
despite AMTSO's efforts there is never going to be a perfect test. there is never going to be a complete absence of bias or a complete absence of measurement error. there will always be some grounds upon which any test can be criticized and testing organizations who can't take criticism would do well and would serve the public's interests if they got over themselves and learned to take that criticism as an opportunity to improve - because they should always be improving.
0 comments:
Post a Comment