why are ethics so undervalued?

why are ethics so undervalued? i honestly don't know the answer to that question but i'd like to explore the topic and explain what i mean.

first i'd like to dispel any fears that i'm about to go on at length about people not understanding the difference between right and wrong - i think most people do understand the difference. that said, i don't think most people appreciate the difference - which is to say i don't think it holds much meaning for people, i don't think it's important to them.

i'll give you an example. not too long ago anton chuvakin posted an article on FUD - specifically one that is, if not an outright endorsement of FUD, at the very least an argument that sometimes it's a good thing. i'm not going to pick too much on the notion of endorsing the use of manipulation in the workplace, what interests me in this discussion was something he wrote in response to a blog post criticizing his stance:

personally, I think that “trumping with ethics” is a low card in intellectual arguments! IMHO it is one step above name calling

i don't think there can be any question that this statement represents a remarkably low valuation of the topic of right and wrong.

by way of contrast, i would place ethical right/wrong one step below technical right/wrong - and those of you who know me know how highly i value technical accuracy (hint: i make enemies simply by correcting people).

so where does such a huge difference in values come from? and what does it mean for the security community that anton is not only not an outlier but in all likelihood far closer to the norm than i am. have we become an "ends justify the means" sort of society? is security as a goal something we need to promote at all costs?

i suppose i need to better understand why it means as much as it does to me, so i guess i've got some soul searching ahead of me, but nowhere in that search do i expect to find why it's so much easier for others to put aside. i don't get many comments on my posts (since normally i know the answer to the question i'm asking) but in this case i'm hoping to hear what others think so please feel free to comment.

samples, variants, and signatures - oh my

last week anton chuvakin characterized the new av-test.org results as "widely ridiculed"... this struck me as odd because i hadn't seen any ridicule so i asked for links... that didn't get me anywhere so i took to googling for the answer and trying to guess what keywords might match this nebulous ridicule anton was speaking of...

now, while i didn't find ridicule, per se, i did find some significant criticism in a thread at wilderssecurity.com by none other than paul wilders himself (i really need to find a way to aggregate web forums so i can actually follow these sorts of discussions instead of finding out about them after the fact)... the criticisms weren't totally unreasonable, in fact i even agree with one (that a good test needs thorough supporting documentation on things like test methodology, sample acquisition, test bed maintenance, etc - how i wish we could see documentation as thorough as the vtc used to put out) but some i think were a little off...

one such criticism had to do with comparing apples to oranges, or in this case the number of samples used in the test to the number of signatures in a product's database - which, when taken with some of the confusion that lead to, makes me think a discussion of what these things are might be in order...

a sample is just a file containing a piece of malware... you distinguish one sample from the next by simple binary comparison - if the files are different then the samples are different, otherwise they're just copies of the same sample...

a variant is a member of a malware family... one variant is distinguished from another by being sufficiently different from that other variant that it makes sense to consider it a different thing, but that is a subjective determination usually made in the context of what makes sense for a particular detection technology... there is no objective, technology-neutral standard by which we can say sample X represents a different variant than sample Y, therefore the distinction between variants is a technology-dependent logical construct... as an example, for a scanning engine that looks at every single byte of a piece of malware it would make sense to be a lot more granular in drawing distinctions between variants than for one that only looks at at most N bytes somewhere within the body of the malware... more generally, technologies that use fundamentally different approaches to malware recognition can likewise result in different delineations between variants... this is one of the reasons why they say each vendor counts viruses/malware differently...

signatures are chunks of data that tell a scanner what a particular piece of malware looks like... they can be a selection of bytes from the actual malware (ie. a scan string), a hash value (for non-parasitic, non-polymorphic malware), a regular expression, or something written in a proprietary virus(malware) description language... the conceptual ideal is that there is a 1-to-1 correlation between signatures and variants but in practice multiple signatures may be required for a single variant (as has sometimes been the case for handling polymorphism) or alternatively a single signature might handle multiple variants or even an entire family...

now obviously, given the above descriptions, comparing samples to signatures or variants doesn't make a lot of sense, but a variant-centric focus when evaluating a test of anti-malware products is not altogether unfamiliar... in a detection test it's important to make sure you include all (or as many as possible) significant members of the malware population and also to make sure that no member is included more than any other so that detection of each is given equal weight/importance... back in the days when viruses ruled the earth it was actually necessary to try and look at things from a variant-centric perspective (subjective as it was) because of how viruses worked... the way file infecting viruses operated, it was trivial to imagine many samples containing not just the same variant of a virus but instances that were all offspring of a common parent... this made it necessary to try to tag samples with variant id's in an effort to enforce good coverage and uniformity...

these days the kind of complications that parasitic self-replication create in malware detection tests are much less of an issue because malware exhibiting those properties make up a much smaller proportion of the malware population... malware that exists as a stand-alone file is now the norm so the simpler sample-centric approach (at least for the non-viral samples) becomes more reasonable...

one potential complication that might still make a variant-centric approach necessary is packed malware or more generally malware incorporating server-side polymorphism... at least that's an argument one might bring up (and i think packed malware was mentioned in the thread) but i think there's a stronger argument to be made for server-side polymorphism in the general case producing distinctly new variants in spite common root malware that they all came from... this is because the resulting malware is the product of both relatively old malware (though the root malware may not technically qualify as known) and unknown and unguessable transformations... likewise with packing in particular, not all scanning engines deal with repacked malware equally well (even though we know many packers, the packers themselves can be altered to produce something we couldn't have guessed) so arguing that many samples created from the same root piece of malware are all the same variant may not be reasonable in light of the subjectivity of the variant concept...

given all this, from the information available in the test results in question, there really doesn't seem to be anything all that wrong... that said, without the thorough supporting documentation, we can't really verify that things are all that right either so those who really can and do read more than just the executive summary are stuck taking the results of this test with a grain of salt (in addition to the grain of salt one takes any detection test in isolation with, since the relative effectiveness of various products can change pretty quickly over a short time)...

(and anton will just have to believe what he wants to believe about whether the results that seem worthy of ridicule point to a cherry-picked test bed or not - which would be some feat, considering it's size)

i know there's no panacea but i still want one, darnit!

have you read this post by lonervamp about the silver bullet syndrome? forgetting his specific example of jericho forums for the moment, i agree whole-heartedly with his observation that although we all seem to agree that there is no panacea people still seem to talk and act as though they expect their to be one...

need an example besides the jericho one? well anton chuvakin has been kind enough to provide us with one this evening that involves anti-virus... he's abandoned known-malware scanning completely after a friend of his had to rebuild their system despite being 'protected' by a major brand av....

now i'm not going to debate the merits of the decision itself, if he wants to use a whitelist instead of a blacklist, that's his decision and it's certainly a workable one... the problem is the motivation - getting fed up because of an instance of av failing (or even many instances) points to (as mike rothman would put it) mismatched expectations... if you know and agree that there is no panacea then you shouldn't be overly bothered by instances of failure, you should be expecting failure...

so what's going on here? i think that although we more or less all agree that there is no panacea, people don't seem to appreciate what that really means... i suspect that the use of the term panacea itself may be obscuring the real implications so i'm going to put it in simple terms that everyone, expert and novice alike, can understand:

all preventative measures fail

that's right, each and every single last one of them... that is what it means to truly accept that there is no panacea, you have to accept that there will be failures... getting fed up with the failures is an emotional rather than rational reaction, and if you base your decisions on it you are likely to be disappointed in the future when it turns out that the next big thing fails too...

people don't like failure, however, and they certainly don't want to accept it... this is a shame because if you're going to develop a successful security strategy you have to not only accept failure, you have to anticipate it... anticipating failure is really a cornerstone of strategic thinking, without it there would be no impetus to devise contingency plans, and without those a strategy is nothing more than a basic plan and a lot of poorly founded hope... in short you need to learn to succeed by planning for failure rather than running blindly from it...

when is a bad test not a bad test?

when it's FUD'ing snake oil...

there have been a number of posts about anti-virus/anti-malware testing recently... even i posted about testing in response to what has become a series of posts about anti-virus testing over on anton chuvakin's blog (1, 2, 3, 4)... well this post is a follow-up because anton has managed to post the original test paper that his series of posts were based on...

to say that i was unimpressed would be an understatement... lets start with the number of samples - you may recall from my previous post that i said that the minimum number samples needed to account for the 2% detection rate that was being claimed was 50... according to the actual paper

Of the 35 malware files, three invalid files were removed from the sample set, leaving 32 malware binaries used in the final tests and performance calculations

so if only 32 samples were used, how is it that the lowest scoring product only detected 2% of the samples? detecting just a single sample gives a detection rate of 3%, not 2%, and all products tested detected more than just one sample... it can't be blamed on anton misremembering the figure he was told either, since the actual test paper states

the lowest was tied between ClamAV and FileAdvisor with a 2% detection rate

in one place and

two products tied for the lowest detection rate at 2%

thankfully the chart with their results clears this up - it's 2 raw detections (not a 2% detection rate) which means a 6% detection rate (which was also correctly reported in that same chart)... now, you'll have to forgive me for calling a spade a spade, but this level of mathematical incompetence (recognizing that you can't have a 2% detection rate with only 32 samples is grade school math and simply reading the column marked "percent" in a chart takes even less skill than that) is inexcusable for people who wish to have their test taken seriously... given such a complete lack of mathematical acumen, it's almost understandable that they failed to realize that a test bed of 32 samples isn't anywhere near large enough to give statistically significant results...

bias also figured heavily in the test... not only because they only used samples that got through the layers of protection already present on the systems they culled the samples from (thereby missing a potentially huge chunk of what's really posing a threat to users and irrevocably compromising the results and the integrity of the test itself), but on a deeper level the test was written from the perspective of an incident response technician... what is the perspective of an incident response technician? well these are the people who spend their days dealing with the after effects of the failure of security software and/or preparing for the next failure so as to make cleaning up after that event easier than cleaning up after the previous one... all they see are the security product's failures because that's their job, and if it weren't for the immutable fact that all security products fail they wouldn't have that job and they'd have to find some other form of employment (such as performing and publishing dubious tests)... just as police (who deal with criminals on a daily basis) are prone to developing an imbalanced view of society if they're not careful, so too are incident response technicians prone to developing an imbalanced view of the efficacy of security products if they aren't exposed to the security product's successes (which are generally invisible by design)... this acute form of perceptual bias taints the entire test at fundamental levels - including the design and methodology of the test (as evidenced by their belief that they need only collect samples that successfully compromised production systems that already had protective measures in place)...

so far these problems would seemingly be attributable to the testers simply being inexperienced and/or suffering from false authority syndrome... it's time to shine a light on a part of the test that can't easily be attributed to that... there was one product in the test that didn't do too badly - in fact it did better than all other products, it detected 50% more than the next best product, and it was one of the few products in the test that weren't used through virustotal... that product was asarium by proventsure (no link for reasons that are about to be made clear)... go and read the paper carefully (it's only 4 pages) and tell me if you can see something a little off about it... yes, that's right - the product that did the best, the product that was better than all others by a wide margin was the product made by the company whose president helped write the paper and is the contact listed in the abstract... the test, which is titled "Antiviral shortcomings with respect to 'real' malware" by gary golomb, jonathan gross, and rich walchuck, is NOT independent... one or more of it's authors has a clear vested interest in making one product look good at the expense of all others... this puts all the other problems with this test into a new and decidedly unfavourable light... the bad math, the sample selection bias, the insignificant sample size, etc. - in light of this revelation they all point to a cooked test designed to make all products other than asarium look worse than they really are (FUD) in order to make asarium look better than it is in comparison to them (snake oil)... the test, therefore, becomes little more than a marketing stunt by a disreputable company whose product should probably be given a wide berth...

and poor anton chuvakin - though widely regarded as a security expert, not only is he clearly not an authority on malware himself but apparently he also can't recognize a fraud/pretender when he sees one... that doesn't bode well for average folks' ability to do the same, does it...

the myth of meaningful informal anti-malware tests

bad tests are not necessarily a problem that is unique to the anti-malware field but it is one that those in the anti-malware community have encountered countless times... it's not that average folks are physically incapable of performing good tests, it's that they simply don't do it...

an informal test lacks strict adherence to established testing protocols, often because the standards set for good tests are so high that most people can't be bothered to go to all that trouble... only those who are dedicated to the process of anti-malware testing have ever done a half decent job of it because it requires a substantial investment of time and effort, not to mention a certain degree of expertise to ensure that the testbed is free of garbage/scanner fodder and duplicates...

i could stop there, i suppose, and hope that what i've said isn't too abstract, but i really object lessons and one sort of fell into my lap... dr. anton chuvakin has used the the results of one such bad test to support the now popular opinion that anti-virus is dead... i saw where this was going early on (the original question was obviously loaded)... i guessed the results would be bad and suggested the most likely reason was that the malware was too new - to which i was told that some samples were weeks old, though weeks can technically still be new enough if no one else has reported it yet... additionally, if a particular piece of malware isn't affecting many people the av companies may down-prioritize it in order to focus on samples that are affecting more people...

the test, as described so far, worked like this: someone who does incident response at a public university collected samples from compromised machines for a period of time and then when the batch was big enough (weeks after the collection was started) this person submitted them to virus total and took note of the results... the results were as follows: the best detection rate was 50%, the worst was 2%, and the average was 33%...

now, i can see a lot of problems with such a test but lets start with the big ones:

1. statistical significance - the sample size was too small... it may sound picky to those who want to believe, but even an unskilled labourer intuitively knows that asking 5 people their opinion on X does not give you a result that can be generalized to the entire population... this test (or at least the interpretation of it that we've been given) pretends to represent the ability of anti-virus products to protect against malware that is currently in the wild affecting people but most likely has a sample size of about 50 (50 is the minimum necessary for a product to get a 2% detection rate, but larger sample sizes make a perfect 50% detection rate increasingly unlikely)... since the wildlist has over 1600 different pieces of malware on it and since that only represents the self-replicating malware (which are believed to be in the minority now), a sample size of 50 (or even 100) just doesn't cut it...
2. sample selection bias - the samples came from compromised machines in a production environment... maybe that sounds reasonable to you but let me ask you this - if you test a variety of lie detectors against people who have proven themselves to be good at fooling lie detectors, are you really measuring how good those lie detectors are in the real world? the answer is no, you're only measuring how good they are against an artificially constructed set of people... the same goes for malware from compromised production machines - the malware you find there is the malware that has proven itself to be good at evading anti-malware scanners, not in-the-wild malware in general...
3. sample selection bias - the samples came solely from computers in an academic institution... sorry to say but universities are not a typical environment... if anything, they are the place where the incidence of truly malicious insider threats are the greatest (uni students practicing their l33t skills)... home users don't intentionally compromise their own computers, enterprise users may consider it (or even try it) but there are more/stronger controls in place (stricter logical access controls, more personally significant administrative controls, etc.) to prevent and/or deter it...
4. test-bed integrity - the test was carried out by incident response personnel... we really have no way of knowing how clean (in the sense of being free of detrimental garbage) the test-bed was... even if it was an incident response technician who was capable of reverse engineering each sample of malware (and had the free time in which to do it), simply reverse engineering the samples would not be enough to determine whether the samples were sufficiently different enough from one another to be considered distinctly different pieces of malware... and since many incident response technicians simply follow a script, there's a serious question as to whether all the samples really were malware instead of scanner fodder and whether any of them were duplicates...

now as i've already said, those of us who have been in the anti-malware community for a while have seen countless examples of these kinds of tests so we know to take them with more than just a single grain of salt... unfortunately the average person doesn't know why they should be wary of such tests and when security experts use these same sorts of tests to prop up FUD that scares people away from anti-virus products it doesn't help anyone... it's bad science and a doctor of physics should know better...