thoughts on "In Search of Evidence-Based IT-Security"

Christopher Soghoian brought to my attention a video of a talk by Hanno Böck at the 33rd Chaos Communication Congress. in it Hanno puts forward the claim that IT security is largely science-free, so let's follow a staple of the scientific process - peer review.

Hanno introduces himself as a journalist and hacker and says that he prefers to avoid the term "security researcher" and that he hopes the audience will see why.  for those who are relatively well versed in the field of anti-malware it should definitely become obvious why he prefers to avoid that term and i'll return to this near the end.

Hanno is a skeptic, and far from the only one, his talk ultimately expresses the same sentiments that are now common-place in the perennially misinformed information security community. the difference is that Hanno has found a novel way of expressing them, couched in scientific jargon and easily mistaken for insight. he spends altogether too long and dives too deeply into the medical analogy upon which computer viruses and by extension anti-virus software is named. the analogy has long been recognized as deeply imperfect and limited. that's why, in reality, there are relatively few references to this analogy in the anti-malware field other than "computer virus", "anti-virus", and "infection" (all three of which date back virtually to the beginning of the field). his call towards the end of his talk for blinded or even double blinded studies, aside from being prohibitively expensive to perform, seem to cling to this medical paradigm in spite of the fact that the subject of such experimentation (ie. the computer, since we're interested in whether AV can prevent computers from becoming compromised) cannot be psychologically influenced by knowledge of which (if any) anti-virus is being used.

when he FINALLY leaves the topic of medical science to return to security products (about 14 minutes into his half hour talk) he harps on the absence of one very particular kind of experiment being performed on security products - what he calls a randomized controlled trial. it turns out this is a hold-over from his preoccupation with medical science. when Hanno says that IT security is largely science-free it is the absence of this particular kind of scientific experiment that he is referring to, but that doesn't actually make it science-free because science has a variety of different ways to study and experiment on things that aren't people.

there is in fact good scientific evidence for the efficacy of anti-virus software and it's provided by none other than Microsoft:

now it's true that this is data is from an observational study and that it only shows correlation rather than causation, but that's not the end of the world. observational studies are still science. showing correlation may not be definitive evidence but it's still strong evidence, especially considering the scope of the study (hundreds of millions of computers around the world out of a total estimated population of 1.25 billion windows PCs). in this particular case A may not be causing B but B definitely can't cause A and if anyone can think of a confounding variable that might be present on hundreds of millions of systems then maybe let Microsoft know so that they can try to account for it in the future.

another source of scientific evidence (oft derided in information security circles because the results don't match experts' anecdata) are the independent testing labs like av-test.org or av-comparatives.org. they eliminate the influence of confounding variables and so are capable of showing causation rather than just correlation. unfortunately Hanno believes their methodology is "extremely flawed". let's look at his complaints:

• "If a software detects a malware it does not mean it would've caused harm if undetected."
• this is trivially false. anyone who actually reads the testing methodology at av-comparatives (for example) can find right at the beginning a statement about first testing the malware without the AV present and eliminating any that don't work in that scenario. therefore every sample that is detected by AV in their tests would have caused harm if it had gone undetected.
• "Alternatives to Antivirus software are not considered." (the talk gives "regular updates" and "application whitelisting" as examples)
• the example of "regular updates" is frankly a little bit bizarre given Hanno's earlier references to confounders. not controlling for this scenario would actually introduce a confounding variable and make it more difficult to show a causal relationship between the use of a particular AV and the prevention of malware incidents.
• the example of "application whitelisting" underscores a serious problem in Hanno's understanding of what he's critiquing. application whitelisting isn't an alternative to AV, it's a part of AV. many products include this as a feature. Symantec's product, for example, has what they call a reputation engine which alerts when it encounters anything that doesn't have a known good reputation (which means new/unknown malware, traditionally the bane of known-malware scanning, will get alerted on because it hasn't been seen before and thus no reputation, good or bad).
• "Antivirus software as a security risk is not considered."
• when malware exploiting vulnerabilities in anti-virus software is found in the wild then perhaps the test methodologies should be updated to include this possibility. until then, changing the methodology to account for malware that doesn't seem to exist outside a lab has no real benefit.
• "None of these tests are with real users."
• again, this would introduce a confounding variable. maybe the lack malware incidents is because of something the user did rather than because of the AV. alternatively maybe the failure to stop malware incidents is because of something the user did rather than because of a failure of the AV. if you want to establish causation you have to control your variables (something our scientifically-minded speaker Hanno should know all too well). does the anti-virus prevent malware incidents? the tests say yes. can a user preempt or compromise that prevention? also yes. is there any prevention a user can't preempt or compromise? sadly (or perhaps thankfully) no. if you want a study that includes users and thus eliminates the ability to establish a causal link between AV use and prevention of malware incidents, see the study by Microsoft, but even with the inclusion of the users it still suggests AV prevents malware incidents.

when Hanno addressed the paucity of scientific papers dealing with security i found myself confused. using Google Scholar to find the most cited scientific papers? surely he doesn't think the realm of security is so narrowly focused that he'll find what he's looking for that way. security is in fact incredibly broad, covering many different quasi-related domains, and looking at a handful of the most popular scientific papers across all of security is in no way representative of the corpus of available works related to any one particular field (like security software). perhaps i'm biased, having previously (in the very distant past) maintained a reference library of papers related specifically to anti-virus, but it doesn't seem like Hanno showed much evidence that he knew how to find evidence-based security. is it really that hard to add the term "malware" to his search query? could he not find a few and then use them as a seed in an algorithm that crawls backwards and forwards through scientific papers by citation? did he even bother to look at Virus Bulletin? does he even know what that is?

security isn't the only thing that is incredibly broad - so too is the practice and discipline of science itself. there are many different fields and each one does things in their own particular way. we do not perform randomized controlled trials on the cosmos. as a general rule we do not intervene in volcano formation. the work being done at the large hadron collider does not follow exactly the same methodologies that are used in medical science. are we to judge cosmology, volcanology, or particle physics poorly because of this? no of course not. a question you might well ask is what kind of science should logically be used when it comes to studying computer security and, while i suspect multiple scientific disciplines could be useful, the one that springs immediately to mind is computer science. does computer science look anything like medical science? as someone with a degree in computer science i can tell you the answer is emphatically no. we do many things in computer science but randomized controlled trials are not among them (because computers are not people). while Hanno may style himself as "scientifically minded" he doesn't seem to demonstrate an appreciation for the breadth of valid scientific research methodologies and one is left to wonder if he's familiar with any kind of science outside of medicine.

when it comes right down to it, it's this apparent lack of familiarity with the subject matter he's talking about that i found most troubling about Hanno's talk.what is anti-virus software really? what is av testing methodology really? what does science really look like? where do you look for scientific research into malware and anti-malware? these all seem to be questions Hanno struggles with, which brings us back to the subject of why he likes to avoid the term "security researcher". if i had to venture a guess i'd say it's because he doesn't do research, even the basic research necessary to understand the subject matter. as such i would say avoiding the term "security researcher" is probably appropriate (for now).

i'm not sure what one can say in a talk about a subject one hasn't done one's homework on, but hopefully that can improve in the future. Hanno referenced Tavis Ormandy during his talk (as people who criticize AV like to do). Tavis' work on AV also suffered from a lack of understanding in the beginning, but he improved over time and, while he still has room for more improvement, now has arguably done some good work in finding vulnerabilities in AV and holding vendor's accountable for the quality of their software. i'm certain Hanno can also improve. i know there are real criticisms to be made of AV software and the industry behind it, but they have to be informed, they have to come from a place of real knowledge and understanding. i look forward to Hanno reaching that place.

the anti-virus harm balance

anti-virus software, like all software, has defects. sometimes those defects are functional and manifest in a failure to do something the software was supposed to do. some other times the defects manifest in the software doing something it was never supposed to do, which can have security implications so we classify them as software vulnerabilities. over the years the software vulnerabilities in anti-virus software has been gaining an increasing amount of attention by the security community and industry - so much so that these days there are people in those groups expressing the opinion that, due to the presence of those vulnerabilities, anti-virus software does more harm than good.

the reasoning behind that opinion goes something like this: if anti-virus software has vulnerabilities then it can be attacked, so having anti-virus software installed increases the attack surface of the system and makes it more vulnerable. worse still, anti-virus software is everywhere, in part because of well funded marketing campaigns but also because in some situations it's mandated by law. add to that the old but still very popular opinion that anti-virus software isn't effective anymore and it starts looking like a perfect storm of badness waiting to rain on everyone's parade.

there's a certain delicious irony in the idea that software intended to close avenues of attack actually opens them instead, but as appealing as that irony is, is it really true? certainly each vulnerability does open an avenue of attack, but is it really doing that instead of closing them or is it as well as closing them?

if an anti-virus program stops a particular piece of malware, it's hard to argue that it hasn't closed the avenue of attack that piece of malware represented. it's also hard to argue that anti-virus software doesn't stop any malware - i don't think anyone in the anti-AV camp would try to argue that because it's so demonstrably false (anyone with a malware collection can demonstrate anti-virus software stopping at least one piece of malware). indeed, the people who criticize anti-virus software usually complain not about set of malware stopped by AV being too small but rather that the set of malware stopped by AV doesn't include the malware that matters most (the new stuff).

so, since anti-virus does in fact close avenues of attack, that irony about opening avenues of attack instead of closing them isn't strictly true. but what about the idea that anti-virus software does more harm than good? well, for that to be true anti-virus software would have to open more avenues of attack than it closes. i don't know how many vulnerabilities any given anti-virus product has so i can't give an exact figure of how many avenues of attack are opened. i doubt anyone else can do so either (though i imagine there are some who could give statistical estimates based on the size of the code base). the other side of the coin, however, is one we have much better figures for. the number pieces of malware that better known anti-virus programs stop (and therefore the number of avenues of attack closed) is in the millions if not tens of millions and that number increases by thousands each day. can the number of vulnerabilities in anti-virus software really compare with that?

it's said that windows has 50 million lines of code. if an anti-virus product were comparable (i suspect in reality it would have fewer lines of code) and if that anti-virus product only stops 5 million pieces of malware (i suspect the real number would be higher) then in order for that anti-virus product to do more harm than good it would need to have at least one vulnerability for every 10 lines of code. that would be ridiculously bad software considering such metrics are usually estimated per 1000 lines of code.

now one might argue (in fact i'm sure many will) that those millions of pieces of malware that anti-virus software stops don't really represent actual avenues of attack because for the most part they aren't actually being used anymore. they've been abandoned. counting them as closed avenues of attack isn't realistic. the counter-argument to that, however, is to examine why they were abandoned in the first place. the reason is obvious, they were abandoned because anti-virus software was updated to stop them. the only reason why malware writers continue making new malware instead of resting on their laurels and using existing malware in perpetuity is because once anti-virus software can detect that existing malware it generally stops being a viable avenue of attack. so rather than the abandonment of that malware counting against anti-virus software's record of closing avenues of attack it's actually closer to being AV's figurative body count.

there is still malware out there that anti-virus software hasn't yet stopped, and as that set is continually replenished it's unlikely that anti-virus software will stop all the malware. it has stopped an awful lot so far, however, so the next time someone says anti-virus software does more harm than good (due to it's vulnerabilities) ask them for their figures on the number of vulnerabilities in anti-virus products and see how it compares with the number of things anti-virus software stops. i have a feeling you'll find those people are full of it.

AV complicity explained

earlier this week i wrote a post about the idea of the AV industry being somehow complicit in the government spying that has been all over the news for months. some people seemed to really 'get it' while others, for various reasons, did not; so i thought i'd try to be a little more clear about my thoughts on the subject.

the question that the EFF et al have put towards the AV industry (besides having already been asked and answered some years ago) is a little banal, a little pedestrian, a little sterile. real life is messy and complicated and things don't always fit into neat little boxes. i wanted to try to get people to think outside the box with respect to complicity, what it means, what it would look like, etc. but i think some people have a hard time letting go of the straightforward question of complicity that has been put forward so let's start by talking about that.

has the NSA (or other organization) asked members of the AV industry to look the other way and has the AV industry (or parts thereof) agreed to that request? almost certainly the NSA has not made such a request, for at least a couple of reasons:

1. telling people about your super-secret malware is just plain bad OpSec. if you want to keep something secret, the last thing you want to do is tell dozens of armies of reverse engineers to look the other way.
2. too many of the companies that make up the AV industry are based out of foreign countries and so are in no way answerable to the NSA or any other single intelligence organization.
3. there's quite literally no need. there are already well established techniques for making malware that AV software doesn't currently detect. commercial malware writers have been honing this craft for years and it seems ridiculous to suggest that a well-funded intelligence agency would be any less capable.

now while it seems comical that such a request would be made, to suggest that the AV industry would agree to such a request would probably best be described as insulting. whatever you might think of the AV industry, there are quite a few highly principled individuals working in that would flat out refuse, in all likelihood regardless of what their employer decided (in the hypothetical case that the pointy-haired bosses in AV aren't quite as principled).

now please feel free to enjoy a sigh of relief over the fact that i don't think the AV industry has secretly agreed to get into bed with the NSA and help them spy on people.

done? good, because now we're going to take a deeper look at the nature of complicity and the rest of this post is probably not going to be nearly as pleasant.

here's one of the very first things wikipedia has to say about complicity:

An individual is complicit in a crime if he/she is aware of its occurrence and has the ability to report the crime, but fails to do so. As such, the individual effectively allows criminals to carry out a crime despite possibly being able to stop them, either directly or by contacting the authorities, thus making the individual a de facto accessory to the crime rather than an innocent bystander.

in the case of government spying we may or may not be talking about a crime. the government says they broke no law and observers speculate that that may be because they've subverted the law (much like they subverted encryption algorithms). so let's consider a version of this that relates to ethical and/or moral wrong-doing instead of legal wrong-doing:

an individual is complicit in wrong-doing if he/she is aware of it's occurrence and has the ability to alert relevant parties but fails to do so. as such, the individual effectively allows immoral or unethical people to carry out their wrong-doing despite possibly being able to stop them either directly or by alerting others who can, thus making the individual a de facto accessory to the wrong-doing rather than an innocent bystander.

in this context, could the AV industry be complicit with government spying? perhaps not directly, not in the sense that they saw what the government was doing and failed to alert people to that wrong-doing. however, what about a different wrong-doing by a different entity but still related to the government spying?

hbgary wrote spyware for the government. this became public knowledge in the beginning of 2011. by providing the government with tools to perpetrate spying they become accessories to that spying.

hbgary was and is a partner of mcafee. now what is the nature of this partnership? hbgary is an integration partner. they make technology that integrates into mcafee's endpoint security product to extend it's functionality. mcafee does marketing/advertising for this technology and by extension for hbgary, giving them exposure, lending them credibility, and generally helping them make money. that money is almost certainly re-invested into research and development of hbgary's products, which includes governmental malware that's used for spying on people/organizations. there are mcafee customers out there right now whose security suite includes components that were written by known malware writers and endorsed by mcafee (although they make sure to weasel out of responsibility for anything going wrong with those components with some fine print). mcafee didn't break off the partnership when hbgary's status as an accessory to government spying became known, and since they didn't break off the partnership you can probably make a safe bet that they didn't warn those customers that part of their security suite was made by people aiding the government in spying either. even if we ignore the fact that mcafee aids a business that writes malware for the government, mcafee's failure to raise the alarm about the possible compromising nature of any content provided by hbgary makes them accessories to hbgary's wrong-doing. by breaking ties with hbgary and warning the public about what hbgary was up to they could have had a serious impact on hbgary's cash flow and hurt their ability to win contracts and/or execute on their more offensive espionage-assisting projects. they didn't do any of that and that makes them complicit in the sense discussed a few paragraphs earlier.

the rest of the AV industry may not be directly aiding hbgary's business but, like mcafee, they have failed to raise any alarm about hbgary. they could have done much the same as mcafee by warning the public, with the added bonus that they would have hurt one of the biggest competitors in their own industry while they were at it and that would have benefited all of them (except mcafee, of course). again, failing to act to help prevent wrong-doing makes them a de facto accessory to that wrong-doing. the AV industry as a whole is complicit in the sense discussed earlier.

of course, the AV industry isn't alone in being accessories to an accessory to government spying, and that brings up a consideration that should not be overlooked because there is a larger context here. historically, the culture of the AV industry has been one that values being very selective in things like who to trust, who to accept into certain groups, etc. add to that a very narrowly defined mission statement (to fight viruses and other malware) and it's little wonder that the ethical boundaries that developed in the early days were so dead-set against hiring, paying, or doing anything else that might assist malware writers or possibly promote malware writing. heck, i knew one member who wouldn't even engage viruses writers in conversation, and another who said he was wary of hiring anyone who already knew about viruses just in case they came by that knowledge through unsavoury means. aiding malware writers, turning a blind eye to their activities, etc. are things that normally would have violated AV's early ethical boundaries.

by contrast, the broader security industry is highly inclusive and has long viewed the AV industry's selectivity as unfair elitism. that inclusivity means that the security industry isn't actually just one homogeneous group. there are many groups, from cryptographers to security operations personnel to vulnerability researchers to penetration testers, etc. each one has it's own distinct mission statement and it's own code of ethics. what do you think you get from a highly inclusive melting pot of security disciplines? well, in order for them to tolerate each other, one necessary outcome is a very relaxed ethical 'soup'. many quarters openly embrace the more offensive security-related disciplines such as malware creation. in order for AV to integrate into this broader security community (and they have been, gradually, over time), AV has to loosen it's own ethical restrictions and be more accepting.

so while the AV industry failed to raise the alarm about hbgary, the broader security industry failed as well. the difference is that ethics in the security industry don't necessarily require raising an alarm over what was going on. hbgary is a respected company in security industry circles and it's founder greg hoglund is a respected researcher whose proclivity for creating malware has been known for a long, long time. as far as the security industry is concerned, hbgary's activities don't necessary qualify as ethical wrong-doing. there will probably be those who think it does, but in general the ethical soup will be permissive enough to allow it, and without being able to call something "wrong-doing" there can be no complicity. this is where AV is going as it continues to integrate into the broader security community. in fact it may be there already. maybe that's the reason they didn't raise the alarm - because they've become ethically compromised, not as a result of a request from some intelligence organization, but as a result of trying to fit in and be something other than what they used to be.

in the final analysis, if you were hoping for a yes or no answer to the question of whether AV is in any way complicit in the spying that the government has been doing (specifically, the spying done using malware), i'm afraid you're going to be disappointed. it depends. based on AV's earlier ethics the answer would probably be yes. based on the security community's ethics the answer may well be no. where is the AV industry now? somewhere between what they were and what the broader security community is. ethical relativity is unfortunately a significant complicating factor. then again, i'm an uncompromising bastard, so i say "yes" (after all, i did grow up with those old-school ethics).

what would AV's complicity in government spying look like?

as you may well have heard, the EFF and a bunch of security experts have written an open letter to the AV industry asking about any possible involvement by them in the mass spying scandal that has been in the headlines for much of this year. at first i thought this was old news for AV, since the issue of government trojans has actually been around a lot longer than the current spying revelations. i thought these people had simply failed to do their homework but, as time passed, the wheels began to turn and i started thinking differently. now i think the question we should all be asking ourselves is, what would AV's complicity look like?

some background, first. the subject of government trojans have been around for over a decade. magic lantern, for example, dates back to 2001 (or at least public awareness of it does). so it should come as little surprise that the question of whether the AV industry looks the other way has come up before. in 2007 cnet ran a story where 13 different vendors were asked about this very thing. they all more or less denied being a party to such shenanigans, but i suggest you read the article and pay careful attention to the answers.

now earlier this year one of the first controversial spying revelations to come about was about a program called PRISM which a whole bunch of well known, big name internet companies (including google, microsoft, yahoo, facebook, etc) were apparently involved with. the companies all denied it of course, and it turns out they may be legally required to do so.

that adds an interesting wrinkle to the question now being put towards the AV industry; would they be allowed to admit to any complicity that might be going on? they say actions speek louder than words, so maybe we should look for something other than the carefully crafted assurances of multi-million dollar corporations. maybe what we should be looking for is the same thing that alerted us to the mass spying in the first place - a leak. maybe then we can get a glimpse of their actions.

back in early 2011 a rather spectacular breach occurred. security firm hbgary was breached by some members of anonymous, and one of the things that leaked out was the fact that hbgary wrote malware for the government. in fact, it doesn't take much imagination to suppose that this would be the very type of malware the EFF et al are concerned the AV industry may have been asked to ignore.

it's unknown whether any AV vendor actually did field such a request. i have my doubts since traditional commercial malware writers seem to be perfectly capable of creating undetected malware without making such requests. that being said, one fact that became rather suspicious in light of the revelations about hbgary was the fact that they were partners with mcafee, one of the biggest AV vendors around and certainly one of the best known names in AV. i wrote about this apparent ethical conflict back in february of 2011, and then again in march of 2011 to note the tremendous non-reaction from the industry. i even went so far as to create a blog specifically for keeping an eye on the industry (though as an outsider myself there was little i could do on my own).

the EFF and others want to know if the AV industry has been complicit in the government's spying. well, one AV vendor was notably evasive when asked by cnet in 2007 about their handling of governmental trojans/police spyware. that same AV vendor was and still is partnered with a company that wrote government malware (in all likelihood for very purpose in question).  furthermore, in the intervening years, nothing has come of it. no other vendor has said anything or done anything to call attention to or raise awareness of this partnership. even after the mass surveillance controversy started earlier this year, not a one bothered to raise the alarm and suggest that mcafee might at least in principle be compromised by that partnership, even though they certainly could have benefited from disrupting mcafee's market share. no one thought they could profit from it? no one thought it was their duty to warn people of a potential problem? to raise concerns that the protection mcafee's customers receive may suffer in some way because of their close ties with government malware writers? to give voice to the doubts this partnership creates even after publicly wringing their hands over how wrong what the government themselves were doing was?

AV vendors may or may not have been asked to turn a blind eye to government malware - we may never know, and it's impossible to prove a negative. but they've done a heck of a job turning a blind eye to the people who make government malware and to those in their own ranks who got in bed with government malware writers. i asked at the beginning what AV complicity would look like and i think when it comes to those whose job it is to raise an alarm, complicity would probably have to look like silence (and something about silence makes me sick).

(2013-10-29 13:21 - updated to change the open letter link to point to the blog post that includes the list of intended recipients as well as a link to the letter itself)

debating AV effectiveness with security experts

a rather disheartening conversation took place on twitter over the weekend. as public conversations sometimes do, it grew beyond any capability i have to do it justice through description, so instead i'll provide some screenshots and links to a couple of branches of the discussion.

because i don't follow either dan kaminsky or robert graham, i knew nothing about this discussion until someone retweeted the tweet pictured below (i included as much context as i could):

what first made me take interest in this was that robert graham seemed to be talking about 2 different things as though they were the same. the AV that's only 4% effective (or 0% when he's done with it) is different than the AV that organizations pay 40% of their budget on.

the apparently ineffective AV is actually the scanner component of the AV; as you can see he describes his methodology for bypassing it - a methodology that essentially amounts to malware q/a, which happens to be a countermeasure against heuristic detection, which is a feature of scanners.

the AV that organizations pay 40% of their budget on (assuming that's an accurate figure, i wouldn't know) is the enterprise security suite, which includes other things beyond just the scanner. for example the tweet by dan kaminsky that seems to have started the entire conversation alludes to the failure of symantec's product to stop 44 out of 45 pieces of malware in the recently publicized attack on the new york times. but as symantec rightly pointed out, their product included a reputation system which, for all intents and purposes, behaves much like a whitelist - if something doesn't have a good reputation (and new things have no reputation at all) then it will be flagged. that is about as different from a traditional scanner as one can imagine and bypassing it isn't nearly as straightforward.

talking about 2 different "AV"s as though they were the same is symptomatic of not being able to see beyond the abstraction that is AV. conceptually AV is an abstraction that encompasses a variety of disparate preventative, detective, and recovery techniques. most people, however, just see AV as a magic box that you turn on and it just protects things. the only component that behaves anything like that is the real-time scanner, but it is not the only component in a security suite (especially an enterprise security suite) by any stretch of the imagination.

failing to see beyond the abstraction means, unfortunately, that you will fail to argue intelligently about the subject. it also means you will probably fail to make effective use of AV. if you don't know how a tool works, how can you possibly hope to use it to your fullest advantage? furthermore, how can you value something you don't understand? just as you can't price a car based on the effectiveness of the wheels, you shouldn't value AV based on the supposed effectiveness of the scanner.

one of the things that also became clear as i read some of the subsequent tweets was that robert seems to think there's nothing special about his attacks. but the fact is his attacks are special. as a penetration tester he launches targeted attacks. targeted attacks take more effort, more human capital to execute, and he himself described some of that extra effort. this basically means targeted attacks are more expensive to launch than the more automated variety and that they don't scale quite as well. consequently targetted attacks represent a minority of the overall attacks being performed. note, however, that that doesn't necessarily mean targetted attacks are a minority of the attacks a particular organization sees, as it's entirely possible that an organization may be a juicy enough target to receive a great deal of attention from targeted attackers.

from what i can tell, dan kaminsky also has difficulty seeing beyond the abstraction of AV. much of what he says above reflects the idea that AV is a magic box that you simply turn on and it should protect you. in the preceding example it appears he thinks there's an expectation that organizations solve all their security problems with scanners when in fact the expectation is simply that they have AV suites in their toolbox and that they use the appropriate tool (which may or may not be part of the suite) for the job (as rik ferguson attempted to explain).

dan also quoted the DBIR as showing AV to only be effective 3% of the time. i wondered about that so i looked a little deeper. DBIR stands for Data Breach Investigations Report. let the meaning of that phrase sink in a little bit. a data breach investigations report is a report about data breach investigations. data breach investigations are investigations of data breaches, and data breaches only occur when all the effort you put into preventing them failed.

you can't judge how successful something is by only looking at it's failures.

one of the consequences of this is that dan has actually failed to understand the statistic he's reported. the 3% where AV detected the breach still represents a failure because it was a detection after the breach had happened. this can happen due to things like signature updates (a scanner can detect more today than it could detect yesterday).

another consequence is that trying to use the DBIR to evaluate the effectiveness of AV represents a self-selected sample bias because the failure itself causes the event to be included in the study. a success would have excluded the event from the study. now one might have entertained the possibility that dan simply wasn't familiar with selection bias, but as we will see, that appears to not be the case.

it appears that dan has in fact heard of selection bias before, not to mention the WildList too (bravo). unfortunately it doesn't appear that he can use them properly.

• AV testers don't define the WildList, WildList reporters do (in a sense, but it's probably more accurate to say the the WildList is a result of a particular type of sampling)
• AV testers typically don't do WildList testing, although virus bulletin does offer a WildList-based certification in addition to larger, more inclusive tests
• if a product only detects 90%+ of the WildList, it's generally considered to be crap, because the WildList is the absolute bottom of the barrel of performance metrics. anything less than 100% is an embarrassment.
• AV testers don't define the set of malware they're going to test against, they cull samples from as wide a variety of real-life sources as they can and describe them as being 'in-the-wild' so as to distinguish them from malware that only exists in a 'zoo' or malware that was whipped up in a lab for testing purposes (something that's generally frowned on)
• defining what you're going to measure is not actually selection bias. "Selection bias occurs when some part of the target population is not in the sampled population" ("Sampling: Design and Analysis" by Sharon L. Lohr)  (now THAT's a textbook definition of selection bias - good thing was minoring in statistics in university). if testers defined their target to be the samples they already had then by definition there couldn't possibly be any selection bias because the target population and the sample population would be the same set.

this isn't to say there isn't selection bias in the tests performed by AV testers. it's entirely possible that some classes of malware (perhaps even targeted malware, for example) are harder to find samples of due to circumstances outside the testers' control. that being said, that bias is a lot more subtle than looking exclusively at failures.

now, it just so happens that i continued digging into the DBIR beyond just figuring out what went into it, and came across a rather interesting chart.

i highlighted the part that should really be interesting here. just to be clear, this only covers organizations that have to be compliant with PCI, but unless organizations that are legally obligated to run up-to-date AV are somehow magically more stupid than the rest of the organizations, the rest of the organizations actually have less motivation to run up-to-date AV and so their numbers are probably as low if not lower. 

now what this means is that there is really no way at all to use the DBIR to evaluate the effectiveness of AV because it appears that most of the organizations included in the report can't even follow the most basic of AV best practices. it also suggests that dan hasn't read his own sources thoroughly enough. if i'm not mistaken his 3% figure comes from the year when only 53% of PCI-bound organizations were running up-to-date AV. the subsequent year it was 1% of breaches discovered by signature-based anti-virus, and in the most recent one, AV doesn't appear to have helped after the fact at all.

that ever decreasing percentage of organizations running up-to-date AV is actually kind of disturbing, and it makes you wonder what's hurting organizations more; the amount of money they pay to AV vendors or the amount of attention they pay to security experts pontificating on subjects they are demonstrably ignorant of?

i anticipate that there will be those thinking that all i do is criticize and that i have nothing constructive to offer, so let's think about how we'd really measure AV effectiveness. the independent AV tests are apparently not good enough - in fact dan kaminsky went so far as to say this:

so how would we measure effectiveness really? like effectiveness in the field where AV is actually getting used? well first of all we stop limiting our data collection to just those instances where AV failed, because that's just ridiculous. no, we'll need to collect data on each and every time the AV raises an alert as well, to supplement our failure statistics. oh, and we'll have to follow up each of those alerts to make sure they aren't false positives because those certainly don't contribute to the effectiveness of AV. we'll also have to use all of the AV suite, rather than just parts of it, in order that people can determine if the effectiveness justifies the money they pay for the entire suite. additionally, we'll need to control for variables - different organizations have different security measures and controls that they deploy in conjunction with AV that may stop some malware before the AV gets a chance. that's not a bad thing, of course, and if they all used the same security measures then we could collect data on how effective AV is under those particular circumstances. but because each organization has different measures, they'll affect the AV results to differing degrees and that will skew the measurement. so we'll have to get organizations to either all use the same complementary measures or get them all to stop using any complementary measures. neither of which seem very likely in production environments, so that leaves us with trying to simulate what happens in the field - but then we get back into the AV testing lab territory which apparently 'no competent soul on the planet' trusts.

the reality is that it doesn't matter what kind of test you do, it's never going to match people's anecdotal experience. that's because testing is inherently designed around the idea of arriving at a single set of results representing how effective AV can be - it's never going to be able to reflect what happens when variables such as complementary controls, sub-optimal operation, targeting motivation, etc. aren't controlled for - and in the real world they aren't controlled for. tests necessarily reflect ideal circumstances and your mileage may (probably will) vary.

were i to be overly judgmental i might sign off this post with this little gem by none other than robert graham himself that i found yesterday will going through my RSS backlog:

The problem with our industry is that it's full of self-styled "experts" who are adept at slinging buzzwords and cliches. These people are skilled at tricking the masses, but they have actually zero expertise in cybersecurity.

but i prefer the school of thought from my own post about security experts from 2006 - security is just too big for anyone to be an expert in all parts of it. it seems to me that the expertise of dan and robert lie elsewhere.

it's important for people to recognize their own limitations, and i believe it's also important to recognize the limitations of the authorities you're listening to as well, lest you give credence to well meaning but uninformed experts. anti-malware is a complicated field, more complicated than i think either dan or robert realize, and if they have difficulty seeing beyond the AV abstraction imagine, how many other people do as well. i hope someday dan and robert and other experts like them can gain a deeper appreciation for how complex it is so that they can pass that along to those who depend on them to do the heavy cognitive lifting.

tavis ormandy's sophail presentation

at the black hat security conference this year tavis ormandy presented his research into the way an anti-virus product, namely sophos' product, operates in order to shine a light on something seems like it would be an important subject to consider when judging the efficacy of a product. the paper associated with the presentation can be found here.

tavis was not particularly kind in his evaluation of the code (which he apparently performed by reverse engineering the product). sophos' response is very measured and diplomatic, which is pretty much the perfect way (from a PR perspective) to respond to the kind of criticism being leveled at them. as usual, however, i don't have to be diplomatic.

tavis' paper betrays a conceit that i suspect is more common in those who break things than it is in those who make things. developers, upon dealing with someone else's code, inevitably learn an important lesson: the code tells you what the software does, but it doesn't tell you why it does it that way. tavis thinks he knows all he needs to know, but he only had the code to go by. so when it comes to why certain things were done the way they were, the only thing he could reasonably do is make educated guesses. in some cases those guesses may well have been quite good, but in others they were not.

i first realized this was going to be the case on the second page of the paper where he describes how weak the encryption used on the signatures was, often an XOR with an 8 bit key. if you were to guess that such encryption was to protect the signatures from an attacker, as tavis seems to have, then you'd be dead wrong. the primary purpose encrypting signatures serves is to prevent other anti-virus products from raising an alarm on the signature database (something that used to happen in the very early days).

on page 3 it's mentioned that the heavy use of CRC32 in the signatures means it's easy to maliciously create false alarms by creating files that have the same CRC32 values that a particular signature is looking for and in the same places. now i ask you, the reader, if someone is maliciously planting files on your network that are designed to raise an alarm, is that alarm really false? it may be a false identification, but there really is something malicious going on that an administrator needs to investigate.

also on page 3 he criticizes the quality of the signatures by stating they appear to ignore the context of the programs they come from, that they're for irrelevant, trivial, or even dead code. perhaps tavis expanded on this in his live presentation, but the paper doesn't make clear whether or not he actually looked at the malware the samples were supposed to be for. if he didn't, then the criticism about ignoring context would be particularly ironic. let's assume then that he did. how many malware samples did he examine? if only a handful then there's a not insignificant chance that he was dealing with bad examples that aren't really representative of the overall signature quality. did he ensure that his samples were actually malware? did he ensure that his samples were being identified by the right signatures? his previous criticism (on the same page!) about false identifications should highlight the fact that hey may have been looking at the wrong code when judging the quality of the signatures. but more importantly than that, there isn't a 1-to-1 relationship between signatures and samples. one signature may be intended to detect many (tens, hundreds, even thousands of) related samples - and however pointless tavis may think those sections of the malware code are, they may represent the optimal commonality between all those samples.

around about page 8 or so, tavis makes a point of highlighting the fact that the emulation the product does in order to let malware reveal it's unencrypted/unpacked form only goes for about 500 cycles. this highlights a failure to understand one of the core problems in malware detection: the halting problem. for any other criteria the code might look for in deciding it's seen enough, there's no guarantee that criteria will ever be encountered on a particular input program. there has to be a hardcoded cut-off point or the process runs the risk of never stopping - and that would severely impact the usability of the AV software. likewise if the hardcoded cut-off point isn't reached soon enough it also impacts the usability of the AV software.

there may yet be other examples of poor guesswork that i didn't see. my own knowledge of why certain design choices might be made is limited as i've never actually built an anti-virus product. i have considered it, of course, since i am a maker of things. perhaps tavis ormandy would benefit from more experience making things rather than breaking them. perhaps this was an unorthodox expression of the oft' repeated concept that the skills needed to attack are not the same as the skills needed to defend.

careful who you let write the history books

over the course of the previous two weeks, kevin mcaleavey has been publishing a series of blog posts on the infosecisland site (parts one, two, three, four, five, and six) about the history of the anti-virus industry. rob lewis thought this might be a subject that would interest me and he was right. unfortunately, rather than finding it mostly informative, i found with each passing part an increasing desire to post a serious critique. i'll take them one part at a time.

part one

to start with i think we need to pay attention to how the author presents himself. each part refers to him as a long time industry insider, but doesn't go into anything more verifiable or specific than that. with such vague credentials (and those are meant as credentials - there's no reason to put them there except to try to convince the reader that the author is an authority on the subject he's writing about) it really feels like he's saying "trust me, i'm a security expert". now i've never heard of him but that alone doesn't say that much, the anti-virus industry employs hundreds if not thousands of people and i'm really only familiar with a comparative handful of them. in part three we get to find out more specifics, but for the time being let's just say that such nebulous credentials makes me very suspicious.

to be fair i should point out my own credentials, so that i'm not being a hypocrite. unlike kevin, i am not now, nor have i ever been an industry insider. i have never been employed by any company in the anti-virus industry, i have never received financial remuneration for anything i've said or done involving the anti-virus field, and frankly i've only ever met a handful of people who were part of the industry. that being said, what i am (in terms relevant to this discussion) is a long time observer of the anti-virus industry. i've always been on the outside looking in, but from about the age of 15 onwards i basically grew up interacting with security hobbyists, security professionals, security software engineers, security researchers, and even some of the big names that kevin mentions in part two.

in theory, being an outsider seems like it should mean i have a more superficial view of the anti-virus industry.  we shall see.

the first part of kevin's series focuses mostly on where we are today so a lot of the things he mentions should sound familiar to people following the security news. one of the things he mentions is the rustock botnet. he presents it as a single piece of malware with a 5 year half-life. this is a bit misleading for a couple of reasons, the first being that rustock is actually a family of malware - there have been many versions since 2006, in part because the anti-virus industry keeps interfering with the utility of existing versions by detecting them. additionally, the term "half life" has a very specific meaning which doesn't really apply that well to a botnet that has for all intents and purposes been killed now that microsoft controls the command and control server.

subsequently he made the following quote:

"TDL4" however has publicly caused the security industry to transition into full panic mode and literally throw in the towel

this may seem pedantic, but in order to literally throw in the towel, there has to be an actual physical towel, and someone has to throw it. an argument could be made, i suppose, for saying that they figuratively threw in the towel, or metaphorically threw in the towel. maybe even virtually threw in the towel - but if he insists that they literally threw in the towel then i have five words - pics or it didn't happen.

now there was some early misinterpretation of the use of the word "indestructible" that got posted in less knowledgeable media circles, specifically that the malware was indestructible rather than the observation that malware authors were trying to create an indestructible or bulletproof network of compromised computers, but a knowledgeable industry insider should have been able to see through that. furthermore, from my perspective the only panic i saw was the panicky feeding frenzy in the media over a statement which, like so many more purposeful scams, was simply too spectacular to be true.

i got the distinct impression upon reading part one that he tends to take mainstream media as gospel when it suggests that all is lost in the fight against malware. that seems strange to me. why is an industry insider putting so much credence in what the mainstream media says? he acknowledges that the industry has called this interpretation wrong, and that they're trying to correct that misinterpretation but he seems to suggest that the "corrections" are a product of public relations rather than a genuine attempt to correct factually erroneous statements that spread fear, uncertainty, and doubt.


another statement i'd like to draw attention to is the following:

To see this public admission that 1980's technology has utterly failed is nothing short of breathtaking.   

admission by whom? where? what are the details? none are given and we must take him on his word that such an admission actually took place. or perhaps he thinks the misinterpretations spread by mainstream media represent that admission. could it be that he's unfamiliar with the degree to which they botch things up on a regular basis? it really makes me question his credibility if he placing his preference on the words of reporters over the words of researchers.

part two


part two is where he actually starts talking about history. he starts right at the beginning but there's a problem. he mistakenly thinks the brain virus 'destroyed' many hard disks. brain came out at a time when hard disks were a rarity. moreover, brain specifically avoided infecting hard disks. the link kevin himself provided says this so i can only assume that he's not actually reading the sources he's providing to the reader.

he also seems to mistakenly think brain required a reinstall to recover from. this is, of course, false. there have always been less drastic ways of restoring boot sectors. the main problem was information about doing so wasn't as easy to come by back then as it is now. still, thinking there wasn't a way to do it is essentially a form of ignorance that you wouldn't expect to find in a long time industry insider.

yet another point on the brain virus; he seems to think the backlash from it forced the developers who made it out of business. i guess he never saw this video by f-secure's mikko hypponen, where mikko found the developers of the virus still working at exactly the same address they were at 25 years ago when they originally wrote the virus.

brain isn't the only piece of malware whose details he gets wrong, though. he mistakenly thinks popureb requires a windows reinstall to remove, much like brain. in reality what's required is to restore the MBR with the recovery console. this seems to be another example where kevin has taken mainstream media at their word instead of digging deeper and getting actual inside information, the way you'd expect an insider to do.

he also seems to think that stoned was the first virus to go memory resident and infect any disk that was inserted in the drive. stoned was a boot sector infector, however. all boot sector infectors infect (more or less) every disk inserted into the drive, it's kinda how they spread, and since it wasn't the first of that sort of virus i can't imagine how he got the impression it was the first one to go memory resident.

more generally he seems to think the majority of viruses in the 90's were jokes and pranks and programs designed to delete files. my recollection is that most actually gave no outward indication of infection whatsoever. virus writers quickly decided they liked the notoriety that came with one of their viruses spreading far and wide, and the best way to help that to happen was to make sure it didn't make itself known by messing up people's computers. the ones that stuck out in people's memories were oftentimes jokes or pranks or had destructive payloads, however.

it's not just malware he seems to get wrong, however. he appears to be under the impression that VIRUS-L was a private echomail conference available to SysOps (system operators) of BBSes that carried FidoNet. VIRUS-L was actually an internet mailing list which was gated into usenet in the form of the newsgroup comp.virus. i have no doubt it was further gated into FidoNet - i know alt.comp.virus was as my first encounter with it was through a usenet-to-FidoNet gateway. i have a feeling i encountered comp.virus the same way (without ever being a SysOp). that said, it's possible kevin could have been thinking of something else; maybe VIRUS or VIRUS_INFO (or even VIRUS_NFO) which were in fact echomail conferences on FidoNet (the first two of which have me as their most recent moderator). they also aren't private, however; i really can't seem to figure out where that part of his recollection might have come from.

another curious thing i noticed is that he seems to think eugene kaspersky is the only member of the old guard of virus hunters still in the field after john mcafee, alan solomon, and peter norton (?) left. he also mentioned frisk software international but sort of as an afterthought, and somehow failed to mention frisk the person. frisk the person (and the company) are still out there, still working. they may not be attracting attention but they're still there. as such, kaspersky isn't the last so kevin's depiction of the industry as having been lobotomized seems all the more inaccurate. what's more, however, is that kevin only seems to be acknowledging a handful of the icons in the anti-virus industry. there were and are a lot more people from the old days out there. a great example would be aryeh goretsky, who was from the sounds of it like john mcafee's right hand man, and who currently is with eset. then there's the venerable vesselin bontchev, once a well respected virus expert at university of hamburg and now a well respected virus expert at FSI. then there's jimmy kuo of symantec and then mcafee and now microsoft. there are many great minds from the early days that are still with us today, so this focus on just a handful of icons and the suggestion that since most of those are gone that the industry has been lobotomized, seems like a decidedly superficial view.

part three

the third part in kevin's series was about the demise of the anti-virus industry (it's not dead, it's just sleeping). it's also where we finally find out about his actual credentials. he was part of privacy software corporation, the company behind BOClean; what was apparently originally a cleanup tool for the back orifice remote access trojan, but was later expanded to cover more malware. apparently this company was bought by comodo in 2007, but from the sounds of things it doesn't sound like kevin stayed on with comodo for all that long, and even if he was still with them today 4 years isn't exactly "long term".

he presents himself as a long term industry insider in a discussion about anti-virus vendors, but his actual first hand inside experience with the kinds of companies he talks about and criticizes in this series of posts only went on for 2 1/4 years according to his linkedin profile. that seems rather underwhelming. oh sure, before that he spent a long time in a different part of the anti-malware industry, but he spends very little time actually talking about the part of the industry where most of his actual experience is from - except to extoll the virtues of the techniques used by BOClean, of course. in any event, it's clear by this point why he was so vague about his credentials in earlier parts.

back to the demise of the anti-virus industry, however. kevin focuses on the point in which trojans started to take off as the death knell of the industry. it was, after all, the point at which his company was forced to start dealing with malware. he seems to think that the early failures to handle trojans reflected inability on the industry's part. just to be clear, when trojans began to take off the industry did hesitate, and i don't mean for a second or a minute or an hour or a day or week or a month or even a year. the anti-virus industry hesitated to redefine itself. it took a lot of doing to overcome the philosophical inertia that kept them out non-viral malware detection. trojans were not viruses, remote access 'tools' arguably had legitimate uses, and the industry was gun-shy with regards to litigation. i hope kevin can appreciate that last point as it was his revered dr solomon who explained that one to me. apparently they couldn't even add generic detection for MtE based polymorphic viruses using the existence of the MtE algorithm as an indicator because some brainiac went and put the algorithm in a 'legitimate' code obfuscation tool. the industry did eventually overcome their philosophical qualms about dealing with non-viral malware but it took the escalation of the trojan problem to make people (both inside and outside the industry) realize that the industry would be justified in going after this new kind of threat that was previously outside of their purview.

a great deal of kevin's characterization of the failure of the industry seems to focus on what his company handled better. the reader is left to take his word for it that his company did a better job. he's clearly not an impartial 3rd party, his perceptions should be taken with a grain of salt. he talks about a number of techniques used by the anti-virus industry and what their failings are - a number of which are almost certainly exaggerated. i've seen my share of alternative anti-malware approaches whose proponents go on ad nauseum about how their approach is the superior one (zvi netiv stands out as a remarkable example of this). better is always subjective, there's always good and bad points.

he describes BOClean's approach of only looking for the malware in memory, rather than trying to find it on disk like traditional anti-virus products do, as superior. there are benefits, of course. packers and cryptors have no effect on detection because that transformation is undone at runtime. however, there are also drawbacks. once the malware is in memory it is active. it's possible for active malware to interfere with security software in any number of ways. it could use stealth, it could shut down the security software (which i gather was a problem for BOClean), or it could even use the security software's own filesystem enumerating behaviour to find new host programs to infect (if you're dealing with a file infecting virus). once the malware is active the security software has lost an important tactical advantage. imagine you're in a street fight - do you wait for your opponent to strike first or do you try to strike before they're even in a position to defend themselves? and remember, i'm talking about a street fight here, not some fair fight with referees and judges. you do not wait for the malware to go active in memory if you can avoid it. the code that gets control first wins, and once the malware is active in memory, it's gained control. it's only by kevin's good fortune (or BOClean's low profile) that malware creators didn't try to stomp out or otherwise interfere with BOClean. the AV industry has already faced that, going back all the way to brain (which had stealth).

some of his exaggerations devolve into more factual errors. for example he calls win32/ska a simple file infector and criticises the industry for taking months to handle it. the fact is that win32/ska's infection technique was nothing like the traditional file infection technique kevin laid out elsewhere. win32/ska was not some appending file infector, it didn't simply insert itself into the winsock DLL, it carried a modified copy of the winsock DLL with it and replaced the original with the modified version. as such, recovery of the original winsock DLL was not comparable to recovering from a traditional file infector. it may have taken months to build the capabilities to recover from this type of infection into their general purpose tools, but that in part was because there were special purpose tools available to mitigate the problem and lower the priority of getting it into their general purpose products. there also happened to be manual instructions for removing win32/ska (i know because i have given those instructions to people) which also mitigated the problem and thus lowered the priority.

he also points out that even now systems are still getting compromised with back orifice 12 years later and characterizes that as an example of the anti-malware industry's failure. the fact is that his BOClean is just as much a failure in that as the rest of the anti-malware industry is. that is if you can really call it a failure. stopping that trojan from ever getting used again, making it become extinct, that's just not something anyone can actually do. and it doesn't really have that much to do with whether the malware can be detected or removed by software tools, but more with the heterogeneity of the computer population and the mind-share of the malware in question amongst those who would use it. old viruses never die, and apparently old trojans don't either.

he places the blame for why all windows malware works at all on microsoft. unfortunately it's not that simple. any general purpose computer is capable of supporting malware, no matter what the operating system. in this way malware can be said to be a 'feature' of general purpose computing. no matter how tightly you try to control things, there will always be the potential for software to maliciously do something you don't want it to do.


kevin seems to think that AV industry has for the past 30 years (which is longer than the industry has actually existed) simply been coming up with signatures and not looking for patterns that could be useful for future variants. the fact that we have the concept of malware families at all invalidates this line of reasoning. it wouldn't be possible to classify new samples as belonging to an existing family without looking for such patterns. this line of reasoning also ignores heuristic alerts that say "possibly modified variant of X". it also ignores the practice of consolidating many specific signatures into a few generic signatures.

he also calls automated analysis 'cheating' and characterizes it as simply creating an MD5 or SHA1 hash. this seems to ignore the fact that you can't logically identify that a sample is a variant of another malware family by simply creating a hash. it also clearly ignores the concept of automated classification, whereby a sample is compared with many other samples in order to determine which ones it's most similar to and thus which family it belongs to. one also has to wonder how exactly vendors are supposed to deal with 100,000 new samples a day without some kind of automation. even if you had a way to detect most of those new samples ahead of time with generic signatures, you still need to test and make sure each of those new samples is actually malware and is actually detected.

rather unsurprisingly at this point, he describes heuristics as bad because they emulate the malware while trying to catch it in the act of doing something bad. this is only (somewhat) true of dynamic heuristics, not static heuristics. it's also hypocritical coming from someone championing the idea of letting the malware become active in memory before trying to do anything about it.

he also referred to heuristics as creating "massive" false positive problems. in reality false positives are relatively rare. a single false positive can, of course, cause problems for many people if it's a system file, but those are rare even amongst regular false positives. those really high profile failures are something that has only happened a handful of times.

part four

the fourth part of kevin's series focuses on operating systems, mostly windows, but covering mac os x and linux too. i actually agree with a lot of what kevin has to say in this part (which makes me wonder if my knowledge of OS security might be a little too shallow), but there are a few things that are worth pointing out.

i tend not to agree with his suggestion that the concept of file associations based on file extensions are a bad thing and should be replaced by some kind of intelligent parsing. for one thing i think the parsing idea is logically infeasible. intelligent parsing would require windows to know about every file format, even the one i just invented yesterday, which it obviously can't. he also envisioned that the parser could generate a warning for the user to indicate what sort of action windows was about to take, but the warning could just as easily been done with file associations. the warning is based on the outcome and has little to do with how the outcome is decided upon (be it parsing or association).

on the matter of unix, kevin appears to be of the opinion that pure unix was secure. this would seem to ignore the fact that the original academic treatment of computer viruses (where the term actually got coined) had a virus spreading on a professionally administered unix system without the admin's assistance. how secure is that?

he also seems to believe that linux used to be secure in the beginning, but seems to forget or not care that rootkits worked on linux way back close to it's beginning. he admits that there was malicious software but contends that it got removed from distros quickly after being introduced. this ignores the fact that once end users start using such an OS, malware doesn't need to be bundled into the distro. if a user can run it, they will, and linux didn't and doesn't prevent running malware

part five

the fifth and penultimate part of the series deals the concept of defense in depth, or as kevin prefers layered security. he spends an inordinate amount of time talking about firewalls and how awful they are. he sort of presented defense in depth as a caricature, focusing only on the layers that gained widespread adoption and suggesting that anything else was outside of the regular user's price range. frankly i don't think integrity checking was ever priced that way, nor behaviour blocking, nor any number of other techniques that he conveniently ignores.

what's more, he criticizes each layer for being insufficient. it's as if he doesn't understand the purpose of having multiple layers. each one has flaws and weaknesses, sure, all security measures do, but in aggregate those weaknesses are diminished. they're never completely eliminated, of course, because there's no such thing as perfect security, but it's certainly something that we can approach. one might argue that one can approach that by re-engineering one particular layer instead of having many, but some of those weaknesses are inherent rather than being design or implementation flaws. such inherent weaknesses can only be braced by other additional, complementary layers.

part six

the final part of keven mcaleavey's series was meant to present solutions to the problems mentioned in the previous 5 parts. when i read the final part in the series all the pieces finally clicked into place. all the confusion i previously had was gone, all the WTF moments finally made sense. i finally knew what he was on about and what the series' true purpose was. kevin's got a product to sell and the series was an extraordinarily long sales pitch.

it followed a familiar pattern too, now that i think about it. first he dished out FUD about his competitors (and since he's now a secure systems provider his competitors include both the anti-malware software vendors and the operating system vendors), and then at the end hype up his own product and make it look like the blatantly obvious choice to avoid the horrible, horrible problems with everything else. he even threw in a smattering of snake oil phrases like "absolute security and protection" for good measure.

the product and/or service in question (KNOS) appears to my untrained self to be similar to some sort of freeBSD-based LiveCD composed entirely out of carefully audited modules, and with the provision that you can request custom configurations from to suit your needs. the idea appears to be that the code that is allowed to run is carefully controlled and limited by them (rather than being something the user can add code to him/herself) and that exploitable vulnerabilities have supposedly been eliminated. this is supposed to keep the users safe from both malware and from them themselves.

unfortunately, kevin seems to have fallen prey to the engineer's conceit - the (often mistaken) belief that a particular problem can solved through carefully engineered technology alone. KNOS does not offer absolute protection, and to say it does offers users a false sense of security. it may well stop all the malware that kevin can imagine, but (to paraphrase something that's often said in cryptographic circles) it's easy to come up with a security system so advanced that you yourself can't figure out a way around it - what's hard is figuring out one that other people can't figure out a way around either.

obviously i don't know enough of the details of KNOS to suggest specific scenarios where it's security can fail, but i do know enough about computation in general to see what kevin (and others who aim to ensure only good/safe code is allowed to run) has missed. what many people don't realize, what society's conventional thinking about computers fails to hint at, is that data is code. the distinction we draw between the two is little more than a mental construct that simplifies the task of building systems. it doesn't represent how computation actually works, and it isn't necessarily adhered to by the people who break systems.

unless and until someone can come up with a way to control which data gets processed as scrupulously as they control which code gets executed, data will remain an open window through which systems with locked down code can be attacked. and since determining the safety of data will ultimately require the data to be processed in some way, we arrive at a catch-22.

i fully expect that KNOS probably has some impressive security capabilities, but absolute security is a fantasy, and the more traditional security layers that kevin derides have managed to keep me essentially malware free (except for an externally non-addressable RAT installation on a sacrificial secondary system) for over two decades. beware security historians who are trying to sell you something - chances are they'll rewrite history to suit their sales objective.

security: it's almost like it isn't there

one of the ideas i continue to encounter over and over again throughout the years is the idea of liking a particular security product because it seems like it's not even there. it's amazing where one can find that idea being expressed. panda security's own luis corrons said the following about his wife's impression of panda's product:

My wife’s computer also have it, and she loves it, mainly because she doesn’t realize that it is installed :)

liking a security product because it seems like it's not even there strikes me as suggestive that the person in question likes to ignore security or not be bothered by security concerns. for most people this is going to be a recipe for eventual disaster. luis' wife, however, has luis on hand to take care of any malware incidents, so i guess for her it's ok. it's an interesting and probably effective strategy - well played, mrs. corrons, well played.

most people can't marry an anti-malware expert, however, so placing value in product's ability to shut up is the wrong way to think about things for most of us. don't get me wrong, if a security tool is too 'chatty' then certainly that poses a usability problem, but the quest for complete transparency is a symptom of mismatched expectations.

the predominant expectation among consumers is that if they install the 'right' product or combination of products then they can forget about all those nasty threats because they'll be protected. that's just not true, though, and it's never, ever going to be true.

people will actively defend this line of thinking, however, often they say they just want to do X and don't want security getting in the way. imagine if i said i just wanted to get to mcdonald's and didn't want traffic safety to get in the way - would that sound reasonable? not so much, i imagine. part of the reason for that is that most of us realize that following certain procedures on the road actually does keep us safer than we would otherwise be; but another part is that we also recognize that when others don't follow those procedures they put us and everyone else at risk, not just themselves.

what if i were to tell you the same principles apply in computer security? there are procedures you can follow that not only allow you to reach your goal in a reasonably secure way (whether that goal is getting work done or enjoying online entertainment or whatever else you use your computer for). not only that but by not following those procedures, by ignoring security, one actually does put other computer users at risk as well. i'm not just talking about other people who use the same computer, either. back in the days of viruses, when a virus infected a computer that computer joined the set of computers from which that virus could further it's spread. essentially it enlarged the platform from which the virus could attack still other systems. today, in the age of the botnet, the same principle applies. when a machine becomes compromised it get's added to the attack platform and assists in attacks on other systems, whether those attacks are simply sending out spam or sending out more malware or performing distributed denial of service attacks. by pretending like security isn't a concern a user puts not only themselves but all other computer users at risk as well.

now, likening secure computing practices to safe driving does not mean i'm trying to argue in favour of requiring users to have a license to operate a computer (though there are those who suggest that). the fact is that day to day life is full situations where you have to take precautions to increase your safety. just crossing the street calls for the precaution of looking both ways first. even toasters (which i bring up because some people literally think computers should be as simple to use as toasters) have safety precautions you need to follow - unplug the thing before you try to retrieve that piece of toast or bagel that's stuck inside.

i often criticize the security industry for perpetuating the myth of install-and-forget security, but the consumer shouldn't be thought of as blameless. people need to wake up and take responsibility for their own safety and security online, as well as being good online citizens and not putting others at undue risk. seriously, folks, computing without the need for taking active precautions is pure fantasy and it's time you started living in the real world. if you don't take responsibility for keeping yourself safe and secure, you won't be safe and secure - period.

pity the anti-virus naysayer

pity the anti-virus naysayer, for when one decries the failure of anti-virus one reveals the failure in oneself

i don't think it's necessarily all that interesting to talk about the AV is dead movement anymore - saying anti-virus (or anything else security-wise) is dead is a pretty obvious cry for attention. instead in this post i want to look at the popular notion of "the failure of AV".

when one talks about the failure of anti-virus, what has anti-virus failed to do in the most general sense? failed to stop malware XYZ? failed to protect the endpoint from a specific attack? no, those are all reasonable failures not really worthy of being harped on if you accept that no preventative measure is perfect. in the most general sense, when one talks about the failure of anti-virus one is talking about the failure of anti-virus to live up to one's own expectations.

but are those expectations reasonable? in all likelihood they aren't. they are expectations born not out of an understanding of AV, but rather out of listening to marketing (stop listening to marketing!). if you truly understood AV then your expectations would be a pretty close match to reality, so incidental failures wouldn't surprise you or be a cause for concern. if you really understand AV then those incidental failures should be anticipated and planned for.

therefore, when one decries the failure of AV, it is because one doesn't actually understand it, one hasn't anticipated the incidental failures and made plans for them. it is a failure of understanding that happens all too often, where one tries to use marketing bullshit as a substitute for actual knowledge but only winds up with mismatched expectations. actual knowledge has no substitute and can often be hard to come by. "the failure of AV" may get you brownie points in populist crowds, but it's too facile a conclusion to be useful in the larger scheme of things.

well mine's infinity big

so sophos seems to be a little late to the mine's bigger party, but at least they're following the go big or go home philosophy to make sure everyone notices them. which is unfortunate - for sophos.

see, while the numbers game that most vendors play sometimes has the faint aroma of snake oil, kris braun's bold statement that sophos 'currently detects an infinite number of malicious files' (with a big infinity graphic, no less) has an overpowering stench of snake oil.

it's one thing to use big but meaningless and impossible to verify numbers, it's another thing to claim things that are literally false. sophos can't currently detect an infinite number of malicious files because an infinite number of malicious files do not currently exist.

claiming to detect not only more than you can verify you detect, but more than actually exist is ridiculous. you might as well say you recognize 15 different boolean outcomes or can write out all 57 letters of the english alphabet.


sophos may, arguably*, have the potential to detect an infinite number of things, but that's a lot different than being able to currently detect an infinite number of things, and you can bet that the average person not only can't tell the difference but will look at the claim of infinite detection much like they would a claim of perfect detection (in spite of kris' explanations to the contrary).

so congratulations go to sophos for the ignominious feat of innovating in the field of snake oil. now please do yourselves a favour and stop being so boastful.

(*technically, an infinite number of files would require that some of those files be infinitely long, since limiting them to a finite length would also limit them to a finite number. since there is not yet a computer capable of holding an infinitely large file let alone indexing one, there's strong evidence to suggest that sophos cannot actually detect those malicious files that happen to be infinitely large and thus cannot currently detect an infinite number of malicious files.)

bad advice: disable your av

i'm sure you can probably think back and remember at least one example of a piece of software you've installed in the past that came with directions suggesting you disable your anti-virus before you install.

that little pearl of anti-wisdom has been with us a long, long time and has always irked me. the software industry, whether too lazy, too hurried, or to ignorant to know any better, decided that rather than resolve whatever conflict they may encounter with security software it would be better to train users to drop their defenses.

what could possibly go wrong?

plenty could go wrong, of course, not the least of which being that once the users are effectively trained to drop their defenses when told it would be trivial for malware authors to distribute their wares with instructions telling people to disable their av. who needs to develop technical measures to bypass anti-virus when you can simply tell users to turn it off for you and have them do it like good little trained monkeys.

well, nowadays more and more applications are being delivered to users over the internet, specifically over the web, and along with that, anti-malware products are focusing increasingly on web-borne threats. as such you can probably guess this was bound to happen eventually:

yeah, this WTF moment was brought to you by a facebook application development company called chainn who thought it would be a good idea to tell users to disable (or even remove) norton anti-virus. hey, it worked well enough back in the days of desktop applications, right? why not facebook applications too? i mean, really, what's more important, online safety or finding out how you fit in amongst your friends?

apparently they also have some problems with adblock plus - i wonder why?