thoughts on "In Search of Evidence-Based IT-Security"

Christopher Soghoian brought to my attention a video of a talk by Hanno Böck at the 33rd Chaos Communication Congress. in it Hanno puts forward the claim that IT security is largely science-free, so let's follow a staple of the scientific process - peer review.

Hanno introduces himself as a journalist and hacker and says that he prefers to avoid the term "security researcher" and that he hopes the audience will see why.  for those who are relatively well versed in the field of anti-malware it should definitely become obvious why he prefers to avoid that term and i'll return to this near the end.

Hanno is a skeptic, and far from the only one, his talk ultimately expresses the same sentiments that are now common-place in the perennially misinformed information security community. the difference is that Hanno has found a novel way of expressing them, couched in scientific jargon and easily mistaken for insight. he spends altogether too long and dives too deeply into the medical analogy upon which computer viruses and by extension anti-virus software is named. the analogy has long been recognized as deeply imperfect and limited. that's why, in reality, there are relatively few references to this analogy in the anti-malware field other than "computer virus", "anti-virus", and "infection" (all three of which date back virtually to the beginning of the field). his call towards the end of his talk for blinded or even double blinded studies, aside from being prohibitively expensive to perform, seem to cling to this medical paradigm in spite of the fact that the subject of such experimentation (ie. the computer, since we're interested in whether AV can prevent computers from becoming compromised) cannot be psychologically influenced by knowledge of which (if any) anti-virus is being used.

when he FINALLY leaves the topic of medical science to return to security products (about 14 minutes into his half hour talk) he harps on the absence of one very particular kind of experiment being performed on security products - what he calls a randomized controlled trial. it turns out this is a hold-over from his preoccupation with medical science. when Hanno says that IT security is largely science-free it is the absence of this particular kind of scientific experiment that he is referring to, but that doesn't actually make it science-free because science has a variety of different ways to study and experiment on things that aren't people.

there is in fact good scientific evidence for the efficacy of anti-virus software and it's provided by none other than Microsoft:

now it's true that this is data is from an observational study and that it only shows correlation rather than causation, but that's not the end of the world. observational studies are still science. showing correlation may not be definitive evidence but it's still strong evidence, especially considering the scope of the study (hundreds of millions of computers around the world out of a total estimated population of 1.25 billion windows PCs). in this particular case A may not be causing B but B definitely can't cause A and if anyone can think of a confounding variable that might be present on hundreds of millions of systems then maybe let Microsoft know so that they can try to account for it in the future.

another source of scientific evidence (oft derided in information security circles because the results don't match experts' anecdata) are the independent testing labs like av-test.org or av-comparatives.org. they eliminate the influence of confounding variables and so are capable of showing causation rather than just correlation. unfortunately Hanno believes their methodology is "extremely flawed". let's look at his complaints:

• "If a software detects a malware it does not mean it would've caused harm if undetected."
• this is trivially false. anyone who actually reads the testing methodology at av-comparatives (for example) can find right at the beginning a statement about first testing the malware without the AV present and eliminating any that don't work in that scenario. therefore every sample that is detected by AV in their tests would have caused harm if it had gone undetected.
• "Alternatives to Antivirus software are not considered." (the talk gives "regular updates" and "application whitelisting" as examples)
• the example of "regular updates" is frankly a little bit bizarre given Hanno's earlier references to confounders. not controlling for this scenario would actually introduce a confounding variable and make it more difficult to show a causal relationship between the use of a particular AV and the prevention of malware incidents.
• the example of "application whitelisting" underscores a serious problem in Hanno's understanding of what he's critiquing. application whitelisting isn't an alternative to AV, it's a part of AV. many products include this as a feature. Symantec's product, for example, has what they call a reputation engine which alerts when it encounters anything that doesn't have a known good reputation (which means new/unknown malware, traditionally the bane of known-malware scanning, will get alerted on because it hasn't been seen before and thus no reputation, good or bad).
• "Antivirus software as a security risk is not considered."
• when malware exploiting vulnerabilities in anti-virus software is found in the wild then perhaps the test methodologies should be updated to include this possibility. until then, changing the methodology to account for malware that doesn't seem to exist outside a lab has no real benefit.
• "None of these tests are with real users."
• again, this would introduce a confounding variable. maybe the lack malware incidents is because of something the user did rather than because of the AV. alternatively maybe the failure to stop malware incidents is because of something the user did rather than because of a failure of the AV. if you want to establish causation you have to control your variables (something our scientifically-minded speaker Hanno should know all too well). does the anti-virus prevent malware incidents? the tests say yes. can a user preempt or compromise that prevention? also yes. is there any prevention a user can't preempt or compromise? sadly (or perhaps thankfully) no. if you want a study that includes users and thus eliminates the ability to establish a causal link between AV use and prevention of malware incidents, see the study by Microsoft, but even with the inclusion of the users it still suggests AV prevents malware incidents.

when Hanno addressed the paucity of scientific papers dealing with security i found myself confused. using Google Scholar to find the most cited scientific papers? surely he doesn't think the realm of security is so narrowly focused that he'll find what he's looking for that way. security is in fact incredibly broad, covering many different quasi-related domains, and looking at a handful of the most popular scientific papers across all of security is in no way representative of the corpus of available works related to any one particular field (like security software). perhaps i'm biased, having previously (in the very distant past) maintained a reference library of papers related specifically to anti-virus, but it doesn't seem like Hanno showed much evidence that he knew how to find evidence-based security. is it really that hard to add the term "malware" to his search query? could he not find a few and then use them as a seed in an algorithm that crawls backwards and forwards through scientific papers by citation? did he even bother to look at Virus Bulletin? does he even know what that is?

security isn't the only thing that is incredibly broad - so too is the practice and discipline of science itself. there are many different fields and each one does things in their own particular way. we do not perform randomized controlled trials on the cosmos. as a general rule we do not intervene in volcano formation. the work being done at the large hadron collider does not follow exactly the same methodologies that are used in medical science. are we to judge cosmology, volcanology, or particle physics poorly because of this? no of course not. a question you might well ask is what kind of science should logically be used when it comes to studying computer security and, while i suspect multiple scientific disciplines could be useful, the one that springs immediately to mind is computer science. does computer science look anything like medical science? as someone with a degree in computer science i can tell you the answer is emphatically no. we do many things in computer science but randomized controlled trials are not among them (because computers are not people). while Hanno may style himself as "scientifically minded" he doesn't seem to demonstrate an appreciation for the breadth of valid scientific research methodologies and one is left to wonder if he's familiar with any kind of science outside of medicine.

when it comes right down to it, it's this apparent lack of familiarity with the subject matter he's talking about that i found most troubling about Hanno's talk.what is anti-virus software really? what is av testing methodology really? what does science really look like? where do you look for scientific research into malware and anti-malware? these all seem to be questions Hanno struggles with, which brings us back to the subject of why he likes to avoid the term "security researcher". if i had to venture a guess i'd say it's because he doesn't do research, even the basic research necessary to understand the subject matter. as such i would say avoiding the term "security researcher" is probably appropriate (for now).

i'm not sure what one can say in a talk about a subject one hasn't done one's homework on, but hopefully that can improve in the future. Hanno referenced Tavis Ormandy during his talk (as people who criticize AV like to do). Tavis' work on AV also suffered from a lack of understanding in the beginning, but he improved over time and, while he still has room for more improvement, now has arguably done some good work in finding vulnerabilities in AV and holding vendor's accountable for the quality of their software. i'm certain Hanno can also improve. i know there are real criticisms to be made of AV software and the industry behind it, but they have to be informed, they have to come from a place of real knowledge and understanding. i look forward to Hanno reaching that place.

the anti-virus harm balance

anti-virus software, like all software, has defects. sometimes those defects are functional and manifest in a failure to do something the software was supposed to do. some other times the defects manifest in the software doing something it was never supposed to do, which can have security implications so we classify them as software vulnerabilities. over the years the software vulnerabilities in anti-virus software has been gaining an increasing amount of attention by the security community and industry - so much so that these days there are people in those groups expressing the opinion that, due to the presence of those vulnerabilities, anti-virus software does more harm than good.

the reasoning behind that opinion goes something like this: if anti-virus software has vulnerabilities then it can be attacked, so having anti-virus software installed increases the attack surface of the system and makes it more vulnerable. worse still, anti-virus software is everywhere, in part because of well funded marketing campaigns but also because in some situations it's mandated by law. add to that the old but still very popular opinion that anti-virus software isn't effective anymore and it starts looking like a perfect storm of badness waiting to rain on everyone's parade.

there's a certain delicious irony in the idea that software intended to close avenues of attack actually opens them instead, but as appealing as that irony is, is it really true? certainly each vulnerability does open an avenue of attack, but is it really doing that instead of closing them or is it as well as closing them?

if an anti-virus program stops a particular piece of malware, it's hard to argue that it hasn't closed the avenue of attack that piece of malware represented. it's also hard to argue that anti-virus software doesn't stop any malware - i don't think anyone in the anti-AV camp would try to argue that because it's so demonstrably false (anyone with a malware collection can demonstrate anti-virus software stopping at least one piece of malware). indeed, the people who criticize anti-virus software usually complain not about set of malware stopped by AV being too small but rather that the set of malware stopped by AV doesn't include the malware that matters most (the new stuff).

so, since anti-virus does in fact close avenues of attack, that irony about opening avenues of attack instead of closing them isn't strictly true. but what about the idea that anti-virus software does more harm than good? well, for that to be true anti-virus software would have to open more avenues of attack than it closes. i don't know how many vulnerabilities any given anti-virus product has so i can't give an exact figure of how many avenues of attack are opened. i doubt anyone else can do so either (though i imagine there are some who could give statistical estimates based on the size of the code base). the other side of the coin, however, is one we have much better figures for. the number pieces of malware that better known anti-virus programs stop (and therefore the number of avenues of attack closed) is in the millions if not tens of millions and that number increases by thousands each day. can the number of vulnerabilities in anti-virus software really compare with that?

it's said that windows has 50 million lines of code. if an anti-virus product were comparable (i suspect in reality it would have fewer lines of code) and if that anti-virus product only stops 5 million pieces of malware (i suspect the real number would be higher) then in order for that anti-virus product to do more harm than good it would need to have at least one vulnerability for every 10 lines of code. that would be ridiculously bad software considering such metrics are usually estimated per 1000 lines of code.

now one might argue (in fact i'm sure many will) that those millions of pieces of malware that anti-virus software stops don't really represent actual avenues of attack because for the most part they aren't actually being used anymore. they've been abandoned. counting them as closed avenues of attack isn't realistic. the counter-argument to that, however, is to examine why they were abandoned in the first place. the reason is obvious, they were abandoned because anti-virus software was updated to stop them. the only reason why malware writers continue making new malware instead of resting on their laurels and using existing malware in perpetuity is because once anti-virus software can detect that existing malware it generally stops being a viable avenue of attack. so rather than the abandonment of that malware counting against anti-virus software's record of closing avenues of attack it's actually closer to being AV's figurative body count.

there is still malware out there that anti-virus software hasn't yet stopped, and as that set is continually replenished it's unlikely that anti-virus software will stop all the malware. it has stopped an awful lot so far, however, so the next time someone says anti-virus software does more harm than good (due to it's vulnerabilities) ask them for their figures on the number of vulnerabilities in anti-virus products and see how it compares with the number of things anti-virus software stops. i have a feeling you'll find those people are full of it.

there's a quality problem in the anti-malware industry

if you follow infosec news sources at all, by now you've probably heard about the claim made by an anonymous pair of ex-kaspersky employees that kaspersky labs weaponized false positives.

more specifically, the claim is that engineers at kaspersky labs were directed to reverse engineer competing products and use that knowledge to alter legitimate system files by inserting malicious looking code into them so that they would both seem like files that should be detected and be similar enough to the original file that the competing product will also act on the legitimate file and in so doing cause problems for users of those competing products.

i've heard this described as fake malware, but for the life of me i can't see why it should be called fake. the altered files may not do anything malicious when executed, but they're clearly designed to exploit those competing products. furthermore, there is clearly a damaging payload. this isn't fake malware, it's real malware. it may launch it's malicious payload in an unorthodox and admittedly indirect manner, but this is essentially an exploit.

some consider the detection of these altered files to be false positives because the files don't actually do anything themselves, but since they have malicious intent and indirectly harmful consequences, i think the only real false positives in play here are the original system files that are being mistaken for these modified files.

by all accounts, this type of attack on anti-malware products actually happened. what's new here is the claim that kaspersky labs was responsible at the direction of eugene kaspersky himself. there's a lot of room for doubt. the only data we have to go by so far, besides the historical fact of the attack's existence, is the word of anonymous sources (who potentially have an ax to grind) and some emails that, quite frankly, are easily forged. circumstantially there's also an experiment kaspersky participated in around the same time frame that has similar earmarks to what is being claimed except for the part about tricking competing products into detecting legitimate files as malware.

i don't expect we'll ever know for sure if kaspersky was behind the attacks. doubts have been expressed by members of the industry, but frankly i've seen too many things whitewashed or completely ignored (like partnerships with government malware writers) to take their publicly expressed statements at face value. there are certainly vendors i'd have a harder time believing capable of this but there just doesn't seem to be sufficient evidence that the claims are true. the problem is that i can't imagine any kind of evidence the anonymous sources are likely to have that isn't easy to repudiate. had they taken a stand at the time (like someone with actual scruples would have done) they would have been able to put their names behind their claims - they may have lost their jobs but they surely would have been able to find employment with a different vendor because hiring a whistle-blower would have been good PR.

however, as it stands now, the anonymous sources have to remain anonymous. if they're telling the truth then they are complicit in kaspersky's wrong-doing, and if they're lying they are throwing the entire industry under the bus for no good reason (because this claim fans the fires of that old conspiracy theory about AV vendors being the ones who write the viruses). Either way, to have this claim linked to their real identities now would make them radioactive in the industry. no one would touch them, and for good reason.

long ago it used to be that the industry only employed the highest calibre of researchers. people who were beyond reproach. naturally, in order to grow, the industry has had to hire ever increasing numbers of people and old safeguards against undesirable individuals joining the ranks don't scale very well. increasingly people who aren't beyond reproach are being found amongst the industry's ranks and there appears to be no scenario where these two anonymous sources don't fall into that category. the inclusivity that the general security community embraces (and that the anti-malware industry is increasingly mimicking) has the consequence that blackhats are included. the anti-malware industry is going to have to either figure out if they're ok with that or figure out a way to avoid what the general security community could not.

no, bromium will not kill all malware forever

over the weekend a discussion broke out on twitter (as discussions are want to do) about a somewhat overly optimistic article concerning the new anti-malware apple of the security community's eye: bromium.

the primary tactic that bromium uses (or at least the primary one that people focus on) is isolation/sandboxing. bromium's vsentry product uses virtualization on a per-process basis to isolate every process from the system and from each other. that level of granularity for isolation is a lot higher than most sandboxing efforts can give you. while there are certainly benefits to that granularity, there are also drawbacks.

perfect isolation is actually not desirable, we want and even need to be able to use the results of one process inside another one. the more sandboxes you have, he harder this is to manage. the folks at bromium have opted to address this issue using rule-based systems to decide what something in a sandbox can access as well as what to do with any changes that are left when the sandboxed process is finished. rules which, in all likelihood, the administrator can modify to suit their needs.

now, while the article in question is reasonably good at explaining what bromium's vsentry does, the author (jason perlow) takes the arguably naive view that this sandboxing technique can stop all possible malware (as evidenced by the article's headline: "Bromium: A virtualization technology to kill all malware, forever"). the reality, however, is that there are limits to what sandboxing can do, and as clever as the folks at bromium are, they aren't clever enough to deliver on the promise that headline makes.

that's a problem, because people are going to read that headline, see nothing in the article to actually contradict it, and believe that it's actually true. have we seen claims like that before? sure we have - saying it can kill all malware forever is not intrinsically different from claiming 100% protection. it's classic snake oil, only in this case it's not the vendor that's spreading it (as far as we know - we don't know exactly what the folks at bromium may have said to mr. perlow, only that they say the headline is his words, not theirs).

i suppose that should mean there's no problem, right? the vendor's hands are clean, after all. the snake oil is being spread by a third party. the vendor isn't doing anything about it in this case or previous cases that have arisen because, let's face it, they benefit from it. it's good for bromium's business if people think vsentry is better than it actually is, at least in the short term. in the long term, the kinds of mismatched expectations that creates are the same kind that the AV industry struggles with daily.

it is bromium's responsibility to control how their products are perceived, and by failing to take action they are giving tacit approval to the snake oil being spread on their behalf. their hands are not actually clean, they are dirty through negligence. however, i didn't really expect any better of them (though i did give them an opportunity to surprise me) and you probably shouldn't either. tread carefully - caveat emptor.

debating AV effectiveness with security experts

a rather disheartening conversation took place on twitter over the weekend. as public conversations sometimes do, it grew beyond any capability i have to do it justice through description, so instead i'll provide some screenshots and links to a couple of branches of the discussion.

because i don't follow either dan kaminsky or robert graham, i knew nothing about this discussion until someone retweeted the tweet pictured below (i included as much context as i could):

what first made me take interest in this was that robert graham seemed to be talking about 2 different things as though they were the same. the AV that's only 4% effective (or 0% when he's done with it) is different than the AV that organizations pay 40% of their budget on.

the apparently ineffective AV is actually the scanner component of the AV; as you can see he describes his methodology for bypassing it - a methodology that essentially amounts to malware q/a, which happens to be a countermeasure against heuristic detection, which is a feature of scanners.

the AV that organizations pay 40% of their budget on (assuming that's an accurate figure, i wouldn't know) is the enterprise security suite, which includes other things beyond just the scanner. for example the tweet by dan kaminsky that seems to have started the entire conversation alludes to the failure of symantec's product to stop 44 out of 45 pieces of malware in the recently publicized attack on the new york times. but as symantec rightly pointed out, their product included a reputation system which, for all intents and purposes, behaves much like a whitelist - if something doesn't have a good reputation (and new things have no reputation at all) then it will be flagged. that is about as different from a traditional scanner as one can imagine and bypassing it isn't nearly as straightforward.

talking about 2 different "AV"s as though they were the same is symptomatic of not being able to see beyond the abstraction that is AV. conceptually AV is an abstraction that encompasses a variety of disparate preventative, detective, and recovery techniques. most people, however, just see AV as a magic box that you turn on and it just protects things. the only component that behaves anything like that is the real-time scanner, but it is not the only component in a security suite (especially an enterprise security suite) by any stretch of the imagination.

failing to see beyond the abstraction means, unfortunately, that you will fail to argue intelligently about the subject. it also means you will probably fail to make effective use of AV. if you don't know how a tool works, how can you possibly hope to use it to your fullest advantage? furthermore, how can you value something you don't understand? just as you can't price a car based on the effectiveness of the wheels, you shouldn't value AV based on the supposed effectiveness of the scanner.

one of the things that also became clear as i read some of the subsequent tweets was that robert seems to think there's nothing special about his attacks. but the fact is his attacks are special. as a penetration tester he launches targeted attacks. targeted attacks take more effort, more human capital to execute, and he himself described some of that extra effort. this basically means targeted attacks are more expensive to launch than the more automated variety and that they don't scale quite as well. consequently targetted attacks represent a minority of the overall attacks being performed. note, however, that that doesn't necessarily mean targetted attacks are a minority of the attacks a particular organization sees, as it's entirely possible that an organization may be a juicy enough target to receive a great deal of attention from targeted attackers.

from what i can tell, dan kaminsky also has difficulty seeing beyond the abstraction of AV. much of what he says above reflects the idea that AV is a magic box that you simply turn on and it should protect you. in the preceding example it appears he thinks there's an expectation that organizations solve all their security problems with scanners when in fact the expectation is simply that they have AV suites in their toolbox and that they use the appropriate tool (which may or may not be part of the suite) for the job (as rik ferguson attempted to explain).

dan also quoted the DBIR as showing AV to only be effective 3% of the time. i wondered about that so i looked a little deeper. DBIR stands for Data Breach Investigations Report. let the meaning of that phrase sink in a little bit. a data breach investigations report is a report about data breach investigations. data breach investigations are investigations of data breaches, and data breaches only occur when all the effort you put into preventing them failed.

you can't judge how successful something is by only looking at it's failures.

one of the consequences of this is that dan has actually failed to understand the statistic he's reported. the 3% where AV detected the breach still represents a failure because it was a detection after the breach had happened. this can happen due to things like signature updates (a scanner can detect more today than it could detect yesterday).

another consequence is that trying to use the DBIR to evaluate the effectiveness of AV represents a self-selected sample bias because the failure itself causes the event to be included in the study. a success would have excluded the event from the study. now one might have entertained the possibility that dan simply wasn't familiar with selection bias, but as we will see, that appears to not be the case.

it appears that dan has in fact heard of selection bias before, not to mention the WildList too (bravo). unfortunately it doesn't appear that he can use them properly.

• AV testers don't define the WildList, WildList reporters do (in a sense, but it's probably more accurate to say the the WildList is a result of a particular type of sampling)
• AV testers typically don't do WildList testing, although virus bulletin does offer a WildList-based certification in addition to larger, more inclusive tests
• if a product only detects 90%+ of the WildList, it's generally considered to be crap, because the WildList is the absolute bottom of the barrel of performance metrics. anything less than 100% is an embarrassment.
• AV testers don't define the set of malware they're going to test against, they cull samples from as wide a variety of real-life sources as they can and describe them as being 'in-the-wild' so as to distinguish them from malware that only exists in a 'zoo' or malware that was whipped up in a lab for testing purposes (something that's generally frowned on)
• defining what you're going to measure is not actually selection bias. "Selection bias occurs when some part of the target population is not in the sampled population" ("Sampling: Design and Analysis" by Sharon L. Lohr)  (now THAT's a textbook definition of selection bias - good thing was minoring in statistics in university). if testers defined their target to be the samples they already had then by definition there couldn't possibly be any selection bias because the target population and the sample population would be the same set.

this isn't to say there isn't selection bias in the tests performed by AV testers. it's entirely possible that some classes of malware (perhaps even targeted malware, for example) are harder to find samples of due to circumstances outside the testers' control. that being said, that bias is a lot more subtle than looking exclusively at failures.

now, it just so happens that i continued digging into the DBIR beyond just figuring out what went into it, and came across a rather interesting chart.

i highlighted the part that should really be interesting here. just to be clear, this only covers organizations that have to be compliant with PCI, but unless organizations that are legally obligated to run up-to-date AV are somehow magically more stupid than the rest of the organizations, the rest of the organizations actually have less motivation to run up-to-date AV and so their numbers are probably as low if not lower. 

now what this means is that there is really no way at all to use the DBIR to evaluate the effectiveness of AV because it appears that most of the organizations included in the report can't even follow the most basic of AV best practices. it also suggests that dan hasn't read his own sources thoroughly enough. if i'm not mistaken his 3% figure comes from the year when only 53% of PCI-bound organizations were running up-to-date AV. the subsequent year it was 1% of breaches discovered by signature-based anti-virus, and in the most recent one, AV doesn't appear to have helped after the fact at all.

that ever decreasing percentage of organizations running up-to-date AV is actually kind of disturbing, and it makes you wonder what's hurting organizations more; the amount of money they pay to AV vendors or the amount of attention they pay to security experts pontificating on subjects they are demonstrably ignorant of?

i anticipate that there will be those thinking that all i do is criticize and that i have nothing constructive to offer, so let's think about how we'd really measure AV effectiveness. the independent AV tests are apparently not good enough - in fact dan kaminsky went so far as to say this:

so how would we measure effectiveness really? like effectiveness in the field where AV is actually getting used? well first of all we stop limiting our data collection to just those instances where AV failed, because that's just ridiculous. no, we'll need to collect data on each and every time the AV raises an alert as well, to supplement our failure statistics. oh, and we'll have to follow up each of those alerts to make sure they aren't false positives because those certainly don't contribute to the effectiveness of AV. we'll also have to use all of the AV suite, rather than just parts of it, in order that people can determine if the effectiveness justifies the money they pay for the entire suite. additionally, we'll need to control for variables - different organizations have different security measures and controls that they deploy in conjunction with AV that may stop some malware before the AV gets a chance. that's not a bad thing, of course, and if they all used the same security measures then we could collect data on how effective AV is under those particular circumstances. but because each organization has different measures, they'll affect the AV results to differing degrees and that will skew the measurement. so we'll have to get organizations to either all use the same complementary measures or get them all to stop using any complementary measures. neither of which seem very likely in production environments, so that leaves us with trying to simulate what happens in the field - but then we get back into the AV testing lab territory which apparently 'no competent soul on the planet' trusts.

the reality is that it doesn't matter what kind of test you do, it's never going to match people's anecdotal experience. that's because testing is inherently designed around the idea of arriving at a single set of results representing how effective AV can be - it's never going to be able to reflect what happens when variables such as complementary controls, sub-optimal operation, targeting motivation, etc. aren't controlled for - and in the real world they aren't controlled for. tests necessarily reflect ideal circumstances and your mileage may (probably will) vary.

were i to be overly judgmental i might sign off this post with this little gem by none other than robert graham himself that i found yesterday will going through my RSS backlog:

The problem with our industry is that it's full of self-styled "experts" who are adept at slinging buzzwords and cliches. These people are skilled at tricking the masses, but they have actually zero expertise in cybersecurity.

but i prefer the school of thought from my own post about security experts from 2006 - security is just too big for anyone to be an expert in all parts of it. it seems to me that the expertise of dan and robert lie elsewhere.

it's important for people to recognize their own limitations, and i believe it's also important to recognize the limitations of the authorities you're listening to as well, lest you give credence to well meaning but uninformed experts. anti-malware is a complicated field, more complicated than i think either dan or robert realize, and if they have difficulty seeing beyond the AV abstraction imagine, how many other people do as well. i hope someday dan and robert and other experts like them can gain a deeper appreciation for how complex it is so that they can pass that along to those who depend on them to do the heavy cognitive lifting.

i wandered lonely as a cloud

relax, i'm not about to start waxing poetic about daffodils. rather i'm thinking about cloud-based anti-malware software.

it's something i've been thinking about for a little while now but i've finally decided to commit my thoughts to a more permanent format and share them with others.

for the past couple of years the major anti-malware vendors have been deploying cloud technology to improve the effectiveness of their products. often this has been an optimization specifically for their known malware scanners, although some have also taken the opportunity to build reputation systems.

it occurred to me that the cloud could be used for a great deal more than just that. think about what those reputation systems are doing. the user is faced with a complex question - is file X safe - and the cloud answers. the cloud can do this either because there are experts feeding the cloud it's answers or because there's a community feeding the cloud it's answers (or both, come to think of it). the point is that the cloud reduced the complexity for the user.

now think for a moment about all those technologies that have sprung up and then fallen by the wayside over the years. how many of them fell out of favour because they required too much knowledge, because they asked too much of the user? do you see where i'm heading yet? the cloud as a complexity reducing technology (alright it technically transfers and collates that complexity, but from the user's perspective it reduces it) seems like it actually has the potential to breathe new life in virtually all of those other techniques, be they sandboxing, whitelisting, behaviour blocking, or even integrity checking.

and of course, as i was originally coming up with that list i was reminded of the fact that many of them have actually been augmented with some kind of cloud technology to help take the complexity out of their operation. those efforts simply haven't been particularly mainstream. the biggest vendors have been slow to recognize the opportunity to augment these technologies (which can be superior in the right hands) with complexity reduction as a service. the smaller vendors that are taking a chance with this don't necessarily have the stability to keep it going. it would be nice if those other options saw more more mainstream deployment and adoption.

the covenant is broken

one month ago i published a blog post excoriating mcafee for being involved with a firm that creates and sells malware. for one month i've been waiting - not for a reaction to my own post, but a parallel reaction from the industry to the revelation that mcafee was involved with malware creators. i have been underwhelmed by the response (or lack thereof), and somewhat overwhelmed by the implications.

take a moment to let the AV industry's silence on this matter sink in. what does it mean? does it mean that they can't say anything because they've all got similar skeletons in their closet? or does it mean they're just not interested in capitalizing on that sort of thing anymore?

you see, for a long time there's been a persistent rumour that AV vendors don't just partner with malware writing companies, they hire malware writers outright. the AV vendors, of course, claim that that doesn't happen - they claim to have a policy against hiring malware writers and they say not to just take their word for it because their competition would take advantage of such ethical lapses if they were ever to occur.

they weren't just blowing hot air, either. making an example of an anti-malware company that hired a virus writer has happened in the past (thank you f-secure), so we know that such self-correcting controls have previously been in place. we were supposed to trust AV companies because they were financially motivated to do the right thing. every company had something to lose if they misbehaved and every other company had something to gain if they caught someone misbehaving.

but now the revelation that mcafee works with a malware writing company comes along and nobody has anything to say. well, to be specific, no company has anything to say (since i know there are individuals who felt strongly about this but may not have been able to speak for their employers). it was the job, the duty, of every member company in the anti-malware industry to act as a watchdog for the industry in case things like this happened, and you all failed. each and every one. one month later is too late to strike - the opportunity has passed - the iron is no longer hot.

the industry was supposed to be policing itself, but that no longer seems to be happening. without that, all we have is their word that they should be trusted, but those are just words. nothing but sweet, sweet words that turn into bitter orange wax in my ears (to quote futurama's philip j. fry). without action it means nothing.

the industry's accountability is gone. we can't honestly believe they're still policing themselves now. they used to adhere to and enforce the anti-malware community's standard of ethical behaviour. it's important to draw a distinction between the anti-malware community and the anti-malware industry at this point. although there has always been significant overlap, there has also always been those who were part of one set but not the other. obviously i'm not in the anti-malware industry, and i can think of a number of people who were members of the community long before they became part of the industry. on the other side, do you think HR is staffed by anti-malware community members? the legal department? upper management may have a few here or there, but for the most part they're just ordinary business folks. increasingly, the anti-malware industry is representing business interests instead of the values and ideals of the anti-malware community. the community's influence in the industry has been gradually waning up to this point where there's no one left who can realistically hold them accountable for violations of the community's standard of ethical behaviour.

they can still be held accountable on technical grounds, i suppose, but for how long? anti-malware testing was in a bad state for a while - AMTSO has been helping to elevate the quality of testing, but does the anti-malware industry (which is increasingly losing touch with the anti-malware community) have too much influence over the goings-on there? ideally the inclusion of both the anti-malware industry and anti-malware testing industry should create a balance. the testing industry has an understandable bias towards the more practicable approaches to testing (they have limited resources, after all) and a strong motivation to not appear to be going to easy on the vendors. the vendors, on the other hand, have insights into the inner workings of their products which are sometimes necessary to understanding and eliminating certain sources of testing bias and a strong motivation to perform well on tests. this should create a balance that forces both sides to take harder but ultimately superior paths. as the industry moves away from ethical accountability in favour of business concerns, it stands to reason that they may start to move away from embracing technical accountability as well - and realizing their input in AMTSO feeds back into a system that enables technical accountability, they may try to game the system for their own ends.

this highlights yet another problem. not only does the industry itself become suspect, so does everything it touches. it's not just accountability that's gone, it's credibility as well, and that lack of credibility can be toxic to others.

when the anti-malware industry no longer represents the values and ideals of the anti-malware community, when the bottom line takes priority over everything else, the result is bad for everyone. it's bad for the users because they will eventually have little left but to choose between crappy products in pretty boxes. it's bad for the anti-malware community within the industry because their jobs will cease to be fulfilling and they will be increasingly disturbed by the actions of their employers. it's even bad for the anti-malware community outside the industry simply because of association and the failure of most people to recognize any distinction between the industry and community. this isn't something that happens overnight. this isn't something that started one month ago. it's been going on for a while and you community members in the industry are all frogs in a pot that is being slowly brought to a boil.

i don't know how to correct this. i don't even know if it can be corrected (damage has already been done, and you can't always go back to the way things were). i don't pretend to have those kinds of answers. if i had to guess, i would guess that turning the industry around and getting back on course would take as much influence as the community can muster. full recovery may not be possible, but is not trying really an option? an alternative may be to restore accountability through external sources, but given the particulars in play (a company that sells malware to a nation state), that would involve scrutiny from other nation states and being investigated by scores of foreign nations on an ongoing basis doesn't sound appealing.

there can be no credibility without accountability and there can be no accountability without consequences. that house of cards depends on consequences and as near as i can tell there have been none.

security: it's almost like it isn't there

one of the ideas i continue to encounter over and over again throughout the years is the idea of liking a particular security product because it seems like it's not even there. it's amazing where one can find that idea being expressed. panda security's own luis corrons said the following about his wife's impression of panda's product:

My wife’s computer also have it, and she loves it, mainly because she doesn’t realize that it is installed :)

liking a security product because it seems like it's not even there strikes me as suggestive that the person in question likes to ignore security or not be bothered by security concerns. for most people this is going to be a recipe for eventual disaster. luis' wife, however, has luis on hand to take care of any malware incidents, so i guess for her it's ok. it's an interesting and probably effective strategy - well played, mrs. corrons, well played.

most people can't marry an anti-malware expert, however, so placing value in product's ability to shut up is the wrong way to think about things for most of us. don't get me wrong, if a security tool is too 'chatty' then certainly that poses a usability problem, but the quest for complete transparency is a symptom of mismatched expectations.

the predominant expectation among consumers is that if they install the 'right' product or combination of products then they can forget about all those nasty threats because they'll be protected. that's just not true, though, and it's never, ever going to be true.

people will actively defend this line of thinking, however, often they say they just want to do X and don't want security getting in the way. imagine if i said i just wanted to get to mcdonald's and didn't want traffic safety to get in the way - would that sound reasonable? not so much, i imagine. part of the reason for that is that most of us realize that following certain procedures on the road actually does keep us safer than we would otherwise be; but another part is that we also recognize that when others don't follow those procedures they put us and everyone else at risk, not just themselves.

what if i were to tell you the same principles apply in computer security? there are procedures you can follow that not only allow you to reach your goal in a reasonably secure way (whether that goal is getting work done or enjoying online entertainment or whatever else you use your computer for). not only that but by not following those procedures, by ignoring security, one actually does put other computer users at risk as well. i'm not just talking about other people who use the same computer, either. back in the days of viruses, when a virus infected a computer that computer joined the set of computers from which that virus could further it's spread. essentially it enlarged the platform from which the virus could attack still other systems. today, in the age of the botnet, the same principle applies. when a machine becomes compromised it get's added to the attack platform and assists in attacks on other systems, whether those attacks are simply sending out spam or sending out more malware or performing distributed denial of service attacks. by pretending like security isn't a concern a user puts not only themselves but all other computer users at risk as well.

now, likening secure computing practices to safe driving does not mean i'm trying to argue in favour of requiring users to have a license to operate a computer (though there are those who suggest that). the fact is that day to day life is full situations where you have to take precautions to increase your safety. just crossing the street calls for the precaution of looking both ways first. even toasters (which i bring up because some people literally think computers should be as simple to use as toasters) have safety precautions you need to follow - unplug the thing before you try to retrieve that piece of toast or bagel that's stuck inside.

i often criticize the security industry for perpetuating the myth of install-and-forget security, but the consumer shouldn't be thought of as blameless. people need to wake up and take responsibility for their own safety and security online, as well as being good online citizens and not putting others at undue risk. seriously, folks, computing without the need for taking active precautions is pure fantasy and it's time you started living in the real world. if you don't take responsibility for keeping yourself safe and secure, you won't be safe and secure - period.

scareware and human frailty

about two ago the folks at trend published a blog post about the persistence of fake av that really got under my skin. it was around the time of my birthday, which is ironic because the following quote really takes the cake:

Online, however, FAKEAV is a good example of a social engineering “success story.” By leveraging human weakness, FAKEAV effectively utilizes social engineering techniques such as blackhat search engine optimization (SEO) to trick users.

if there's one time a vendor should not be laying the blame for users being fooled on "human weakness" it's when talking about scareware.

scareware generally presents itself to the user in very much the same way legitimate security products do. vendors should consider that maybe scareware purveyors can be so effective while imitating legitimate security vendors is because of how close legitimate security vendors' messaging is to being an outright scam in and of itself.

use our software. we'll protect you.

worry free computing

we take care of X so you don't have to

you need our solution

the virus problem is solved

etc.

while those wearing vendor-coloured glasses may see the average user's propensity to believe the messaging put forward by illegitimate security vendors as nothing out of the ordinary (and certainly nothing to do with them themselves), i see over 2 decades of marketing and media training the populace to be as unquestioning, as unthinking as a pack of lemmings in a mindless frenzy when it comes to what security vendors say (whether they're really security vendors or not).

it's not human frailty at work here, it's bad guys figuring out how to exploit the one thing that security vendors are loathe to change: their marketing and business practices. legitimate (or so-called legitimate) security vendors made the market for scareware. the scareware purveyors are just showing up to the party, putting their hands out, and having a slice of the pie handed to them on a silver platter.

if the security industry really wants to do something about scareware purveyors, they should stop acting so much like them and start fostering skepticism amongst the populace - not only skepticism in what others say but also in what you yourselves say. stop creating an environment where scareware flourishes. stop doing their market development for them and actually start dismantling that blind-trust based market in spite of the fact that it's paid you so well in the past.

the bad guys are milking your cash cow, vendors. it's time to stop treating customers like cattle. it's time for you to lead rational critical thinkers rather than herd livestock. it's time for you to stop being part of the problem.

well mine's infinity big

so sophos seems to be a little late to the mine's bigger party, but at least they're following the go big or go home philosophy to make sure everyone notices them. which is unfortunate - for sophos.

see, while the numbers game that most vendors play sometimes has the faint aroma of snake oil, kris braun's bold statement that sophos 'currently detects an infinite number of malicious files' (with a big infinity graphic, no less) has an overpowering stench of snake oil.

it's one thing to use big but meaningless and impossible to verify numbers, it's another thing to claim things that are literally false. sophos can't currently detect an infinite number of malicious files because an infinite number of malicious files do not currently exist.

claiming to detect not only more than you can verify you detect, but more than actually exist is ridiculous. you might as well say you recognize 15 different boolean outcomes or can write out all 57 letters of the english alphabet.


sophos may, arguably*, have the potential to detect an infinite number of things, but that's a lot different than being able to currently detect an infinite number of things, and you can bet that the average person not only can't tell the difference but will look at the claim of infinite detection much like they would a claim of perfect detection (in spite of kris' explanations to the contrary).

so congratulations go to sophos for the ignominious feat of innovating in the field of snake oil. now please do yourselves a favour and stop being so boastful.

(*technically, an infinite number of files would require that some of those files be infinitely long, since limiting them to a finite length would also limit them to a finite number. since there is not yet a computer capable of holding an infinitely large file let alone indexing one, there's strong evidence to suggest that sophos cannot actually detect those malicious files that happen to be infinitely large and thus cannot currently detect an infinite number of malicious files.)

no i will not disable my anti-malware...

... or any part thereof  just because you think it's causing your software to crash on my system.

and who is giving users the 'lower your guard for a better experience' advice today? why google of course.

Other software has incompatibilities with Google Chrome, but you may be able to solve your crash issues without disabling the entire software.

If you have Internet Download Manager (IDM), disable the 'Advanced browser integration' option within IDM (go to Options > General).

If you have NVIDIA Desktop Explorer, try removing the nvshell.dll library using the steps on this site: http://www.spywareremove.com/security/how-to-remove-dll-files/

If you have FolderSize, try the fix on this site: http://sourceforge.net/tracker/?func=detail&aid=2900504&group_id=127365&atid=708425

If you have NOD32 version 2.7, upgrade to the latest version of NOD32 or disable internet monitoring in NOD32 2.7.

now, to be fair they did  also suggest upgrading, and that is a good option - in fact it should have been the only option they offered when it came potential anti-malware software conflicts for a couple of reasons.

1. turning things off is a lot less hassle than upgrading so if people think it'll work they're more likely to take that option
2. the current landscape is such that anti-malware software is most people's only defense against malware. 

ideally i'd like to see people put up additional defenses, but at the very least we should try to avoid telling them to lower the only defense they've got.

and yes, i did notice that they only suggested disabling the internet monitoring; well guess where most new malware is coming from these day - yeah, that's right, the internet. some parts of an anti-malware product may be a little less useful than others but internet monitoring, when the internet is a major malware vector, is really kind of important.

i know google has a few smart folks when it comes to security, but they were also rather famously compromised using ancient microsoft software. one wonders if they disabled anti-malware protection in that case too. i'd like to see some of the security know-how that a few of them have distributed more broadly across the entire company so that well-meaning but misguided engineers don't give out dangerous advice in the name of improving the user experience with the browser.

bad advice: disable your av

i'm sure you can probably think back and remember at least one example of a piece of software you've installed in the past that came with directions suggesting you disable your anti-virus before you install.

that little pearl of anti-wisdom has been with us a long, long time and has always irked me. the software industry, whether too lazy, too hurried, or to ignorant to know any better, decided that rather than resolve whatever conflict they may encounter with security software it would be better to train users to drop their defenses.

what could possibly go wrong?

plenty could go wrong, of course, not the least of which being that once the users are effectively trained to drop their defenses when told it would be trivial for malware authors to distribute their wares with instructions telling people to disable their av. who needs to develop technical measures to bypass anti-virus when you can simply tell users to turn it off for you and have them do it like good little trained monkeys.

well, nowadays more and more applications are being delivered to users over the internet, specifically over the web, and along with that, anti-malware products are focusing increasingly on web-borne threats. as such you can probably guess this was bound to happen eventually:

yeah, this WTF moment was brought to you by a facebook application development company called chainn who thought it would be a good idea to tell users to disable (or even remove) norton anti-virus. hey, it worked well enough back in the days of desktop applications, right? why not facebook applications too? i mean, really, what's more important, online safety or finding out how you fit in amongst your friends?

apparently they also have some problems with adblock plus - i wonder why?

fred cohen says anti-virus doesn't work

the EICAR (european institute for computer antivirus research) conference was held earlier this week and something which may at first blush seem quite curious happened... fred cohen (a keynote speaker, the father of computer viruses, and the man behind the seminal academic treatment of the subject) said that anti-virus doesn't work...

i wish i'd been there to see that (actually no i don't, i almost certainly would have rolled my eyes, crossed my arms and gone to sleep), but since i'm not part of the industry and flying over to berlin on my own dime is kinda out of the question i had to miss out... it seems i wouldn't have even heard about it if not for randy abrams posting about it here... i've waited patiently for a couple of days now for anyone else to write something about it so i could get more details of what was actually said, but that doesn't seem to be forthcoming... even eicar's site only tells me that the title of his talk was supposed to be "Computer Virology: 25 Years From Now"...

so, since i don't have first hand experience of the conference to give me a clue as to what on earth he was thinking, i went and did some research to see if anything he's said or written in the past might give me a clue and i believe i've found what i was looking for...

in an article on his site titled "Unintended Consequences" it starts to become clear that he feels the computer security industry is in the practice of detecting and reacting to bad things instead of preventing and deterring them, and he paints that in a rather negative light... when i read this i was struck - if he thinks anti-virus (probably one of the most mainstream parts of the computer security industry) is about detection and not prevention then he's doing it wrong!... in my earlier post on defensive lines in anti-malware i illustrate pretty clearly how anti-virus technologies (both the ones that security numpties consider av and the ones they don't) are employed as preventative measures, preventing such things as access to threats, execution of threats, and behaviour of threats (among other things)...

the thing is, fred cohen is really not the sort of person you'd expect to be using av wrong... thankfully i found a pair of interviews with him - one called "Three Minutes With Fred Cohen, Virus Trends Tracker", and the other "Fred Cohen - Not all virus writing is a crime"... although anti-virus is certainly used for prevention, it's underlying mechanisms are largely detection-based - however when cohen talks of prevention he's referring to the type where the underlying mechanism is sort of like an immunity... it all became clear to me when he said we should be using systems that are less susceptible to viruses in the first place - systems that have more limited functionality... this relates back to his 1984 paper Computer Viruses - Theory and Experiments where, in the section on prevention (i really should have looked there earlier) he discusses the 3 key properties of a system that allow viruses to spread - sharing, transitivity of information flow, and the generality of interpretation...

systems with limited functionality refers specifically to the third of these properties - the generality of interpretation - by limiting the functions that a system can perform you limit what can be done as a result of interpreting data as code... cohen poses the examples of turning off macros in ms word, or turning off javascript in the browser - i would follow those up with more contemporary examples such as microsoft recently announcing the disabling of autorun for flash media, or the increasing trend (at least among security conscious users) to use alternate PDF viewers as opposed to Adobe's own Acrobat Reader which has been bloated with unnecessary (and frankly unsafe) functionality for a long time...

i'm not going to try and detract from his arguement by saying the idea doesn't have merit - it does... i even considered whether my defensive lines post needed another line regarding system immunity, though i later realized that limited functionality is already covered by existing lines (as it's just another method of preventing exection and/or particular behaviours)... the problem isn't that his argument in favour of using this technique has merit, the problem is figuring out how he could think anti-virus is broken and this technique isn't - or what he thinks the security industry can do about limiting functionality...

the very existence of the security industry underscores the fact that there has historically been no (and largely still is no) place for security in the rest of the computer industry... limiting functionality is the domain of the original product developers, not 3rd party security companies - the norton's and mcafee's of the world don't have the freedom to cripple Acrobat Reader for example (at least not without getting sued)... and the original product developers by and large aren't limiting functionality because that goes against consumer expectations for technology... technology is supposed to empower us to do more, to do things better/faster, etc - so it's a given that when we choose something we expect to empower us, we aren't going to go with the less powerful option... this is born out by history as well because there have always been less functional/powerful options available and the market has consistently selected against them in favour of the bloated product suites and application platforms...

furthermore, limiting applications is all well and good but there's plenty of malware out there that exists as native binary executables, and limiting the operating system's functionality (or worse, that of the hardware itself) to prevent things like self-replication or botnet command and control communication is a far trickier proposition...

so given the difficulty in getting more adoption of limited systems and given the fact that even cohen isn't suggesting limiting things to the point of systems with fixed first order functionality, i can't help but wonder how he could consider this any less flawed an approach in practice than anti-virus... it is clear to me that it will be just as possible to work around the limitations he's proposing as it is to work around anti-virus technologies, and limited functionality defenses are a lot less agile than things like known-malware scanning so when those special cases come along it will cost a great deal more to redesign a system to further limit functionality so as to deal with such special cases...

that should give you a clue as to my opinion of limited functionality-based defenses - they have promise as a complement to anti-virus techniques... anti-virus may not work in the sense that it's not completely effect at preventing malware infestation, but limited functionality systems aren't either... anti-virus may not work in the sense that it's a computationally expensive approach to prevention, but limited functionality has logistical costs for development, marketing, and maintenance that we may never completely overcome... i fully endorse the idea of selecting software that only has as much power and functionality as you need and no more, but i still think you should use anti-virus right along side that...

frankly, at the end of the day, the main difference between systems that are designed to have more limited functionality and ones where anti-virus is deployed is that the former's limitations are baked in while the latter's limitations are bolted on (because like it or not when an anti-virus stops a piece of malware from executing on your system it is effectively limiting your system's ability to perform the function that malware represents)... that appears to be what cohen's real contention is about - that baked in is better than bolted on... in some ways that may be true, but in others (like the matter of agility) it's definitely not (because it's easier to change the bolt than it is to change the whole system)...

defensive lines in anti-malware revisited

now, i realize i've covered the topic of defensive lines in anti-malware before, about 2 years ago, but i've refined my thinking since then and now i've found the motivation to share (guess i was just being lazy before)... this thread on wilders security forums made me realize that there are some misconceptions about defensive lines in anti-malware... specifically the ideas of what can be the first or last line of defense seem to require some refinement so here goes...

if you're thinking about multiple lines of defense then it makes sense to consider that there is a sequence of defensive lines at various 'distances' from the final outcome of complete system compromise... that is to say that there are various opportunities (some earlier, some later) during the course of a piece of malware's progression towards compromising a system at which you can prevent a system from becoming compromised, and each of these points a defense can be mounted... the first line of defense, therefore, should be the one furthest away (or at the earliest opportunity) from that possible outcome... likewise the last line of defense necessarily must be the last chance for preventing that outcome...

so what is the first possible line of defense then? well, first opportunity you have for avoiding having your computer get infested with malware is the point of initial exposure - if you don't encounter the malware in the first place then there's nothing else to say about it... when it comes to defending oneself the main defense here is actually risk aversion behaviours like not going to so-called 'dodgy' sites, not plugging strange flash drives you found on the ground into your computer, etc... when it comes to defending others, any malicious site take-down effort will also help prevent people from coming across the malware on malicious sites that get taken down... somewhere in the middle are tools like mcafee's site advisor, or google's malware warning functionality that can help users determine which sites to avoid...

the next logical point at which you can prevent compromise is preventing the transfer of the malware onto the machine we're trying to protect... for automatic transfers (such as drive-by downloads or vulnerability exploiting worms) the primary defense is closing the avenues by which such automated transfer can take place, usually by applying security patches (at least where browser exploits are concerned)... a basic NAT enabled router can also protect machines behind it from certain types of automatic transfer by virtue of preventing unsolicited traffic to those machines... for manual transfers, risk aversion behaviours again play an important role - only downloading software from reputable sites (preferably direct from the vendor's site) being a prime example... a somewhat after-the-fact but related defense is on-demand scanning all incoming materials (whether you think they could carry malware or not) before allowing them to be accessed for any other purpose (sort of like a mandatory quarantine period)...

after potential malware has successfully been transferred onto the system the next opportunity to prevent it's progression to full compromise is preventing access to it... this is where on-access scanners come into play... if the user can't access the malware then there should be no way they can trigger it...

if your malware access controls fail to prevent access (say because the malware is packaged within some obfuscating wrapper, or perhaps the malware is just new) then the following point at which full compromise can be avoided is by preventing execution of the malware... on-access known-malware scanning can prevent execution as well, at least for known malware (if it's in some obfuscated wrapper like a dropper, then when the malware is 'dropped' it will no longer be obfuscated and that camouflage will no longer be protecting it from the scanner)... application whitelisting can also prevent a great deal of malware (even that which is too new to be considered 'known' yet) from executing (by virtue of preventing everything that isn't already explicitly allowed to execute from executing) so long as the user doesn't give the malware permission to run...

after malware has successfully begun to execute on a system, any certainty about preventing that malware from compromising that system is gone, but that doesn't mean there aren't still possibilities for prevention... it may still be possible to prevent compromise by preventing one or more of the malware's bad behaviours using behavioural blacklisting, behavioural whitelisting, or some combination thereof (all of which falling under the general umbrella of behaviour blocking)... in so far as certain behaviours involve accessing system resources that can have their access restricted, operating as a user who doesn't have access to those resources (ie. running as a non-administrative user, following the principle of least privilege) will prevent a great deal of existing malware from being able to operate properly and thus can prevent compromise...

finally, if malware gets past all those previously described defensive lines, there's still one possible opportunity to full system compromise... if you can avoid the consequences of the malware's behaviour by being lucky enough (or by managing the circumstances well enough) that the malware was actually running in a sandbox rather than on the main host system then it will be the sandbox that gets compromised rather than the main host system... sandboxes are such that their compromise is usually inconsequential because they can be regenerated easily - though extrusion of sensitive data may still be a problem, if the malware was contained withing a sandbox then compromise of the host will have been avoided... otherwise the system will have become compromised, prevention will have completely failed, and it would now be an issue of detecting the preventative failure, diagnosis to determine the extent of that failure, and recovery from it...

you can make any one of these stages your own first line of defense, but only by ignoring earlier opportunities to defend yourself... likewise, you can make any of these your last line of defense, but only by ignoring subsequent opportunities to defend yourself... think about what that statement means: ignoring opportunities to defend yourself... is that really something you want to do? does it sound wise? it certainly doesn't sound that way to me...

nuggets of misinformation

over the weekend martin mckeay published a post asking people what free av they used at home... the story is ordinary enough, i'm sure a lot of people out there have faced the problem of what anti-malware software to choose, whether a free one, one of the big name for fee ones, or none at all (and for the record, i'm not in any of those camps)... martin is a well known security blogger and podcaster, he knows about a lot of security and privacy related subjects, but from this fairly informal posting i now know that martin does not know av...

what caught my eye about his post came near the end where martin pointed towards this proactive detection test report as showing how ineffective av really is... for everyone's benefit, tests of proactive protection capabilities are tests specifically designed to bypass the signature-based portion of an anti-malware product so as to test only the heuristic components... that one word - "proactive" - all on it's own would tell someone familiar with this field that the test does not measure the overall effectiveness of products but rather just the effectiveness of a subset of the technologies in those products - and that word was right in the main heading for the report...

reading further (ie. reading the introduction) reveals that the subset of technologies tested is further constrained... the test only measures the effectiveness of static heuristic techniques only, no dynamic heuristics, nothing involving run-time behavioural detection or anything like that... it should be clear that when you're only testing a small part of a product your results won't indicate it's overall effectiveness...[EDIT dec. 16, 2008: turns out i read the intro wrong, however it's still only a test of the heuristic components of anti-virus products rather than of the entire products, and thus not a reflection of their overall effectiveness]

of course if you don't understand the terminology being used and only look at the numbers and the graphs then of course you might think this represents the overall effectiveness - that's probably why martin thinks the effectiveness of av is somewhere between 60% and 80% (not too different from the numbers on the report he points to) when the latest on-demand tests (which still don't include run-time behavioural detection, but do include a broader range of the detective capabilities of the products) performed by both av-comparatives.org and av-test.org place the effectiveness of most products well above 90%...

sadly of all the people who responded to his post, none of them seem to have noticed this interpretation error so far... i'm sure everyone has heard the idiom that there are lies, there are damn lies, and there are statistics... since numbers can be so misleading, it behooves one to familiarize oneself enough with a topic to at least properly interpret those numbers so that you can't be so easily fooled by them...

does av really suck that badly?

while looking through my rss feeds today i saw this comment rich mogull posted about a week ago (don't ask why it took so long to reach my reader, i don't know)...

primarily it's his observation about malware, anti-malware, and the mac platform and community, but he ends the comment much more generally with this:

To be honest, I think desktop AV sucks in general and isn't nearly as effective as everyone would like us to think.

this is probably a common enough sentiment among the more technically savvy crowd... i wouldn't go so far as to say this is part of the anti-av movement, but rather a consequence of the mismatched expectations people have with regards to anti-virus software and the persistent mischaracterization of av as being solely about virus scanners...

i can understand where the opinion is coming from - if you look just at scanners, and more to the point if you look just at what populist media reports about scanners then the image you get is of something that fails a lot... but here's the main problem with this line of thinking (besides the issue of what av is, which i think i covered adequately before) - no preventative measure is an island complete unto itself...

as i mentioned in my post about the blacklist value proposition, the primary benefit of a scanner is to take care of the exceptions that aren't covered by other measures... scanners have never, ever been the sole preventative measure in play, they've always been complementing something else... even when the only technological measure present was a scanner, there were still procedural measures, there was still common sense, there was still (in the distant past at least) the relative disconnected nature of the computing ecosystem, etc... judging a scanner's effectiveness in isolation as though it were supposed to take care of the entire problem all by itself is like judging how well table salt satisfies your appetite...

the problem is that people think the scanner is supposed to take care of the threat all by itself, and they think that because av marketing departments have been feeding them that line of rubbish for something like two decades now and they aren't really taking many steps to correct the imbalance in the image they're creating and the mismatched expectations they're giving the public... this is why i often frown on marketing, why i've accused those who overuse the concept of protection as snake-oil peddlars, and why i cringe when someone calls a set of security tools a solution...

the problem isn't the technology, the problem is what people understand (or fail to understand) about the technology, and by extension the thing that causes the misunderstanding... as mark linton points out, there is a definite false sense of security being fostered here, and as cd-man suggests in pointing to that same post, that false sense of security is causing harm - possibly even more harm than a scanner can make up for... av companies need to wake up and realize that by allowing their own marketing departments to subtly lie to the public they're going to be shooting themselves in the foot in the long run... by operating in bad faith they are increasingly losing the faith of consumers - and not only will that accelerate when the idea that av sucks makes it into mainstream public consciousness, but it is also very hard to win back once lost...

but back to rich's opinion - i don't think he's entirely wrong, av isn't nearly as good as it's often made out to be, but rich and probably a lot of other people out there are being so profoundly affected by the reality-distortion field put up by av marketing that when they finally start to see a glimmer of reality through a thin spot in the fog bank they see a stark contrast between it and the marketing message and start rejecting everything in the marketing message, even though the best lies are those that are hidden among truths, and come to equally imbalanced conclusions... the opinion is one that smacks of not seeing the whole picture... as i keep saying, what most people call av (the scanner) should be part of a larger whole, not abstracted out on it's own.. further, it's not everyone who wants you to believe av is so good, it's really just marketing (stop listening to marketing; seriously don't even bother rejecting what they're saying, just don't let them affect your thinking at all) and the corporate big-wigs who care more about market share than they do about actually contributing to their customers security and well being (ahem john thompson ahem)... there are plenty of honest, ethical, technical people in the anti-malware industry trying to spread a more balanced message, but they may not be as easy to find as the pitch on the outside of the product's packaging...

why perform virustotal-based av tests?

probably most people with any familiarity with the anti-malware field has heard of virustotal.com - for those that haven't, it's an online service that runs the commandline version of a collection of av scanners against submitted samples in order to perform static analysis on them and determine if they're known malware (or perhaps close enough to known malware to be picked up by static heuristics)...

as has been well stated by others - virustotal is for testing samples not for testing anti-malware software... unfortunately that doesn't seem to stop everyone and their grandmother (apparently) from performing comparative and/or effectiveness testing on anti-virus products using the virustotal service...

there are a number of reasons why you shouldn't perform av tests using virustotal, including:

• those of us who know better will laugh at you - no, seriously, we will
• virustotal doesn't (can't) include the full detective capabilities of the av products they're using and therefore tests based on their service misrepresent the effectiveness of those products
• even the people who run virustotal say such testing methodologies are bogus right on their own site
• retrospective testing already provides results on the effectiveness of av products against new/unknown malware (and it already makes av look pretty bad)

those seem like pretty compelling reasons not to do this kind of testing and yet the practice persists... here are a couple reasons why people might still do it regardless of the reasons not to:

• it costs too much to do things the right way (proper testing takes a lot of work, time, and resources)
• people are lazy and virustotal can appear to be a convenient short-cut to getting things done, even though it's really just a short-cut to irrelevance
• some people seem to be genuinely ignorant of the irrevocable problems with test designs that use virustotal to compare scanners or gauge anti-virus technology
• related to ignorance but on a grander scale, some people may simply not be capable of designing a scanner test that even flirts with validity, nevermind one that is actually somewhat valid
• there are some pervasive misconceptions about anti-virus products/technology/vendors/industry that some people have an irrational need to affirm

of course that's just for individual people, when a security company (or worse, an anti-malware company) uses virustotal for quick and dirty av testing then it raises serious questions about the competency of that company's staff... although i have hinted before at the connection between innovation and not being constrained by the 'this is the way we've always done things' mentality, that isn't a license for the security industry to throw scientific rigor out the door...