expert misuse of the term "virus"

in the past, the argument many experts have used against putting in the effort to actually correct people's misconceptions about what a computer virus is and what it does and how it's different from other forms of malware have centered on the idea that it doesn't really matter to a victim what kind of malware they have. when they have malware on their computer, all they care about is getting rid of it. i can understand this school of thought, even though i've never agreed with it. however, the world has changed. 

this is a post stuxnet world and defenders/victims aren't the only ones we need to consider anymore. we now have people barking orders to create digital weapons, and the details of those orders matter. if they say make me a virus then someone will make them a virus even if they didn't understand what they were asking for - and there's evidence to suggest they don't. 

stuxnet is generally believed to have been a targeted, covert operation, but it used a noisy, untargetable* type of malware - a computer virus (or more specifically a worm). there were and are better ways to achieve the ends we believe the creators of stuxnet were after, so one can only assume that high level decisions were made in ignorance of the differences between malware types and the consequences those differences would have on that type of operation.

the question, then, of whether to put in the effort to educate people about the proper use of the term virus can no longer be answered by looking exclusively at the victims who want their computers to be clean. it is now also necessary to consider aggressors who want to use malware as a weapon to serve national interests. as misguided as such behaviour is, we have to accept that it's happened and will continue to happen, and actually knowing the differences between malware types may mean the difference between a surgically precise operation, or one with a lot of collateral damage. 

this isn't to say that i think we should start helping aggressors create and/or launch their digital weapons. i still don't believe in helping the bad guys, and even if i believed such nationalistic aggressors weren't bad guys (which i don't), i don't believe there's any way to help them without also helping those who are much more unambiguously bad. what i am saying, however, is that this particular form of ignorance that experts have been too lazy to address can cause real harm and ignoring it means ignoring the opportunity to reduce the unintended harm such people will cause.

(*many believe stuxnet was highly targeted, but there's a distinction to be made. while it's destructive payload was highly targeted to a very specific environment, it's self-replication was not - it spread far beyond it's intended target)

more on bromium and snake oil

in my previous post about bromium i looked at claims that a technology reporter made about their technology (that it would kill all malware forever), noting that it was for all intents and purposes snake oil, and suggesting that the folks at bromium were doing the public a disservice by failing to dispel the false sense of security that that sort of reporting/opinion generates.

bromium's tal klein took exception to this based on what he believed to be my misunderstanding of their technology, and suggested i go read their white paper, among other things. here's some friendly advice for all you vendors out there: when someone calls you out for snake oil and you tell them to go read your white paper because they don't know what they're talking about, you better make sure they don't find more snake oil in your white paper - especially not in the second paragraph. and i quote:

It defeats all attacks by design.

that's a rather bold claim, don't you think? sort of suggests perfect security, doesn't it? but wait, there's more in the third paragraph:

It revolutionizes information protection, ensures compliance – even when users make mistakes, eliminates remediation, and empowers users – saving money and time, and keeping employees productive.

the emphasis above is mine. "eliminates remediation" or "eliminates the need for remediation" is something of a recurring theme in bromium's marketing materials. you can find it in their introductory video, and even hear a version of it from the mouth of simon crosby himself in their video on isolating and defeating attacks.

the only way you can eliminate remediation is if prevention never fails. but there is no such thing as a prevention technique that never fails. all preventative measure fail sometimes. if you believe otherwise then i've got a bridge to sell you (no, not really). perfect prevention is a pipe-dream. it's too good to be true, but people still want to believe and so snake oil peddlers seize on it as a way to help them sell their wares.

so it would appear that things are actually worse than i had originally thought. not only is bromium letting 3rd party generated snake oil go unchallenged, they're actively peddling their own as well. now just to be clear, i'm not saying that vsentry isn't a good product, from what i've read it sounds quite clever, but - even if you have the best product in the world, if you make it out to be better than it is (or worse still, make it out to be perfect) and foster a false sense of security in prospective customers, then you are peddling snake oil.

customers may opt to ignore the possibility for failure and the need to remediate an incident, but i wouldn't suggest it. to re-iterate something from my previous post, it's an isolation-based technique. although they often like to gloss over the finer details, their isolation is not complete - they have rules (or policies if you prefer) governing what isolated code can access outside the isolated environment, as well as rules/policies for what can persist when the isolated code is done executing. this is necessary.  you're probably familiar with the aphorism about if a tree falls in forest and no one is around to hear it, does it make any sound? well:

 if code runs in an environment that is completely isolated, does it do any useful work?

the answer is a resounding no. useful work (the processing of data to attain something that can itself be used by something else) has not occurred. all isolation-based techniques must allow for exceptions because of the nature of how work gets done. we divide labour, not just between multiple people, but also between multiple computers, multiple processes, multiple threads, and even multiple subroutines. we need exceptions to isolation, paths through which information can flow into and out of isolated environments, so that the work that gets done in isolation can be used as input for yet more work. this transcends the way the isolation is implemented, it is an inescapable part of the theory of isolating work.

and that is a weakness of every isolation-based technique - the need to breach the containment it affords in order to get work done. someone or something has to decide if the exception being made is the right thing to do, if the data crossing the barrier is malicious or will be used maliciously. if a person is making the decision then it boils down to whether that person is good at deciding what's trustworthy or not. if a machine is making the decision then, by definition, it's a decidability problem and is subject to some of the same constraints as more familiar decidability problems (like detection - after all, determining if data is malicious is as undecidable as determining if code is malicious). in the case of vsentry, a computer is making the day to day decisions. the decisions are dictated by policies written by people, of course, but written long before the circumstances prompting the decision have occurred, so people aren't really making the decision so much as they're determining how the computer arrives at it's decision. the policies are just variables in an algorithm. the decisions made by people involve what things vsentry will isolate (it only isolates untrusted tasks, not all tasks), but people deciding what to trust and what not to trust is basically the same thing that happens in a whitelisting deployment or when people think they're smart enough to go without anti-virus software, and we already know the ways in which that can go awry.

vsentry may have scored a perfect score in an evaluation performed by NSS Labs using malware and an operator utilizing metasploit, but that doesn't mean it's perfect anymore than receiving the VB100 award makes an anti-virus product perfect. they weren't able to find a way past vsentry's defenses because vsentry is still new and still novel. it will take time for people to figure out how to effectively attack it, but eventually they will. the folks at bromium need to tone down their claims and take these famous words to heart:

Don't be too proud of this technological terror you've constructed - Darth Vader

no, bromium will not kill all malware forever

over the weekend a discussion broke out on twitter (as discussions are want to do) about a somewhat overly optimistic article concerning the new anti-malware apple of the security community's eye: bromium.

the primary tactic that bromium uses (or at least the primary one that people focus on) is isolation/sandboxing. bromium's vsentry product uses virtualization on a per-process basis to isolate every process from the system and from each other. that level of granularity for isolation is a lot higher than most sandboxing efforts can give you. while there are certainly benefits to that granularity, there are also drawbacks.

perfect isolation is actually not desirable, we want and even need to be able to use the results of one process inside another one. the more sandboxes you have, he harder this is to manage. the folks at bromium have opted to address this issue using rule-based systems to decide what something in a sandbox can access as well as what to do with any changes that are left when the sandboxed process is finished. rules which, in all likelihood, the administrator can modify to suit their needs.

now, while the article in question is reasonably good at explaining what bromium's vsentry does, the author (jason perlow) takes the arguably naive view that this sandboxing technique can stop all possible malware (as evidenced by the article's headline: "Bromium: A virtualization technology to kill all malware, forever"). the reality, however, is that there are limits to what sandboxing can do, and as clever as the folks at bromium are, they aren't clever enough to deliver on the promise that headline makes.

that's a problem, because people are going to read that headline, see nothing in the article to actually contradict it, and believe that it's actually true. have we seen claims like that before? sure we have - saying it can kill all malware forever is not intrinsically different from claiming 100% protection. it's classic snake oil, only in this case it's not the vendor that's spreading it (as far as we know - we don't know exactly what the folks at bromium may have said to mr. perlow, only that they say the headline is his words, not theirs).

i suppose that should mean there's no problem, right? the vendor's hands are clean, after all. the snake oil is being spread by a third party. the vendor isn't doing anything about it in this case or previous cases that have arisen because, let's face it, they benefit from it. it's good for bromium's business if people think vsentry is better than it actually is, at least in the short term. in the long term, the kinds of mismatched expectations that creates are the same kind that the AV industry struggles with daily.

it is bromium's responsibility to control how their products are perceived, and by failing to take action they are giving tacit approval to the snake oil being spread on their behalf. their hands are not actually clean, they are dirty through negligence. however, i didn't really expect any better of them (though i did give them an opportunity to surprise me) and you probably shouldn't either. tread carefully - caveat emptor.

know your enemy: security vendors

just to be clear, i'm not suggesting that vendors are waging some kind of war against their own customers - they aren't (usually) that kind of enemy. but by the same token, vendors are not your friends either. when it comes to laying out strategies for protecting yourself and your stuff, it's important to know what category to place the various players involved, and vendors are best thought of as adversaries.

to better explain what i mean, imagine you're sitting around a table with your friends playing the classic board game monopoly. although these people really are your friends, in the context of the game, their goal is to win at everyone else's expense. in serving their own interests, they act in ways that don't serve yours and in fact may sometimes be in direct opposition to your interests. in this way it can be said that you and your friends have competing interests.

the customer and the vendor are generally not competing with each other in the conventional sense, but their interests are not aligned and in some cases the interests do compete. you as a customer have an interest in keeping your computers, intellectual property, banking credentials, etc. safe and secure. vendors also have an interest in that to a certain extent, but protecting you and your stuff is not a vendor's highest priority.

vendors are companies. as such their highest priority is the bottom line. without the bottom line, the company ceases to be. companies don't just start up out of thin air, they need money; which means they have investors and those investors expect a good return on their investment, or else it's not a good investment and they might not invest anymore in the future, or maybe even pull out their stake in the company. companies also have operating expenses. they need to pay to keep the lights on and the machines running, and they need to pay their employees who themselves have expenses (families they need to feed and put roofs over their heads). therefore the company has to make profit it's priority. the way vendors make money is by vending - they sell a product and the more product they sell the more money they make.

in theory if the product is good then they'll sell more of it, but it doesn't need to be good enough to stop all the threats to you or your stuff - vendors aren't competing with the bad guys, they're competing with each other, so they only need to be better than other vendors. what's more, since technical 'goodness' is difficult for customers to accurately quantify, the vendor only needs their product to be perceived to be good. technical quality is still required up to a point, of course, because you can't fool all the people all the time. but, since your buying decisions as a customer are based on perception, and that perception can be altered/manipulated more cheaply through marketing than through technological advancement, companies engage in this kind of shortcut to help them maintain or even advance their market position.

how does this compete with your interests as a defender of yourself and stuff? well, in a few different ways, actually:

1. by conventional falsehood, they make their product out to be better than it is and so draw you away from something that may actually suit your needs better (example: look at any vendor that's ever claimed to be able to take care of all/100% of any kind of threat)
2. by omission, they make solving your security problems seem easier than they really are because nobody wants to make the customer swallow a bitter pill about how much work is really involved in staying safe, especially when their competitors aren't doing it (example: how many vendors will tell you about what you need to do when their product doesn't work? how many will even talk about that scenario?)
3. by framing the issue, they make the customer think about the customer's security issues in the vendor's terms, thereby favouring the vendor's proposed 'solution' rather than formulating strategies to meet the customers own unique, individual needs (example: a number of anti-malware vendors used to provide generic detective controls in the form of integrity checkers, but those seem to be mostly gone now and vendors instead talk about technologies based on having varying degrees and types of knowledge about threats, while 'generic detection' (of a different sort) has become a glossed over, value added feature of their scanners)

all of these work against your interests in protecting yourself and your stuff. they work against you finding the best tool for your job, or figuring out everything you need to do, or even knowing there's more to it than just using the vendor's product.

before you get the wrong idea, i don't want you to think this is a condemnation of the people who work for vendors. individually, many of them may well be much closer to being your friend and being on your side than the company they work for as a whole is. their interests are never perfectly aligned with yours, of course. you won't see them sacrificing their own interests (their families, their money, their jobs) for your benefit, and you wouldn't really expect them to, would you? some of them (a scant few when you consider the total number that security vendors employ) will sacrifice some of their time and energy to help people (whether their company's customers or no) learn about the threats that are out there and thus be better armed against those threats. just because someone works for a vendor doesn't mean their character is a reflection of the character of the corporate entity that employs them. yes, companies are run by people but it's their collective behaviour that makes the character of the company. the phrase "none of us are as cruel as all of us" doesn't just apply to anonymous, nor does it just apply to cruelty. 

i also don't want you to think this is a condemnation of vendor companies either. remember, they're not exactly enemies in the conventional sense, but rather adversaries. as much as i tend to refer to them as bad actors, or irresponsible, or any number of other judgmental labels, i can't really see how they could work any other way. the judgments are really just a way of highlighting the divergence of interests between the vendor and the customer. there is some variation in the degree to which they do the the things that they do, of course. smaller companies are more easily influenced by noble ideals, in part because of size and in part because they have less at stake and so can afford to be more 'innovative' in how they operate. it doesn't always work that way, and it doesn't mean their bottom line isn't still the bottom line, but some take a more scenic route to their goals.

that being said, the fact remains that vendors' interests do not align with those of their customers (i.e. you). that means it's important to take what they say with a grain of salt and to evaluate whether the things they say or do or produce are really of actual benefit to you. pick over what they have to offer, take what you can use and throw away the rest. in essence, forage on the enemy.

the abc's of security

over the years i've found myself becoming increasingly dissatisfied with the boiler plate advice i formulated when i was younger, as well as all the other boiler plate advice i've seen/heard given by other people, and even the very concept of boiler plate advice itself. this includes things like best practices (aren't you done practicing yet?) and really any simple, prescriptive answer to questions involving how to keep oneself secure. more and more they seem like incomplete or obsolete anachronisms that aren't suited to the diverse and ever changing circumstances in the real world. never mind the fact that everyone's values (and thus their priorities) are slightly different from each other so boiler plate advice is rarely a really good fit - and of course people's priorities change over time, too.

i've grown and evolved as a security user (a user of security), and no boiler plate seems capable of reflecting my reality anymore. it's just not how i think about or approach the problem of keeping myself secure anymore and i find it difficult to direct others down such fixed, one dimentional paths.

and yet i know people still need advice and direction in order to grow themselves. the subject of first principles and fundamentals occasionally comes up and so i thought to myself what is the most fundamental thing in all of security? if there was just one thing about security that i could impart to another human being, what would it be? the answer is surprisingly simple, surprisingly complex, and surprisingly not limited to just security but in fact really a life lesson that happens to have meaning within security.

the most important thing for anyone to remember when it comes to defending yourself and the things and people you care about is this:

when i say this, i don't mean changing mindlessly like some derivative of the crazy ivan maneuver from the movie hunt for red october (although being unpredictable certainly has tactical advantages) but rather that you change what you do to protect yourself in intelligent, mindful ways. you should always be learning, always growing, always evolving, always adapting, always improving. don't stand still because your adversaries certainly won't be and you don't want to fall behind (or at least any further behind).

there are no easy answers, no matter how many people may be offering them (it seems like everyone does), and no matter how well-intentioned they may be.

debating AV effectiveness with security experts

a rather disheartening conversation took place on twitter over the weekend. as public conversations sometimes do, it grew beyond any capability i have to do it justice through description, so instead i'll provide some screenshots and links to a couple of branches of the discussion.

because i don't follow either dan kaminsky or robert graham, i knew nothing about this discussion until someone retweeted the tweet pictured below (i included as much context as i could):

what first made me take interest in this was that robert graham seemed to be talking about 2 different things as though they were the same. the AV that's only 4% effective (or 0% when he's done with it) is different than the AV that organizations pay 40% of their budget on.

the apparently ineffective AV is actually the scanner component of the AV; as you can see he describes his methodology for bypassing it - a methodology that essentially amounts to malware q/a, which happens to be a countermeasure against heuristic detection, which is a feature of scanners.

the AV that organizations pay 40% of their budget on (assuming that's an accurate figure, i wouldn't know) is the enterprise security suite, which includes other things beyond just the scanner. for example the tweet by dan kaminsky that seems to have started the entire conversation alludes to the failure of symantec's product to stop 44 out of 45 pieces of malware in the recently publicized attack on the new york times. but as symantec rightly pointed out, their product included a reputation system which, for all intents and purposes, behaves much like a whitelist - if something doesn't have a good reputation (and new things have no reputation at all) then it will be flagged. that is about as different from a traditional scanner as one can imagine and bypassing it isn't nearly as straightforward.

talking about 2 different "AV"s as though they were the same is symptomatic of not being able to see beyond the abstraction that is AV. conceptually AV is an abstraction that encompasses a variety of disparate preventative, detective, and recovery techniques. most people, however, just see AV as a magic box that you turn on and it just protects things. the only component that behaves anything like that is the real-time scanner, but it is not the only component in a security suite (especially an enterprise security suite) by any stretch of the imagination.

failing to see beyond the abstraction means, unfortunately, that you will fail to argue intelligently about the subject. it also means you will probably fail to make effective use of AV. if you don't know how a tool works, how can you possibly hope to use it to your fullest advantage? furthermore, how can you value something you don't understand? just as you can't price a car based on the effectiveness of the wheels, you shouldn't value AV based on the supposed effectiveness of the scanner.

one of the things that also became clear as i read some of the subsequent tweets was that robert seems to think there's nothing special about his attacks. but the fact is his attacks are special. as a penetration tester he launches targeted attacks. targeted attacks take more effort, more human capital to execute, and he himself described some of that extra effort. this basically means targeted attacks are more expensive to launch than the more automated variety and that they don't scale quite as well. consequently targetted attacks represent a minority of the overall attacks being performed. note, however, that that doesn't necessarily mean targetted attacks are a minority of the attacks a particular organization sees, as it's entirely possible that an organization may be a juicy enough target to receive a great deal of attention from targeted attackers.

from what i can tell, dan kaminsky also has difficulty seeing beyond the abstraction of AV. much of what he says above reflects the idea that AV is a magic box that you simply turn on and it should protect you. in the preceding example it appears he thinks there's an expectation that organizations solve all their security problems with scanners when in fact the expectation is simply that they have AV suites in their toolbox and that they use the appropriate tool (which may or may not be part of the suite) for the job (as rik ferguson attempted to explain).

dan also quoted the DBIR as showing AV to only be effective 3% of the time. i wondered about that so i looked a little deeper. DBIR stands for Data Breach Investigations Report. let the meaning of that phrase sink in a little bit. a data breach investigations report is a report about data breach investigations. data breach investigations are investigations of data breaches, and data breaches only occur when all the effort you put into preventing them failed.

you can't judge how successful something is by only looking at it's failures.

one of the consequences of this is that dan has actually failed to understand the statistic he's reported. the 3% where AV detected the breach still represents a failure because it was a detection after the breach had happened. this can happen due to things like signature updates (a scanner can detect more today than it could detect yesterday).

another consequence is that trying to use the DBIR to evaluate the effectiveness of AV represents a self-selected sample bias because the failure itself causes the event to be included in the study. a success would have excluded the event from the study. now one might have entertained the possibility that dan simply wasn't familiar with selection bias, but as we will see, that appears to not be the case.

it appears that dan has in fact heard of selection bias before, not to mention the WildList too (bravo). unfortunately it doesn't appear that he can use them properly.

• AV testers don't define the WildList, WildList reporters do (in a sense, but it's probably more accurate to say the the WildList is a result of a particular type of sampling)
• AV testers typically don't do WildList testing, although virus bulletin does offer a WildList-based certification in addition to larger, more inclusive tests
• if a product only detects 90%+ of the WildList, it's generally considered to be crap, because the WildList is the absolute bottom of the barrel of performance metrics. anything less than 100% is an embarrassment.
• AV testers don't define the set of malware they're going to test against, they cull samples from as wide a variety of real-life sources as they can and describe them as being 'in-the-wild' so as to distinguish them from malware that only exists in a 'zoo' or malware that was whipped up in a lab for testing purposes (something that's generally frowned on)
• defining what you're going to measure is not actually selection bias. "Selection bias occurs when some part of the target population is not in the sampled population" ("Sampling: Design and Analysis" by Sharon L. Lohr)  (now THAT's a textbook definition of selection bias - good thing was minoring in statistics in university). if testers defined their target to be the samples they already had then by definition there couldn't possibly be any selection bias because the target population and the sample population would be the same set.

this isn't to say there isn't selection bias in the tests performed by AV testers. it's entirely possible that some classes of malware (perhaps even targeted malware, for example) are harder to find samples of due to circumstances outside the testers' control. that being said, that bias is a lot more subtle than looking exclusively at failures.

now, it just so happens that i continued digging into the DBIR beyond just figuring out what went into it, and came across a rather interesting chart.

i highlighted the part that should really be interesting here. just to be clear, this only covers organizations that have to be compliant with PCI, but unless organizations that are legally obligated to run up-to-date AV are somehow magically more stupid than the rest of the organizations, the rest of the organizations actually have less motivation to run up-to-date AV and so their numbers are probably as low if not lower. 

now what this means is that there is really no way at all to use the DBIR to evaluate the effectiveness of AV because it appears that most of the organizations included in the report can't even follow the most basic of AV best practices. it also suggests that dan hasn't read his own sources thoroughly enough. if i'm not mistaken his 3% figure comes from the year when only 53% of PCI-bound organizations were running up-to-date AV. the subsequent year it was 1% of breaches discovered by signature-based anti-virus, and in the most recent one, AV doesn't appear to have helped after the fact at all.

that ever decreasing percentage of organizations running up-to-date AV is actually kind of disturbing, and it makes you wonder what's hurting organizations more; the amount of money they pay to AV vendors or the amount of attention they pay to security experts pontificating on subjects they are demonstrably ignorant of?

i anticipate that there will be those thinking that all i do is criticize and that i have nothing constructive to offer, so let's think about how we'd really measure AV effectiveness. the independent AV tests are apparently not good enough - in fact dan kaminsky went so far as to say this:

so how would we measure effectiveness really? like effectiveness in the field where AV is actually getting used? well first of all we stop limiting our data collection to just those instances where AV failed, because that's just ridiculous. no, we'll need to collect data on each and every time the AV raises an alert as well, to supplement our failure statistics. oh, and we'll have to follow up each of those alerts to make sure they aren't false positives because those certainly don't contribute to the effectiveness of AV. we'll also have to use all of the AV suite, rather than just parts of it, in order that people can determine if the effectiveness justifies the money they pay for the entire suite. additionally, we'll need to control for variables - different organizations have different security measures and controls that they deploy in conjunction with AV that may stop some malware before the AV gets a chance. that's not a bad thing, of course, and if they all used the same security measures then we could collect data on how effective AV is under those particular circumstances. but because each organization has different measures, they'll affect the AV results to differing degrees and that will skew the measurement. so we'll have to get organizations to either all use the same complementary measures or get them all to stop using any complementary measures. neither of which seem very likely in production environments, so that leaves us with trying to simulate what happens in the field - but then we get back into the AV testing lab territory which apparently 'no competent soul on the planet' trusts.

the reality is that it doesn't matter what kind of test you do, it's never going to match people's anecdotal experience. that's because testing is inherently designed around the idea of arriving at a single set of results representing how effective AV can be - it's never going to be able to reflect what happens when variables such as complementary controls, sub-optimal operation, targeting motivation, etc. aren't controlled for - and in the real world they aren't controlled for. tests necessarily reflect ideal circumstances and your mileage may (probably will) vary.

were i to be overly judgmental i might sign off this post with this little gem by none other than robert graham himself that i found yesterday will going through my RSS backlog:

The problem with our industry is that it's full of self-styled "experts" who are adept at slinging buzzwords and cliches. These people are skilled at tricking the masses, but they have actually zero expertise in cybersecurity.

but i prefer the school of thought from my own post about security experts from 2006 - security is just too big for anyone to be an expert in all parts of it. it seems to me that the expertise of dan and robert lie elsewhere.

it's important for people to recognize their own limitations, and i believe it's also important to recognize the limitations of the authorities you're listening to as well, lest you give credence to well meaning but uninformed experts. anti-malware is a complicated field, more complicated than i think either dan or robert realize, and if they have difficulty seeing beyond the AV abstraction imagine, how many other people do as well. i hope someday dan and robert and other experts like them can gain a deeper appreciation for how complex it is so that they can pass that along to those who depend on them to do the heavy cognitive lifting.

imperva's anti-virus study is garbage

Enough is enough! I have had it with these motherf#$%ing flakes on this motherf#$%ing train of thought - (what i imagine samuel l. jackson might say if he were following this nonsense about imperva)

in case you are unfamiliar, imperva (a security vendor of some sort) commissioned a bunch of students from the technion-israel institute of technology to perform an evaluation of the efficacy of anti-virus (all anti-virus as a whole, apparently, rather than comparing them to each other) by uploading 82 found samples to virustotal. yes, you read that right, it's another virustotal-based test.

these days i have a number of alternative avenues to express myself that i didn't have when this blog was still young, and that can often sate my need to express my feelings on some topic. i can make snide comments on twitter, or even parody tweets from a satirical twitter account. in fact i can even make memes about it. unfortunately none of that has proven sufficient in this case because the hits just keep coming.

you see, imperva keeps shopping this quackery out to more and more media outlets where it gets gobbled up and regurgitated uncritically by writers/editors (who really ought to know better if reporting on this sort of topic is part of their actual job) and thus gets put in front of more and more eyeballs of those who realistically can't know better. along the way it can even collect somewhat supporting voices from venerated members of the security community like robert david graham

or wim remes

let me be clear, however - this is all wrong. as has been repeated over and over again, virustotal is for testing samples, not anti-malware. they say it themselves in their about page

The reason is that using VirusTotal for antivirus testing is a bad idea.

and

BAD IDEA: VirusTotal for antivirus/URL scanner testing

those statements alone should be enough but, because virustotal later talks specifically about comparative tests, imperva (and others) have tried to argue that imperva's test is OK because it doesn't compare products to each other. however...

VirusTotal's antivirus engines are commandline versions, so depending on the product, they will not behave exactly the same as the desktop versions: for instance, desktop solutions may use techniques based on behavioural analysis and count with personal firewalls that may decrease entry points and mitigate propagation, etc.

this makes it pretty clear that the product a customer installs is very much a different thing from the program that virustotal uses - they will in most cases behave very differently and so the results that virustotal spits out cannot be considered representative of what actual users of anti-malware products will experience.

(ironically,  a product that appears to fare best in a virustotal-based test may actually be the worst because a higher focus on the type of static (often signature-based) detection that virustotal best measures could be to cover for a weakness in (or absence of) more generic/dynamic detection capabilities.)

but don't just take my word for it, let's hear from a couple of people who actually work at virustotal

yes, that's right, imperva's study is a joke. this shouldn't be surprising to long time readers of this blog since when i first wrote about this problem four years ago the first reason i gave for why you might want to avoid performing virustotal-based tests was that those of us who know better will laugh at you. i'm sure a number of people are laughing at imperva's gross incompetence (hanlon's razor makes me choose this explanation over the more sinister alternatives) but i'm afraid i can't consider the mess they're making to be a laughing matter.

promulgating ignorance in a security context has the potential to do real harm, and that is where i draw the line. that's why i'm writing this, that's why the title gets straight to the point, and that's why i'm going to start naming some names of people/organizations who have helped make this mess and who really ought to have known better. imperva has behaved like a dung beetle, persistently rolling this turd around, but somehow it keeps getting bigger like some katamari damacy of bullshit, and i think it's important to see the scale and scope of it and hold the people responsible accountable. it's worth noting, however, that somewhere deep down someone at imperva must also have seen the potential for their message to do harm - that's why the caveat that they weren't advising eliminating AV was added (as an apparent afterthought).

a non-exhaustive list of people/orgs who really should have known better, tried harder, and ought to be held to account for this growing mess is as follows:

• the technion-israel institute of technology because they presumably performed the test so that the test could have an air of objectivity it would otherwise lack if imperva performed it themselves
• the students who actually did the work because apparently no one bothered to read virustotal's about page
• the students' supervisor(s) who also failed the reading test and whose job it is to look out both for their students and the reputation of the learning institution that employs them
• imperva for coming up with this cockamamie idea in the first place and for later trying to defend it
• rob rachwald who posted the defense of the test on imperva's blog
• tal be'ery who helped with the defense of imperva's test
• amichai shulman (imperva cto) who is quoted far and wide by uncritical reporters
• clinton karr who seems to have written the press release that many others have simply regurgitated as their own reporting, perhaps as a result of getting that press release onto reuters
• the register which took the bait imperva offered
• richard chirgwin who posted an uncritical account of imperva's research
• techworld which took the bait imperva offered
• john e dunn who posted an uncritical account of imperva's research
• cso online which unthinkingly reprinted the techworld article
• the new york times (yes, that's right) who once again took the bait imperva offered
• nicole perlroth who posted an uncritical account of imperva's research
• the wallstreet journal who apparently uncritically summarize the work of others
• michael hickens who ran a summary of nicole perlroth's article as if it was gold inside an aggregate of summarized stories
• dark reading which posted a regurgitation of imperva's press release - is that what dark reading is supposed to be for?
• ihotdesk which took the bait imperva offered
• steven gaskill who posted an uncritical account of imperva's research
• pc magazine which took the bait imperva offered
• damon poeter who posted an uncritical account of imperva's research
• business computing world which took the bait imperva offered
• christian harris who posted a regurgitation of imperva's press release
• security bistro which took the bait imperva offered
• anthony m freed who posted an uncritical account of imperva's research

(i'm aware there's a lot more than this that you can simply find by googling sentences from the press release, i wish i had the time to make this list exhaustive - that said: reuters, the new york times, and the wallstreet journal... that definitely caught a lot of eyeballs)

now, perhaps you're thinking i'm being too hard on the journalists involved here. after all, they aren't experts. frankly, however, they don't have to be experts to see what's wrong with this test. if you're the type of reporter who reports on this type of technology then you should already know about virustotal and about how it can and can't be used. this isn't rocket science, or even some obscure nuance that only matters every 5th wednesday - not in the context of reporting on this subject. this is something reporters covering security technology ought to know. it's table stakes. you need to be this tall to get on the ride.

perhaps you think i'm being too hard on the students and their supervisor(s)? but this is academia we're talking about. they're expected to do their research, and i don't just mean the experimental research, i mean looking up and reading about the issues involved in designing and performing tests on anti-malware products. and their supeverisor(s) should have made sure they were doing their due diligence in this regard. frankly, in my time i've seen lone rank amateurs perform better tests than this with fewer resources. this is not acceptable academic performance.

and as for imperva themselves, well... if you intend to occupy part of the security industry that hopes to steal some of the AV industry's market, then you better know this stuff like it's the back of your hand. the institutional incompetence going all the way up the chain of command to the chief technology officer is astonishing and i'm surprised they managed to find someone with too many dollars and too little sense to give them funding, but i guess p.t. barnum was right about there being one born every minute.

imperva - do yourselves a favour and put a stop to this mess before it gets any bigger. you can't defend this junk computer science, the truth will eventually come out (it seems to have already started). you can't sweep it under the rug either, you've let things get too out of hand. the kind of smear campaign you're currently running was already attempted by the whitelisting industry years before you, and while that industry itself is still around and may even still be pumping out this same kind of junk, it didn't stop them from drifting back into obscurity. the way i see it the only way you can move forward sustainably is

1. admit your error
2. publicly retract your study
3. reach out to the journalists whose reputations have been tarnished by listening to you and apologize
4. assist the students you dragged into this in learning the error of the experimental methodology they followed (you can probably find a lot of good info either on or linked to from the anti-malware testing blog)
5. start over with a more intelligent methodology and try to make your case again with valid data

and if you can't manage to follow these steps then i'll be glad to watch you fade away or get swallowed up in a few years time, because the kind of incompetence you've been proudly displaying so far is not the path to success.

formula for the future

while reading over some of the prediction-posts that tend to spring up towards the end of the year it struck me that many of these predictions follow from pretty basic axioms. sometimes they're particular to the malware world but other times they're even more general and are simply combined with a malware concept to make a malware prediction. at any rate, i thought perhaps i should point out some of these things and help prediction-makers in the future, as well as to offer a different perspective on predictions of these types.

• the end of something as we know it - the only constant is change, you can never step into the same river twice (because having stepped in it the first time changed it), etc, etc. things are always different in the future than they were in the past. note that the prediction is rarely about the end of something but rather the end of something as we know it. all that means is that that something is going to change in a way that some people will consider significant.
• the dynamic equilibrium between focusing on software exploits and social engineering will continue to be dynamic - when exploits are hard to come by the bad guys will focus on social engineering to get the job done, because it is a job and they want to get paid. when exploits are easy to come by, focus on using them will increase because it's easier to fool an automaton consistently than it is to fool people consistently.
• software that has been popular to exploit in the past will continue to be popular to exploit in the future - bad guys will continue to focus on many of the same pieces of software because that's what their victims use and because they've had so much success there in the past. it's where the money is. good guys will continue to focus on many of the same pieces of software because that's what many people use and thus where the greatest impact can be made. in essence, so long as frequently exploited software remains popular among users it will continue to be a valuable target.
• we will learn more things about more things because of attacks - attacks are generally disruptive in one way or another. even if it's an attack that simply compromises confidentiality rather than availability, it still disrupts the process of using a system or service even if the system or service isn't itself disrupted. disruption has a tendency to highlight things that we would never have known if everything continued to run smoothly and the disruption had never taken place. disruption gives us an indirect view into things that might otherwise remain opaque.
• trends that are increasing will continue to increase, trends that are decreasing will continue to decrease - inflection points are rare. if they weren't it would be much more difficult to recognize trends in the first place. as a corollary, emerging trends will go mainstream. 
• the world is increasingly being made out of computers, and as new marriages of old-tech and computers become mainstream the resulting new-tech will become a target for attack - everything new opens up new possibilities for attack, and everything that is made new by sticking a computer in it opens up new possibilities for attacking that computer. furthermore, the more popular something is, the more profit can be had by attacking it, and so the more tempting a target it becomes.
• people will grow tired of defending themselves the same way they always have and try to find new alternatives - it is a quirk of human nature that we are always looking for new things. it is also true that we are largely unsatisfied with our current defensive capabilities (for whatever reason).
• new defenses will be developed to ward off new attacks and those defenses will be met with new offensive countermeasures - this is just the same offensive vs. defensive cat and mouse game that it's always been and always will be.
• new platforms will offer promise and seem secure, until they stop - figuring out how to successfully attack something without a lot of prior knowledge is difficult and time consuming, but it eventually happens, and the more it happens the faster additional attacks can be formulated until eventually we recognize that the platform wasn't the promised land we had hoped for.
• everything old will become new again - over time, as new people shift into a population and old people shift out, that population will collectively forget the past. at least for a while until someone figures out how to do new things using old concepts and then a renaissance occurs. we've already seen that occur with stealth, as well as boot sector infection/modification.

if you see the shadow of a prediction you've made in the list above, congratulations, you probably made a formulaic prediction. don't feel bad, you get better results using a formula than by trying to pluck the future out of thin air.