the myth of in-the-wild prevalence

upon reading this article at ghacks.net about scanning linux systems for viruses i became aware that there are some misunderstandings over the meaning of the term 'in the wild'.

the article in question is not the only place i've seen these misunderstandings and i don't want to knock it too hard because it does advise scanning your linux systems, but the statement that

Linux is immune to viruses right? Well…mostly. Even though a proof of concept virus has been discussed, and nothing has actually made it into the wild…you still have email on your system.

fairly clearly indicates both a lack of awareness of the threat linux faces as well as a lack of understanding of what constitutes as 'in the wild'.

so let's get this out of the way early in the discussion. 'In the wild' means literally that the malware in question is active and victimizing someone or some group, somewhere in the real world. that seems like an obvious and natural definition but what isn't obvious is the implication that that has for most people. you see many people equate 'in the wild' with epidemic. they think that if something were really in the wild it would have affected a lot of people and they would have seen it personally or known someone who had seen it. they think that they can use their own experience as a measure of whether something is 'in the wild' or not. the reality is that something being 'in the wild' does not mean that that something is common enough for you to have stumbled across it - there is a wide spectrum of prevalence possibilities for 'in the wild' malware.

to that end, there have of course been linux viruses in the wild. are there still some in the wild? well given that old viruses never really die, i'm going to have to say yes. remember, rare and 'in the wild' are not mutually exclusive concepts - something can be both at the same time. once something goes into the wild it's subsequently very difficult to conclusively show it has left the wild. in fact you could say it's equivalent to proving a negative (which, as we all know, is impossible).

(note: just to be clear, i'm not talking about the wildlist from wildlist.org. things that are on the wildlist are definitely 'in the wild' but not everything 'in the wild' gets to go on the wildlist. the wildlist is a much more narrowly defined set than what's 'in the wild')

lifehacker's mac anti-virus poll

if there's one thing that never fails to disappoint me it's the failure of the wisdom of crowds principle to work when it comes to malware-related topics, and this ask-the-reader style post on lifehacker lives down to that standard quite well...

you've got some people like astrosmash saying "There are no OS X viruses" - which ignores both the fact that there are in fact os x viruses (osx/leap.a is an overwriting file infecting virus, among other things) and the fact that anti-virus software targets non-viral malware too (of which there has been more than a few for the os x platform)...

you've also got people like texizboy saying:

I don't run A/V on some of my windows machines. All boils down to common sense in my opinion. Webmail services have helped out on this front also, to give credit where it's due, I believe there are less viruses getting around due to them.

despite the fact that email is just one of many different attack vectors that malware have been known to use for some time now, and despite the fact that not all malware is obvious enough for common sense to help (nevermind what they say about common sense)...

then there's people like kilianamphitrite saying:

The real strength of the Mac is that in general, when a Mac is running an untrusted bit of code, it is not doing so with system management privileges. Most of the time (and especially for home systems) Windows users run untrusted code as privileged users.

which incorrectly assumes that you need privileges to do bad things... a lot of windows malware depends on privileged access not because it's necessary for the ultimate goal of the malware, but rather just because such privileged access was almost always there so malware authors didn't have to think of alternatives...

on top of that you've got people like insomniac who says:

The idea of "Mac/Linux/Unix do not have enough market share so people don't develop a virus for them" is only partially true. Unix and Linux based systems are just a lot more difficult to infect because of their architecture and security design than a Windows machine (Vista does a much better job than previous versions of Windows).

which ignores the fact that the first academic treatment of the computer virus phenomenon back in the early-to-mid 80's had viruses successfully spreading in a professionally administered unix environment without aid from privileged users like root...

or how about sverrip who says:

I mainly surf around pages I trust, and don't download and open setup files like "Free-XXX.exe" on my Windows machine.

apparently ignorant of the fact that there is no such thing as safe/trustworthy sites, not to mention ignorant of the existence of the drive-by download vector... even the cbs website can serve malware to unsuspecting victims... and it's not like macs are immune to drive-by downloads - remember the safari carpet bombing flaw?

sad, isn't it? that people believe these fantasies about why they don't need anti-virus software on their mac (or in some cases even pc) computers... i have my doubts as to whether apple's quiet urging of people to use av is going to do anything at this stage of the game... the baseline level of ignorance about malware issues was bad enough but add to that apple's previous arrogance (which no doubt resonated with a lot of their fans) about security and the damage done is all but complete - the only thing left to do is wait for the fallout...

is anti-virus software falling behind?

as readers of my blog are probably aware i have a bit of a penchant for trying to dispel a variety of popular myths... one of the most popular ones i've dealt with is the notion that anti-virus software can't keep up - or as i put it before the myth of overwhelming numbers...

when i look at my previous treatments of this myth, however, i don't see something i could point a complete newbie at and have a reasonable expectation that they'd get it so i'm going to try and make this as simple as possible - there is no publicly available evidence that points conclusively to anti-virus software falling behind...

the only kind of evidence that would conclusively point to av vendors failing to keep up is a growing backlog of undetected malware... some people think the growing numbers of people who get hit with undetected malware while using up-to-date av products or the growing number of malware samples that are undetected at any given time is equivalent to this growing backlog but it isn't...

let's use an analogy to demonstrate this... let's say i have a dog... this dog shits on the ground and then i come along and clean up the shit... someone may (unfortunately) step in the shit before i get to it, but it does get cleaned up and so long as i clean up the same amount of shit my dog produces i'm not falling behind... now let's say i get a second dog - all of a sudden the amount of shit that hits the ground on any given day doubles, the chance of someone stepping in it before it gets cleaned up doubles, but so long as i'm still picking up as much as those two dogs drop i'm still keeping up with them and not falling behind...

this can go on and on with an ever increasing number of dogs and at some point i may eventually reach a point where i have so many dogs that i can't keep up, where they produce more shit in a day than i can clean up in a day and that leftover portion is a backlog which will build up day after day and become a real mess before too long... now, unless you're keeping track both of when the dogs shit and when i clean it up you have no way to determine if the increasing amount of shit currently on the ground getting stepped on represents a backlog or simply an increasing amount of shit being produced by an increasing number of dogs... if i were getting close to that undesirable point where the dogs produce more than i can clean up i would hire help to help me pick up after them (and/or maybe i'd build a robot or various other tools to help speed up the process)... with that help there would be more breathing room (figuratively) and we'd be able to increase the number of dogs and still not fall behind...

this is pretty much the same thing that goes on in av companies - as the amount of crap the malware writers produce increases the companies hire more analysts and develop better automated tools so that they can deal with an increasing amount of malware per day... that doesn't mean that the amount of undetected malware won't grow, it will... preventing that set from growing would require known-malware scanning to be able to detect malware before it was released (ie. before it was known)... known-malware scanning generally can't do that anymore than i can catch dog shit before it hits the ground... so long as they're analyzing as many as the malware writers are producing, though, they aren't falling behind - an increasing amount of undetected malware is not the same as a growing backlog of undetected malware... the growing pile of crap is an unavoidable consequence of the increasing production of crap and has nothing to do with whether or not anyone can keep up...

(yes, i did just compare malware to dog shit)

the myth of optimal malware

this is going to be an interesting myth-debunking post because the object of the myth actually exists...

there are any number of various optimal properties that a given piece of malware can have, such as novelty (new/unknown), rarity (targeted), stealth, polymorphism, anti-debugging tricks, security software termination, automatic execution through exploit code, etc...

likewise, there are any number of optimizations a malware purveyor can adopt, such as continually updating the malware, targeting the malware to a small group of people, spreading it with botnets, only using malware which is itself optimal, etc...

there really is malware using some or perhaps most of those tricks and there really are malware purveyors using some or all of those techniques so you may be wondering where the myth comes in... the myth comes in when we start considering these optimizations as being universal or at least close to it - that most if not all malware is as optimal as it can possibly be and that most if not all malware purveyors use the the most optimal deployment techniques they possibly can...

while each of those optimizations on their own seem to have become common place, few instances of malware can truly be considered optimal... for example, most malware is not targeted - sure the instances we hear about make for a great story and draws lots of readers, but they're a drop in the bucket next to the total malware population... novelty is seemingly even more popular since all malware starts out as new at some point, but as i've said before novelty is a malware advantage that wears off... polymorphism attempts to keep that novelty going indefinitely but it, along with novelty and targeting, were really only ever effective against known malware scanning - they hold no particular advantage against anti-malware techniques that don't operate by knowing what the bad thing looks like...

the same holds true for malware purveyors as well, few do what it really takes to get the most out of malware... otherwise malware would be much more successful than it is... even security conscious folks like you and i would be getting compromised left right and center because our anti-malware controls would just not be effective...

but that doesn't stop people from believing or falling into the logical trap that is optimal malware... i'm sure you've seen and perhaps even constructed arguments based on this fallacy... the anti-virus is dead argument is based in this as it posits that scanners are not effective because of new/unknown malware despite the fact that that malware doesn't stay new/unknown for long and that the effectiveness of known-malware scanning is precisely the reason the malware creators have to keep churning out new versions of their wares... the school of thought that says software firewalls are useless because malware can just shut them down or tunnel through some authorized process is likewise based on the myth of optimal malware because although some malware certainly does bypass software firewalls, not all do, and so they remain at least somewhat effective as a security control... in fact, any similar argument that says security technology X is useless because malware can just do Y to get around it is based on the myth of optimal malware as there is plenty of malware that doesn't do Y... i think i've even fallen prey to this fallacy on occasion when constructing arguments (so there's no need to point examples out to me, i know i'm not perfect)...

so keep this in mind the next time you run across a school of thought that attributes near supernatural abilities to malware - with truly optimal malware the malware purveyors would be able to get past most if not all our anti-malware controls all of the time (not unlike fooling all of the people all of the time), and since that isn't happening we can conclude that most malware is in fact not optimal...

the myth of what anti-virus is

if you're like most folks the term "anti-virus" elicits images of a virus scanner methodically checking each and every file on a system for something that matches one of it's hundreds of thousands of signatures... obviously that is the most well known aspect of the anti-virus field, but if you (like many others in this day and age) thought that that was all there was to anti-virus then you'd be dead wrong...

it is an exceedingly popular misconception that can be found underlying such pejorative statements as "anti-virus is a fundamentally flawed technology" and the new favourite "anti-virus is dead"... the simplest way to express the idea is that 'anti-virus == known virus scanning', but it's an idea that betrays a profoundly superficial view of the field... for example, it ignores the fact that the first anti-virus tools weren't scanners at all - flu_shot (which predates virtually any anti-virus product you've ever heard of) was a behaviour monitor/blocker and that was just one of many generic tools developed in the early days of the virus problem...

of course known virus scanning was developed before too long and became hugely popular (largely because it required and still requires the least amount of know-how from the operator) but although it thoroughly eclipsed generic techniques in market penetration it never completely displaced generic techniques from the spectrum of anti-virus technologies as evidenced by tools such as integrity master, advanced disk infoscope, chekmate, and invircible (all primarily integrity based tools)... then of course there was the rather visionary product (for it's day) called thunderbyte anti-virus which was probably the first instance of what would today be recognized as an anti-virus suite, containing a known virus scanner (with one of the most instructive examples of an early heuristic engine), a behaviour monitor/blocker, an application whitelist, and more (if memory serves, frans veldman and the gang at thunderbyte also had hardware designed for virus detection but obviously that was separate from their software offering)... later thunderbyte's technology was bought by norman data defense which then went on to become known for their sandboxing approach to virus detection... i should also note that throughout the 90's a number of what most would consider conventional anti-virus vendors included integrity checkers in one offering or another in part due to it's theoretically perfect ability to detect the effects of viruses that got past their scanners (assuming the operator was capable of using an integrity checker to it's full potential)...

so what was anti-virus really? what did the av community/industry consider it to be? basically anything that was intended to fight (and i don't just mean prevent) viruses... blacklisting (scanners), whitlisting, sandboxing, forensic integrity checking, etc... in other words, the basis for virtually every anti-malware technology today...

now there were a couple of developments in the late 90's (and just beyond) that bear some attention... one is that the incumbent anti-virus industry was slow to jump on the non-replicative malware problem, which obviously created a market opportunity for things like anti-trojan and anti-spyware tools... another is that some new entrants to the emerging anti-malware field realized that, although they could do the generic technologies as well or possibly even better than the existing anti-virus industry, there was no way they were going to be able to compete in what was still very much the anti-virus market without comparable scanning technology and that developing comparable scanning technology from scratch at that point was probably next to impossible... so instead they had to try to differentiate themselves not just from anti-virus companies but from the entire anti-virus industry (while still applying essentially the same techniques) which eventually lead to an apparently fractured market - but not before the av industry finally committed itself to non-viral malware and became fully anti-malware...

of course we still call them anti-virus products in spite of the fact that they are intended to fight a lot more than just viruses now... this is because the concept of a computer virus has a far better foothold in the public's psyche than this new term 'malware' has... and why not? the computer virus has had decades to penetrate into the public's awareness... so much so that the term 'infect' frequently gets used in the context of any and all malware, even by otherwise knowledgeable security experts... we also still call them anti-virus products in spite of the fact (often completely ignoring the fact) that they are increasingly embracing the comprehensive suite approach that thunderbyte anti-virus took over a decade ago (which, by the way, is the real reason for the interest in new testing methodologies, as the old methods are still perfectly reasonable for testing known malware scanning on it's own) and thus broadening the technological footprint of individual products to be closer to that of the field in general...

so no, anti-virus is not just scanning... it was anything intended to fight viruses and has now become an archaic reference to (and the root of) what is now properly referred to as anti-malware (anything intended to fight malware)...

the myth of meaningful informal anti-malware tests

bad tests are not necessarily a problem that is unique to the anti-malware field but it is one that those in the anti-malware community have encountered countless times... it's not that average folks are physically incapable of performing good tests, it's that they simply don't do it...

an informal test lacks strict adherence to established testing protocols, often because the standards set for good tests are so high that most people can't be bothered to go to all that trouble... only those who are dedicated to the process of anti-malware testing have ever done a half decent job of it because it requires a substantial investment of time and effort, not to mention a certain degree of expertise to ensure that the testbed is free of garbage/scanner fodder and duplicates...

i could stop there, i suppose, and hope that what i've said isn't too abstract, but i really object lessons and one sort of fell into my lap... dr. anton chuvakin has used the the results of one such bad test to support the now popular opinion that anti-virus is dead... i saw where this was going early on (the original question was obviously loaded)... i guessed the results would be bad and suggested the most likely reason was that the malware was too new - to which i was told that some samples were weeks old, though weeks can technically still be new enough if no one else has reported it yet... additionally, if a particular piece of malware isn't affecting many people the av companies may down-prioritize it in order to focus on samples that are affecting more people...

the test, as described so far, worked like this: someone who does incident response at a public university collected samples from compromised machines for a period of time and then when the batch was big enough (weeks after the collection was started) this person submitted them to virus total and took note of the results... the results were as follows: the best detection rate was 50%, the worst was 2%, and the average was 33%...

now, i can see a lot of problems with such a test but lets start with the big ones:

1. statistical significance - the sample size was too small... it may sound picky to those who want to believe, but even an unskilled labourer intuitively knows that asking 5 people their opinion on X does not give you a result that can be generalized to the entire population... this test (or at least the interpretation of it that we've been given) pretends to represent the ability of anti-virus products to protect against malware that is currently in the wild affecting people but most likely has a sample size of about 50 (50 is the minimum necessary for a product to get a 2% detection rate, but larger sample sizes make a perfect 50% detection rate increasingly unlikely)... since the wildlist has over 1600 different pieces of malware on it and since that only represents the self-replicating malware (which are believed to be in the minority now), a sample size of 50 (or even 100) just doesn't cut it...
2. sample selection bias - the samples came from compromised machines in a production environment... maybe that sounds reasonable to you but let me ask you this - if you test a variety of lie detectors against people who have proven themselves to be good at fooling lie detectors, are you really measuring how good those lie detectors are in the real world? the answer is no, you're only measuring how good they are against an artificially constructed set of people... the same goes for malware from compromised production machines - the malware you find there is the malware that has proven itself to be good at evading anti-malware scanners, not in-the-wild malware in general...
3. sample selection bias - the samples came solely from computers in an academic institution... sorry to say but universities are not a typical environment... if anything, they are the place where the incidence of truly malicious insider threats are the greatest (uni students practicing their l33t skills)... home users don't intentionally compromise their own computers, enterprise users may consider it (or even try it) but there are more/stronger controls in place (stricter logical access controls, more personally significant administrative controls, etc.) to prevent and/or deter it...
4. test-bed integrity - the test was carried out by incident response personnel... we really have no way of knowing how clean (in the sense of being free of detrimental garbage) the test-bed was... even if it was an incident response technician who was capable of reverse engineering each sample of malware (and had the free time in which to do it), simply reverse engineering the samples would not be enough to determine whether the samples were sufficiently different enough from one another to be considered distinctly different pieces of malware... and since many incident response technicians simply follow a script, there's a serious question as to whether all the samples really were malware instead of scanner fodder and whether any of them were duplicates...

now as i've already said, those of us who have been in the anti-malware community for a while have seen countless examples of these kinds of tests so we know to take them with more than just a single grain of salt... unfortunately the average person doesn't know why they should be wary of such tests and when security experts use these same sorts of tests to prop up FUD that scares people away from anti-virus products it doesn't help anyone... it's bad science and a doctor of physics should know better...

the myth of overwhelming numbers

one of the myths put forward by the anti-virus is dead crowd is that anti-virus companies just can't keep up with the malware authors... this article about the death of anti-virus, balanced though it may be, contains the following quote that demonstrates just what i'm talking about:

"The traditional, signature-based technologies are simply not able to keep up with the sheer volume of malware that's out there," Jaquith told TechNewsWorld. "There are over 200,000 unique pieces of malware out there. Some host intrusion vendors say that number is closer to a million.

200,000 indeed - that does seem like an awfully big number, how could anti-virus companies possibly keep up? in actuality, this is an example of lying with numbers... although the 200,000 figure is fairly accurate (at least it's in line with what mcafee and f-secure were saying last time i checked), it is not something you keep up with... keeping up is something you do with a rate (a certain number of things per minute or hour or day or year) and 200,000 isn't a rate, it's a total... if we were to turn that total into a rate we'd need to include the time period over which it occurred, and although 200,000 sounds like a pretty big number, 200,000 over the course of 20 years doesn't sound quite so overwhelming...

i could end there, but that would be misleading in the opposite direction... 200,000 over 20 years works out to about 27 per day but that's not the right figure right now, that assumes a constant rate of malware production over those 20 years and it has been anything but constant... in this article stanislav shevchenko (the head of kaspersky labs virus lab) is quoted as saying that november 2006 saw 10,000 malware samples added to their database - that works out to about 333 per day... what's more, it's apparently 5 times the number from january 2006...

now you're probably thinking we're back in the overwhelming numbers range again, but wait - stanislav has more figures, specifically the time it takes for a good analyst to process an average malware sample is 5 minutes... if you add in the stated constraint of a 12 hour shift a good analyst should be able to process 144 samples per day so november's incredible figures could probably have been handled by a grand total of 3 analysts with time to spare assuming there weren't too many samples that were significantly more complicated than average - and even if that assumption is false, anti-virus labs tend to have more than just 3 analysts...

so clearly, right now anti-virus companies are not being overwhelmed by samples... that only leaves the future to worry about... if the numbers increased 5 fold from january to november in 2006, what are they going to be like this time next year or the year after that?...

the first graph shown here depicts the increase in the malware production rate over time for the past several years and admittedly it looks pretty grim but i saw a graph with a nearly identical shape (though obviously with very different numbers) ten or more years ago... somehow the world hasn't ended yet, anti-virus hasn't died yet, and life goes on... the more you do a repetitive task, the more patterns start to emerge and with those patterns comes the potential for more automation which in turn leads to faster malware analysis... advances in automation have sped up and will continue to speed up the analysis process so the future doesn't really look all that bleak either...

[edited to fix the year specified - apparently i'm still getting used to this whole 2007 thing]

the myth of AV's failure

i'm sure you've heard it said before that the idea of conventional anti-virus techniques are broken or obsolete, that av is failing... i've touched on the topic before here and here but i think it's time to tackle this myth head on...

people say and believe that conventional av is outdated because of all the reports of it's failure... there are lots of examples where program X failed to stop malware Y from infesting computer Z so could forgive some of the greener minds for concluding that those reports are representative of reality, that anti-virus products don't do a good job...

what anyone with any degree of training in statistics will notice, however, is the obvious selection bias in those reports - you hear about av products failing but you don't hear much in the way of success stories, not because there are no successes but because there's no reason to report them... the failure reports give an entirely lopsided view of the efficacy of conventional anti-virus techniques, leading to the negative perceptual bias the security community now suffers from...

conventional anti-virus (known virus, or known malware scanning) does fail, there's no denying that... specifically it fails to detect new/unknown malware, and that is an accepted and acceptable limitation... acceptable because no single technique can ever be effective against everything (thus necessitating other, complementary techniques) and new/unknown malware, though good for capturing media attention, is not quite as big a deal as it's made out to be... known malware vastly outnumbers new/unknown malware and new/unknown malware (especially that which affects many people, thereby posing a significant threat) generally does not stay new/unknown for long...

now, if known malware scanning really was as broken a model as some people like to think it is, why does a piece of malware's population growth start decreasing once it becomes known malware? it doesn't magically stop trying to spread (or in the case of non-replicating malware, the malware spreader doesn't automatically stop trying to spread it)... the population growth starts decreasing because it begins to fail to spread (or be spread)... the fading out of a particular piece of malware can take years, a decade or more in some cases (usually for reasons unrelated to a scanner's efficacy), all the while it's trying and usually failing to spread... potentially failing orders of magnitude more times than it ever succeeded - and those failures represent successes for known malware scanning...

if people thought seriously for a moment about the life-cycle of malware, about the relative deployment of various anti-malware technologies, and the implications they have for the notion that scanners are failing, falling behind, or just not working they'd realize that in fact scanners must be working far more than they're failing otherwise the population of a given piece of malware wouldn't go down... this is still very much an av world, most people use nothing more than scanners (if they even use that much), there aren't enough deployments of alternative technologies to stem the tide of the malware's spread...

scanning isn't broken or outdated or failing - malware would be a much bigger problem if it were... the notion that it is failing is just an erroneous perception based on an interpretation of incomplete facts...