more on whitelisting

or perhaps moron whitelisting... yes dave, i'm taking the bait with the emperor's robin bloor's new clothes article about the decline of anti-virus and the rise of whitelisting... and it's pretty good bait too, since i wrote the rise of whitelisting (anything look familiar?) over a year ago in response to something else robin wrote when his misguided anti-virus-is-dead campaign was still new (a day old judging by the posting dates)...

i haven't really seen much new material that i didn't cover myself in that first response - exotic execution, accuracy of local whitelist management, and scalability of global whitelist management are all outstanding problems with whitelisting... finding out that the whitelist vendors themselves admit to the unmanageable scope of their own efforts is one of the few new things that have come up since then...

robin doesn't address any of this, of course, and like the emperor's new clothes there really isn't anything there in his new article... 2 whitelist companies merged and became bigger than either was before - big woop...

the only interesting things were the comments on the register's mirror of the article... vesselin bontchev hit the nail on the head several times (and i'd expect nothing less, though i wouldn't have expected vess to be first to post on the register - ugh), but what interested me most was the suggestion that robin bloor's 'research' is funded by the whitelist industry...

clearly this would be some segment of the whitelist industry that is misguided enough to think that anti-virus companies are actually their competitors... it's apples and oranges though, av companies won't be competitors until they start offering whitelist technology of their own... what's so misguided about this, though, is that if they actually manage to take av's place on peoples' desktops they're ultimately going to wind up also taking av's place in peoples' cross-hairs... if they displace anti-virus then people will necessarily become just as disenfranchised with whitelisting as they are with anti-virus because whitelisting has failings too...

in the comments robin does admit that whitelisting won't stand on it's own and that additional technologies will be needed to complement it but weasels out of saying what... i suspect intuitively he knows the answer: at the highest, most abstract level, what complements a whitelist? that's right, a blacklist... though he expressly derides the idea as "positioning" in his article and says that you should nevermind it, the fact is that known malware scanning and application whitelisting are natural companions...

i want to call robin bloor a troll, i really do, but if he's being funded by whitelisting companies then what he really is is a shill... either way, i'm done with him - when my arguments (and most other peoples' arguments) now are the same ones i posted if march of '06 there really is little point in continuing with this... besides, considering the uptick i get in hits based on searches involving the term whitelist when he posts more nonsense, i'm pretty sure the counter-arguments are being heard without additional effort on my part so i can put that you are being trolled feeling to rest...

the slow death of someone's credibility

robin bloor doesn't show up on my radar very often, but when he does it always seems to be about how anti-virus is dead, dying, or doomed in some way... this time it's about the slow death of anti-virus technology...

dave lewis thought it worth pointing to (thus leading to the blip on my radar) and the folks over on the authentium blog went against their better instincts and actually responded to it...

it's a good response, too, and it touches on a point i've also made about whitelisting - specifically that the numbers of good programs (ie. those that would be put on the vendor supplied whitelist) far exceed those of malware... why this is interesting is because one of the main arguments against anti-virus technology is a fundamental disdain for enumerating bad things on the premise of it being too big of a problem... clearly, if enumerating good things is several orders of magnitude harder than enumerating bad things then this particular argument is garbage... it's nice to see that a whitelist vendor is owning up to the enumeration cost, even if some whitelist proponents are unwilling... to the unwilling i would say imagine if the TSA used a whitelist instead of a blacklist - do you think telling most people they can't fly because they aren't on the fly list would work better than telling a few people they can't fly because they are on the no-fly list?...

this time around mr. bloor wants us to believe that, because whitelisting is gaining traction in the market, his prognostications about whitelisting replacing blacklisting are gradually coming true... seemingly choosing not to consider the possibility that the trends he's noticed are nothing more than the maturation of the whitelisting market, and unperturbed by history's numerous examples of the av industry adding technologies to their portfolios rather than replacing the old with the new, mr. bloor's arguments seem rather ridiculous...

but then again, they're coming from the same guy who thinks traditional polymorphism and server-side polymorphism are comparable - that because traditional polymorphism (where the transformation function responsible for the polymorphism is carried with the self-replicating malware and therefore open to attack by the av'ers) has been around for 16 years that the av industry should have developed something capable of dealing with the newly emerged server-side polymorphism (where the transformation function isn't carried by the malware and thus isn't generally open to attack) by now...

so should anyone take him seriously when he says whitelisting is better than av and that it's going to replace av? no, because whitelists are not going to replace blacklists, whitelists aren't better than blacklists, they're just different from blacklists... whitelists don't have the same weaknesses blacklists do just as blacklists don't have the same weaknesses that whitelists do... the notion that we would or should use only one type of technology to protect ourselves from malware is antiquated and fundamentally broken - there is no silver bullet, no panacea, and if you don't employ a multi-layered strategy (aka defense in depth) then you're just setting yourself up for unnecessary failure... i have to wonder, when av technology is still here 10 years from now, will robin bloor be eating crow or will he still be forecasting av's death like some inverted version of monty python's parrot skit...

anti-virus is not a faulty burglar alarm

wow, what a great analogy robin bloor makes here... too bad it's his reasoning that is faulty...

one of my favourite turns of phrase lately is mismatched expectations - robin bloor is quite clearly suffering from mismatched expectation by likening anti-virus to a faulty burglar alarm... anti-virus is nothing like a burglar alarm, faulty or otherwise, nor is the problem it is trying to solve amenable to a burglar alarm type of approach...

burglar alarms are pretty simple things - you have one or more sensors that detect basic, easily quantifiable environmental conditions (broken window, open door, motion, etc) and an alarm goes off when the sensor is triggered... this is pretty dumb, all things considered, but it works well enough when the thing you're trying to detect can be broken down to such simple events and even better when the home owner can easily decide whether something is a false alarm...

the malware problem, by comparison, cannot be broken down into simple elements quite so easily and end users are largely incapable of deciding whether an alarm from a malware detector is false or not... this is why behavioural detection techniques (which have been around for over a decade) are still not receiving the same kind of mainstream attention that known-virus scanning receives... instead of going the burglar alarm route, anti-virus incorporates a considerable amount of knowledge about the viruses (now more generally, malware) that the vendor has seen in order to minimize false alarms... anti-virus still has false alarms, of course, but just imagine how bad it would be if av were made to be as dumb as a burglar alarm... if we're going to stick to crime-related analogies, anti-virus is really much more like the criminal databases that have proven so useful in the past to keep known felons out of places they're not supposed to go or out of job positions they shouldn't have... not that those databases are perfect, but they sure do help...

of course, bad analogies are not the only thing robin has up his sleeve in that article... he spreads some clever FUD about the malware pandemic using carefully selected figures from a microsoft study i wrote about once before... yes, the malicious software removal tool removed malware from 5.7 million computers, but what robin fails to tell you is that it scanned over 270 million computers... that gives a malware penetration just 2.1% - hardly a pandemic...

all this is to build up to his conclusion that application whitelisting is a superior technology that should be used in place of anti-virus... maybe it is superior, maybe it isn't, it depends on things i've mentioned elsewhere, specifically whether the user can accurately decide what is safe to add to the whitelist and whether the whitelist can cover enough of the various types of program execution (it's not just *.exe's out there)... should whitelisting be used in place of blacklisting (anti-virus)? no, in reality blacklists and whitelists complement each other, they partially mitigate each others weaknesses, so they really ought to be used together...

and just as a point of correction - contrary to mr. bloor's assertion, application whitelisting is not relatively new... the basic idea dates back at least a decade as an anti-virus (gasp!) suite known as thunderbyte anti-virus included a rudimentary form of whitelisting in it's tbcheck module... tbav was a fairly well known product in it's day and there was plenty of opportunity for whitelisting to become popular, but it didn't... others have languished in obscurity too... anti-virus seems to have remained the most popular in part because it required the least amount of knowledge and/or thinking from the end user - and when it comes to malware, that can actually be a very good thing...

the rise of whitelisting

application whitelisting, the practice of keeping a list of known good applications (a whitelist, as opposed to a blacklist) and only allowing those applications to run, is not a new idea... it's a good idea certainly, but not new... nick fitzgerald, for example, has been calling for the development and deployment of just this type of technology for years now... i myself utilize the application launch control functionality (a sort of whitelist) in kerio personal firewall (which is more interesting to me than the actual firewall part)...

today, however, there was a post on the nod32 news blog about whitelisting that i think underlines the need to delve into the strengths and weaknesses of the technology...

at the most fundamental level an application whitelist needs to know 2 things, what sorts of things are programs and which of them are known to be good...

determining what is a program and what isn't is not as simple as it sounds - in fact it's an intractable problem (unless someone solved the halting problem without telling me)... that's not to say we can't make that determination accurately most of the time... basically we're limited to detecting known program types - new types of script programs and that sort of thing are always going to exist and always going to be left out because they're too new... at any rate, this level of detection is always hard-coded into the whitelist system by the developers - there's no real way to add new program type detection on-the-fly on the end user's machine...

where things really get interesting is when we're trying to identify the known good programs... there are 2 main paradigms we can follow: either the 'goodness' is determined by the end user and is specific to his/her machine only, or the 'goodness' is determined by some central body somewhere that decides what's good for everyone... it's also possible to combine these two but let's look at them individually first...

in decentralized whitelist management (where the end user decides what's good) the list of known good things is relatively small and can be updated whenever the user needs to but there is a serious problem trying to enable end users to accurately determine the trustworthiness of new software... most people just don't have the expertise or inclination to perform the necessary research to find out if something should be allowed to run on their machine... frankly, if they were any good at making those kinds of decisions, email worms (and just about every other type of malware out there) would never have become as big a problem as it is today...

in centralized whitelist management (where a central body decides what's good) it's possible to have lots of expertise on hand to determine whether a specified program is good or not but the list of known good things needs to be huge in order to cover everything that everyone might use... one of the arguments against known virus scanning and in favour of whitelisting is that keeping track of all the known bad things (essentially a known virus scanner implements a blacklist) is that the number of known bad things is very large and ever-growing, but a centralized whitelist has this same problem in spades as determining if something is good is just as hard as determining if something is bad, there are far more good programs out there than there are bad ones, and the list grows even faster... in addition, the central organization will undoubtedly have finite resources with which to work so adding updated programs (a number of programs out there do tend to have regular security updates) to the global whitelist may not happen in as timely a fashion as we might like... finally, it puts the central organization in a position where they could exercise undue influence over which legitimate applications get to run on end users machines (presence on the whitelist would effectively be a certification and has many of the same undesirable properties that the system where a certifying authority digitally signs the known good programs so that people would know which ones they could trust)...

combining these two into a hierarchical whitelist management system (where the global whitelist serves to help the end user determine the trustworthiness of programs most of the time) would seem like it would give the best of both worlds but that's not necessarily the case... giving the end user the power to add a program to his/her local whitelist gives him/her the power to make the same mistakes s/he'd be making with a fully decentralized whitelist system - with the understanding that the global whitelist isn't always going to be completely up to date a piece of malware need only advertise itself as the latest security update of some widely deployed program (like part of the operating system, for example) in order to get the user to add it to the local whitelist... also, giving the end user the power to add programs to his/her local whitelist doesn't obviate the need to make the global whitelist as thorough and up-to-date as possible, nor does it make the task any easier or less time-consuming/resource-intensive... further, the absense of a program on the global whitelist could still communicate a lack of trustworthiness to the end users and bias them against a particular legitimate application...

even if you could solve all these problems, it's still possible to make good programs do bad things... and i'm not just talking about buffer overflows and other software exploits, the whitelist can't know that you want Format.com to run sometimes but not others... it can't know that Regedit.com should import the *.reg file with your personal configuration preferences in it but the *.reg file that disables the whitelist on the next reboot and causes the download and execution of some piece of malware... it can't know that your SQL server should run the query that selects all your customer table but not the one that self-replicates all through that same table... it also does nothing to stop malware that exists only in memory, nor malware that executes outside the scope of the operating system (like bootsector malware)

all that said, it should be fairly obvious that if you only let known good programs execute, then the bad programs (known and unknown alike) generally won't be able to run on your system... with a whitelist in place monitoring the attempted launching of applications you'll also get alerted when something you don't know about tries to run... finally, with a relatively small and static set of applications allowed to run, the security of the system is more easily analyzed and modelled...

application whitelisting, like integrity checking/change detection before it, may offer a technically superior security technology in theory, but in practice it is not a panacea... it's not a magical cure-all and it really doesn't eliminate the need for conventional anti-virus/anti-malware scanning... like all anti-malware technology, it is best when used in conjunction with complementary technologies and techniques...