don't be too proud of this technological terror you've created

lot's of folks have been posting about the 'here you have' mass mailing email worm that's been making the rounds. it's strange that such an old-school technique should inspire so much discussion, but it has and some of it's actually interesting.

one of the discussions comes from the enterprise application whitelisting blog, in other words, it comes from application whitelisting vendor bit9. they are, perhaps understandably, quite bullish about the fact that their technology would have stopped the threat before it could have spread while the anti-virus software vendors were supposedly left to scramble to get detection added after the fact.

while it's true that a classical blacklist or known-malware scanner would require updating after the threat becomes known, it seems that at least some of the anti-virus software vendors that harry sverdlove was taking a shot at were actually able to detect the threat heuristically (see f-secure's post or kaspersky lab's post for example).

it also deserves to be said that many anti-virus software vendors are bundling whitelisting in their suites these days, so people using that feature of those offerings would have been just as safe as if they'd been using bit9's.

most importantly, though - if social engineering can be used to get people to extract malware from a password protected archive sent as an attachment and then run that malware (and we have historical examples of successful email worms that used precisely this technique), social engineering can be used to get people to add that malware to the whitelist.

whitelists do not make you magically immune to this threat. i'm not even convinced they raise the bar a significant amount when you consider how easily people can be tricked into doing all sorts of dumb things. perhaps an enterprise would be in a better position because relatively few would (in theory) have access to modify the whitelist, but administrative users aren't above doing dumb things.

a whitelist user's perspective on windows update

as i may have mentioned before, i use whitelists - for web content (via noscript), for basic network traffic (via my router's port forwarding functionality), and also for traditional applications (via the application launch control functionality of my software firewall). whitelisting is an important part of my security strategy but if there was one thing that made me wish i didn't use application whitelisting it would be windows update.

you see, i don't add the programs involved in software updates to my whitelist. in fact, i don't add the majority of programs on my system to the whitelist. i don't want them being able to run without my knowledge or authorization, especially those that give no indication that anything is running in the first place, and even more so for components that have any role in installation/update. i'm not about to enable silent installs using existing components. and frankly i expect updates to be new anyways so there's little point adding those. for the most part that's not a big deal, i just take note of what program is trying to run (so that i can catch anything that looks suspicious) give a one-time authorization and let the program do it's thing. even for updating most software that's not a big deal, but when it comes to windows update it's a very big deal.

microsoft, for reasons that i can't begin to fathom, are clearly cobbling the update procedure together from as many bits and pieces as they can. i know this because of the number of program executions i need to permit (it's a lot) and how big of an interruption it is. on my older, slower system that i only power on on the weekends i started permitting various parts of windows update on saturday. i don't really have the time (or patience) to sit around clicking every couple of minutes for hours on end so the confirmation dialog only gets clicked when i have to time to go back and check on it's progress (and i sit there for a little bit, clicking here and there until i have to leave again). as of tuesday morning the update was still going. it took until monday just to get to the point of saying "there are updates ready to install". i can only hope that by the time this post is published the update will finally be finished and i can power the machine down.

some people might consider this a failure of application whitelisting, but i don't.  microsoft has designed their update procedure to involve far, far too many invocations of far, far too many separate executables. they need to stop expecting that they can run whatever system components they want, whenever they want, however many times they want with impunity. it's wasteful of resources and it's wasteful of my time.

the blacklist value proposition

how do you defend the use of blacklists in the face of seemingly stronger defensive mechanisms like whitelists?

no matter what defensive technology you use there will always be some holes in those defenses... there will always be exceptional cases that your defenses don't currently handle and/or are unsuitable for handling... what's the fastest/easiest way to deal with exceptional cases you want to avoid? yeah, you guessed it, with a blacklist...

let me give you an example: lets say that we have an application whitelist... application whitelists control the execution of some subset of known program types... they're limited to the known types because, well, how do you intercept a kind of execution that's never been seen before?... they're also usually limited to a subset of the known types because developing the technology to intercept and block programs of an arbitrary type (such as script programs for a particular interpreter, or the unanticipated programs that exploit code represents) is not necessarily easy or cheap and for the more obscure types it's often just not worth the investment...

now lets say that a piece of malware is created that exploits this partial coverage of the set of program types... when there's only one such piece of malware, or even just a handful, the benefits of re-engineering the application whitelisting software to be able to cope with this additional type don't justify the costs in terms of time, money, and effort required to do it... when there are so few instances (relative to the billions of programs out there) it is faster, easier, and cheaper to just look for those particular instances (via a blacklist) than to re-engineer the whitelist to handle them... it won't be until the program type in question becomes mainstream that it becomes worth it to add capabilities for it to an application whitelist...

similar scenarios can be constructed for any other type of preventative measure... as such there will always be a need for blacklisting regardless of what other defenses are in play because there will always be a need to deal with emergent exceptional cases as fast and cheaply as possible... even malware blacklists (ie. known malware scanners) themselves have exceptional cases that they can't deal with - that being new/unknown malware... however, as i've stated in the past, novelty is an advantage that wears off, and as far as i can tell it's the only one that does...

whitelist opinion smackdown

i realize i've been rather quiet as of late - not sure why, perhaps i lost my mojo... anyways, you can all thank cdman for rousing this ogre out of slumber...

in a recent post, cdman lays out his response to randy abrams' post on whitelisting... perhaps it was the hint at the possibility of an ad hominem attack against a fairly well known and long-standing member of the av community (randy was, for a long time, the voice of av from within the belly of the beast - aka microsoft) that piqued my interest, but that wasn't cool so let's move on...

cdman's first substantive beef is the suggestion that whitelisting companies can't do their job without anti-virus software... ignoring the fact that in practice this is actually true (whitelisting companies currently depend on anti-virus software to determine if something is safe to add to their whitelist) lets look at the hypothetical alternatives he suggests - specifically that whitelist vendors could rely on reputation or building the generic malware equivalent of marko helenius' automatic and controled virus code execution system...

relying on reputation offloads the problem of keeping bad software off the whitelist onto the very people providing the bad software... sure people who provide bad software consistently will get a bad reputation and not be trusted, but what about people who only do it once in a blue moon? microsoft releases tons of legitimate and safe software but they have on occasion also distributed virus infected materials... you'd be hard pressed to justify not whitelisting code from microsoft if you were relying on reputation but if you did whitelist all their code you would eventually whitelist something you shouldn't have... furthermore, relying on reputation is precisely the method that customer-generated whitelists are primarily made with, which would make a vendor-generated whitelist using the same technique rather pointless...

next is the idea of building a system to automatically execute samples and perform baseline comparisons to see if the sample compromised the system... and of course this has to be done on a scale sufficient to handle the rate at which sample files are produced (otherwise whitelist vendors wouldn't be able to keep up, much like av vendors supposedly aren't able to)... but have you looked at bit9's (a whitelist vendor) figures? av companies already augment their small armies of malware analysts with automated methods of determining what's bad, and old methods like this are almost certainly among them... if the av vendors can't keep up with the malware then what hope do whitelist vendors have in keeping up with the goodware when it's production rate is (necessarily) several orders of magnitude greater than that of malware? there are all kinds of capabilities peculiar to traditional av companies that whitelist vendors could try to replicate in-house, but the scale of the samples they have to deal with make it impractical for them to do anything other than to replicate the blacklisting capabilities in full in-house and that would mean they would still be using what the general population considers av - it would just be their own...

a third option cdman mentions is using technology like that developed by mandiant... whitelist vendors are unlikely develop such capabilities in-house when it's almost certainly cheaper to buy products/services from others who've already developed those same capabilities, but lets hope in this case they stay away from such ethically questionable companies as mandiant... bad enough that mandiant hires people whose marketability in security is thanks in no small part to their past efforts at making the problem worse, but to then turn around and have some of those same people do essentially the same thing in the company's name at an event like race-to-zero smacks of not just some lapse in HR's judgment but rather of an alignment of moral compasses... perhaps i'm in the minority here, but if a whitelist vendor gets in bed with a company like mandiant i wouldn't touch them with a 10 foot barge pole...

second to the beef about what whitelist vendors would do without av software was cdman's beef with randy's understanding of what actually constitutes a whitelist... i have to admit that my first impression on reading the statement that the TSA implements a whitelist was one of confusion... the most widely known (and reviled) measure the TSA implements is the no-fly list, which is fairly obviously a blacklist... i actually left a comment on randy's original post expressing my confusion but literally as i was writing it it dawned on me that there were other measures implemented by the TSA such as the newly revised rules for flights which basically require one to be granted permission in a 2-stage process before you can fly... of course, as i write this i'm reminded of the various trusted traveler programs that schneier has written about on occasion - those are also whitelists...

despite all the disagreement, though, in the end cdman and randy are actually in agreement about the role of whitelisting - it's simply another layer... both think it's got it's strengths and it's weaknesses, areas where it's more applicable than others, etc.. however, i think randy has once again distilled a complicated topic to a simple analogy when he compares the folks who say whitelists are the end of av with airbags calling seatbelts obsolete... what a clever way to say they're full of hot air...

if i have a whitelist, do i still need AV?

on reading this dark reading article about a texas bank dumping AV in favour of application whitelisting technology i was reminded of an email conversation i had with a reader who was dealing with almost the exact same issue only days earlier... i've come to realize this is a question more and more people are going to be wrestling with as time goes on so instead of expecting them to contact me privately i'm going to answer the question of whether you need AV software if you have a whitelist here... the glib answer is no you don't need it...

of course that answer continues with: you don't need the whitelist either, or your computer for that matter - people survived perfectly fine before any of that stuff existed...

clearly, the glib answer isn't all that useful...

the real answer is that i can't decide for anyone whether they need AV... in practice security has a lot to do with making trade-offs and which trade-off is right for any particular person or organization is best decided by those who would have to live with the consequences of that decision... what i can do, however, is point out weaknesses in both individual technologies and describe some of the benefits of using them together...

it's no secret that known-malware scanners are ill-suited to detection of new/unknown malware, updates can be a hassle for large organizations, and the ongoing subscription fees have in the past seemed like a necessary evil but are increasingly seeming less necessary as application whitelisting works it's way into the mainstream...

of course, as we know from figures provided by bit9, the set of good software is larger and growing faster than the set of malware so a vendor-supplied whitelist will almost certainly be worse from an update point of view (and maybe subscription-wise too)... the customer-editable whitelist is much more palatable in that regard because you only put on it those things you actually need in your production environment... coming up with the initial whitelist (not to mention modifying it if/when the needs of that environment change or when software needs updating) puts the customer in the position of having to decide what's safe or not... certainly one should only trust software from known, reputable sources but those sources aren't perfect (even microsoft has accidentally distributed malware in the past) so the first benefit of continuing to use known-malware scanner even though you've chosen to use a whitelist is that you can check the software you intend to whitelist (as whitelist vendors often do for whitelists they provide)... this is an example of the axiom "trust but verify"...

that alone may not seem like it's enough to warrant keeping the desktop av, however, so consider this: just as known-malware scanners don't recognize everything that's malware, application whitelist software doesn't recognize everything that's a program... will the whitelist software block office macros if the office binaries are on the whitelist? will it block batch, perl, kixstart, etc. scripts if their respective interpreter is on the whitelist? will it block javascript from the myriad of ways it can be launched on the system? as an example, i use the application launch control functionality of sunbelt personal firewall as a whitelist and i discovered quite by accident recently that it does not block individual batch files... a known-malware scanner would detect known malware in a batch file, however, and in an office macro, etc... that's a second benefit of keeping known-malware scanners around in a whitelist deployment, and it's one that applies even at the desktop level...

that's not all though... there's also the issue of exploit code that exploits vulnerabilities in whitelisted applications... if the exploit needs to launch additional applications that aren't on the whitelist then the whitelisting software will have interfered with and potentially blocked the exploit, but what if all the applications it needs are on the whitelist? the whitelisting software would be helpless to stop it but a known-malware scanner might still have at least some chance to do so...

known-malware scanners are weak against that which is novel while application whitelists (assuming no malware gets whitelisted) are weak against that which is exotic... together they're only weak against that which is both novel and exotic... this is the essence of what defense in depth means when it comes to the anti-malware world - not using multiple different scanners but rather using multiple and entirely different types of technology so that the second (or third, or fourth, etc) line of defense can stop at least some of what gets through the cracks in the previous line(s) of defense... it's still up to the people affected to decide whether the benefits of a multi-layered strategy warrant the cost but they should definitely consider it as no technology is an island, perfect unto itself...

debunking the mythology of whitelist practicality

while i don't normally listen to podcasts, it does occasionally happen and the risky business podcast episode 66 mentioned on the tenable security blog was one of those times... one of the topics discussed was the practicality of whitelists over blacklists and it amazed me (again) that people actually think this way...

why, when the number of developers making good software far outnumbers those making bad software, do people insist on believing there's more bad software than good and it's easier for vendors to keep track of good software than it is to keep track of bad software...

it's a pretty popular belief these days that it's not practical to keep track of all bad software anymore and vendors should be keeping track of the good software instead because that's somehow more practical but that belief starts to look a little ridiculous when you start considering the origins of the good and bad software in the world... just like most people in the world are actually good people (police states would be a necessity otherwise), most programmers are good people too so they're not writing malware... if most of the programmers in the world are writing good software rather than malware then it stands to reason that the production of good software out-paces the production of malware and since it has always been this way it should also outnumber malware...

as such, good software far outnumbers malicious software and is produced at a faster pace than malicious software... however big the set of malicious software seems and however fast it seems to be growing you need to ask yourself how much more aware you are of those stats for malware than for good software (a lot less attention is paid to those figures for good software)... i've mentioned before (and i'll probably mention again) that bit9 actually has some figures related to both the total number and rate of production of good software and it's shocking the degree to which it dwarfs those same measures for malicious software... billions of good programs while there were still less than a million malicious ones, and millions more good programs produced each day while malicious software is still in the range of thousands for the same period...

for the average person this may not seem intuitive; indeed, how could microsoft alone produce 500,000 new files each day - they certainly don't have that many products... the reason for the discrepancy is at least 3-fold... 1) the average person doesn't understand how many different things actually qualify as programs and would need to be kept track of if vendors were to supply whitelists, 2) the average person doesn't realize how many programs go into a single product, and 3) the average person doesn't actually have any idea how many products a company like microsoft actually produces because microsoft produces software for such disparate sets of people... you think ms word is just one program? it's not, it's many different programs that inter-operate to give you the functionality and user experience you're used to... if it were a single program there would be little or no need to install it, you could just run it as a stand alone application... the same holds for excel, and powerpoint, and outlook, and so on and so forth... do you think the hundreds (if not thousands) of megabytes that windows takes up is all because of data? what data does an operating system need? it's mostly programs...

a common refrain these days is that blacklisting just isn't working, but the problem with common notions is that they're often over simplified... blacklisting just isn't working well enough all on it's own... it is a challenge to keep up with the malware production rate so just imagine how much more of a challenge it is to keep up with the good software production rate... sure whitelisting companies like bit9 seem to be able to do it but you wanna know how? by using the same blacklists people think are failing in order to determine what's safe to put on their whitelist... it shouldn't take a rocket scientist to figure out that such a whitelist will be no more accurate than the blacklist it's based on - anything the blacklist misses will get onto the whitelist and then what will you do?

bad really is in the minority

from liam tung's article signature-based antivirus is dead: get over it:

However, there is a problem with the use of blacklists, said Turner. "When the majority of stuff you're handling is malicious, it makes more sense to use a white list because that deals with the exception — blacklists only work if 'bad' is in the minority."

i totally agree with this statement... there's just one thing that turner and just about every other av detractor out there fail to realize... bad really is in the minority... bit9 (an application whitelist vendor) has shown that there are several orders of magnitude more good stuff (on the order of billions) than bad stuff (about a half million at the time) and that microsoft alone produced as many good binaries in a day as there had been bad binaries produced in the previous 20+ years combined (from the bit9 presentation at the international anti-virus testing workshop in 2007)...

furthermore, most of the stuff anyone (other than the anti-malware industry) handles is non-malicious (unless you're looking only at email and are considering spam)... most web pages are safe, most binaries are safe, the majority of stuff most regular people encounter on a day to day basis is safe so if you're going to advocate a security technology that focuses on the exceptions you're going to have to get over your perceptual biases and realize that bad stuff is the exception so blacklists make more sense (at least by that logic)...

you've heard the argument that blacklisting is inferior to whitelisting because the list of all bad things is growing too big too fast, but we have quantifiable proof that the list of all good things is far, far bigger and growing far, far faster... that doesn't mean blacklists don't have serious problems (they do) or that whitelists are unusable (they aren't), it simply means that particular argument is fundamentally flawed and if people took the time to become familiar with the reality of the situation they'd know that...

what is application whitelisting?

application whitelisting is, as the name suggests, an example of whitelisting whereby only applications that are authorized get to be executed/interpreted/run...

in the case of application whitelisting the whitelist itself is the list of authorized programs (generally with some integrity information like a cryptographic hash to ensure that a given program is actually the same as the one on the list rather than just having the same name)...

if the whitelist itself is provided by a vendor then it will necessarily be very large (as they have to account for all good programs that anyone might use) and may not include everything a user needs... if, on the other hand, the whitelist is provided by the user him/herself then it will likely only contain the programs that are relevant to the user, but the user will have to decide for him/herself which programs are trustworthy enough to put on the list... either way, the whitelist would need to be updated at least as often as the user's set of authorized software changes...

the anti-malware applications of this technique rely on malware never getting on the whitelist... so long as the whitelist remains malware-free then (barring exotic execution) malware shouldn't be able to run... a second, lesser understood requirement is that the application whitelist be able to recognize everything that qualifies as a program, lest some program types get executed without whitelist-based oversight...

back to index