race to zero in the security special olympics

news has started floating around about a contest in virus obfuscation being held at defcon this year... there have been a couple of mentions of it elsewhere as well, such as robert graham's "race to zero" post, sunnet beskerming's post "defcon competition has antivirus vendors complaining", and even an ars technica post titled "antivirus vendors pan free research from defcon contest" by david cartier...

now i'm obviously opposed to this and think it's irresponsible and unethical... not to mention they've chosen just about the worst malware type to play with - viruses... note to contest organizers, participants, and proponents: in the event that something goes wrong, self-replicators have a tendency to go on and on and on long after they are released or spread (old viruses never die)... they aren't called viruses because they make your computer feel bad, they're called viruses because they spread by themselves just like any infectious biological pathogen... this contest and those in it aim to play with fire...

with that out of the way, it's time i got to debunking a lot of the wrong thought surrounding this contest...

contest organizers (as quoted by pcworld):

Contest organizers say that they're trying to help computer users understand just how much effort is required to skirt antivirus products

if i'm not mistaken one of the key aspects to presenting things is to know your audience... the computer users at defcon already know it's easy to evade a scanner...

race to zero website:

The event involves contestants being given a sample set of viruses and malcode to modify and upload through the contest portal. The portal passes the modified samples through a number of antivirus engines and determines if the sample is a known threat.

this is a logical failure... a modified sample is by definition not a known threat until such time as a signature has been added to the scanner... if you're going to put on this kind of contest you might want to make sure you know what the difference between a known and unknown threat are...

Signature-based antivirus is dead, people need to look to heuristic, statistical and behaviour based techniques to identify emerging threats

this represents a complete failure to understand what the heck they're testing... changing a single bit can often foil signature detection - for contestants to get anywhere in this contest they will have to foil the heuristic, the statistical (also arguably a heuristic), and the behaviour based (heuristic here too) techniques that the scanners also implement... there are no naked known-malware scanners anymore (at least not as far as consumer products go), they all have some kind of heuristic engine in them...

Antivirus is just part of the larger picture,

and this is failure to understand their own messaging/marketing... how can anti-virus be part of the picture if it's dead?... on the one hand they want to tear down the practice of using anti-virus (anti-virus is dead) and on the other they expect people to keep it around as part of the picture... someone needs to make up their mind...

We are not creating new viruses

contrary to their mistaken belief, modifying existing viruses until such time as they're different enough from the original that neither known-malware scanning nor heuristics can recognize them is creating new viruses...

modified samples will not be released into the wild

one wonders how precisely they're going to keep that from happening? they're presenting contestants with samples, not a locked down environment that prevents them from taking their samples with them at the end of the day and doing something stupid/careless/malicious with them after the fact... how do i know this? because providing that kind of environment would be prohibitively expensive and restrictive... it might work in an educational institution (where the costs are offset by student tuition) but not as a contest..

robert graham:

The 'protectors" (product vendors) have big marketing
budgets to tell us their side of the story

it's not their side of the story, it's their attempt to get people to buy their product... on some level we all know that marketing and advertising is just another kind of lying... you don't honestly believe a whopper looks as perfect and juicy in real life as it does on tv, do you? then stop being disingenuous by treating av marketing as anything more factual than that...

We only get one side of the story

that is bullshit... we hear about the failures of anti-virus all the time, we hear about people giving up entirely on anti-virus, and we hear about anti-virus being dead... we get a lot more than just the vendor's side...

Yet, such contests also help customers

defcon isn't a customer education setting, this contest isn't going to teach them anything because they aren't going to be there...

The educating needed here is that the mainstream anti-virus technologies are easily evaded, and that such evasion happens a lot

if it happens so much then why do people need to be educated about it? surely they'll have seen it for themselves or they'll know someone who's seen it and has related the story to them?

Anti-virus vendors publish tests "proving" a 99% detection rate

this is the point where one realizes that robert graham is playing the part of an uninformed crank... anti-virus vendors do not publish tests like that... that kind of self-serving behaviour would absolutely NOT be tolerated by competitors or by the community... vendors point to tests carried out and published by independent 3rd parties...

However, that doesn't apply to customers. Often, the best way to test an anti-virus product is to create your own virus.

this is beyond stupid... you don't test your airbags by smashing your car into things and you don't test your anti-virus product by making new viruses... you leave such testing to the people with the expertise and resources to do it properly... customers generally have neither...

sunnet beskerming:

It should also show up the antivirus tools that
are making use of poor signature detection mechanisms

just like the race to zero website, this is a failure to understand what the contestants will really be bypassing... in order to not be detected by the scanners the samples will have to bypass the heuristic engines in those scanners... getting past a good signature scanner can be as easy as changing a single bit (because a good signature scanner will be very exacting so as to avoid false alarms)...

and those that are using weak heuristics to detect previously unknown malware.

heuristics have to be weakened in order to reduce the number of false alarms to an acceptable level... customers are generally unprepared to resolve potentially false alarms...

It is strange, though, how competitions like CTF, or the recent 0-day competition at CanSecWest, do not attract much complaint, but as soon as antivirus or antimalware tools are targeted it is too much for people

an interesting point, but one that highlights the fact that those other contests revolve around software flaws, whereas showing that new malware doesn't get picked up by blacklists is no more a flaw than notepad's inability to act as a hex editor...

david chartier/ars:

Instead of trying to deride Race to Zero, the AV industry could have a chance at working with the contest to harness what, in reality, could turn out to be some of the best research available on new malicious techniques. "You get what you pay for," as the old saying goes, but in the case of Race to Zero, the AV industry could be passing up a veritable gold mine of free ideas on how to better fight new threats.

except there's nothing for the av industry to learn from this contest... it's already known that malware can be modified, it can be modified in a countably infinite number of ways and if you protect against one the bad guys will just choose (not even find, choose) another... uninformed people think that things would be different if we used heuristics or behaviour blockers or application whitelisting, but the reality is that those can be bypassed in an equally numerous ways... their failures aren't discussed as much because because few people make use of them except for heuristics... and as far as heuristic failures
go people just misinterpret that as a failure in signature scanning, as most people involved in or commenting on this contest have already done...

to repurpose a train of thought from the riskanalys.is blog, the chance of any malware authors coincidentally creating the same malware or using the same modifications as the participants in this contest is basically 0 (n/infinity) so the value of trying to anticipate malware creation/modification techniques and use that knowledge for prevention is also generally 0... conversely, the chance of malware creators/users making use of what is revealed by this contest is greater than zero (because they're lazy, just look at eeye's bootroot to see an example of this having happened) so the value in adding detection for these new samples after the fact (or better still, avoiding the creation of those samples in the first place) is greater than zero...

ultimately i'm reminded once again of something david harley wrote some time ago... he hit the nail on the head when he said the rest of the security industry still doesn't understand av technology, practice, or issues - what people (including the contest organizers) have been saying about this contest proves that much... worse still, robert graham's maligning the credibility of vendors with false statements underscores one of david harley's other points; that the av industry and community remain hugely untrusted...

one has to wonder about the security industry when it doesn't understand one of it's oldest segments (and it's not like the information and people involved aren't available)... the mistrust, on the other hand, is completely understandable under these circumstances - you fear/hate what you do not understand, after all... schneier recently stated that the security industry would be coming to an end and i'll admit i had a bit of a knee-jerk reaction to that (though not so much that i wrote about it - if i did that everytime i disagreed with schneier . . .) but rothman (i think) made a subtle change by saying that security as we know it will come to an end... that's a possibility i almost look forward too - not because i think security should be subsumed by other things, but because the things i'm seeing make me think the security industry has become fundamentally broken (and/or gone mad)... of course, long time readers might recall i have my own prediction about security...