i wouldn't bet on it

last year cryptography professor matthew green made a bet with mikko hypponen that by the 15th of this month there would be a snowden doc released that showed that US AV companies collaborated with the NSA. he has since accepted that he lost the bet to mikko, but should he have?

i mentioned to matthew the case of mcafee being in bed with government malware writing firm hbgary and mikko chimed in that hbgary wasn't an AV company and being partners with them wasn't enough to win the bet. aside from the fact that this is the first time after all these years that i've seen a member of the AV industry publicly comment on the relationship between mcafee and hbgary (i guess managing matthew's perception of AV is more important than managing mine), something about mikko's response rang hollow.

one way to interpret the situation with hbgary is to view them as government contractors whom mcafee endorsed, advertised, and helped get their code onto the systems of mcafee's customers (hbgary makes a technology that integrates with mcafee's endpoint security product). that certainly would have given hbgary access to systems and organizations they might have had difficulty getting otherwise. i have no idea if that access was ever used in an offensive way, though, so this line of thought is a little iffy.

another way to interpret the situation is to directly contradict mikko and admit that hbgary is a member of the AV industry. after all, they make and sell technology that integrates into an endpoint security product. they may only be on the fringe of the industry, but what more do you have to do to be a member of the industry than make and sell technology for fighting malware? the fact that they also made malware for the government makes them essentially a US AV company that collaborated with the government in one of the worst ways possible.

i feel like this should be enough to have won matthew green the bet, at least in spirit, but the letter of the bet was apparently that a snowden doc would reveal it and the revelation about mcafee and hbgary actually predates snowden's leaks by a number of years. 

so, the question becomes are there any companies that happen to be members of the AV industry and also happen to have been fingered by a snowden leak? it turns out there was (at least) one. they were probably forgotten because they're not just an AV vendor, but AV vendor does happen to be one of the many hats that microsoft wears (plenty of security experts were even advising people to drop their paid-for AV in favour of microsoft's offering at one point in time), and microsoft was most certainly fingered by snowden docs. the instances where microsoft helped the government may not have involved their anti-malware department, but the fact remains that a company that is a member of the AV industry was revealed by snowden documents to have collaborated with the government.

i imagine mikko could find a way to argue this doesn't count either - i admit it's not air-tight - but given how close it meets both the spirit and (as i understand it) the letter of the bet, i think mikko should match the sum he had matthew pay to the EFF and pay it to an organization of matthew's choosing. i won't bet on that happening, though.

AV complicity explained

earlier this week i wrote a post about the idea of the AV industry being somehow complicit in the government spying that has been all over the news for months. some people seemed to really 'get it' while others, for various reasons, did not; so i thought i'd try to be a little more clear about my thoughts on the subject.

the question that the EFF et al have put towards the AV industry (besides having already been asked and answered some years ago) is a little banal, a little pedestrian, a little sterile. real life is messy and complicated and things don't always fit into neat little boxes. i wanted to try to get people to think outside the box with respect to complicity, what it means, what it would look like, etc. but i think some people have a hard time letting go of the straightforward question of complicity that has been put forward so let's start by talking about that.

has the NSA (or other organization) asked members of the AV industry to look the other way and has the AV industry (or parts thereof) agreed to that request? almost certainly the NSA has not made such a request, for at least a couple of reasons:

1. telling people about your super-secret malware is just plain bad OpSec. if you want to keep something secret, the last thing you want to do is tell dozens of armies of reverse engineers to look the other way.
2. too many of the companies that make up the AV industry are based out of foreign countries and so are in no way answerable to the NSA or any other single intelligence organization.
3. there's quite literally no need. there are already well established techniques for making malware that AV software doesn't currently detect. commercial malware writers have been honing this craft for years and it seems ridiculous to suggest that a well-funded intelligence agency would be any less capable.

now while it seems comical that such a request would be made, to suggest that the AV industry would agree to such a request would probably best be described as insulting. whatever you might think of the AV industry, there are quite a few highly principled individuals working in that would flat out refuse, in all likelihood regardless of what their employer decided (in the hypothetical case that the pointy-haired bosses in AV aren't quite as principled).

now please feel free to enjoy a sigh of relief over the fact that i don't think the AV industry has secretly agreed to get into bed with the NSA and help them spy on people.

done? good, because now we're going to take a deeper look at the nature of complicity and the rest of this post is probably not going to be nearly as pleasant.

here's one of the very first things wikipedia has to say about complicity:

An individual is complicit in a crime if he/she is aware of its occurrence and has the ability to report the crime, but fails to do so. As such, the individual effectively allows criminals to carry out a crime despite possibly being able to stop them, either directly or by contacting the authorities, thus making the individual a de facto accessory to the crime rather than an innocent bystander.

in the case of government spying we may or may not be talking about a crime. the government says they broke no law and observers speculate that that may be because they've subverted the law (much like they subverted encryption algorithms). so let's consider a version of this that relates to ethical and/or moral wrong-doing instead of legal wrong-doing:

an individual is complicit in wrong-doing if he/she is aware of it's occurrence and has the ability to alert relevant parties but fails to do so. as such, the individual effectively allows immoral or unethical people to carry out their wrong-doing despite possibly being able to stop them either directly or by alerting others who can, thus making the individual a de facto accessory to the wrong-doing rather than an innocent bystander.

in this context, could the AV industry be complicit with government spying? perhaps not directly, not in the sense that they saw what the government was doing and failed to alert people to that wrong-doing. however, what about a different wrong-doing by a different entity but still related to the government spying?

hbgary wrote spyware for the government. this became public knowledge in the beginning of 2011. by providing the government with tools to perpetrate spying they become accessories to that spying.

hbgary was and is a partner of mcafee. now what is the nature of this partnership? hbgary is an integration partner. they make technology that integrates into mcafee's endpoint security product to extend it's functionality. mcafee does marketing/advertising for this technology and by extension for hbgary, giving them exposure, lending them credibility, and generally helping them make money. that money is almost certainly re-invested into research and development of hbgary's products, which includes governmental malware that's used for spying on people/organizations. there are mcafee customers out there right now whose security suite includes components that were written by known malware writers and endorsed by mcafee (although they make sure to weasel out of responsibility for anything going wrong with those components with some fine print). mcafee didn't break off the partnership when hbgary's status as an accessory to government spying became known, and since they didn't break off the partnership you can probably make a safe bet that they didn't warn those customers that part of their security suite was made by people aiding the government in spying either. even if we ignore the fact that mcafee aids a business that writes malware for the government, mcafee's failure to raise the alarm about the possible compromising nature of any content provided by hbgary makes them accessories to hbgary's wrong-doing. by breaking ties with hbgary and warning the public about what hbgary was up to they could have had a serious impact on hbgary's cash flow and hurt their ability to win contracts and/or execute on their more offensive espionage-assisting projects. they didn't do any of that and that makes them complicit in the sense discussed a few paragraphs earlier.

the rest of the AV industry may not be directly aiding hbgary's business but, like mcafee, they have failed to raise any alarm about hbgary. they could have done much the same as mcafee by warning the public, with the added bonus that they would have hurt one of the biggest competitors in their own industry while they were at it and that would have benefited all of them (except mcafee, of course). again, failing to act to help prevent wrong-doing makes them a de facto accessory to that wrong-doing. the AV industry as a whole is complicit in the sense discussed earlier.

of course, the AV industry isn't alone in being accessories to an accessory to government spying, and that brings up a consideration that should not be overlooked because there is a larger context here. historically, the culture of the AV industry has been one that values being very selective in things like who to trust, who to accept into certain groups, etc. add to that a very narrowly defined mission statement (to fight viruses and other malware) and it's little wonder that the ethical boundaries that developed in the early days were so dead-set against hiring, paying, or doing anything else that might assist malware writers or possibly promote malware writing. heck, i knew one member who wouldn't even engage viruses writers in conversation, and another who said he was wary of hiring anyone who already knew about viruses just in case they came by that knowledge through unsavoury means. aiding malware writers, turning a blind eye to their activities, etc. are things that normally would have violated AV's early ethical boundaries.

by contrast, the broader security industry is highly inclusive and has long viewed the AV industry's selectivity as unfair elitism. that inclusivity means that the security industry isn't actually just one homogeneous group. there are many groups, from cryptographers to security operations personnel to vulnerability researchers to penetration testers, etc. each one has it's own distinct mission statement and it's own code of ethics. what do you think you get from a highly inclusive melting pot of security disciplines? well, in order for them to tolerate each other, one necessary outcome is a very relaxed ethical 'soup'. many quarters openly embrace the more offensive security-related disciplines such as malware creation. in order for AV to integrate into this broader security community (and they have been, gradually, over time), AV has to loosen it's own ethical restrictions and be more accepting.

so while the AV industry failed to raise the alarm about hbgary, the broader security industry failed as well. the difference is that ethics in the security industry don't necessarily require raising an alarm over what was going on. hbgary is a respected company in security industry circles and it's founder greg hoglund is a respected researcher whose proclivity for creating malware has been known for a long, long time. as far as the security industry is concerned, hbgary's activities don't necessary qualify as ethical wrong-doing. there will probably be those who think it does, but in general the ethical soup will be permissive enough to allow it, and without being able to call something "wrong-doing" there can be no complicity. this is where AV is going as it continues to integrate into the broader security community. in fact it may be there already. maybe that's the reason they didn't raise the alarm - because they've become ethically compromised, not as a result of a request from some intelligence organization, but as a result of trying to fit in and be something other than what they used to be.

in the final analysis, if you were hoping for a yes or no answer to the question of whether AV is in any way complicit in the spying that the government has been doing (specifically, the spying done using malware), i'm afraid you're going to be disappointed. it depends. based on AV's earlier ethics the answer would probably be yes. based on the security community's ethics the answer may well be no. where is the AV industry now? somewhere between what they were and what the broader security community is. ethical relativity is unfortunately a significant complicating factor. then again, i'm an uncompromising bastard, so i say "yes" (after all, i did grow up with those old-school ethics).

what would AV's complicity in government spying look like?

as you may well have heard, the EFF and a bunch of security experts have written an open letter to the AV industry asking about any possible involvement by them in the mass spying scandal that has been in the headlines for much of this year. at first i thought this was old news for AV, since the issue of government trojans has actually been around a lot longer than the current spying revelations. i thought these people had simply failed to do their homework but, as time passed, the wheels began to turn and i started thinking differently. now i think the question we should all be asking ourselves is, what would AV's complicity look like?

some background, first. the subject of government trojans have been around for over a decade. magic lantern, for example, dates back to 2001 (or at least public awareness of it does). so it should come as little surprise that the question of whether the AV industry looks the other way has come up before. in 2007 cnet ran a story where 13 different vendors were asked about this very thing. they all more or less denied being a party to such shenanigans, but i suggest you read the article and pay careful attention to the answers.

now earlier this year one of the first controversial spying revelations to come about was about a program called PRISM which a whole bunch of well known, big name internet companies (including google, microsoft, yahoo, facebook, etc) were apparently involved with. the companies all denied it of course, and it turns out they may be legally required to do so.

that adds an interesting wrinkle to the question now being put towards the AV industry; would they be allowed to admit to any complicity that might be going on? they say actions speek louder than words, so maybe we should look for something other than the carefully crafted assurances of multi-million dollar corporations. maybe what we should be looking for is the same thing that alerted us to the mass spying in the first place - a leak. maybe then we can get a glimpse of their actions.

back in early 2011 a rather spectacular breach occurred. security firm hbgary was breached by some members of anonymous, and one of the things that leaked out was the fact that hbgary wrote malware for the government. in fact, it doesn't take much imagination to suppose that this would be the very type of malware the EFF et al are concerned the AV industry may have been asked to ignore.

it's unknown whether any AV vendor actually did field such a request. i have my doubts since traditional commercial malware writers seem to be perfectly capable of creating undetected malware without making such requests. that being said, one fact that became rather suspicious in light of the revelations about hbgary was the fact that they were partners with mcafee, one of the biggest AV vendors around and certainly one of the best known names in AV. i wrote about this apparent ethical conflict back in february of 2011, and then again in march of 2011 to note the tremendous non-reaction from the industry. i even went so far as to create a blog specifically for keeping an eye on the industry (though as an outsider myself there was little i could do on my own).

the EFF and others want to know if the AV industry has been complicit in the government's spying. well, one AV vendor was notably evasive when asked by cnet in 2007 about their handling of governmental trojans/police spyware. that same AV vendor was and still is partnered with a company that wrote government malware (in all likelihood for very purpose in question).  furthermore, in the intervening years, nothing has come of it. no other vendor has said anything or done anything to call attention to or raise awareness of this partnership. even after the mass surveillance controversy started earlier this year, not a one bothered to raise the alarm and suggest that mcafee might at least in principle be compromised by that partnership, even though they certainly could have benefited from disrupting mcafee's market share. no one thought they could profit from it? no one thought it was their duty to warn people of a potential problem? to raise concerns that the protection mcafee's customers receive may suffer in some way because of their close ties with government malware writers? to give voice to the doubts this partnership creates even after publicly wringing their hands over how wrong what the government themselves were doing was?

AV vendors may or may not have been asked to turn a blind eye to government malware - we may never know, and it's impossible to prove a negative. but they've done a heck of a job turning a blind eye to the people who make government malware and to those in their own ranks who got in bed with government malware writers. i asked at the beginning what AV complicity would look like and i think when it comes to those whose job it is to raise an alarm, complicity would probably have to look like silence (and something about silence makes me sick).

(2013-10-29 13:21 - updated to change the open letter link to point to the blog post that includes the list of intended recipients as well as a link to the letter itself)

ethical conflict in the anti-malware domain

forgive my silence over the last little while. motivation to blog sometimes isn't easy to find. as time wears on, fewer and fewer things get under my skin enough to drive me to rant (is that what it means to mellow with age?). but since you're reading this i think you can guess what this post infers.

five years ago i wrote a post about what i perceived as an ethical conflict in the anti-'rootkit' domain. it detailed the actions of two of the most notorious names in stealthkit research, jamie butler and greg hoglund, and how they were profiting from making a particular niche of the malware problem more popular (and thus, inevitably a bigger problem).

one of the things i pointed out was that symantec was working with a start-up company (komoku) that had jamie butler (author of what was at one time one of the most widely deployed stealthkits around) as it's chief technology officer. i thought the fact that an anti-malware company was in bed with a company that hired such a high profile malware writer deserved at least a moment of reflection, considering the hard-line stance anti-malware companies take on hiring malware writers themselves. at the end of the day, mind you, that start-up was focused on prevention so maybe the argument could be made that mr. butler had or was trying to reform in some way. (mr. butler has since moved on to mandiant, along with his disciple {the FU2 to butler's FU} peter silberman)

when i read earlier this past week that another anti-malware company (mcafee) had been working with greg hoglund's company (hbgary) i thought it an interesting historical footnote but paid little attention to it beyond that (though, if i had remembered that mcafee had once been pointing fingers at rootkitDOTcom, maybe the hypocrisy would have stood out more). after all, little attention seemed to be paid to such connections five years ago so why should this time be any different? well, that was before i knew what hbgary was in to.

apparently, on top of the legitimate work that one can find out about by visiting the hbgary website (which of course i won't link to), it appears that hbgary also writes and sells malware for fairly large sums of money. the customers for their malware include the government/military but might not stop there. even if that set of customers does stop there, hbgary appears to be in the high-end commercial malware business.

so where does that leave mcafee? it leaves them in bed with commercial malware writers. while AV companies have been proclaiming for decades that they don't and won't hire malware writers, apparently they don't have to. they can simply partner with the boutique security shops that do. clearly they are not picking their business associates as carefully as they are their actual employees.

and then there's the claim that surfaces from time to time that AV companies won't make special provisions to keep malware deployed by the authorities from getting detected. what's the point of making such a claim if you're just going to turn around and do business with the company that may very well be making said malware?

how many other AV companies, besides mcafee, were or are in bed with hbgary? how many are in bed with companies LIKE hbgary? where's their ethical high horse when it comes to partnerships? why wasn't the "malware writers need not apply" policy updated when commercial malware became the norm and presented the loophole we see before us today?

some AV companies are rewarding malware writers financially. it may not be in the ways we traditionally thought of, but with the #2 company in the industry involved in this practice (and arguably the #1 company as well, depending on where you want to draw the line), the end result is AV companies contributing to the commercial success of malware writers, and that is not ok at all.

ethical conflict in the anti-'rootkit' domain

not long ago i blogged about how anti-virus companies don't hire virus writers... now of course there was a somewhat high profile exception involving low profile company (ie. small start-up so far removed from the av industry they didn't know how badly they'd be shooting themselves in the foot) but this example is the exception that proves the rule...

but now i'd like to shed some light on a similar example in the so-called 'rootkit' domain... in a securityfocus.com article from last year we have the following:

Jamie Butler: I am a kernel developer and contributor at rootkit.com, where I go by the name Fuzen. I really enjoyed working with Greg on the book Rootkits: Subverting the Windows Kernel. Most of my time is spent at the bit and byte level.

and

Jamie Butler: I am not sure of the exact time frame that I learned about rootkits. I guess I knew of their existence since the early UNIX Trojan system file replacements. However, I did not become active in the research until 2001 when I was working on my Master's degree. At that time, I was looking for modifications to the operating system in memory used to hide things. After that research, I realized that you could alter data structures directly in kernel memory to hide without modifying any operating system code. That is what the FU rootkit, which I wrote, is intended to demonstrate. It is not malicious but more proof of a premise.

and finally

Greg Hoglund: Not really - I don't spend much time trying to detect rootkits. But, I do know that FU is one of the most widely deployed rootkits in the world. [It] seems to be the rootkit of choice for spyware and bot networks right now, and I've heard that they don't even bother recompiling the source - that the DLL's found in spyware match the checksum of the precompiled stuff available for download from rootkit.com. I think that is kinda lame, these spyware guys don't even bother to use the source.

did you catch that? jamie butler (aka fuzen) created what became one of the most widely deployed 'rootkits' in the world (not the only 'rootkit' he's authored, by the way)... and then he co-authored what some have described as the only book on 'rootkits'...

so? how does that relate to av companies who hire or don't hire virus writers? well, how about this - the same jamie butler is the CTO of komoku, a company that has received $2.4 million from various branches of the US government to develop a solution to the 'rootkit' problem in the form of CoPilot and Gamma... Gamma, being a software product, is intended to be priced similarly to anti-virus products but won't be as successful against 'rootkits' because it's software only... CoPilot, on the other hand, is a hardware product that will probably be going for about $1000 a pop so they'll be pulling down some major cash when they finally get to market... oh, and did i mention they've partnered with symantec?

but wait, there's more - while working for hoglund's security company (hbgary) butler developed a 'rootkit detection technology called VICE and now he and peter silberman (who has also worked at hbgary and authored the FUTo 'rootkit' which is also freely available on hoglund's site) have developed yet another 'rootkit' detection technology called RAIDE - it's still too new for a business to be made out of it yet, we'll have to wait and see...

let me boil that down for you - jamie butler created the FU 'rootkit'... jamie butler and greg hoglund made FU available for free download on greg hoglund's 'rootkit' site... jamie butler and greg hoglund wrote the only book on this new threat, marking them both as experts in the field... FU, a 'rootkit' written by one of the foremost experts in the 'rootkit' field became one of the most popular 'rootkits' amongst those who deploy them, in fact the exact binaries that jamie butler and greg hoglund make available are the same ones getting deployed and greg hoglund admits it... jamie butler and greg hoglund get money for a book about a problem they actively contribute to... jamie butler, while working for greg hoglund, develops anti-'rootkit' technolog - free to use but still it draws people to hoglund's other products and so they both get money as a result of technology meant to thwart a problem they actively contribute to... jamie butler, being an expert in the field, is hired and made CTO by an anti-'rootkit' company called komoku which in turn gets millions from the government and so jamie butler gets still more money for working on technology meant to thwart a problem he actively contributes to... symantec, who correctly refuses to hire virus writers, partners with a company that has a 'rootkit' writer and distributor as their CTO... jamie butler and peter silberman, both 'rootkit' authors whose works are freely downloadable from hoglund's site, develop yet another technology meant to thwart a problem they both actively contribute to ....

so there you have it - people creating, distributing, advertizing, and evangelizing 'rootkits' - basically contributing to the problem by creating the tools, popularizing the threat, and arming the bad guys... and then some write a book about the problem or actually create products to solve the problem and in so doing rake in lots of dough... if this were going on in the anti-virus industry people would be raising a big stink, and it's not like anyone would trust an anti-spam company that hired spammers, so what the heck is going on here? how are these people getting away with this?

VICE? RAIDE? CoPilot or Gamma? i certainly won't be touching any of those with a 10' barge pole - give me sysinternals rootkit revealer or f-secure's blacklight, thanks... i'd rather not support people who are actively part of the problem... it's a shame the US government doesn't feel the same way... also a shame symantec doesn't feel that way - not sure why they seem to have less problems with 'rootkit' creators than with virus writers...