metasploit open letter revisited

"what we have here is a failure to communicate"

at least, that's what it feels like after fielding a bunch of comments on my last metasploit related post. hd moore specifically felt the need to clarify his view and since it seems that my position could stand some clarification itself i'd like to reiterate my response to him here.

my open letter to the metasploit community was meant to drive home 3 points:

1. AV detection of metasploit output is a problem for anyone trying to use metasploit legitimately and not necessarily appropriate.
2. when people go around saying there's something wrong with AV because it fails to detect metasploit output (in other words, there's something wrong with AV because it fails to do something we've already established would be a problem if it did do) then that actually contributes to the problem by generating market pressure on the AV vendors to correct this supposed problem and add detection.
3. since (1) is a consequence of (2), therefore discouraging (2) should help minimize (1).

(2) is what i saw when i watched john strand's video, but as it happens i've also had a chance to communicate with john and gain a better understanding of where he was coming from (i'd like to think he also gained a better understanding of where i was coming from but that's immaterial for this discussion). i won't divulge the contents of a private communication but i will say that i came away with the impression that he was more interested in showing that there is something wrong with defenses that rely too heavily on what is traditionally considered AV.

if my interpretation is correct then i am in complete agreement with him and i think you'd be hard-pressed to find anyone in the anti-malware community or industry who would disagree. i'd even go so far as to say that his video was an appropriate way of demonstrating that point if that point had been presented in such an unambiguous manner. unfortunately that's not what i took away from it when i first viewed it, and as some of the comments on the 'open letter' post show there are those with a far less nuanced understanding of the topic. i just watched the video again and although i found the phrase that triggered my previous response, in light of my new found understanding i no longer see it the same way.

there isn't anything wrong with AV technology per se (and there are certainly some vendors with more non-signature-related bells and whistles than others), but rather the way people use it (as their one and only defense). it's a poor craftsman who blames his tools, so stop trying to drive screws with a hammer and stop trying to block new/unknown threats with a known-malware scanner.

open letter to the metasploit community

dear metasploit community,

first, please direct your attention to the following video as it demonstrates the very thing i'd like to speak to you about:


as members of the metasploit community, you are no doubt aware of the various legitimate uses for metasploit and the exploits it generates. validating the efficacy of a patch, testing patch deployments, maybe even some penetration testing.

i imagine that you're also aware that some in your ranks have embarked on a wholly misguided ideological quest to highlight the supposed shortcomings of the anti-virus industry using metasploit and it's output.

now let's be clear about something here; we are all in agreement that there are legitimate uses for metasploit's output, therefore that output in general can only be classified as potentially unwanted programs. let's also be clear that the only proper way to address the vulnerabilities that metasploit's output exploits is by patching the vulnerable software. as such, the argument put forward that there's something wrong with anti-virus products that don't detect metasploit output is fallacious on 2 counts: 1) the output isn't necessarily malware (usually only greyware), and 2) anti-virus products are not the proper defense against known exploits (patching is).

there are some concepts in the anti-malware community that some members of the metasploit community may be less aware of than others. one is that there is a countably infinite number of possible programs, and of those a countably infinite number that can do bad things. it is beyond the realm of possibility to detect an infinite number of things so the anti-malware industry has wisely limited it's focus to those that actually exist, to threats that are real, at least as far as malware scanning goes.

another concept which may not be all that apparent is that, while we've faced users of malware creation toolkits for a long time, we have generally lumped them in with the script kiddies due to the complete lack of skill requirements for using such toolkits.

now, even though metasploit isn't a malware creation toolkit (since it's output generally can't be considered malware), it's being used like one in the above video (with the concomitant implication for the user in question). furthermore, this non-malware is being uploaded to virustotal, effectively abusing the service by wasting it's bandwidth on known non-malware, in order to test scanners in a manner that the people who run virustotal themselves say is invalid.

this in turn is putting pressure on av vendors to add detection for metasploit's output, even though that output isn't technically malware, and you know what? as vendors add better and better detection for metasploit's output files, metasploit becomes less and less capable a tool for the legitimate purposes we already identified because anti-virus products will interfere with it's use. how can you use output files from metasploit to validate patch efficacy, test patch deployment, or perform pen-tests if the anti-virus is blocking them? the argument could be made, i suppose, that anti-virus impeding pen-tests is actually a good thing, but it's clear that anti-virus is a fragile defense against exploits when compared to the proper defense of actually patching the vulnerabilities and if AV is interfering with your ability to test if the proper defenses are in place, that really can't be considered a good thing.

with all of this in mind, i would like to ask the metasploit community to do whatever you can to help discourage people from engaging in the behaviour demonstrated in this video. it's arguably worse for the metasploit community than it is for the anti-malware community, but it is definitely bad for both communities and, i think, bad for security in general. no one is helped when people burn one perfectly good tool in order to shame another simply because the tool they're targeting doesn't behave the way they naively think it should.

anti-virus usage fail

how do you top a virustotal usage fail? you attempt to commit virustotal usage fail but use samples that aren't even malware in the first place...

that's what john strand did in a video embedded in daniel miessler's post "Metasploit 3.2 Makes AV Look Silly | DiD is the Only Answer"...

the premise may sound ok - you can create executables using metasploit that won't be detected by the (perhaps severely) cut-down versions of anti-virus products that is used by virustotal... supposedly this points to a problem with anti-virus technology in general, but ask yourself this - is it really a problem that you can create executables that virustotal can't detect? and if so, why?

is the output from metasploit malware? if it is then hdmoore is a bad man and should be stopped (and it's not like we can't find him)... i don't see a lot of people calling him a bad man, though, or suggesting he needs to be stopped - that says to me that metasploit and it's output are not malware or at least occupy that gray area between malware and benign software... as such, if these things aren't malware then why are we expecting anti-malware programs to detect them? the av world knows that if it ain't bad then you shouldn't be catching it and if metasploit output is bad then why aren't we doing more about it?...

if it's not malware then stop expecting anti-malware apps to do anything about it... if it is malware then go after the root cause (isn't that bejtlich is always talking about being the more effective strategy dealing with the malware problem?)...

ultimately, the video and the post it's in make a good point - that you shouldn't be relying on av as your sole form of protection - but the argument would be better served by using a legitimate av failure as an example instead... better still would be to take an approach that doesn't seek to tear down a largely successful anti-malware control in the first place - you can promote defense in depth without erroneously trying to make av look like it's useless... tearing av down does not actually promote defense in depth, it promotes the search for the next great anti-malware hope that we can replace av with - and that's not going to help anybody because all preventative measures (even whatever people replace av with) fail...