Thursday, April 22, 2010

mcafee's catastrophic false alarm

mcafee recently published a signature update which, when applied, caused their anti-malware software to falsely identify a critical windows component as malware and quarantine it. hilarity did not ensue. boot failure, on the other hand did.

what the hell, mcafee? i thought you folks knew better. this is not the first time a catastrophic false alarm has happened - i don't think it's even the first time it's happened to you. this is a problem that was identified years and years ago and i would have thought that every major AV vendor would have figured out how to avoid this.

i guess i was wrong, so i'll spell it out:
  1. maintain a library of every version of every critical operating system component
  2. before you release a new signature update, use a system with that update applied to scan the aforementioned library of critical operating system components
  3. if there are any alerts, DO NOT RELEASE
this isn't rocket science - it's not even all that computationally expensive because it's just the critical files and just for the OS. sure it would be great if you could do this kind of testing to ensure you don't break a bunch of other things, but preventing the bricking of customer computers should be the bare minimum.

i'm not sure what kind of QA you're doing on your signature updates, but if you're not including this test (or worse if you are and this still happened) then you're doing it wrong.

EDITED 2010/04/22 19:59: it occurred to me after posting the above that there was in fact another way this problem could have been avoided that is perhaps a little more sophisticated.

for years now spam filters have been using whitelisting as a means of avoiding falsely classifying correspondence from important contacts as spam. if mcafee's known malware scanner were outfitted with a small whitelist of critical OS components that must not be false alarmed on under any circumstances then svchost.exe wouldn't have gotten quarantined and mcafee customers would have been saved a great deal of hassle.

this is a perfect example of how blacklists and whitelists can complement each other. i've been saying for a couple years that blacklists, whitelists, and sandboxing were complementary technologies and i've felt that anti-malware products would be even better if they incorporated all 3 of those preventative paradigms. to date the only vendor i'm aware of who actually does that is kaspersky (apparently they introduced sandboxing into their internet security suite this year), but as i haven't tried the product i can't comment on the efficacy of their execution. i'm also not so sure any of their whitelisting is integrated into their blacklist in the manner described above.


Unknown said...

MaCafee completly screwed up my PC on April 22, never again will I use their crap. Thanks. MaCaffee for NOTHING. Rick

kurt wismer said...

well, to mcafee's credit, it appears that they're going to be implementing some of the ideas i presented in my post (see here). that should prevent this from ever happening again.

Peachy said...

@kurt wismer stopping it from happening again is (of course)a good move that fact that it happened is a rookie move.

Not that I rate Mcafee very high to begin with. I recently decided to buy it as there was a special on at my local pc superstore even though every fibre in my body told me not to. With AV as CPU heavy as their latest beast is, I would have expected it to keep my system far safer than what I had eventually decided was best for my set up. It wasn't the case; and I am back (quite happily I might add) to a free AV and safe from this horrendous mistake by the Mcafee team. which, no doubt has set back the trust built up between users who will never want to use anything with the Mcafee stamp of "excellence" on again

kurt wismer said...

i agree, it did seem like amateur hour to me too, hence the tone of the original post (before the edited bit).

i can't really speak much about the product, though, as i haven't used it since before they had dat files.

Peachy said...

@kurt wismer the week i had it installed; it when something like this:

false positive, false positive, false positive, one of those was when it told me that my IM client was in fact a virus (im pretty sure it wasnt :D) uninstall
TBH I can cope with false positives every now and again it is more the resource hog it is for such little payoff, plus if it came up with a file that i had never heard of and i was feeling too lazy to google it i would have just done what all those other poor people did and would now be without a PC

Anonymous said...

good times!


Didier Stevens said...

Checking the AuthentiCode digital signature of Microsoft executables is another way to whitelist. But calculating the cryptographic hash is resource intensive.

kurt wismer said...

@didier stevens:
while digital signatures give you many of the same benefits as a whitelist, it's not quite the same.

i guess it could be considered a highly distributed whitelist where the record of a particular executable's inclusion on that whitelist is stored with that executable, but from a whitelisting perspective there are a number of problems with this model - not the least of which being that you have to trust the signing party which is often the same folks who made the executable in the first place. also the signature is technically only meant to prove authenticity, not fitness for use or safety. whitelists don't prove these very well either, but at least they're usually generated with those concepts in mind.

in the case of the particular suggestion i made above (using a whitelist to prevent false positives on critical system files) digital signatures probably would have sufficed if one were to limit oneself to checking only for microsoft signatures - but that assumes that all critical files are signed. i hope they are, but maybe some aren't, i don't know for sure.

Didier Stevens said...

@kurt I just realized there's a way to limit the resource consumption with digital signatures checking.

Check the signature only after the AV engine has flagged the file as positive. If the signature is a valid Microsoft signature, don't take action.

Almost all MS Windows executables are signed, but most don't contain the signature in the PE file itself, but in a separate catalog file.

Windows File Protection uses digital signatures to check (and preserve) the integrity of critical files. WFP is the feature in Windows XP that automaticaly restores a deleted critical file.