are good spyware simulators still a bad idea

so apparently someone came up with the idea of creating spyware simulators collectively known as spycar (apparently a play on the eicar test file) for testing the effectiveness of anti-spyware applications...

the idea, basically, is that a bunch of these tiny applications could be written that each do some small spyware-like thing... supposedly this is meant to test how well an anti-spyware app can detect spyware they don't already know about - the more of these spyware simulators the anti-spyware detects the better it is (in theory)...

this idea came up before in the virus realm years ago and i'll direct you now to sarah gordon's paper are good virus simulators still a bad idea... i suggest reading the entire thing through but for now just scroll down to the conclusion and replace "virus" with "spyware" and it will be about as valid...

essentially, for testing the effectiveness of your security app, using simulators only tells you how good the app is at detecting simulations rather than the real thing... to assume that anti-spyware vendors won't simply add signatures for the simulations is naive, especially when their work starts being judged on how well it detects the simulations... there's no way this effort on it's own is ever going to force the vendors to improve their generic detection techniques, it's just not a cost effective way for the vendors to deal with the simulations...

as for using these things as test files to see if your anti-spyware app is installed correctly (as their namesake the eicar standard anti-virus test file was intended for) the anti-spyware vendors could just as easily use the eicar test file directly for that purpose... there's no need to implement any particular behaviour in the test file if all you're doing is checking if the app can detect something it already knows about; it's not like the eicar test file has any virus-like behaviours, nothing of the sort was required to test the sanity of an installation, all it does is display a string of text on the screen (at the command prompt) when executed...

clearly, if folks in the anti-spyware field are just now coming up with an idea that was debunked in the anti-virus field over a decade ago, the anti-spyware field isn't anywhere near as mature... the question is are the anti-spyware principals looking into learning from what the anti-virus community and industry have already figured out?...