AV-TEST certificates can no longer be trusted
Eugene Kaspersky believes certifications from AV-TEST have become meaningless
How do you go about picking the best of any particular product? And how do you know it’s the best? Well, you would probably start by looking at the results of comparative testing in a specialist magazine, or the online equivalent. I’m sure this is not news to you. The same goes for AV solutions – there are a number of test labs that evaluate and compare a huge variety of antivirus products and then publish the results.
Now, for some unknown reason (below I’ll try and guess why exactly) the renowned German test lab AV-TEST has quietly (there was no warning) modified its certification process for home user products. The changes mean that the certificates produced by the new rules are, to put it mildly, pretty useless for evaluating the merits of different AV products.
Yes, that’s right. I officially declare that AV-TEST certification of AV solutions for home users no longer allows product quality to be compared adequately. In other words, I strongly recommended not using their certificate as a guide when choosing a solution to protect your home PC.
Now let’s take a closer look at what happened and why – or a crash course in interpreting AV-TEST results.
The formula for the ideal AV solution was thought up a long time ago. It goes something like this:
1. 100% protection and 0% false positives.
2. Zero impact on system resources.
3. And no questions to the user.
(4. And, if we want to get into the realms of fantasy, then all that has to be provided absolutely free.)
Obviously, that ideal is unattainable, but we can at least aspire to come as close as possible to it and in particular to:
• Catch as many malicious programs as possible, and if something does get through – be able to treat the infection (and be able to install protection on an infected computer).
• Minimize the risk of false positives – and if they do occur, get rid of them ASAP.
• “Our integral knows no bounds,” claims a good friend of mine, and there’s no limits to the work that goes into optimising use of system memory, processor operation time, the number and size of updates via the Internet. And of course, none of that should impact on the level of security.
That all sounds very straightforward. But what does an average user do when he/she looks at dozens of antivirus products? Which is better, and why? Who can rank them in terms of how closely they come to the “ideal AV solution”? (And remember, all those products make convincing claims to being the best there is.)
So, who can we trust to tell us the truth? Independent testers, of course. And that includes AV-TEST.
A few years back the AV-TEST team created a very good method for testing products and an equally good system of certification was based on that method. Products were tested on three criteria: protection (prevention of infections), repair, and usability (ease of use, performance and number of false positives). A certificate is issued, or not as the case may be, based on the results (number of points accumulated). We always supported this system and held it up as an example to the other testers in the “premier league” of comparative testing.
So what metamorphosis has taken place at AV-TEST? Why can we no longer trust its certification system?
First of all, the necessary criteria for obtaining a certificate have changed. The important repair parameter has been discarded. And why not? I mean, what’s the point of an AV solution that’s capable of detecting an infection but is incapable of treating it? (Just imagine at the dentist’s: “Oh, you’ve got tooth decay, but we won’t treat it – we can’t treat it!”) Not so long ago we found out that about 5% of all computers in the world with AV installed are infected! Every twentieth!
We found that the same proportion of our new users required the treatment of an active infection immediately after installing our product. In other words, the ability to treat active infections is essential for every twentieth user, while the other 19 may well need it at some point in the future if they suddenly find that they themselves are in that one-in-20.