Is it the end of the line for antivirus signatures?

One fact leaps out of the Symantec Intelligence Report for July 2011: The sharp rise in “aggressively polymorphic” malware delivered by email over the last six months. This form of malware adopts a range of techniques to disguise itself to avoid detection, even changing its code each time it starts up.

“This new aggressive approach to distributing generic polymorphic malware on such a scale should be concerning for many businesses, particularly those who rely solely on more traditional security countermeasures, which this type of malware is designed to evade,” Symantec wrote in the report.

The report’s author, Paul Wood, a senior intelligence analyst at Symantec, concludes: “Technology cannot rely on signatures and heuristics alone, and must also take into account the integrity of an executable based on knowledge of its reputation and circulation in the real world.”

It is a problem that has been bubbling up over several years, but the surge in recent months is worthy of note. Polymorphic malware accounted for 23.7% of all email-borne malware intercepted in July, more than double the rate detected in January. In the same six-month period, according to the Symantec report, the number of variants, or different strains of malware involved in each attack, has also grown dramatically, by a factor of 25 times.

Traditionally, whenever antimalware companies have detected a new piece of malware, they take a sample of the code – its signature – and, in an early version of cross-industry cooperation, share that with the rest of the antimalware industry. Thus, when an antivirus program next recognised that piece of code by its signature, it would know to reject it.

But, if you need to write a new signature for every new variant of every piece of malware, you end up with a large and unwieldy signature database; one which is constantly lagging behind the malware authors.

Generic signatures need to be specific in one sense, but also fuzzy enough to find variants within a group

David Emm
Kaspersky Lab

Faced with this fast-changing threat landscape, antimalware companies have had to supplement signatures with other techniques, such as reputation scoring, heuristics, code analysis and whitelisting. But, signatures have remained a cornerstone of antimalware defences up to now.

Some companies, however, have started to question the need for antivirus signatures, arguing that, no matter how fast the AV companies react, many users will be affected by new variants of malware before they reach the signature database. And, by that time, the malware will have morphed into something new anyway.

One such company is Colorado-based Webroot Software, which has just launched Webroot SecureAnywhere, a technology that dispenses with signatures altogether and relies on a bundle of other techniques to spot malware before it has a chance to do any damage. The result is a piece of security software that occupies just 580K of memory and can conduct a complete virus scan on an endpoint in a couple of minutes.

According to Webroot’s vice president of product management, Brian Czarny, security companies, including Webroot “have been miserable at protecting users online, offering solutions still focused on yesterday’s threat landscape.”

Webroot has done a complete rewrite of its own software, incorporating technologies from recent strategic acquisitions – cloud-based antivirus from PrevX, and categorisation of Web content from Brightcloud.

“We now focus on file behaviour as the file executes, rather than having to rely on having a copy of the malware, creating a definition for it, and then pushing it out to the end user’s machine,” Czarny said. “We rely heavily on the cloud, which gives us a small footprint on the machine. And we use the overall intelligence gathered from all our users, consumers and businesses, to protect everyone.”

Webroot is not alone. FireEye, a California-based company, has been courting customers with a range of security products designed to deal with zero-days threats, which by definition, have no recognisable signature. The company published a report in September showing that 99% of organisations suffered at least one malicious infection in their networks each week, and 80% suffered more than 100 infections per week. It concluded signatures and what it described as “crude heuristics” were no longer suited to the more dynamic threat landscape.

However, even though most companies still cling to signatures, the nature of signatures is changing. According to David Emm, a senior security researcher at Kaspersky Lab, the signature now tends to be most useful in cleaning up infections rather than stopping them. “We’re not at a point where signatures will disappear,” he said. “They are valuable for remediation, where a business is dealing with an outbreak. The signature is a great way of finding infections, and helping to clean up the systems.”

Greg Day, CTO for Symantec EMEA, agreed: “Signatures help in investigation and forensics,” he said. “It doesn’t matter that we get signatures after the fact. Other technologies will become more predominant in blocking threats, but we will always want to put a name to the problem, and then understand what the malware was trying to do.”

Even so, the sheer size of the signature database is still a problem, and companies have to find ways of keeping it manageable. As Day admits, many customers already complain about the size of the current database and ask how it can be cut down, possibly by stripping out signatures for old DoS-based viruses.

While most researchers are uneasy about doing that, vendors have found ways of building generic signatures that seek to help ease signature database-size complaints. “We do manage to rationalise signatures, and try to write a generic signature that covers a collection of viruses or variants,” Day said. “We are always trying to consolidate. As much as we keep adding more in, we equally try to take things out."

Emm said similar techniques have been so successful at Kaspersky that the size of the company's signature database began to fall at one point last year, thanks to the creation of more generic signatures. “Generic signatures need to be specific in one sense, but also fuzzy enough to find variants within a group,” Emm said. “But there’s no guarantee you’ll find every one, and if another variation comes along that is sufficiently different, it may get by, but then you can retroactively go back and rewrite the signature to detect it. Signatures are not fixed in stone for all time.”

Size and frequent changes in the signature database have also sparked debate among researchers about where it should sit: on the client PC or in the cloud somewhere. Symantec’s Day said a lot of people favour taking the database off the client altogether, as it can be better maintained and stored in the cloud.

But as he said, most endpoints work offline sometimes, and while that might save them from Internet-borne threats, there is still the danger of a dodgy USB stick (remember Conficker?) introducing an infection. And, for that reason, he thinks signatures will still sit on the endpoint for some time to come.

Ron Condon is UK bureau chief for SearchSecurity.co.UK. Send comments on this column to mailto:[email protected].