A
Tool to Detect Malware
By Erick Engelke
Updated: February 5, 2007
erick@uwaterloo.ca
We are facing a growing problem with viruses and other
malware: they are appearing at our site days before
commercial antivirus products can catch them.
These are called day zero
attacks.
Antivirus products use several methods to detect malware, but mostly they rely on virus signatures, or familiar sequences of bytes in the file that
identify known viruses or some of their variants. Virus signatures work like familiar phrases
in English which make you think “wait, I’ve heard that phrase before”, they are
quite effective.
The antivirus vendors are constantly monitoring
suspicious software from around the world.
For each piece of suspected software, they analyze it, confirm it is malware or not, and if it is dangerous, they add a portion
of the file to their databases of virus signatures. The new signatures are then sent to test
sites, to make sure they don’t cause problems to valid software, and eventually
the signatures are made available to the public. Antivirus clients on personal computers call
back to the vendors’ sites, or to their corporate
antivirus server every day and download the latest signatures. Only then are we protected.
The whole process can take several days from the time
the hacker releases malware; this is the attack window, or the time during which
the malware is best able to spread.
Our site saw several recent incidents when multiple
computers were hacked and joined botnets, or armies of compromised computers which can be
told to do bad things such as doing further attacks. There are certainly other risks, data on the
compromised computers can be destroyed by deletion or by changes, or sensitive
data can be read.
The Othello program is one of several technologies
which can help us defend against day zero attacks, or at least help us identify
problem files so we can alert the antivirus vendors. It helps during the attack window.
Othello uses cryptography to verify important programs
in the computer are actually from trusted vendor and not from hackers. How can Othello verify that programs were
written by someone trustworthy? At first
it would appear we only need to know three things:
-
who in the world wrote
the software
-
was the author
trustworthy
-
has the software been
hacked since… making it no longer trustworthy
A reasonable concern is that there are more than six
billion people on Earth, so it would appear to be a difficult problem to
automatically determine if an author is trustworthy.
To solve these problems, we will rely on
cryptography. Most people know
cryptography involves the use of secrets in communications. In this case we will rely on a special type of
cryptography called public keys, the
same type used for secure web pages.
In public key systems, there are two keys. One is called the public key, because it is published for anyone to read, sort of
like a phone number in a phone book. The
other key is a private key that the
software author will always keep secret.
In reality, both are very long
numbers, so long that they are impossible to guess.
A popular way to describe these keys is to compare
them to the old fashioned wax seal on envelopes. The inside message can be opened and read by
anyone (it’s not encrypted), but only the person with the special embosser –
the private key - can create the seal.
In that crude description, the public key is a phonebook-like list of seal
pictures, and the seal on the envelope verifies who sent the message. Simply possessing one of the sealed envelops
or looking at the photograph does not enable anyone else to exactly fake the
wax seal and impersonate the sender. The
analogy is not perfect, but the basic outcome is that anyone can verify the
sender without gaining the ability to impersonate the sender.
In our case, the keys are used to produce a very hard
math problem that only the person knowing the private key can answer. But what is clever is that anyone can use the
public key to verify that his answer is correct. We call this a digital signature and we know that it verifies who signed the
message.
The specific math problem involves all the bytes in
the file, so if a hacker were to change even a single byte or switch two bytes,
the digital signature would no longer match.
Then we would know it had been tampered and would not trust the file.
In our solution we will rely on the fact that many
companies (like Microsoft, IBM, Symantec and others) place digital signatures
in their software. If you agree that you
trust Microsoft and these other companies, we can prove with a digital
signature that the software you are running comes directly from these companies
and so that software is trustworthy.
For our public key digital signatures to work, we need
a way to reliably publish the public key – like the phone book we
mentioned. If someone could just make up
a key and get it published without further checks, the security would break
because bad people would publish fake keys and claim they were people they were
not.
Instead, we need a public
key infrastructure, which is a bunch of totally trustworthy clearinghouses
that assign and store public keys. In
computer jargon, they are called certificate
authorities.
To understand their role, consider whom you
trust. You trust police in emergencies
and you entrust bankers with your money (perhaps grudgingly). In both cases, you did not personally check
out the individuals. Instead, you know
that someone did background checks before they were hired. And they both will be held responsible for
any misdeeds – they risk penalties if they make poor choices. It is that security infrastructure that
allows you to trust the individual whom you do not even know.
Certificate authorities take similar steps which make
public keys hard to register, and even harder for a hacker.
They require several sources of credentials to prove
the person assigned a public key is really who he says
he is, and that he represents the company he claims. Hackers will not do this – they risk going to
jail if their identities are known, they thrive on anonymity.
The next step is a steep fee of about $500 US to
register. Hackers are rarely willing to
part with money like this.
Before certificate authorities allow someone to
digitally sign programs, they are required to have the person take an oath that
they will not digitally sign viruses or other evil software.
Finally, the certificate authority has a very powerful
weapon. If they believe the oath was
broken, or if the private key was compromised, they can repeal that person’s
trusted status by effectively deleting their public key from the registry. In an instant, anything that was signed by
the user is no longer trusted. And in
case of a hacker buying a key, $500 is gone and the police are knocking at the
door.
It is worth pointing out that few individuals have
these public keys, usually there is one individual in the company who has the
responsibility to sign the software. This
is similar to the certified accountant who signs off on all the financial
reports. There is real responsibility
and the job is not taken lightly.
Microsoft, for example, has thousands of programmers, but only a small
number of individuals can digitally sign programs on its behalf.
All this infrastructure changes our requirements: we only need to know that the
software was signed by someone the certificate authority trusts, on behalf of
some company we trust. The certificate authorities
have already done the background work and hold the power.
Othello is a program administrators and help desks can
run to look for viruses. It first
inspects itself and all files on which it depends, to ensure that it was not
tampered and that the computer is trustworthy.
It then goes to great lengths to find all the system programs running a computer. These system programs or services are integral to Windows, but are also what the hackers
install to make use of their illegally acquired computers. It also collects a list of all the user
applications running, and all the startup programs.
Othello researches each of these programs, computing
the digital signature and verifying it with the appropriate certificate
authority. It then categorizes programs
as either signed or unsigned.
The signed programs are organized by company and
displayed. Generally the signed programs
are somewhat trustworthy. An iPod owner, for example, might find the Apple iPod service installed and waiting for an iPod to be connected.
These services are trusted to be from the companies displayed, but it is
still up to the computer administrator to decide if that software is wanted on
this computer. Hackers, for example,
might try to install a commercial VNC client, which may be signed, but in the
wrong hands acts as a back door for the hackers to take over the machine at any
future time.
The unsigned programs are considered suspect and are reported to the user for
closer inspection. It also supplies an
easy way to submit these files to multiple antivirus engines to see if any
antivirus company has already flagged the file as malware. On day zero attacks, we often find that a few
of these engines detect something suspicious, but not all.
Cerberus is a popular UW-written system program or
service. It runs in the computer all the
time, looking for unusual events.
Cerberus has been extended with the Othello technology so it can keep a
vigilant eye on running processes. As
soon as a suspect service is started, Cerberus Emails the administrator to warn
that the system may be compromised and details exactly why. Cerberus is told to only trust programs from
a well defined list of vendors. Signed
products from other vendors are flagged as signed but still untrusted. Cerberus, as installed on some critical
machines, will usually detect new services within two seconds of their
installation.
There is no magic security bullet that can be used to
totally secure a computer attached to a network. Instead, security is applied in many
protective layers which combine to give good total protection. These layers include firewalls (software
and/or hardware), antivirus technologies, vigilant updates and good management
practices. The tools described here add
another protective layer in our ongoing efforts. They fill a niche for increasingly common
day-zero attacks.