Detect rootkits and rootkit behavior with these techniques

When administrators and security professionals hear the word rootkit, most think first of a UNIX-based system. Unfortunately, this only leads to a false sense of security for Windows-based systems. The fact is that Windows rootkits do exist, and you need to be able to detect them.

What is a rootkit?

To clarify, a rootkit is not an exploit — it’s the code or program an attacker leaves behind after a successful exploit. The rootkit then allows the hacker to hide his or her activity on a computer, and it permits access to the computer in the future. To accomplish its goal, a rootkit will modify the execution flow of the operating system or manipulate the data set that the operating system relies on.

Windows operating systems support programs or processes running in two different modes: user mode and kernel mode. Traditional Windows rootkits such as SubSeven and NetBus operate in user mode.

Also known as backdoors or Trojans, user-mode rootkits run as a separate application or within an existing application. They have the same level of system privileges as any other application running on the compromised machine. Since these rootkits operate in user mode, applications such as antivirus scanners can detect the rootkit’s existence if they have a signature file.

On the other hand, a kernel-mode rootkit is remarkably different — and much more powerful and elusive. kernel-mode rootkits have total control over the operating system and can corrupt the entire system.

By design, kernel-mode rootkits control the operating system’s Application Program Interface (API). The rootkit sits between the operating system and the user programs, choosing what those programs can see and do.

In addition, it uses this position to hide itself from detection. If an application such as an antivirus scanner tries to list the contents of a directory containing the rootkit’s files, the rootkit will suppress the filename from the list. It can also hide or control any process on the rooted system.

Rootkit detection

Methods to detect rootkits fall into two categories: Signature-based and heuristic/behavior-based detection.

• Signature-based detection: As its name implies, this method scans the file system for a sequence of bytes that comprise a “fingerprint” that’s unique to a particular rootkit. However, the rootkit’s tendency to hide files by interrupting the execution path of the detection software can limit the success of signature-based detection.
• Heuristic/behavioral-based detection: This method works by identifying deviations in normal operating system patterns or behaviors. For example, this method could detect a rootkit by determining that a system with 200-GB hard drive that reports 160 GB of files has only 15 GB of free space available.

Rootkits are hard to detect. But there are programs — some free and from reputable companies such as F-Secure and Sysinternals – to help you detect their presence on your systems. Microsoft has even stepped up to the plate with its Malicious Software Removal Tool, designed to detect and remove Windows rootkits.

Final thoughts

If you discover someone has compromised your machine, it’s vital that you take the necessary steps to find out if the attacker has installed a rootkit — and then eliminate the threat. Applying vulnerability patches after someone has installed a rootkit on your machine won’t close the security holes that already exist on your network.

Mike Mullins has served as an assistant network administrator and a network security administrator for the U.S. Secret Service and the Defense Information Systems Agency. He is currently the director of operations for the Southern Theater Network Operations and Security Center.

Worried about security issues? Who isn’t? Automatically sign up for our free Security Solutions newsletter, delivered each Friday, and get hands-on advice for locking down your systems.

Detect rootkits and rootkit behavior with these techniques

Grisoft has developed quite a following with its free (for personal, non-commercial use) security applications, and for good reason. AVG anti-virus and anti-spyware programs use little overhead and have proven effective at combating common malware. The security products have racked up a number of awards and accolades as a result.

Now there's an anti-rootkit utility in AVG's free software stable, too, and for users seeking a minimum of interaction, AVG Anti-Rootkit Free may very well be the Right Tool for the Job. Find out why in this screenshot gallery.

Grisoft makes its free AVG Anti-Rootkit application available for download. Users download the avgarkt.exe setup file, which features simple installation.

You may download a free, trial version of AVG Anti-Rootkit here.

cheers Aurobindo