What Is a Rootkit?

A rootkit is a collection of malicious software designed to conceal an attacker's presence on a compromised system while maintaining persistent, privileged access. The term combines "root," the name for the highest-privilege account on Unix-like systems, with "kit," referring to the set of tools deployed after an initial compromise. As defined in the NIST Cybersecurity Resource Center glossary, a rootkit is a set of tools that an attacker installs on a host after gaining elevated access, enabling the attacker to hide activity and sustain control through covert means.

Rootkits are distinguished from other malware categories by their emphasis on stealth and persistence rather than immediate destructive action. They manipulate the operating system's own data structures, interfaces, or firmware to hide files, processes, registry entries, and network connections from both users and security tools. This self-concealment makes rootkits among the most difficult threats in endpoint security to detect and remove.

Types of Rootkits

Rootkits are classified by the layer of the computing stack they occupy. Kernel-mode rootkits operate at the OS kernel level, inserting malicious code as device drivers or kernel modules. They can modify core structures such as the System Service Descriptor Table (SSDT) on Windows or intercept kernel function calls, allowing them to filter what the operating system reports to user space about processes and files.

Bootkit variants load before the operating system during the boot sequence, persisting across OS reinstallation. Hypervisor-based rootkits place a thin virtualization layer beneath the legitimate operating system, which then runs as a guest unaware of the underlying monitor. User-mode rootkits operate at the application layer by intercepting calls to system libraries, a technique easier to detect but also easier to deploy.

Attack Mechanisms

A rootkit installation generally follows an initial privilege escalation step, in which an attacker exploits a kernel vulnerability or a misconfigured service to obtain root or administrator privileges. With elevated access, the rootkit modifies key system components to redirect normal system calls through attacker-controlled code. Direct Kernel Object Manipulation (DKOM), for example, unlinks a malicious process from the operating system's process list so that standard task-listing utilities report no trace of it.

Firmware rootkits target BIOS or UEFI firmware, embedding code that survives disk replacement and OS reinstallation. Because firmware executes before any OS-level security controls initialize, firmware rootkits require low-level forensic tools and, in some cases, physical hardware replacement to remediate.

Detection and Defense

Detecting a rootkit is inherently difficult because the rootkit can subvert the very tools used to examine it. Effective detection strategies rely on out-of-band observation: inspecting system state from a trusted vantage point that the rootkit cannot reach. Hypervisor-based detectors exploit this principle by running detection logic inside a virtual machine monitor, where the rootkit executing in the guest OS has no ability to tamper with the monitor's view of memory and process state. Research published on IEEE Xplore on rootkit detection at the hypervisor level demonstrates how cross-view differencing between the guest OS's reported state and the hypervisor's direct memory inspection can reliably identify hidden processes and connections.

Analysis published in the ACM Digital Library on Linux rootkit techniques documents how existing detection tools fail against kernel rootkits that manipulate eBPF structures and kernel thread stacks, underscoring the need for ongoing research in this area. Preventive measures include Secure Boot enforcement, kernel module signing, and integrity monitoring tools such as Linux's dm-verity that cryptographically verify the kernel and filesystem at load time.

Applications

Rootkits appear in a range of adversarial and defensive contexts within cybersecurity, including:

• Advanced persistent threat (APT) campaigns targeting critical infrastructure
• Nation-state espionage operations requiring long-term stealth
• Digital forensics and malware reverse engineering research
• Trusted execution environment and secure enclave development
• Endpoint detection and response (EDR) product evaluation