These days all kinds of malware are pervasive on the Internet. Compared to their ancestors that were commonly used for vandalism or demonstration of skills, modern malware, such as Bots, are driven by the underground economics. Often consisting of hundreds to thousands of bots, botnets are one of the most serious threats on the Internet, responsible for various attacks, such as spamming and distributed denial of service (DDoS). As web browsers are the main interface for the majority of Internet users to surf the Internet today, many of such stealthy malware seek to invade via web browsers in the form of browser helper objects (BHO) and browser toolbars.
To defend against Internet malware, existing schemes mainly rely on either signature-based or anomaly-based detection approaches. Signature-based detection is effective for known malware if the malware signature has been generated. However, the effectiveness of signature-based schemes is challenged by polymorphism, metamorphism, obfuscation, encryption, and other techniques. Moreover, signature-based schemes do not work for zero-day (or unknown) malware. On the other hand, anomaly-based detection schemes seek to detect behavior patterns that do not conform to the established normal patterns. Anomaly-based detection schemes do not require malware signatures. However, modern computer software and systems are often complicated, building and analyzing a comprehensive behavior model is time consuming and even impractical.
To overcome these challenges, we propose a novel execution-based approach for stealthy malware detection. In order to facilitate such run-time detection, we aim to design and implement multi-level sandboxing techniques to create controlled running environments to execute testing programs so that their behaviors can be closely observed and analyzed.
First, we leverage virtual machines for OS-level sandboxing to detect bots on individual hosts. By cloning the host image to a virtual machine and screening user input on the virtual machine, the detection noise is significantly reduced. We find that a typical bot exhibits three invariant features along its onset: (1) the startup of a bot is automatic without requiring any user actions; (2) a bot must establish a command and control channel with its botmaster; and (3) a bot will perform local or remote attacks sooner or later. These invariants indicate three indispensable phases (startup, preparation, and attack) for a bot attack. Thus, we propose [special characters omitted] to detect these three phases with the assistance of OS-level sandboxing techniques. To validate [special characters omitted], we implement a prototype of [special characters omitted] based on [special characters omitted]. The results show that [special characters omitted] can successfully detect all the bots in the experiments. However, [special characters omitted] may slightly degrade the user performance. Furthermore, advanced malware could evade [special characters omitted] by performing virtual machine fingerprinting.
Second, to overcome the limitations of OS-level sandboxes, we build [special characters omitted] based on process-level sandboxes for malware detection. The key of [special characters omitted] is to defeat malware anti-detection mechanisms at startup and runtime so that malware behaviors during execution can be accurately captured and distinguished. For analysis, [special characters omitted] always starts a copy, referred to as a shadow process, of any suspicious process in the process-level sandbox by defeating all startup anti-detection mechanisms employed in the suspicious process. To defeat internal runtime anti-detection attempts, [special characters omitted] further makes this shadow process mutually invisible to the original suspicious process. To defeat external anti-detection attempts, [special characters omitted] makes as if the shadow process runs on a different machine to the outside. Since ultimately malware will conduct local information harvesting or dispersion, [special characters omitted] constantly monitors the shadow process behaviors and adopts a hybrid scheme for its behavior analysis. In our experiments, [special characters omitted] can accurately detect all malware samples that employ various anti-detection techniques.
Lastly, to detect and contain malicious browser plugins, we develop [special characters omitted] with intra-process sandboxing techniques. With an intra-process sandbox, only plugins are closely monitored for misbehavior detection without confining the entire process. This further reduces the detection overhead while maintaining transparency to end-users. Based on intra-process sandboxing techniques, we build [special characters omitted] to enhance the security of a browser by enforcing security policies on plugins' accessing requests to the browser's internal objects and external system-level resources, such as file systems and network interfaces. [special characters omitted] deals with both native and .NET-based plugins and its unique design renders it possible to work with commodity web browsers without requiring any modifications to the legacy browser architecture or plugin code. We implement [special characters omitted] in Windows XP and IE8.
|School:||George Mason University|
|School Location:||United States -- Virginia|
|Source:||DAI-B 72/07, Dissertation Abstracts International|
|Subjects:||Computer Engineering, Computer science|
|Keywords:||Computer security, Malware detection, Web browsers|
Copyright in each Dissertation and Thesis is retained by the author. All Rights Reserved
The supplemental file or files you are about to download were provided to ProQuest by the author as part of a
dissertation or thesis. The supplemental files are provided "AS IS" without warranty. ProQuest is not responsible for the
content, format or impact on the supplemental file(s) on our system. in some cases, the file type may be unknown or
may be a .exe file. We recommend caution as you open such files.
Copyright of the original materials contained in the supplemental file is retained by the author and your access to the
supplemental files is subject to the ProQuest Terms and Conditions of use.
Depending on the size of the file(s) you are downloading, the system may take some time to download them. Please be