Abstract
The research on computer worms and their defense has been in full swing in recent years. Still, there are some challenges in understanding the nature of worm propagation because of the complex structure and dynamics of networks, in conducting detailed network worm emulation experiments, and in devising and evaluating new worm defense or containment strategies. In this report, we propose a new virtualization method for running high-fidelity network emulation experiments on a network testbed and present an integrated toolkit we developed for the benefit of experiment specification and data visualization. On the analytical model and computer simulation front, we extend the KMSim worm model, a variation of Kermack-McKendrick epidemic model that takes enterprise networks as the unit of analysis so that various distributions of worm susceptible and network topology characteristics can be accounted for. Extension of KMSim worm model is used to study the self-destructing and removal/death behavior of worms and for the simulation of worm propagation in enterprise networks.
On the defense side, we call attention on the danger of a class of worms which target specific enterprise networks. The efficiency and stealthiness of such local scanning worms with advanced scan strategies would render most existing worm defense schemes ineffective. For this reason, we propose a new dark port detection scheme to actively monitor all intra-enterprise network connections and raise security alerts when a new connection attempt targeting a nonservicing host/port tuple is detected. To compensate possible false positives and balance between worm defense and maintaining normal service, we further explore using a Markov decision process model to quantitatively evaluate various defense strategies and make best defense decisions based on this cost-benefit analysis.
Cost-effective containment requires accurate estimation of worm virulence and extent of network damages under various game scenarios. In this regard, we develop a maximum likelihood estimation algorithm to estimate the size of susceptible population using inter-arrival timing information collected by the dark port detectors. Based on the idea of worm propagation detection and worm virulence estimation algorithm, we propose two collaborative containment strategies which can effectively containment a local scanning worm and cause minimum service disruption to the network.
CHAPTER 1 MOTIVATION AND BACKGROUND
There are three parts in this introduction and outline chapter. The first part introduces some background of worm and worm research and the motivation for the presented research in this thesis. The second part gives a brief literature review on the past and latest research on the related topics. The final part lists major research contributions of the study and outlines the structure of the thesis. Before we proceed into the introduction section, we first define some important terms used in this thesis.
Term definitions
- Worm: A worm is a computer program or a piece of executable code that can replicate itself from one computer system to another computer system through network or other communication channels. A computer system infected by a worm is called a worm victim, an infected host, a worm infective, or a worm instance.
- Propagation: The process of worm spreading from the initial computer system or computer systems to more computer systems is called worm propagation.
- Scanning worm: A scanning worm is one kind of worm that actively sends out probes to new network addresses to find target systems for infection and replication.
- Local scanning worm: A local scanning worm is one kind of scanning worm that limits its scans to a particular network address range, e.g., a /16 address range. Normally, this network address range corresponds to a local network or an enterprise network. We also use term local worm or enterprise worm in this thesis to refer to local scanning worms.
1.1 MOTIVATION AND INTRODUCTION
Recently, the Internet has experienced a series of self-propagating worm attacks (e.g., Code-Red, SQL Slammer, Blaster, etc.) that have caused significant disruption to financial, transportation, and government institutions, among others. The severity of the experienced worm effects clearly shows that worms are a serious threat to the security of our network infrastructure. Worms can be classified by their target acquisition strategies, activation methods, payloads, or by their spreading media. Besides scanning worms, worms could propagate using a pre-generated target list (hit-list worm) or by finding new targets based on information on newly infected hosts (topological worm).
There are many ongoing research efforts to study the worm dynamics and devise ways to combat against worm attacks. Major methodologies for worm research can be classified into the following categories: mathematical modeling, simulation, emulation, and hybrid approach. Numerous mathematical models have been proposed to study the worm propagation on the Internet or hypothetical networks, many based on epidemics models of biology [37, 42] . Normally the results from these modeling and analysis match against the results from real world monitoring rather well, but their underlying assumption of uniform distribution on a simplified Internet topology becomes problematic when the speed of worm scanning is bounded by the limited uplink bandwidth of component networks or when dealing with worm propagation and defense on an enterprise network where topology and hardware/software configurations vary considerably.
Simulation and emulation are also used extensively in studying the propagation and other behaviors of worms. Simulation packages such as Network Simulator (NS) and Scalable Simulation Framework (SSF) can be used to reproduce worm behavior in a more detailed level, but simulation on a simple CPU or multiple CPUs can only handle a rather small topology without losing substantial fidelity. The fidelity of emulation is highest compared with numerical analysis and simulation. Ideally, emulation should be run by a one-to-one fashion between real world hosts and experiment nodes to get the most accurate results, which makes scalability a big concern for emulation. Emulation can be run on a production network, a testbed network, or an overlay network. To balance between fidelity and scalability, various hybrid approaches have been proposed, including the combination of emulation and simulation, simulation with scaling-down, etc.
New proposals on worm defense and containment have to be evaluated to test their effectiveness and other important performance metrics. Evaluation of worm defenses at the enterprise network level could involve an experiment set-up such as that depicted in Figure 1.1 wherein: intra-network worm spread could be recreated using actual malware or simulated programs on a network testbed such as the DETER or Emulab testbed [6, 1] , background traffic would need to be generated to accurately estimate false positives [18] , and external attack traffic H representing scanning activity from outside directed at the network would need to be realistically modeled.
Fig. 1.1. Simulation set-up for a worm defense experiment
When conducting an enterprise simulation/emulation experiment, the modeling of ingress traffic into the enterprise network from the rest of the Internet is crucial in affecting the subsequent worm propagation, network traffic dynamics, and the effect of experimental defense/containment. The data source for the ingress traffic could be from a general mathematical model of worm propagation, or from the real world worm trace collected by various Internet monitors. Reversely, the experimental results of enterprise emulation will reveal important characteristics of egress traffic from the enterprise network under various network configurations and defense strategies, which could be used to verify the results of Internet worm modeling or direct the modeling efforts. From this perspective, the emulation and simulation of enterprise networks and the modeling of worm propagation at the Internet level are closely related and mutually benefiting.
KMSim simulation model extension and virtualization based testbed emulation proposed in this thesis are motivated by the two methodological needs of experimental enterprise network worm defense research: macro or Internet level worm propagation modeling and micro or enterprise-network level worm emulation technology. For the former, we will extend the KMSim model [16] to account for the self-destructing behavior of worms, a special characteristic manifested in the Witty and Blaster worms, and to model network dynamics under various defense strategies so the effectiveness and performance of them can be compared. For the latter, we will investigate various virtualization technologies and the implication of virtualization on the fidelity of testbed emulation experiments. Network emulation and validation experiments need a variety of supporting tools and usability and extent of integration of these tools are important in the conduct of worm experiments and post-hoc analysis. A user-friendly graphical interface software toolkit designed specifically for the purpose of testbed experimentation is highly desired and is on the research agenda as well.
The research to find weapons against worm attack has gained momentum in recent years and a number of detection and defense approaches have been proposed. As more accurate and advanced worm detection and defense techniques have been or are being developed, we believe that worms are also evolving and future worms and malware[1] could have more stealthy and advanced propagation strategies. One possible variant of such more stealthy worms, as we believe and call attention on in this thesis, may be purposefully targeting a local or enterprise network only and adjusting its scan strategy according to the local network information. We will use mathematical modeling and network testbed to explore and compare potential scanning strategies of such worms and demonstrate the danger of this new threat to the enterprise networks.
For the detection and containment of local scanning worms, we will propose a new dark port detection scheme to actively monitor all intra-enterprise network connections and raise security alerts when a new connection attempt targeting a non-existent service host/port tuple is detected. Dark port scan alerts detected by soft firewalls are reported to a central security console, which aggregates these alerts and runs a sequential hypothesis test algorithm to detect a potential local scanning worm.
Multiple containment actions are available when suspicious dark port scans are detected and each has its advantages or disadvantages. To compensate possible detection false positives and balance between worm defense and maintaining normal service, we further propose a Markov decision process model to quantitatively evaluate various defense strategies and make best defense decisions based on this cost-benefit analysis.
A comprehensive quantitative model for defense strategy evaluation entails accurate network and threat information such as current worm virulence level. In this regard, we develop a maximum likelihood estimation algorithm to estimate the size of susceptible host population in the network under three different worm propagation scenarios.
Based on the sequential hypothesis test worm propagation detection and the result of susceptible population size estimation, we propose two containment strategies to illustrate the advantage of proactive and collaborative containment. We will run experiments using simulated data and real world traces to evaluate their effectiveness in containing local scanning worms.
1.2 LITERATURE REVIEW
In this short review, we will discuss some related works around topics that will be covered in this thesis: worm modeling and simulation, worm detection and defense, and defense strategy evaluation. Worm related works are abundant in the literature and this short review will only give a glimpse of them. Additional works will be introduced when we proceed into individual
chapters.
1.2.1 WORM MODELING AND SIMULATION
A number of worm models have been proposed for the study of Internet worm propagation and effects of various defense strategies. Many of such models are based on the original Kermack-McKendrick epidemic model such as [8, 42, 43] . The limitation of KermackMcKendrick model when dealing with Internet worms having special characteristics is made evident in [42] . In [16] , the first KMSim model was introduced to account for the effect of local network access-link saturation caused by fast scanning worms such as SQL Slammer.
As for the simulation of worms, in [34] , proprietary worm simulations are done but only coarse-grained global worm propagation activities are simulated with very high level abstraction of the worm propagation environment. In [22] , SSFNet is used to simulate realistic network worm traffic for worm warning system design and testing, but only at an abstract network level. Recently, network testbeds are becoming available and researchers began to use them to simulate and emulate worms and study their behaviors in the Internet [37] or in enterprise networks [20] .
The KMSim model in general, and the extension for enterprise networks that we will introduce in this thesis, lies between a pure differential-equation based mathematical model and a full topology-based simulation framework. The input and parameters for the KMSim program can be detailed enough for experimenters to run a highly customized network simulation, or can be simply one line of numbers as the feed for a theory-validation computation. The flexibility of KMSim along the spectrum of modeling and simulation makes it ideal for network and security researchers who want to move smoothly and swiftly between the initial proof of concept and a full-blown simulation.
Staniford’s work [31] covered a variety of topics on enterprise worm defense including worm modeling, worm simulation, and containment effectiveness. While its focus was largely on the final worm infection density in the network, we are interested more in the time-efficiency of enterprise worms and the danger it brings to the enterprise defense in our simulation/emulation. The unique structure of KMSim model also enables the use of model to ‘simulate’ the effect of inhomogeneous vulnerability density among sub-networks or cells[2] .
1.2.2 WORM DETECTION AND DEFENSE
Worm defense or containment could be done without an a priori detection, as demonstrated in such defense schemes as worm throttling [41] , Earlybird [30] , etc. However, such defense schemes can only work against fast worms or they are only effective in filtering suspicious packets at the network gateway. For the defense of slower worms or stopping the propagation of worms before a reliable signature can be extracted, a sensitive worm detection scheme is required. Honeypot or honeynet [9] can detect suspicious worm activities by monitoring scans that are targeting at unused or dark network addresses. For detection and defense in local or enterprise networks, a double honeypot can be used to collect worm samples for signature generation [33] . In [11] , an enhanced honeypot based detection, HoneyStat, combines network security alerts with computer OS alerts to improve the detection accuracy and reduce false positives.
Another type of worm detection scheme, failed scan detection, is also taking advantage of worms’ blind scanning behavior. While the early version of failed scan detector used to rely on the returned ICMP packets from the destination host or network [22] , most recent proposals detect failed scans by keeping logs of all connection requests on the sender site and singling out these requests that have not received response after a pre-defined period of time [15, 38] . Compared with these failed scan detection proposals, our dark port detection can detect a failed local scan much faster since the security agent at the destination site can report a failed scan right after its arrival and there is no need to wait for a connection to go time-out at the sending site.
The idea of scan detection at the port level was touched in [10] and explored in [40] in the form of exposure map. A host exposure map includes all the open ports on a host collected in a training period and a network exposure map is the aggregate of all host exposure maps. In comparison, our dark port approach is more flexible in handling dynamic services and concrete in design. More importantly, our sequential hypothesis test can detect a worm’s propagation, beyond detection of individual scanners.
Researchers also looked at the DNS records at the enterprise network to detect suspicious scanning worms [39] . Since the majority of normal network connections are preceded by DNS inquiry for IP address look-up, connections without corresponding DNS visits are then suspicious and may be signs of worm activities in the network.
Worm containment agents can be deployed at end hosts [41] or on the sub-network gateways [38, 44] . While the sub-network firewalls in [38, 44] act on their own, our proposed enterprise soft firewalls will operate cooperatively with a central security console as the central commander. The advantage of cooperative or group defense over independent firewalls has been validated by simulation experiments such as [7, 24] .
1.2.3 DEFENSE STRATEGY EVALUATION
Quantitative analysis on the cost and reward of network security strategies especially on the worm defense field is still in its infancy. We are not aware of similar work on using Markov decision process model for quantitative cost-benefit analysis of worm defense strateg. However, Markov process and Markov decision process have been used in modeling worm propagation process in [23] and in computing the probabilities of intruder success in a sequential network attack scenario in [13] . Markov decision process has also been used in vulnerability modeling of physical infrastructure facilities and other security domains.
For the estimation of worm virulence, Zou et al. [43] proposed a method using a Kalman filter and worm propagation information collected by Internet telescopes. Though the Kalman filter method was effective in estimating susceptible population sizes for Internet worms such as the Code Red and Slammer worms, it may not be as effective in the estimation of susceptible population size in an enterprise network.
1.3 RESEARCH CONTRIBUTIONS AND OUTLINE OF THE THESIS
The contributions of this study can be summarized around five subjects: worm modeling and simulation, virtualization for testbed emulation, dark port detection and worm propagation detection, optimal defense strategy, and estimation of worm virulence and collaborative containment.
C1: KMSim model and its extensions will improve the accuracy and fidelity of worm modeling.
C2: Virtualization will reduce the resource requirement of worm experimentation and maintain a high level of fidelity.
C3: Dark port detection scheme has two advantages in the detection of local scanning worms: early detection and low false positive rate.
C4: Model based strategy evaluation and decision-making will increase the quality of defense decision in terms of network security and service availability.
C5: Aggressive and collaborated worm containment strategy can realize a higher level of security in an enterprise network environment than existing worm containment approaches.
This thesis is organized as follows. Chapter 2 presents our ESVT toolkit (Experiment Specification and Data Visualization) and LAN virtualization emulation method. Chapter 3 is on the topic of KMSim model extension and application of KMSim model on the simulation of the Witty, Slammer, and Blaster worms. The dark port detection scheme, and related sequential hypothesis test for the detection of worm propagation, is presented in Chapter 4. We explore the idea of quantitative strategy evaluation and optimal decision making in Chapter 5. Two models based on Markov decision process model are developed. In Chapter 6, we propose and evaluate an algorithm for the estimation of susceptible host population size. Advantages and potentials of collaborative containment are discussed and two collaborative containment schemes are presented. Chapter 7 concludes the thesis.
The major research methods used in this study are quantitative, computational, and analytical. Computer simulation and network emulation will be used extensively in the exploration and validation of proposed worm models, detection, and containment schemes.
[1] We believe the difference between worms and other malware will be blurred in the future.
[2] Sub-network and cell are inter-exchangeable in this thesis.
ENTERPRISE WORM: SIMULATION, DETECTION, AND OPTIMAL CONTAINMENT