Center for Applied Internet Data Analysis

NGI Cooperative Agreement No. N66001-98-2-8922

Project Title: Predictability and Security of High Performance Networks

Organization: University of California - San Diego

AO Number: G835

ContractNumber: N66001-98-2-8922

Start Date: September 15, 1997

End Date: October 15, 2001

Principal Investigator:

Dr. Kimberly Claffy
9500 Gilman Dr.
CAIDA at San Diego Computer Center UCSD MS#0505
La Jolla, CA 92093-0505
Phone: (858) 534-8333
Fax: (858) 534-5113
Email: kc@caida.org

Level of Participation - Billed: $ 2,721,950

Level of Participation - Unbilled: $ 249,854

Project URL: https://www.caida.org/funding/ngi1998/

Overall Objective: UCSD/CAIDA is focusing on advancing the capacity to monitor, depict, and predict traffic behavior on current and advanced networks through developing and deploying tools to better engineer and operate networks, to identify traffic anomalies in real time, and to better visualize data. UCSD/CAIDA's NGI project is divided into four tasks and one option.

Summary of Previously Completed NGI Tasks

Task 2, Tomography (skitter): skitter monitoring at more than 20 sites is being used to automate the discovery and visualization of macroscopic Internet topology and peering relationships. Discussion of progress with analysis of gathered skitter data is included with Option 1.

Task 3, Security: CAIDA modified Bro security software to use libcoral, giving Bro the ability to read data from Coral monitors. CAIDA also modified CoralReef's crl_filter feature (which provides command line BPF filtering rules) so that it is now possible to use this feature in combination with other existing tcpdump tools. Finally, CAIDA released crl_portmap, a tool that detects suspicious portmapper (RPC) activity, then logs all traffic to and from the probing host in tcpdump format.

Task 4, Data Storage and Analysis: CAIDA makes available daily summaries of skitter data collected between 1 Sep 1999 and 30 June 2001. Daily summaries ( http://sk-summary.caida.org/cgi-bin/main.p ) provide an interface capable of generating a various graphic analyses of collected data. A summary of significant analysis results to date is included with Option 1.

Task 1, Coral OC48mon/Gigabit Ethernet Monitor

Objective:

Collaborate with major Internet service providers and the University of Waikato to develop hardware and analysis tools (Coral Monitors) capable of providing real-time monitoring of optical, high performance networks at OC48 and Gigabit Ethernet speeds. This initiative draws upon state of the art developments in computer processing and data storage to develop tools capable of line-speed monitoring of traffic up to OC48 speeds for research and engineering purposes. Coral Monitors are used to characterize traffic traversing a high-speed link (in bytes, packets or flows), as well as that link's application and protocol workloads.

Approach:

CAIDA and the University of Waikato are designing a capture card with an innovative architecture comprised of Vitesse's ATM and POS OC48 chipsets with Xilinx FPGAs. The resulting DAG 4.1 capture card permits capture of every packet in both directions on a full 2.4 GB OC48 speed link.

An additional part of the Coral project is the development of an array of software tools to enable post-analysis of Coral trace files, traffic characterization, and continuous monitoring. This library includes utilities for flow analysis, Autonomous System matrices, composition of traffic by application or other category, and various protocol-centric analyses.

Recent Accomplishments:

OC48 monitors: The University of Waikato team has succeeded in using their DAG 4.2 card to capture a 60 second trace of OC48 traffic and then perform extensive postprocessing and trace analysis on the collected data. The Waikato team is now addressing several bugs discovered during this test. Waikato's OC48 DAG card has not yet been integrated or tested with CoralReef. Previously, CAIDA tested basic capture operations of the OC48 DAG 4.1 card at SDSC under varying traffic load levels. Although initial testing included synthetic traffic, it was insufficient to adequately test firmware at OC48 line speeds. Additional firmware work was done to place timestamps on trace packet headers in a format that was then captured to disk. Finally, CAIDA has acquired beta OC48 cards and drivers from Lucent in a parallel effort to test CoralReef using commercial OC48 cards. An OC48 capture box from Narus has also been acquired for potential testing.

Gigabit Ethernet monitors: CAIDA has acquired and is testing some beta Gigabit Ethernet cards and drivers from Lucent and Narus. CoralReef access to these cards works well, but the cards and drivers themselves are still buggy and need more work by their manufacturers.

CoralReef: This comprehensive software suite (https://www.caida.org/tools/measurement/coralreef/ consisting of a set of drivers, libraries, utilities, and analysis software, enables passive monitoring of ATM, POS and other high-speed network interfaces. CoralReef's toolbox paradigm offers a consistent API along with extensive tools and utilities to provide multiple ways to address a variety of passive monitoring requirements. Recently, the ability to use live pcap interfaces was added, allowing CoralReef to monitor a wide variety of common commercially available interfaces that support Unix-like operating systems. A paper highlighting CoralReef philosophy, architecture and capabilities was presented at PAM 2001 and can be found at https://www.caida.org/publications/papers/2001/CoralArch/.

Current Plan:

Development and testing of the OC48 DAG cards will continue at the University of Waikato. This NGI task will be completed before the contract period end date. CAIDA also plans to install a Gigabit Ethernet or OC48 CoralReef monitor (using Lucent Gig-Ether cards) on the SD-NAP located at SDSC. CAIDA will help SDSC develop a prototype SD-NAP customer workload profiling service based on the CoralReef report generator.

Technology Transition:

The CoralReef software suite is available for download at https://www.caida.org/tools/measurement/coralreef/status.xml. Both a public package and a CAIDA member package, containing additional features and better performance relative to the public package, are available. In addition, CoralReef licensee CAIMIS (http://www.caimis.com) plans to provide commercial production environment support and documentation for CoralReef software.

Option 1, DNS Root Server Initiative and Visualization of Massive Data Sets

Objective:

Provide ICANN with recommendations regarding optimal locations for current and future root name servers. Also expand visualization initiatives to facilitate aggregation, analysis and layout of massive data sets (tens of gigabytes in size). The Domain Name Server (DNS) technical advisory committee to ICANN includes existing root server operators, institutional representatives (from IESG, IANA, DOC, etc.) and technical measurement experts (CAIDA). One of the committee's responsibilities is to provide ICANN with recommendations regarding optimal locations for root name servers. There are currently 13 root name servers (http://www.wia.org/pub/rootserv.html). RSSAC has asked CAIDA for assistance gathering data to help determine architecturally strategic locations for current and planned root name servers within the Internet.

Approach:

CAIDA has co-located skitter hosts at the A, E, F, K-peer, K-root, L, and M root name servers. Additional skitter monitors for the D, H, and G root name servers will be shipped during July 2001. Finally, the skitter monitor for the I-root name server is in place, but awaits administrative activation. CAIDA has developed a methodology for identifying and depicting sets of destinations with high latency from these instrumented locations. CAIDA uses the skitter tool to measure connectivity and performance of the network between root servers and a subset of their clients.

Recent Accomplishments:

Marina Fomenkov led CAIDA RSSAC investigations which are reported in "Macroscopic Internet Topology and Performance Measurements from the DNS root name servers", a paper to be presented at the USENIX Lisa 2001 conference. The methodology employed makes use of a common skitter destination list for all skitter monitors co-located with each root name server, containing more than 58,000 IP destinations covering 8406 origin Autonomous Systems (ASes) and 184 countries. In addition to providing representative address prefix coverage, use of this common "DNS clients" destination list serves as a yardstick against which performance comparisons can be made. If a set of destinations shows high latency from all root servers and clusters either geographically or topologically without having systematic regional bandwidth problems or other political constraints, this might suggest a region meriting a new root name server. However, collected data cannot be used to decide how well a particular root server responds to its own specific clients, due to an internal BIND load-balancing feature. Even so, knowing which destinations in this list are frequent clients of which particular root server, local subsets of the DNS clients lists can still be used to study individual server-specific issues.

The first set of traces was collected between December 1 through December 30, 2000. The second set was gathered between March 6, and April 4, 2001. In December, M-root monitoring had not yet started. In March, the L-root monitor experienced local connectivity problems and was temporarily disconnected. In March, each monitor probed destinations in the DNS Clients list between 7 and 13 times per day, a rate which is 15-60% higher than probes made during December when an older, slower version of skitter was in use.

Two metrics of connectivity (hop count and round trip time) were calculated from the root name server to the hosts in the target set. The IP hop count distributions for each root server monitor can indicate whether they are near the edge of their local networks and/or near a major exchange point (See peak positions for A, E, F, and L root server monitors in Figure 1) or are further away from their destinations (See peak positions for K and M root servers in Figure 1).

Figure 1. IP path length distributions for DNS root server monitors.

Clusters of hosts having particularly large latencies from all root name servers indicate a potential deficiency in the current Internet infrastructure. A destination is defined as having high latency during a given day if, on that day, it had large RTTs in at least half the probe cycles on all root server monitors. Results are then aggregated over a month to filter out transient problems. In Figure 2, the left side maximum is due to random variations in connectivity while the right-side maximum reflects destinations that consistently have high latency on every (or almost every) day during the 30-day collection period.

Figure 2. The persistence of high latency destinations.

Figure 3 depicts high latency destinations by continents. It shows that Africa, Asia, and South America IP addresses account for over 60% of high latency destinations, but less than 14% of the total DNS client list.

December 1 - December 30, 2000	March 6 - April 4, 2001

Figure 3. High-latency destinations compared to entire target list by continent

CAIDA's skitter measurements can be used with local client lists as a baseline against which to analyze topology and performance characteristics of the network between a root name server and its typical clients. Other placement issues, such as distance to the edge of the local network, peering relationships, and choice of upstream transit providers, can be discerned from the graphs provided by the daily summaries generated automatically from each skitter monitor's data.

Other skitter analysis projects: Several researchers have requested and been granted access to skitter data. Descriptions of their projects can be found at https://www.caida.org/data/skitter/skitter_data_use.xml

Passive and Active Measurement (PAM2001) conference papers: Six CAIDA papers were presented in Amsterdam in April.

1. Brownlee, N., and M. Murray, "Streams, Flows and Torrents". This paper extends the RTFM (RFCs 2720-2724) definition of network traffic as bi-directional flows by adding the concepts of streams and torrents. Example analysis demonstrates useful traffic analysis based on collecting flow data for stream-based flow metrics.
2. Huffaker, B., M. Fomenkov, D. Moore, and k claffy, "Macroscopic Analysis of the Infrastructure: Measurement and Visualization of Internet Connectivity and Performance". The robustness and reliability of the Internet is highly dependent on efficient, stable connectivity and routing among networks comprising the global infrastructure. CAIDA has developed and deployed the skitter tool to dynamically discover and depict global Internet topology and measure performance across specific paths and is developing a systematic approach to visualizing the multi-dimensional parameter space covered by skitter measurements aggregated on a daily basis.
3. Murray, M. and k claffy, "Measuring the Immeasurable: Global Internet Measurement Infrastructure". The cooperative anarchy of the global Internet defies easy characterization or measurement of its behavior, yet it is neither practical nor particularly effective to monitor and measure every single link. Existing public and mission-specific Internet measurement infrastructures are surveyed, comparing them using a variety of criteria, in order to facilitate collaboration and community awareness.
4. Keys, K., D. Moore, R. Koga, E. Lagache, M. Tesch and k claffy, "The Architecture of CoralReef: An Internet Traffic Monitoring Software Suite" CoralReef design philosophy, overall architecture, and capabilities are presented. CoralReef is a package of libraries, device drivers, classes, and applications written in, and for use with, several programming languages. By highlighting design and architectural decisions at all levels in CoralReef, the authors show how CoralReef is a powerful, extensible, efficient, and convenient package for passive data collection and analysis.
5. Shannon, C., D. Moore and k claffy, "Characteristics of Fragmented IP Traffic on Internet Links" While many assertions about fragmented IP traffic are based on folklore, the authors analyzed characteristics of fragmented traffic, and examined the causes of IP packet fragmentation. The effects of NFS, streaming media, networked video games, and tunneled traffic are quantified, as well as the prevalence of machines whose improper configurations were causing excessive amounts of fragmented traffic.
6. Brownlee, N., k claffy, M. Murray and E. Nemeth, "Methodology for Passive Analysis of a University Internet Link". This paper presents two case studies that use publicly available link monitoring tools (CoralReef and NeTraMet) for collecting and analyzing Internet flow data.

Denial-of-Service Attack Identification: A new technique called "backscatter analysis," was used to estimate worldwide denial-of-service activity. Three weeklong datasets have been analyzed, assessing the number, duration and focus of attacks, and characterizing their behavior. David Moore of CAIDA and Geoffrey M. Voelker and Stefan Savage of the UCSD Department of Computer Science and Engineering discuss results of this analysis technique in a paper entitled "Inferring Internet Denial-of-Service Activity" to appear at the Usenix Security Symposium during August 13-17, 2001 in Washington, D.C. (See: https://www.caida.org/publications/papers/backscatter/)

Routing and Connectivity Analysis:Two studies carried out in conjunction reveal that the current common research practice of using routing tables for connectivity analysis is somewhat suspect. CAIDA finds that routing tables capture only a very small fraction of actual connectivity. Instead, CAIDA offers the beginnings of a new calculus for routing and connectivity analysis. "Internet Topology: Connectivity of IP Graphs," by Andre Broido and k claffy, accepted for presentation at the ACM SIGCOMM Internet Measurement Workshop, introduces a framework for analyzing local properties of Internet connectivity by comparing BGP and probed topology data. "Complexity of Global Routing Policies," accepted at the same conference, analyzes BGP connectivity, and evaluates a number of new complexity measures for a union of core backbone BGP tables. Sensitive to engineering resource limitations of router memory and CPU cycles, the authors focus on techniques to estimate redundancy of the merged tables, in particular how many entries are essential for complete and correct routing. The notion of policy atoms is also introduced as part of this new calculus for routing table analysis.

Large graph handling API: CAIDA is also working on a generalized framework for dealing with large graphs. Design and implementation of library libsea (tentative name), which provides functionality for loading, saving, examining, and, to a certain extent, processing large graphs, has begun. Libsea's main purpose is to make graph data easily accessible to programs, and it should be able to handle graphs with around a million nodes, a few million links, and hundreds of paths.

Current Plan:

CAIDA will continue to collect skitter connectivity and performance data, making both data and tools available to CAIDA members and researchers.

Technology Transition:

Dr. Claffy is talking to ICANN and Verisign about leveraging the DARPA investment into the current RSSAC study to continue use of the developed methodology to evalute these now critically strategic components of the name server infrastructure.

The methodology for the DNS root server location evaluation, and associated mechanisms for determining 'central' positions within the Internet will be relevant far beyond the DNS system, applicable to location research for any type data server of strategic infrastructural relevance.