CAIDA NGI Project - Second Technical Report

Predictability and Security of High Performance Networks: Expanding Control through Monitoring, Visualization, and Analysis

Objective

UCSD/CAIDA is focusing on advancing the capacity to monitor, depict, and predict traffic behavior on current and advanced networks through developing and deploying tools and software to better engineer and operate networks and to identify traffic anomalies in real time.

Work on this effort is performed in the following four areas:

Development and deployment of Coral Monitors:
Tools for monitoring traffic flows at OC48 and above speeds currently do not exist. High-speed networks, such as those comprising the Next Generation Internet (NGI) initiative, and ATM-based networks, are left without any means of monitoring and analyzing actual traffic data. The lack of tools and information about traffic behavior limits engineers' abilities to optimize network configurations or assess the performance of emerging hardware/software, network protocols, and practice. In this initiative, CAIDA, MCI Worldcom, and the University of Waikato, New Zealand are collaborating with computer chip/hardware vendors (Vitesse, Xilynx, Agere) and major providers, and are developing hardware and analysis tools (Coral Monitors) capable of providing real-time monitoring of optical, high performance networks at OC48 speeds.

Macroscopic analyses relating to the global Internet infrastructure (skitter):
The majority of today's network measurements are at a microscopic level e.g., lab simulations or measurements of individual networks. CAIDA is currently measuring and analyzing traffic behavior of a large cross-section of the Internet infrastructure by using skitter to send ICMP echo requests to approximately 29,000 destinations. To our knowledge, this initiative (we call it 'Internet Tomography') is the only measurement effort of this scale. It offers unprecedented information and insight into inter-provider connectivity, routing behavior, and Internet performance. Under the tomography task, CAIDA also plans to develop a simple traffic model that can be used for conducting "what-if" scenarios of traffic data based on performance, forward path and BGP data. Development of associated analysis and visualization tools are also planned, as is a web site for disseminating key skitter analyses and data. CAIDA's measurement efforts are intended to help users, providers and researchers understand the complexities in the current Internet and to identify/refine areas for exploration in DARPA's Next Generation Internet initiatives.

Develop security-monitoring capabilities without adversely affecting performance on high performance networks:
Firewalls are now being introduced at up to OC12 speeds, however, many networks, particularly research networks, need security tools that do not compromise performance, especially in the face of new protocols or applications. Under this effort, CAIDA and SDSC's Pacific Institute for Computer Security are developing lightweight monitors that, in both continuous and real-time, will identify anomalous traffic patterns and trigger pre-defined enforcement countermeasures.

Collect, store and analyze massive volumes of Internet-wide traffic data:
Outside of CAIDA and the National Laboratory for Applied Network Research (NLANR) efforts, there are few sources of commercial Internet traffic data available to the network research community. The active measurement data gathered through skitter is the only known source of infrastructure-wide information available to researchers. Through this effort, CAIDA provides an invaluable source of information to the community. CAIDA also plans to correlate active and passive measurements, as well as routing table information from core Internet routers to characterize and model infrastructure dynamics, including tracking global deployment growth of new hardware and software releases. We have already found examples where certain hardware/software versions manifest remarkably different skitter-measured behavior, and the effect of such releases on the larger picture can support network engineers in evolving next generation networking technologies.

UCSD/CAIDA's project is divided into four tasks: Coral monitors, tomography (skitter), security monitoring, and data storage/analysis. Our approach to each effort is described in the program plan, available at https://www.caida.org/funding/progplan/NGIprogplan98.xml, and is briefly summarized below.

Coral OC48mon:
The community's ability to monitor advanced optical and high performance networks has not kept up with the raw development and deployment of fiber and switching capability at those bandwidths. WDM networks, OC48 and OC192 networks are gaining momentum, but the current state of measurement technologies leaves engineers managing these networks handicapped in trying to monitor or even identify the presence of traffic on some links. This initiative draws upon state of the art developments in computer processing and data storage to develop tools capable of constant monitoring of traffic (not sampled) at OC48 speeds for research and engineering purposes. Monitoring a fiber-optic link by collecting information on the flows, (i.e., sequencing of packets between a source and destination application), is a valuable tool for service providers and engineers for immediate troubleshooting as well as tracking traffic trends for future capacity planning. Coral monitors can be used to collect information about the amount of traffic (in bytes, packets or flows) traversing a link, as well as important traffic characteristics such as which applications and transport protocols generate the most traffic, which packet sizes are most common, how many packets of various sizes tend to arrive in clusters, and data aggregated by/across/between traffic between individual networks and autonomous systems. This initiative requires close participation of hardware vendors to develop an innovative architecture comprised of Vitesse's ATM and POS OC48 chipset and full monitor 6664 PCI chipsets. The resulting tool is capable of handling the requisite 528 MB/S bursting, permitting capture of every packet in both directions on a full 2.4 Gb OC48c link. As an additional part of the Coral project, CAIDA has developed an array of software tools to enable post-analysis of Coral trace files, traffic characterization, and continuous monitoring. Such tools offer flows analysis, Autonomous System matrices, BGP routing lookup and various protocol-centric analysis tools.

Tomography
In order to gather more macroscopic information on Internet infrastructure, behaviors, platforms, and evolution, CAIDA is using active measurements and other tools to gather, analyze and visualize traffic and topology data. By the end of year 1 of this project, 15 measurement hosts were deployed monitoring up to 29,000 end-destinations distributed throughout the IPv4 address space. Skitter, a light-weight active monitoring tool developed by CAIDA's Daniel McRobb, is run continuously to gather hop-by-hop connectivity, routing, and performance information, which is stored at SDSC for later analysis, correlation (with other data) and visualization. CAIDA's skitter tool measures the forward IP path to a destination in a manner similar to traceroute: it increments the TTL when sending packets to a destination and records the router that replies at each TTL, until a TTL sufficient to reach the destination is used. skitter uses ICMP echo requests as probes. When skitter, finally receives the ICMP echo reply from the intended destination, it terminates the path probing for that destination and records the round-trip time from the source to the destination and back. Target sites are intentionally selected to pervasively stratify the IPv4 address space, in pursuit of a comprehensive picture of the deployed commodity Internet. skitter also offers promise in potential correlation to BGP data to allow engineers to discern who is announcing what to whom over specific paths. Although such information will not answer why given events occur or if such traffic behavior is optimal, it will provide real-world inputs to traffic models and simulations designed to answer such questions. CAIDA is also developing a simple traffic model that can be used to conduct scenarios of various types of traffic behavior. This model is intended as a proof of concept and will be used primarily in working with commercial vendors who invest millions of dollars annually in developing commercial vendor simulation products. CAIDA will also use skitter to assist gathering data to help determine architecturally strategic locations of DNS root servers within the Internet. For approximately two weeks in July 1999, a skitter host co-located with a root name server will measure connectivity and round trip latency to a comprehensive set of root sever client IP addresses. This will start with the "F" root server, maintained at the Digital Palo Alto Internet Exchange, and will expand to other current or proposed root server locations if circumstances warrant.

Security
The security-related component of this project includes engineering and deployment of low-cost passive security monitors linked to active response modules. The Applied Telecom OC12 card used in the Coral OC12mon is being used as the foundation for development of algorithms supporting integration of low-level, high-performance, real-time packet filtering with security policy enforcement modules. CAIDA is working to enhance the OC12mon passive traffic monitor to facilitate ubiquitous network monitoring at aggregation points, DMZ's and ISP's, by developing dynamic filtering and data collection, security policy compliance monitoring, and security policy enforcement components. Filtering is required to reduce data, isolate suspicious traffic, minimize contention for the peripheral bus, and permit persistent monitoring of heavily loaded links. This is accomplished with two-level filtering: in hardware on the network adapter FPGA and in the host software. Modifications of the FPGA firmware to enable classes of filters are under development. The initial prototype security module will rely on packet re-assembly in the software, but later prototypes will likely move part of this functionality to the kernel. The few extant commercial solutions filter on the first ATM cell of an IP packet, an approach vulnerable to subversion by padding headers. CAIDA's approach procures arbitrary length payload extraction, which requires optimized zero-copy SAR on host and subsequent BPF linkage. It is also essential to support evidence collection and session replay, for which header filtering alone is insufficient. The delivered tool also provides a more general tool for network metrics by providing finer-grained filtering, e.g. by AS or protocol or source address. The tool should also provide networks with a relatively inexpensive means of enhancing security without compromising the actual performance of the network - a critical consideration for all research networks.

Database Storage/Analysis
A raid array (350 Gb) is used to store active (skitter) measurement and routing data. Arts++ , a binary file format library, was developed by CAIDA for storage and analyses of those data, see https://www.caida.org/tools/utilities/arts for code. Researchers using CAIDA's skitter data, such as Notre-Dame, University of Illinois and UCLA, are analyzing these measurements for implications relating to current and next generation networking. CAIDA researchers will also develop techniques for correlating and analyzing various forms of measurement and routing data.

Recent Accomplishments:

Coral OC48mon
The University of Waikato in New Zealand signed a subcontract to develop prototype OC48 monitor cards building upon their successful DAG-3 designs (OC3 and OC12 speeds). The DAG-4 will have both ATM and POS OC48 monitoring capabilities. A meeting of the seven member Waikato DAG development team and CAIDA OC48mon developers occurred at SDSC June 8 and 9, 1999 to discuss file format and design issues. CAIDA selected the Vitesse OC48 chipset for capturing traffic at OC48 speeds. Initial chips have arrived and preliminary schematics based on the Vitesse components are complete. The ATM prototype card is scheduled for testing in late August 1999, the POS prototype card incorporating the 6664 PCI chipset is estimated for late 1999, and a version incorporating the Agere fast-path lookup technology is under consideration for early 2000.

Tomography (skitter)
As of June, 1999, CAIDA deployed a total of 15 skitter machines. The first skitter monitor intended for the analysis of logical placement of DNS root servers in the network infrastructure was sent to the Palo Alto Internet Exchange (PAIX) in June of 1999. This will provide a methodology for evaluating the optimality of DNS server locations based on the metrics of "macroscopic distance" for a comprehensive cross-section of Internet infrastructure. CAIDA recently performed spectral analyses of skitter data using the Lomb periodogram and results were discussed with ARL/DREN. Preliminary analyses are located at https://www.caida.org/tools/measurement/skitter/skping/. CAIDA significantly enhanced a prototype 3D hyperbolic layout/navigation tool for use in visualizing skitter and routing data as part of the Tomography Mapping/Modeling task. The tool, "hypview", was originally developed by Tamara Munzner (Stanford University) for use in visualizing hierarchies of data, e.g., web relationships and is available at https://www.caida.org/tools/measurement/skitter/viz/. CAIDA also used Otter, a general-purpose visualization tool, https://www.caida.org/tools/visualization/otter/ and an algorithm developed by Lucent Technologies, http://www.cs.bell-labs.com/~ches/map/index.html to visualize skitter data.

CAIDA is achieving high levels of collaboration with commercial and research organizations who are participating in the skitter effort. Three major backbones are sponsoring a total of seven machines: AboveNet (4), Qwest (1), and MCI Worldcom (1). Several research networks are also hosting or sponsoring monitors, including Canarie (Canada); the Asia Pacific Advanced Networking group's KAIST (Korea), KDD (Japan); and SingaREN (Singapore). In total, 6 countries have skitter active sources running code. Numerous technical/scientific journals have published stories about the skitter initiative, including Science, Science News, and Nature, as well as magazines in the UK, Germany, Brazil, France, and Venezuela.

Security
The initial development of the prototype dynamic security monitoring code for Coral Monitors is complete. Testing is underway on packet filtering for full packets and triggered packet collection. Memory mapping is used in lieu of kernel filtering and re-assembly is now done in user space instead of being done in the kernel. Testing will provide indication of whether this solution will viable at full line rates or if additional functionality needs to be handled in the kernel.

Database Storage/Analysis
In addition to specific analyses described in Task 2, skitter data is being made available to Notre-Dame, the University of Illinois, and UCLA for their analyses. Acceptable use agreements are being used to ensure privacy of data and encourage publication of the results. Collection of data permitting analysis of region-specific traffic behavior is also underway for several countries in the Asia-Pacific region. Skitter monitors were deployed in New Zealand, Japan, Korea, Singapore and Canada in addition to the U.S. monitors. A European monitor is also based in London. Approximately 80 Gigabytes of skitter data have been transferred back to the CAIDA San Diego disk array as of July, 1999. An alternative to a secure server for CAIDA collaborators to access their skitter data is being explored. Accessing the secure skitter data requires a browser with a valid X.509 client certificate signed by the Certificate Authority (CA). A prototype Certificate Authority was implemented in June 1999 for use by remote CAIDA staff. The CA will insure security and data privacy by requiring the user to obtain a server certificate, which verifies the identity of the server to the client, a client certificate, which verifies the identity of the client to the server, and a Certificate Authority certificate, which is used to test the validity of server and client certificates.

Current Plan:

Coral OC48mon
The first ATM prototype capture card is scheduled for completion in the next quarter. The capture card with full monitor 6664 PCI chipsets and the Vitesse POS chip should be available by the end of 1999. Addition of the Agere Fast Path Processor Chipset in the OC48 card is anticipated in early 2000. MCI testing and evaluation of OC48mons scheduled to begin in late 1999 and last through the project. Preliminary specifications and costs for development of the OC192mon are planned for early 2000.

Tomography (skitter)
Skitter monitors will be deployed at multiple DNS root server locations during the next two quarters in an effort to evaluate the optimal placement of DNS servers in the Internet infrastructure. Briefings to the scientific community, Federal groups, and commercial Internet providers and suppliers on the significance of tomographic analyses of the Internet will increase during the second year. Initial development of a prototype traffic model is anticipated in early 2000 in collaboration with several third party organizations. Additional visualization and analysis tools will be developed during the coming year.

Security
Testing of the first version of the dynamic security-monitoring module for the Coral OC3 and OC12 monitors will start during Quarter 1. Integration of the security compliance monitor software as well as the enforcement implementation will be completed by late1999. FPGA firmware packet header filter design is complete, with assembly of actual firmware scheduled to begin next quarter, followed by testing and evaluation. Testing and evaluation of OC12 card security features in OC3 applications are slotted for late 1999. During 2000, the security module software will be enhanced to work with OC3 and OC12 DAG-3 monitors and feasibility of applying the techniques to the OC48 DAG-4 monitored will be explored. Recommendations for additional security applications/implementations of the Coral monitors for DOD applications will be submitted to DARPA in spring 1999.

Database Storage/Analysis
Preliminary findings on the Asia-Pacific traffic study will be released during the next quarter. Also during the next quarter, CAIDA will either enhance its prototype Certificate Authority (CA) capabilities to cover remote collaborators or migrate to the SDSC CA once it is available. Development of web-based query forms to permit users to access raw data is also planned for Quarter 1. Continued enhancement of the database is scheduled throughout the second year of the project. Correlation of skitter data with passive data and routing tables will begin in the next quarter. Enhancement of analysis code and reporting formats is scheduled for Quarters 3 and 4, and posting of standard analyzed, correlated data to a public CAIDA Web site in Quarter 4.

Technology Transition:

Coral OC48mon
This monitor will permit monitoring and evaluation of traffic data on research networks such as NTON and SuperNet, both funded by DARPA, as well as the vBNS and ABILENE research and education networks. vBNS personnel are collaborating with CAIDA personnel to ensure that specifications for these monitors are responsive to their engineering and operational networking requirements for next generation networks. Several Internet Service Providers have expressed interest in early deployment of Coral OC48 monitors. A large hardware vendor is also working directly with CAIDA's subcontractor, the University of Waikato, to license the OC48 monitoring technology as it becomes available.

Tomography (skitter)
Active measurement data gathered through this initiative will provide the community with a unique and valuable source of infrastructure-wide data. Researchers from several Universities and collaboration partners (APAN/KDD, APAN/KAIST, AboveNet, Qwest, MCI Worldcom, SingAREN, and the University of Waikato) have requested data sets for analysis. Techniques are being developed to analyze and visualize macroscopic Internet-wide data that will provide new capabilities relating to the operation and management of networks as well as the design of hardware, software and architectures associated with emerging networks.

Security
This software allows fine-grained access to broadband network traffic. As such, it permits a wide range of applications including network metrics, firewall control, and security-related evidence collection (e.g. session byte streams). Target communities include DOD installations, major ISP security officers, law enforcement, and network engineers seeking greater control over traffic measurements. Deployment and testing of OC12 security modules at ARL and DREN are also anticipated. Prototype code for the security module will be available on the CAIDA web site by the end of the next quarter.

Database Storage/Analysis
Datasets and analyses results are being made publicly available on the CAIDA web site. Recommendations for additional research by third parties will be identified and collaborations will be encouraged. Techniques and tools for analysis and for correlating various traffic data will be made available during the second year of the project.