CAIDA NGI Project - Third Technical Report

Predictability and Security of High Performance Networks: Expanding Control through Monitoring, Visualization, and Analysis

Objective

UCSD/CAIDA is focusing on advancing the capacity to monitor, depict, and predict traffic behavior on current and advanced networks through developing and deploying tools to better engineer and operate networks, to identify traffic an omalies in real time, and visualize data. Work on this effort is performed acros s five tasks in the following areas:

Task 1, Coral Monitors: CAIDA and the University of Waikato, New Z ealand are collaborating with major Internet Service Providers to develop hardwa re and analysis tools (Coral Monitors) capable of providing real-time monitoring of optical, high performance networks at OC48 speeds.
Task 2, Tomography (skitter): This effort focuses on development of t ools to automate the discovery and visualization of macroscopic Internet topolog y and peering relationships.
Task 3, Security: Under this effort, CAIDA is developing lightweight monitors that identify anomalous traffic patterns continuously in real-time.
Task 4, Storage and Analysis: This effort focuses on storing skitter data on a DEC raid array for use in later analysis. CAIDA is using this data to conduct in-depth analysis on specific topology characteristics such as: distribu tion of Autonomous Systems and IP path lengths, hop counts and models of load ba lancing, topology growth, and applicability of power laws to real topologies.
Option 1, DNS Root Server Initiative and Visualization of Massive Data Se ts: CAIDA is co-locating skitter measurement hosts with select DNS root serv ers to provide ICANN recommendations regarding optimal locations for current and future root name servers. Also under this option, visualization initiatives wer e expanded significantly to develop techniques and code designed to facilitate a ggregation, analysis and layout of massive datasets (tens of gigabytes in size).

Approach

UCSD/CAIDA's project is divided into four tasks and one Option; Coral mon itors, Tomography (skitter), Security monitoring, Data Storage/Analysis, and DNS Root Server/Visualization of massive data sets. The approach to each effort is described in the program plan, available at https://www.caida.org/funding/progpl an/NGIprogplan98.xml, and is briefly summarized below.

Coral OC48mon:
The community's ability to monitor traffic on advanced optical and high perf ormance networks has not kept up with the raw development and deployment of fibe r and switching capability at those bandwidths. WDM networks, OC48 and OC192 net works are gaining momentum, but the current state of measurement technologies ha ndicaps engineers who manage these networks from trying to monitor traffic on su ch links. This initiative draws upon state of the art developments in computer p rocessing and data storage to develop tools capable of constant monitoring of tr affic (not sampled) at OC48 speeds for research and engineering purposes. Coral Monitors are used to collect information about the amount of traffic (in bytes, packets or flows) traversing a link, as well as traffic characteristics such as which applications and transport protocols generate the most traffic, which pack et sizes are most common, how many packets of various sizes tend to arrive in cl usters, and matrices of traffic flows between individual networks and autonomous systems. CAIDA and the University of Waikato are designing a capture card with an innovative architecture comprised of Vitesse's ATM and POS OC48 chipsets and a Xilinx Virtex chipset with data transfer rates of more than 400 Mbytes/second. The resulting DAG 4.1 capture card permits capture of every packet in both dire ctions on a full 2.4 GB OC48 speed link. An additional part of the Coral project is the development of an array of software tools to enable post-analysis of Cor al trace files, traffic characterization, and continuous monitoring. This librar y includes utilities for flow analysis, Autonomous System matrices, composition of traffic by application or other category, and various protocol-centric analys is tools.

Tomography
In order to gather more macroscopic information on Internet infrastructure, behaviors, platforms, and evolution, CAIDA is using active measurements and othe r tools to gather, analyze and visualize topology data. Skitter, a light-weight active monitoring tool, is continuously gathering topology from 19 sources to th ousands of destinations to collect information on hop-by-hop connectivity and pe rformance information, which is stored at SDSC for later analysis, correlation ( with other data such as routing and workload) and visualization. By the end of y ear 2 of this project, the 19 measurement hosts were monitoring several independ ent lists of end-destinations distributed throughout the IPv4 address space. CAI DA's skitter tool measures the forward IP path to a destination in a manner simi lar to traceroute: it increments the TTL when sending packets to a destination a nd records the router that replies at each TTL, until a TTL sufficient to reach the destination is used. Skitter uses ICMP echo requests as probes. When skitter finally receives the ICMP echo reply from the intended destination, it terminat es the path probing for that destination and records the round-trip time from th e source to the destination and back. Target sites are strategically selected to pervasively stratify the IPv4 address space in pursuit of a comprehensive cross -section of the commercial Internet.

Security
The security-related component of this project consists of engineering low-c ost passive security monitors. The Applied Telecom OC12 card used in the Coral O C12mon is used as the foundation for development of algorithms supporting integr ation of low-level, high-performance, real-time packet filtering. CAIDA is worki ng to enhance the OC12mon passive traffic monitor to facilitate ubiquitous netwo rk monitoring at aggregation points, DMZ's and ISP's, by developing dynamic filt ering and data collection, security policy compliance monitoring, and security p olicy enforcement components. Filtering is required to reduce data, isolate susp icious traffic, minimize contention for the peripheral bus, and permit persisten t monitoring of heavily loaded links. This is accomplished on the network adapte r FPGA and in the host software. CAIDA's approach procures arbitrary length payl oad extraction, which requires optimized zero-copy SAR on host and subsequent BP F linkage. Arbitrary length payload extraction is also essential to support evid ence collection and session replay, for which header filtering alone is insuffic ient. The delivered tool also provides a more general tool for network metrics b y providing finer-grained filtering, e.g., by protocol or source address.

Database Storage/Analysis
A DEC raid array is used to store active (skitter) measurement and routing d ata. Arts++, a binary file format library, was developed by CAIDA for storage an d analyses of those data, see https://www.caida.org/Tools/arts++ for code. Also u nder this effort, multiple analyses are being done on the static and dynamic cha racteristics of skitter topology data including application of power laws, dista nces and hop counts, path run lengths, and load balancing, see http://ipn.caida. org/~broido/overview for in-depth analysis.

DNS Root Servers/Visualizations
The Domain Name Server (DNS) technical advisory committee to ICANN includes existing root server operators, institutional representatives (from IESG, IANA, DOC, etc.) and technical measurement experts (CAIDA). One of the committee's res ponsibilities is to provide ICANN with recommendations regarding optimal locatio ns for root name servers. There are currently 13 root name servers. RSSAC has as ked CAIDA for assistance gathering data to help determine architecturally strate gic locations for current and planned root name servers within the Internet. CAI DA is achieving this objective by co-locating skitter hosts with select root nam e servers, and measuring connectivity and round trip latency to a target list of hosts taken from the root's DNS query logs. CAIDA currently has skitter hosts c o-located with 5 of the 13 root name servers; F root in San Jose, E root at NASA Ames, L root at ISI, K root in London, and the K mirror site in Amsterdam. The primary goal of the measurement effort is to assess two metrics of connectivity: round trip time and hop count from the root name server to the hosts in the tar get set. CAIDA is specifically exploring three possible topological results: 1) Clusters of hosts that are particularly far, measured by latency, from all of th e roots, and might thus suggest a region that merits a new root server 2) Insuff icient redundancy in the root server architecture might be reflected in skitter topologies from multiple roots that suggest that the failure of a strategic inte rmediate router or sub-path would render many end hosts unable to reach any root 3) Conversely, excessive redundancy in the infrastructure might be reflected in a set of skitter topologies from different roots where a large set of destinati on hosts are quite close to several of these roots.

Recent Accomplishments

Coral OC48mon
Development and testing of the DAG 4.1 OC48 ATM/POS capture card prototype w as completed during the last year. This card is POS and ATM capable, and uses a 64-bit 66 MHz PCI interface.
NeTraMet software was ported to the CoralReef software suite, providing acce ss to packet headers from live interfaces and from trace files. NeTraMet is an o pen-source implementation of the IETF's RTFM Traffic Measurement system; it prov ides a platform for implementing and testing real-time performance measurement t echniques. The CoralReef NeTraMet meter (monitoring traffic on the SDSC commodit y Internet connection) was used for preliminary studies of root nameserver perfo rmance (response time and request loss rate), stream lifetime (ms) and size (pac kets and kB) distributions, web object size, download time and transfer rates, R ealAudio packet size and interarrival time distributions.
The CoralReef report generator was augmented and is now fully operational. A demonstration of the tool is available at https://www.caida.org/tools/measureme nt/coralreef. The report generator produces graphs and tables for various types of information found on UCSD's incoming commodity Internet link with AT T, inclu ding by protocols, application, and hosts measured in packet, bytes and flow tup les.
Animations describing Coral Monitors and analyses were created and made avai lable to the community, see http://flicks.caida.org.

Tomography (skitter)
A total of 19 skitter measurement hosts were globally deployed as of June 20 00, allowing for comprehensive collection of world -wide topology data.
A 2100 IP address destination topology dataset was created as a result of a successful 10-day data collection (August 29-September 8, 1999) from skitter sou rce machines deployed at the Asia Pacific locations. CAIDA made this destination list available to Asia Pacific collaborators for analysis, e.g., validation of its `representativeness' with respect to Asia Pacific region as well as the larg er Internet.
CAIDA released additional skitter datasets to the community for use in third -party research, see https://www.caida.org/funding/ngi1998/content/reports/skitter_c omuse.xml for details.
Three different skitter destination lists were created for investigation of specific goals: The Web Server List, with a comprehensive collection of web serv ers around the globe; The Intermediate list, to study the lifetime characteristi cs of "non-edge" IP addresses; and the BGP Prefix list, containing a single dest ination in every /24 of the IPv4 space to effectively stratify the Internet infr astructure, see https://www.caida.org/tools/measurement/skitter/lists/ for detail s.
A summary of statistics from each skitter monitor in the field was maintaine d and updated daily, see http://sk-summary.caida.org/cgi-bin/main.pl
An animation describing how skitter works was created, see https://www.caida. org/publications/animations/.

Security
CAIDA completed algorithms permitting efficient re-assembly of packet heade rs and incorporated them into the CoralReef software package.
CAIDA released crl_portmap, a prototype monitoring tool that scans for suspi cious activity to the portmapper (RPC) service.. Once crl_portmap detects suspic ious activity, all traffic to and from the probing host is logged out in tcpdump format. Attackers often probe portmapper early in their attempt to breach secur ity. By detecting this early, it is possible to trigger more complete logging of suspicious activities for future analysis and possible use in prosecution.
Vern Paxon's Bro security software was ported to the CoralReef software suit e. Bro is a stand-alone system for detecting intruders in real time by passively monitoring the link over which the intruder's traffic transits. Bro has provisi ons for real-time notification, clear separation between mechanism and policy, a nd extensibility.

Database Storage/Analysis
Extensive analysis code was developed and preliminary summary graphics on th e Asia Pacific study were made available to Mari Maeda for DARPA presentations i n Europe and at the PITAC review meeting. An abstract describing this research w as also accepted for the INET 2000 conference. This paper is now available at ht tp://www.caida.org/publications/papers/asia_paper/.
CAIDA established a website that collects, monitors, analyzes, and visualize s several forms of Internet traffic data concerning network topology, workload c haracterization, performance, routing, and multicast behavior. The analyses serv e a variety of disciplines/purposes, including research, policy, education, and visualization, see: https://www.caida.org/analysis.
CAIDA currently has 1/3 terabyte (360 gigabytes) of skitter topology data st ored on the RAID array.
CAIDA worked on mechanisms to derive a "Giant Component" of the Internet fro m their collected topology data, allowing depiction of the most well connected c omponents of the Internet infrastructure.
Analysis was also done to investigate the correlation of various metrics for 'distance' between IP addresses with performance (RTT) between those data point s. Metrics of geographic distance investigated were: great circle distance (circ umference around globe); just longitude; longitude + latitude; and distance from +to the U.S. (See http://ipn.caida.org/~bhuffake/skitter/distance/ for details.)
Further analysis continued to characterize the rate and pattern of IP addres s loss characteristics, e.g., destinations that become unreachable. CAIDA is dev eloping a model for this process, which will yield parameters necessary to acqui re a legitimate sample of Internet topology.

DNS Root Servers/Visualizations
A total of 5 skitter measurement hosts were co-located with DNS root server sites over the past year. There is a skitter host co-located with the E root at NASA Ames, the L root at ISI, the F root at Vixie Enterprises in San Jose, the K root at RIPE in London, and K-mirror root in Amsterdam. Traffic measurement is occurring on each root sever.
A visualization of core Internet Autonomous Systems was created, see http:// www.caida.org/analysis/content/visualization/as_core_network/. The graph reflect s 220,533 IP addresses (374,013 links, and 154,104 target destination IP address es) from paths obtained by merging three datasets collected during a sixteen day period in mid-January, 2000, and shows a sample snapshot of the Internet core.

Current Plan

Coral OC48mon
CAIDA expects to have a fully operational OC48 monitor ready for testing by late September 2000. Initial testing will be performed in CAIDA's test lab at SD SC using OC48 capable routers donated by Juniper and Cisco. CAIDA will test basi c capture operations of the card under varying traffic load levels. Testing will include synthetic traffic as well as real network traffic from production netwo rk links. Next steps will involve further testing on CAIDA member or other colla borator OC48 networks. The monitor will then be deployed at a site to be determi ned by CAIDA's DARPA PM.
CAIDA plans to install a Gigabit Ethernet monitor on the SD-NAP located at S DSC and potentially other sites. We will help SDSC develop a prototype customer workload profiling service based on the CoralReef report generator.

Tomography
All additional tomography work will be performed under Option 1.

Security
CAIDA plans to support SDSC's installation of the Coral based security modul es for use in UCSD campus infrastructure protection.
We will continue collaborations with University of Waikato on hardware filte ring techniques for the DAG series capture cards.

Database Storage/Analysis
All work will continue under Option 1.

DNS Root Servers/Visualizations
CAIDA's current plan is to continue deployment of skitter monitors at DNS r oot server sites in order to satisfy the ICANN request to gather data to help de termine architecturally strategic locations for current and future root name ser vers within the Internet infrastructure.
CAIDA will continue storage of skitter data collected from monitors deployed in the field.
Planned analysis for the upcoming year includes providing a metric to deter mine if there is an Internet Core, and if so, does an individual server's "close ness" to that Internet core affect RTT. CAIDA plans to systematically run differ ent destination lists on different sources to compare metrics of "closeness" or "optimal placements" of a given source with respect to a set of destinations.
CAIDA plans to develop a CoralReef module that will provide aggregate inform ation about DNS root server traffic.

Technology Transition

Coral OC48mon
Prototype deployment of the OC48 monitor will occur under the auspices of CA IDA's DARPA PM.
Two different versions of the CoralReef software suite are currently made av ailable to different facets of the Internet community, CAIDA members and the pub lic.
The CoralReef software suite was recently licensed from the University of Ca lifornia to a new software development company; CAIMIS. CAIMIS will provide prod uction environment support and documentation for CoralReef software, see http:// www.caimis.com/about/background.html

Tomography (skitter)
Active measurement data gathered through this initiative provides the commu nity with a unique and valuable source of infrastructure-wide data. Researchers from several Universities and collaboration partners (APAN/KDD, APAN/KAIST, Abov eNet, Qwest, MCI Worldcom, SingAREN, and the University of Waikato) are using da tasets for analysis.
Skitter binaries are made available to CAIDA members, allowing them to colle ct topology data from their own sources in their own Networks.
Skitter was licensed by the University of California to CAIMIS. CAIMIS will provide production environment support and documentation for skitter software, s ee www.caimis.com.

Security
No notable technology transition has occurred in this arena.

Database Storage/Analysis
Select skitter datasets and analyses results are made publicly available on the CAIDA web site. Several researchers have published papers based on these dat asets.
CAIDA members have access to skitter topology data.

DNS Root Servers
The methodology for the DNS root server location evaluation, and associated mechanisms for determining 'central' positions within the Internet will be relev ant far beyond the DNS system, applicable to location research for any type data server of strategic infrastructural relevance.