• HOME
  • About
  • Program Plan
  • Annual Report
  • Joining CAIDA
  • Staff
  • Jobs
  • Legal Agreements
  • How-To
  • Blog
  • Contact Us
• CATALOG
  • Datasets
  • Media
  • Papers
  • Recipes
  • Software
• RESEARCH
  • Topology
  • Security
  • Traffic Analysis
  • Economics and Policy
  • Visualization
• DATA
  • Overview
  • Data Sharing
  • Distribution Statistics
  • How-to
  • Data Usage FAQ
  • Papers using CAIDA Data
  • Acceptable Use Agreement
• TOOLS
  • Overview
• SERVICES
  • AS Rank
  • AS Rank API
  • BGPStream
  • BGPStream API
  • HI-CUBE (login required)
  • IODA
  • MANIC
  • MANIC API
  • Vela
  • Vela web API
  • Vela MIDAR API
  • Vela aliasq API
• PUBLICATIONS
  • Papers
  • Presentations
  • Blog
  • Visualizations
  • - Animations
  • - Posters
  • Networking Bibliography
• WORKSHOPS
  • Overview
  • AIMS
  • DUST
  • IMAPS
  • KISMET
  • WIE
  • WOMBIR
  • Collaborative
  • Archive
• PROJECTS
  • Ark
  • KISMET
  • MANIC
  • Macroscopic Topology
  • Spoofer
  • Network Telescope
  • IMPACT (PREDICT)
• FUNDING
  • PANDA
  • FANTAIL
  • PENMAN
  • SLAM
  • Censorship Outages
  • MapKIT
  • STARDUST
  • MADDVIPR
  • Program Plan

The CAIDA DNS root/gTLD RTT Dataset

This dataset contains DNS roundtrip time (RTT) information useful for studying conditions within the global Internet, and the way it has changed over the last few years. DNS RTTs are influenced by several factors, including remote server loading, congestion within Internet routes, route changes, and local effects such as link or equipment failures.

Domain nameservers such as BIND use various algorithms to select which of the 13 root or gTLD servers they will ask to resolve a top-level domain name. By watching requests to root/gTLD at the boundary of a campus (i.e. large enterprise) network, and the responses to them, one can make passive measurements of the RTT to all of the root/gTLD servers which the site has routes to.

This data was collected using NeTraMet, producing daily data files in NeTraMet's flow data file format, from several different sites. The dataset contains historical data for four sites:

Location	Start date	End date
University of California, San Diego (UCSD)	8 Jan 2002	9 Aug 2003
University of Colorado (CU)	8 Jan 2003	4 Apr 2008
WIDE (Keio Univ. Shonan Fujisawa Campus)	13 Jul 2005	15 Jun 2009
WIDE (The University of Tokyo)	1 Jun 2004	27 Sep 2009

Currently one site is actively collecting data:

Location	Start date	End date
University of Auckland (UA)	11 Nov 2002	05 Jan 2012
	19 Feb 2014	(ongoing)

The data was collected at five-minute intervals using an SRL (RFC 2723) ruleset called dns-root.srl (available with the data).

A web page allowing you to examine 5-minute median RTT values over periods of 1 to 7 days appears at: https://www.caida.org/cgi-bin/dns_perf/main.pl

The NeTraMet flow data file layout is described in https://www.caida.org/tools/measurement/netramet/download/ntm43.pdf

Flow data files contain 2 types of lines:

• control record lines, these start with a # (pound-sign)
• flow data records

Each file begins with a header giving details of when the file was created, and the format of its data records. The format is described by a #Format record, listing the RTFM attributes requested in the dns-root ruleset.

In this dataset, the NeTraMet meters were read every five minutes; flow data for each five-minute interval appears between #Time and #Enddata records in the files.

The data was collected using NeTraMet's ToTurnaround attribute, i.e. each flow record contains an RTT distribution for one root or gTLD, as described in https://www.caida.org/publications/papers/2002/nsrtd/

If there are less than 100 (later 120) data points, their actual values are saved in the flow data record. Otherwise the data is binned, using bounds generated from the data. This approach preserves the maximum accuracy for the RTT values.

NeTraMet versions 51b4 (21 Sep 04) through 51b6 (15 Mar 05) had a bug in its handling of dynamic distributions.

If such a distribution had 120 or less data values, the distribution is written in the flow data file as a type 5 distribution, i.e. a list of actual values. These are correct.

However, if there were more than 120 values, NeTraMet computes lower and upper limits, and uses them to produce a binned distribution, type 6. The values are bin counts, those counts are correct. Unfortunately, the bug corrupted the computed upper limit, producing distributions which always had an upper limit of 7000 (700 ms). To produce actual values for the bin upper edges, one needs correct limit values - these are incorrect, and should NOT be used. The (corrupted) type 6 distributions remain in the .dif files, they have not been removed.

The affected date ranges are:

UC San Diego	(not affected)
Auckland and Colorado	Mon 15 Sep 2004 through Wed 16 Mar 2005
Tokyo and Fujisawa	Mon 15 Sep 2004 through Wed 9 Nov 2005

The Auckland and Colorado data doesn't have very many type 6 distributions, so this isn't too much of a problem. For the WIDE data (Tokyo and Fujisawa), the root distributions seem mostly unaffected, but the gTLD distributions have very high counts per 5-minute reading interval, so there is almost no usable gTLD RTT data from the WIDE NeTraMet meter while the buggy version(s) were running.

A sample perl program, dnsroot_to_dat.pl, is provided to demonstrate one approach to extracting RTT distributions from the flow data files.

Acceptable Use Agreement

Access to these data is subject to the terms of the following CAIDA Acceptabl e Use Agreement (printable version in PDF format )
CAIDA ACCEPTABLE USE AGREEMENT for PUBLICLY ACCESSIBLE DATASETS The following terms comprise the Acceptable Use Policy and Data License Agreement for all publicly accessible datasets (the "Public Agreement") made available by the Center for Applied Internet Data Analysis (CAIDA), a research unit at the University of California San Diego (UCSD) and governed by The Regents of the University of California. LICENSE CAIDA's authorization to access the data grants You a limited, non-exclusive, non-transferable, non-assignable, and terminable license to copy, modify, and use the data in accordance with this Public Agreement. No license is granted for any other purpose and there are no implied licenses in this Agreement. Nothing in this License is intended to limit any rights You may have arising from fair use or due to other limitations on CAIDA's exclusive rights under copyright law or other applicable laws. CAIDA has the authority and reserves the right, in its sole discretion, to discontinue further access and use to anyone who violates this AUA. If You create a publication (including web pages, papers published by a third party, and publicly available presentations) using data from this dataset, You should cite the data as follows: The CAIDA UCSD [DataSet Name] - [dates used], https://www.caida.org/data/[dataset-URL] We encourage You to provide CAIDA with a copy of (or a link to) the publication. We use this information in reports to our funding agencies. DISCLAIMER OF WARRANTIES. CAIDA USES ITS BEST EFFORTS TO PROVIDE DATA IN ACCORDANCE WITH ETHICAL PRINCIPLES AND SCIENTIFIC INTEGRITY. HOWEVER, THE DATA PROVIDED HEREIN IS ON AN "AS IS" BASIS. NEITHER CAIDA, ITS RESEARCHERS, RESEARCH PARTNERS, LICENSORS, AND DATA PROVIDERS, NOR THE UNIVERSITY OF CALIFORNIA AND ITS TRUSTEES, OFFICERS, EMPLOYEES, AND AGENTS MAKE ANYWARRANTY, EITHER IMPLIED OR EXPRESS, OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, INCLUDING, BUT NOT LIMITED TO, THE ACCURACY, TIMELINESS, COMPLETENESS, RELIABILITY, OR AVAILABILITY OF CAIDA DATA, APPLICATIONS, OR SERVICES ACCESSIBLE THROUGH OR MADE AVAILABLE BY CAIDA. LIMITATION OF LIABILITY. TO THE EXTENT ALLOWED BY LAW, IN NO EVENT SHALL CAIDA AND THE UNIVERSITY OF CALIFORNIA BE LIABLE TO YOU OR ANY THIRD PARTY FOR ANY INDIRECT, CONSEQUENTIAL, INCIDENTAL, SPECIAL OR PUNITIVE DAMAGES, ARISING FROM YOUR USE OF THE DATA. If You have any questions about the data or about this Public Agreement, please email data-info@caida.org.

When referencing this data (as required by the AUA), please use:

The CAIDA UCSD DNS root/gTLD RTT Dataset,
https://www.caida.org/data/passive/dns_root_gtld_rtt_dataset.xml

Also, please, report your publication to CAIDA.

Data Access

• Access the publicly available CAIDA DNS root/gTLD RTT Dataset

References

For more information on Netramet, see:

• https://www.caida.org/tools/measurement/netramet/

Acknowledgements

Thanks to:

• The University of Auckland, for their support in developing early versions of NeTraMet and the standards effort leading up to the RTFM system.
• CAIDA for their ongoing support of NeTraMet development, making it a more versatile research tool.
• The WAND group at Waikato University for support in making NeTraMet work with early versions DAG cards.
• Endace Technologies for support with newer versions of DAG cards.

Special thanks (mostly for their help in setting up and maintaining the NeTraMet meters) to :

• Russell Fulton (U Auckland)
• CAIDA (Duane Wessels, Dan Andersen)
• U Colorado (Robert Roybal)
• WIDE (Yuji Sekiya, Kenjiro Cho)