UCSD Network Telescope Aggregated Flow Dataset

This dataset consists of hourly files of unsolicited traffic captured by the UCSD Network Telescope traces and aggregated into the FlowTuple format.

Data Description

The UCSD Network Telescope consists of a globally routed, but lightly utilized /8 network prefix, that is, 1/256th of the whole IPv4 address space. It contains few legitimate hosts; inbound traffic to non-existent machines - so called Internet Background Radiation (IBR) - is unsolicited and results from a wide range of events, including misconfiguration (e.g. mistyping an IP address), scanning of address space by attackers or malware looking for vulnerable targets, backscatter from randomly spoofed denial-of-service attacks, and the automated spread of malware. CAIDA continously captures this anomalous traffic discarding the legitimate traffic packets destined to the few reachable IP addresses in this prefix. We archive and aggregate these data, and provide this valuable resource to network security researchers.

Raw data captured by the UCSD Network Telescope are stored in huge pcap files, each file containing 1 hour of data. In order to enable more efficient data storage, processing, and analysis, these hourly pcap files are post-processed using Corsaro software to extract the most important packet header fields and aggregate data into FlowTuple files. The FlowTuple format includes the following eight fields:
source IP address; destination IP address; source port; destination port; protocol; TCP Flags; TTL; IP length.
In the hourly FlowTuple output files, the data are broken into 60-second intervals. Within a given interval, each unique key (a unique combination of the FlowTuple fields) observed in the raw pcap data is reported on a separate line in the following format:

<src_ip>|<dst_ip>|<src_port>|<dst_port>|<protocol>|<tcp_flags>|<ttl>|<ip_len>,value

where "value" is the number of packets in this interval whose header fields match this FlowTuple key.
Flows are further subdivided into three FlowTuple classes: backscatter, ICMP Request and "other" (not backscatter and not ICMP Request), and the total number of flows in each class is recorded.

One can use "cors2ascii" command to display the FlowTuple output in a human-readable ASCII format:

$ cors2ascii example.flowtuple.cors.gz
#CORSARO_INTERVAL_START 0 1527811200
START flowtuple_backscatter 1012004
xxx.181.55.12|xx.68.45.71|3|3|1|19|0x00|100,20
xxx.85.229.197|xx.4.59.12|3|1|1|19|0x00|159,1
......
END flowtuple_backscatter
START flowtuple_icmpreq 314399
xxx.136.34.2|xx.128.122.196|8|0|1|1|0x00|44,3
xxx.52.83.2|xx.165.41.187|8|0|1|1|0x00|48,2
......
END flowtuple_icmpreq
START flowtuple_other 17964239
xxx.228.34.84|xx.151.31.6|0|0|4|63|0x00|256,6
xxx.228.34.84|xx.151.31.6|0|0|4|63|0x00|512,140
......
END flowtuple_other
# CORSARO_INTERVAL_END 0 1527811259
# CORSARO_INTERVAL_START 1 1527811260
START flowtuple_backscatter 991397
.........

Each hourly flowtuple file contains 60 intervals (0-59) characterized by unique start and end local unix time. More information about the structure of Corsaro global output file can be found at Corsaro Documentation page. This ongoing dataset covering the period from February 2008 till now is stored locally at CAIDA.

Caveats that apply to this dataset

This dataset and the types of worm and denial-of-service attack traffic contained therein are representative only of some spoofed source denial-of-service attacks. Many denial-of-service attackers do not spoof source IP addresses when they attack their victim, in which case backscatter would not appear on a telescope. Attackers can also spoof in a non-random fashion, which will incur an uneven distribution of backscatter across the IPv4 address space, and may cause backscatter traffic to miss any telescope lenses. Note that the telescope does not send any packets in response, which also limits insight into the traffic it sees.

Data Access Policy

These data must be analyzed on CAIDA machines, and cannot be downloaded!

Academic researchers and US government agencies can request access through CAIDA by filling out and submitting the online form. It usually takes about five to ten business days to process your request. We carefully review each application and the decision to grant the data access is based on the merits of your proposed data use.

These data also may be available for corporate entities who participate in CAIDA's membership program. Information on membership levels, services, and rates can be requested by emailing sponsorship@caida.org.

Once users are approved for access to this dataset, they will receive an account on the CAIDA machine that provides direct access to the Telescope data they requested. Accounts are valid for a nominal twelve months in which the research is expected to be completed. CAIDA strictly enforces a "take software to the data" policy for this dataset: all analysis must be performed on CAIDA computers; download of raw data is not allowed. CAIDA provides several basic tools to work with the dataset, including CoralReef and Corsaro. Researchers can also upload their own analysis software.

Acceptable Use Agreement

Access to these data is subject to the terms of the following CAIDA Acceptable Use Agreement (printable version in PDF format)
and the supplemental AUA below:
CAIDA ACCEPTABLE USE AGREEMENT (ver 071814) The following terms comprise the Acceptable Use Policy (AUP) and Data License Agreement (collectively, the "AGREEMENT", or, "Acceptable Use Agreement (AUA)" for all datasets made available to You by the Center for Applied Internet Data Analysis (CAIDA), a research unit at the University of California San Diego (UCSD) and governed by The Regents of the University of California. Certain datasets may have additional Supplemental provisions. References to this AGREEMENT shall include any and all relevant Supplemental provisions. CAIDA has the authority and reserves the right, in its sole discretion, to refuse requests for dataset(s) or discontinue further access and use to anyone. If You feel Your request is inappropriately denied please contact CAIDA by sending a message to data-info@caida.org. In consideration for requesting and receiving access to CAIDA dataset(s), You acknowledge that You understand and agree to be bound by the terms and conditions of this AGREEMENT. Any violation of this AGREEMENT may result in the immediate suspension or termination of this AGREEMENT and/or other action entitled by law such as injunctive or equitable relief. You are individually liable and responsible for compliance with this AGREEMENT. This AGREEMENT is legally binding under the laws of the State of California, United States. You may terminate this Agreement by contacting CAIDA in writing and receiving acknowledgement of such request. 1. LICENSE CAIDA's authorization to access the data grants You a limited, non-exclusive, non-transferable, non-assignable, and terminable license to copy, modify, and use the data in accordance with this AGREEMENT solely for the purpose of non-profit research, non-profit education, commercial internal testing and evaluation of the data, or for government purposes by or on behalf of the U.S. Government. No license is granted for any other purpose and there are no implied licenses in this Agreement. This Agreement shall become effective as of the date of approval by CAIDA and shall remain in force for a period of one year, unless terminated earlier or amended in writing. CAIDA shall have the right to use any of Your feedback received during the license period solely for its non-profit educational and/or research purposes. 2. GENERAL CONDITIONS (i) You will not impersonate any individual or entity, misrepresent any affiliation with another person, entity or association, use false information, or otherwise conceal Your identity from CAIDA at any time for any purpose. (ii) You consent that CAIDA can make public or otherwise disclose Your name as the registered requestor, the name of Your Affiliated Institution, the name of the dataset(s) that CAIDA has made available to You under this AUA, and the brief description of the type of research being undertaken that You provided to CAIDA. (iii) You will abide by any and all modifications. If a modification occurs, it will be explicitly communicated to You via Your registered email address and shall become effective fifteen (15) days after the transmission of such notification. Your continued access to or use of the data after such time shall indicate Your assent to any and all modification(s). If You do not agree to comply with the modification(s), You agree that You will: (a) inform CAIDA immediately, at at which point Your access to the data will be terminated and no longer authorized; and, (b) no longer access or use this data. 3. USE RESTRICTIONS If You have any concerns or questions about these restrictions, You are encouraged to contact CAIDA management via email to data-info@caida.org. To obtain an exemption from any of these restrictions, You will need a written authorization from CAIDA management. (i) While using non-anonymized data set(s), You will respect the privacy of persons that may be identified in the data. For any publication or other disclosure, You will anonymize or de-identify personally-identifiable information, IP addresses, and other data identified in Supplemental provisions (if any) by using commonly accepted techniques such as one of the methods recommended by CAIDA (https://www.caida.org/projects/predict/anonymization/). (ii) While using anonymized data set(s), You will not attempt to reverse engineer, decrypt, de-anonymize, derive or otherwise re-identify anonymized information. (iii) You will not distribute, disclose, transfer or otherwise make available the dataset(s) to any person other than those employed by your institute who are assisting or collaborating with You using the dataset(s). Other entities with whom You are collaborating in research using the dataset(s) must request access to the dataset(s) separately and directly from CAIDA. (iv) All conditions, restrictions and obligations attached to this data shall accompany any and all subsequent uses and disclosures of this dataset by You. Therefore, You are personally and fully responsible for communicating this AUA and ensuring its compliance as to any and all users described above to whom You make the data available. 4. USE OBLIGATIONS (i) If You create a publication (including web pages, papers published by a third party, and publicly available presentations) using data from this dataset, You must provide CAIDA with a copy of (or a link to) the publication and You must cite the data as follows: The CAIDA UCSD [DataSet Name] - [dates used], https://www.caida.org/data/[dataset-URL] (ii) At the end of the research, or semi-annually (whichever is sooner), You will report a summary of the research and any findings/conclusions to CAIDA. This information is used in reports to our funding agencies. (iii) You agree to expunge any and all copies of the received Dataset(s) upon completion or termination of stated research and/or termination of data access or use. Completion of stated research shall allow for a reasonable period of time that You may need to retain the dataset(s) in order to satisfy scientific reproducibility obligations. 5. ACCOUNTABILITY AND ENFORCEMENT (i) You agrees to safeguard any and all sensitive data as required, or may be required, by law by using at least the same degree of care that You uses for its own data of a like nature but no less than a reasonable degree of care, to protect the confidentiality of data and/or the privacy of any identifiable person and to prevent its unauthorized disclosure and use. Data is confidential if it is marked as such, if by its nature or content is reasonably distinguishable as confidential, or if You have reasonable cause to believe that its disclosure to a third party would cause harm or damage. Data is not confidential, and therefore not sensitive if: (a) You already knew the data before it was disclosed to You by CAIDA; (b) You gain subsequent knowledge of the data by either lawfully obtaining it from another source under no obligation of confidentiality, or You develop it independently; (c) the data is or becomes generally available to the public through no wrongful act of You or any other party; (e) it is required to be disclosed under applicable law, regulation or court order provided You notify CAIDA prior to making such a disclosure so that CAIDA may take appropriate action. (ii) You will notify CAIDA immediately of all relevant details if: (a) confidentiality or privacy is compromised; or (b) You receive any legal, investigatory, or other government demand to reverse engineer, decrypt, de-anonymize or otherwise disclose anonymized or confidential data. (iii) If requested, You will provide CAIDA evidence of compliance with this AGREEMENT, such as a written description of Your data protection plan or a written affirmation that You have disposed of any and all copies of the received dataset from all systems reasonably known to you. (iv) DISCLAIMER OFWARRANTIES. CAIDA USES ITS BEST EFFORTS TO PROVIDE DATA IN ACCORDANCE WITH ETHICAL PRINCIPLES AND SCIENTIFIC INTEGRITY. HOWEVER, THE DATA PROVIDED HEREIN IS ON AN "AS IS" BASIS. NEITHER CAIDA, ITS RESEARCHERS, RESEARCH PARTNERS, LICENSORS, AND DATA PROVIDERS, NOR THE UNIVERSITY OF CALIFORNIA AND ITS TRUSTEES, OFFICERS, EMPLOYEES, AND AGENTS MAKE ANY WARRANTY, EITHER IMPLIED OR EXPRESS, OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, INCLUDING, BUT NOT LIMITED TO, THE ACCURACY, TIMELINESS, COMPLETENESS, RELIABILITY, OR AVAILABILITY OF CAIDA DATA, APPLICATIONS, OR SERVICES ACCESSIBLE THROUGH OR MADE AVAILABLE BY CAIDA. TO THE EXTENT ALLOWED BY LAW, YOU WILL INDEMNIFY, DEFEND AND HOLD HARMLESS CAIDA AND THE UNIVERSITY OF CALIFORNIA FROMALL LIABILITIES, CLAIMS, DEMANDS, COSTS, JUDGMENTS, DAMAGES, LOSSES AND EXPENSES ARISING OUT OF OR IN CONNECTION WITH ANY USE OF THE DATA BY YOU. IN NO EVENT SHALL CAIDA AND THE UNIVERSITY OF CALIFORNIA BE LIABLE TO YOU OR ANY THIRD PARTY FOR ANY INDIRECT, CONSEQUENTIAL, INCIDENTAL, SPECIAL OR PUNITIVE DAMAGES, ARISING FROM YOUR USE OF THE DATA. 6. INTELLECTUAL PROPERTY This section applies only if You are licensing the data for the purpose of commercial internal testing and evaluating the data. (i) Except as specifically provided in paragraphs (ii), (iii) or (iv) of this Section, all right, title and interest in and to any "Intellectual Property" any invention, discovery, improvement, computer or other program, related documentation and work of authorship that is developed for commercial use by You or Your employer, employees or agents, either individually or jointly with CAIDA, using any information provided by CAIDA under this Agreement, shall be the joint property of UCSD/CAIDA and You, and that shared portion is hereby assigned to UCSD/CAIDA, solely for its non-profit educational and/or research purposes. As between CAIDA and You, You retain exclusive and all right, title and interest in and to "Your Intellectual Property" the information originating with or provided by You. (ii) You obtain an exemption from the default Intellectual Property provisions in Section (i) for specific developments via a written authorization from CAIDA management; (iii) You obtain a full Commercial License from the UC San Diego Technology Transfer Office that grants alternative rights in the Intellectual Property; or (iv) You make the Intellectual Property freely available and without restrictions to the public, provided such disclosure complies with obligations regarding sensitive information described in Section 5(i). 6.1 UCSD COPYRIGHT Permission to use, print, copy, and modify any copyrightable part of this UCSD CAIDA Data for educational, research and non-profit purposes, as set forth in the Researcher MOA without fee, and without a written agreement is hereby granted, provided that this paragraph and the following copyright notice and paragraphs appear in all copies: Copyright 2006-2014 The Regents of the University of California. All Rights Reserved. If You desire to use or otherwise incorporate any copyrightable part of this data for commercial purposes, You should contact the UC San Diego Technology Transfer Office, (858) 534-5815, fax: (858) 534-7345.

CAIDA ACCEPTABLE USE AGREEMENT -- Supplement You agree that you will not copy from CAIDA servers or otherwise transfer, distribute, or disclose raw NRTT data or any derived data with identifying or sensitive information. You may transfer and use, in accordance with the terms and conditions of the AUA, derivative information obtained from the raw NRTT data only if you ensure that the raw data (including the address of the telescope network) is de-identified or anonymized. You are not permitted to attempt to connect to, probe, or in any other way initiate contact with a machine or machine administrator identified or identifiable, via IP address or otherwise, within the dataset(s), without written authorization from CAIDA management at data-info@caida.org.

When referencing this data (as required by the AUA), please use:
The CAIDA UCSD Network Telescope Aggregated Flow Dataset - < dates used >,
https://www.caida.org/data/passive/telescope-flowtuple.xml
Also, please, report your publication to CAIDA.

UCSD Network Telescope Datasets

Historical and Near-Real-Time Network Telescope Dataset

Aggregated Traffic Data in FlowTuple format

Daily RSDoS Attack Metadata

Two Years of Daily RSDoS Attack Metadata (downloadable paper supplement)

Three Days Of Conficker Dataset

CAIDA UCSD Network Telescope Traffic Samples

Witty Worm Dataset

Code-Red Worms Dataset

Patch Tuesday Dataset

Two Days in November 2008 Dataset

Telescope Educational Dataset

Telescope Dataset on the Sipscan

Telescope Darknet Scanners Dataset

For more information on the UCSD Network Telescope, see:

https://www.caida.org/projects/network_telescope/

For more information on the CoralReef Software Suite, see:

https://www.caida.org/tools/measurement/coralreef/

For more information on the Corsaro Software Suite, see:

https://www.caida.org/tools/measurement/corsaro/

For a non-exhaustive list of Non-CAIDA publications using Network Telescope data, see:

UCSD Network Telescope papers

Backscatter-related papers