ASSISTS - Advancing Scientific Study of Internet Security and Topological Stability

Project Summary

Large-scale Internet cyber-attacks and incidents - route hijacking, network outages, fishing campaigns, botnet activities, large-scale bug exploitation, etc. - represent a major threat to public safety and to both public and private strategic and financial assets. Mitigation and recovery, assessment of impacts and restoration costs, as well as prevention of further attacks of similar nature, are impeded by the fact that such events can remain unnoticed or are hard to characterize, in terms of motivation, infrastructure used by the attacker, and scope. Because of their macroscopic nature, identifying such events and understanding their scope and dynamics requires three critical inputs:

heterogeneous sources and types of data to cross-validate inferences;
a system to enable close to real-time integration and interactive visualization of such data;
a team of experts with varied background and skills to soundly interpret fused data.

We are pursuing these three inputs via strategically planned two-fold participation in the IMPACT program. As a Data Provider, we will continue to provide data sets that have already proven relevant to researchers studying security, stability, and resilience of networks. As a Decision Analytics-as-a-Service Provider, we will support new analytic capabilities that integrate, correlate, and cross-validate multiple sources of measurement and meta-data to enable informed mitigation of and response to attacks and other disruptive events.

Statement of Work

CAIDA performs fundamental research on a reasonable efforts basis and in accordance with UC policy. Technical reports will be submitted triannually.

TTA #1: Supporting Cybersecurity Research through Network Data Collection and Curation

Subtask	Description	Projected Timeline	Status
1. Data Provider Tasks
1.1	Curate and package the Internet Topology Measured from Ark Platform datasets	ongoing	Ark topology datasets indexed in IMPACT
1.2	Curate and package the Internet Topology Data Kits	every 3-6 mo	ITDK CAIDA page
1.3	Curate and package the UCSD Real-time Network Telescope Datasets	ongoing	Telescope datasets indexed in IMPACT
1.4	Collect, process, and archive the U.S. backbone bidirectional traffic data^* ^*as long as conditions permit and links and traffic monitors are available	ongoing	Anonymous Internet Traces Dataset
1.5	Acquire a 100gb packet capture monitor	Year 2	Done
1.6	Deploy the packet capture monitor on a 100gb national backbone link	Year 2	Work in progress
2. Data Host Tasks
2.1	Maintain and expand our hosting capabilities	ongoing	Size of datasets Indexed in IMPACT
2.2	Manage, maintain, and serve previously collected CAIDA data	ongoing	CAIDA Data Overview Table
2.3	Index and share new CAIDA data sets with researchers	ongoing	CAIDA Data available in IMPACT
2.4	Compile statistics of data volumes, requests and download	ongoing	IMPACT datasets access requests stats
3. New Data Sets
3.1	Generate new data sets that are crucial for studying threats, vulnerabilities, and hazards to critical infrastructures	ongoing	List of new datasets indexed in IMPACT
3.2	Generate derivative data sets that reveal signals of connectivity disruptions from active and passive measurement methods	Year 2
3.3	Experiment with which possible data sets are most amenable to live streaming to support HI-CUBE's near-real-time analytic capabilities	Year 2
4. Project Support
4.1	Work closely with other IMPACT project team members	ongoing
4.2	Work closely with IMPACT Portal developers	ongoing
4.3	Update IMPACT MOAs to support new data offerings	as needed
4.4	Host and attend project meetings	as needed	DHS IMPACT PI Meetings/Presentations
4.5	Provide documentation, outreach materials, marketing efforts	ongoing	List of Outreach Publications and Presentations

Deliverables

1	Hosting Infrastructure Description	Anually	Apr 2018
2	Summary of use and utility of CAIDA's IMPACT data	Annually	Summary

TTA #2: Developing HI-CUBE: Hub for Internet Incident Investigation

Subtask	Description	Projected Timeline	Status
1. Development of web services and visual interfaces
1.1	Extend the authorization functionality of the current Charthouse web application to support fine-grained data access control	Year 1	done
1.2	Develop a management interface for users, groups and shared data	Year 2	ongoing
2. Design and development of software infrastructure for data storage, query, and transformation
2.1	Replace our monolithic time series database (DBATS) with a distributed database for time-series analytics (e.g. Apache Kudu, Influx DB Enterprise version)	Year 2	done
2.2	Replace the Graphite back-end that queries DBATS with a data analytics query engine	Year 2	ongoing
3. Integration and testing of HI-CUBE system in operational research environments
3.1	Acquire the hardware needed for hosting the service	Year 1	done
3.2	Migrate current databases and integrate additional datasets developed	Year 1	done
3.3	Deploy, benchmark, and tune the upgraded components of the infrastructure for big data analytics	Year 2	ongoing
3.4	Migrate the time series currently stored in DBATS into the new system	Year 2	ongoing
3.5	Deploy the query engine and the HTTP query server	Year 2	ongoing
3.6	Integrate and test the data analytics query engine	Year 2
4. Community outreach and service
4.1	Collect feedback during meetings and presentations	ongoing
4.2	Interact with the users of the platform to better focus our efforts on the needs of the community of cybersecurity researchers and analysts	ongoing
4.3	Present the HI-CUBE platform in one or more of our CAIDA workshops	May 2020	done

Milestones

1	Deploy SSD cluster machine, storage server, and disk tray	Jun 2018	done
2	Deploy Web Application Server	Sep 2018	done
3	Extend the authentication and authorization functionality	Sep 2018	done
4	Release alpha version of prototype website	Sep 2018	done
5	Migrate time series currently stored in DBATS	Mar 2019	ongoing
6	Deploy second Web Application Server	Mar 2019	done
7	Complete the development of a distributed database for time-series analytics	Mar 2019	done
8	Develop management interfaces for users, groups and shared data	Mar 2019	ongoing
9	Deploy second SSD cluster machine	May 2019	done
10	Complete the tuning of the distributed database system	Jul 2019	ongoing
11	Deploy the query engine and the HTTP query server	Aug 2019
12	Complete the development of the Data analytics query engine	May 2020	ongoing
13	Release beta version of prototype web site	May 2020	done
14	Release as open source the distributed time-series database and query engine	May 2020

Deliverables

1	Capability Design Plan	Feb 2018	done
2	Demonstrate web service at PI Meetings	Triannually	done
3	Open source HI-CUBE software	May 2020	ongoing

Acknowledgement of awarding agency's support

This material is based on research sponsored by Air Force Research Laboratory under agreement number FA8750-18-2-0049. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of Air Force Research Laboratory or the U.S. Government.