Atomised Routing - Final Report

Atomised Routing - Final Report - February 2004

Project Summary

Project Title: Atomised Routing

Organisation:CAIDA, NLnet Labs and Ripe NCC

Start Date: October 2002

End Date: October 2003

Principal Investigator: kc claffy, Patrick Verkaik, Andre Broido

Project URL: https://www.caida.org/funding/atomized_routing/

Overall Objective

In the atomised routing project we are researching and implementing modifications to BGP routing that aggregate prefixes into equivalence classes (policy atoms) based on common AS path from a given topological location. The motivation behind development of BGP atomisation mechanisms is to achieve potential savings in computation and communication costs (by absorbing routing dynamics of prefixes into coarser grained atoms), as well as reduction in BGP table size (there will be far fewer atoms than prefixes). The project proposal is at https://www.caida.org/funding/atomized_routing/proposal/index.xml and https://www.caida.org/funding/atomized_routing/proposal.pdf.

Refining and Releasing Atom Computation Scripts

Objective:

Documenting and releasing Andre Broido's Perl scripts for atoms computation.

Background:

The atoms computation scripts perform two tasks. They derive BGP statistics from the University of Oregon Route Viewsshow ip bgp tables, and they compute BGP policy atoms (groups of prefixes that share the same AS path in each BGP router). Earlier versions of the scripts were used for many of the results in papers authored by Andre Broido et al.:

Results:

We rewrote and code-reviewed the bulk of the original scripts for readability and easy usage, and provided usage documentation. We posted the results on the Web, and made announcements to a number of mailing lists. The scripts are available at https://www.caida.org/funding/atomized_routing/download/. Since then we have maintained the scripts based on bug reports and feature requests.

The output generated by the scripts include:

Atoms: a list of policy atoms, a mapping of each prefix to its atom, and a distribution of prefix counts per atom.
A rich set of statistics about ASes, prefixes, Route Views peers and AS paths.
Various AS graphs in text form, intended for further processing.
The original BGP table dump in a standard, easy-to-parse format. The scripts remove ambiguities and irrelevant information from this file, which will be beneficial for future scripts and programs.

Sample of Results:

A sample of the results of running the scripts on the Route Views dump of 1 February 2003 (noon) follows below:

Number of prefixes	125857
Maximum number of prefixes per peer in /0-/24 range	119400
Number of prefixes shared by 35 peers	113343
Number of ASes with positive indegree	14754
Number of ASes with positive outdegree	2353
Number of BGP atoms	29059
Number of origin-declared atoms	21771 (derived)

We computed the above results over the 35 Route Views peers that carry at least 114,000 prefixes with prefix length 24 or less. The full set of files for this run is also available. The number of origin-declared atoms is actually the number of distinct origin links (first link in the AS path) observed. Declared atoms are a fundamental part of the architecture we discuss below.

The scripts, written in Perl, produce many statistics. Geoff Huston has written an optimised C program that specialises in the task of computing atoms alone, and now includes atom statistics on http://bgp.potaroo.net/. In addition, Yehuda Afek's group at Tel Aviv University have been researching BGP atoms and have written an independent implementation of atoms computation.

Architecture of atomised routing

Objective:

To design a routing architecture based on atoms.

Results:

We have designed and prototyped a routing architecture based on the concept of the declared atom, an aggregation mechanism complementary to CIDR aggregation. In contrast to policy atoms, which are empirically observed, a declared atom is created by an AS as a container of prefixes. While the policy atom is a powerful analysis tool, declared atoms constitute a more practical alternative on which to base a routing architecture. We describe the architecture in detail in [techreport]. The design went through several stages and was driven by feedback from the operator and vendor community. Initially, our routing architecture focused on reducing per-prefix processing by a BGP router on a per-packet basis. However, feedback from the community at IETF-56 indicated that reducing the number of global routing table entries would be a more suitable application of the atoms concept. This led to the current incarnation of the atoms architecture. The current architecture also addresses business relationships among ISPs and their customers, robustness in the face of policy decisions by independent parties, potential inconsistencies as a result of independent signalling protocols, and the potential for misconfiguration.

Implementation of atomised routing

Objective:

To design, implement, and test a router based on the atomised routing architecture.

Results:

We have completed the prototype implementation of our routing architecture as modifications to the Zebra 0.93b code base. We will release the source code in February 2004. The result is a functional, atomised BGP PC router whose atoms extensions are configured using a modified version of the Zebra configuration language. ¹ We describe the implementation in more detail in [techreport].

We did not heavily optimise our implementation, placing more emphasis on portability. We implemented several functions that technically belong in the kernel (encapsulation and decapsulation) in user space for better portability, but at the cost of performance. Although our current solution relies on FreeBSD diverted sockets, porting it to platforms such as Linux should be relatively straightforward.

We have thoroughly tested the atomised routing architecture and router implementation in small topologies (around fourteen routers), using Marko Zec's vimage tool to create virtual PC routers. In doing so, we were able to provide the author of vimage with useful input regarding design improvements and bugs. We are satisfied with the level of robustness of our implementation. We have also performed limited testing of atomised routing in an environment that includes BGP and an intra-domain routing protocol (RIP).

Analysis

Objective:

To empirically discover statistics about policy atoms in the current interdomain routing system, and to verify or discount the effectiveness of an atomised routing architecture.

Background:

In collaboration with NLnet Labs' support for Patrick, CAIDA has contributed significant resources to the Atoms project; the analysis below was pursued by Young Hyun and summer Phd student Ruomei Gao, both supervised by CAIDA senior statisician Andre Broido.

Results: Analysis of Stability of Atoms

Young has performed detailed analysis of the statistics of the number of atoms over a time interval, and taxonomised the types of observed instabilities [techreport]. We will use these statistics as input to simulation, to gain an understanding of the feasibility of atomised routing.

Results: Analysis of Atoms and BGP Communities

Ruomei performed an analysis of the refinement of atoms by BGP communities. In this analysis we compared three ways of grouping prefixes: (a) by policy atom, (b) by community value, and (c) by policy atom and community value. We used the community values visible in Route Views MRT data. The results indicate that atoms group prefixes at a higher level of granularity than do the communities that appear in Route Views. Furthermore, as expected, a grouping by atoms and communities further refines the atom grouping, but not to a great extent (around 8% to 8.5%).

Results: Analysis of Provider-Declared Atoms

We developed a technique that allows ASes to delegate atom declaration to their immediate providers. In certain cases this technique would enable providers to group prefixes belonging to different customers into the same atom, thus reducing the number of atoms. We call these provider-declared atoms. Ruomei has performed an analysis that estimates the reduction in the number of declared atoms using this technique at around 48%. Young has subsequently investigated the stability of provider-declared atoms. We present results in our technical report [techreport].

Results: Traffic Breakdown by Network Objects

Young and Ruomei performed a traffic breakdown analysis by IP address, network prefix, atom, and AS. The results are part of a PAM 2004 paper: "Their share: diversity and disparity in IP traffic". The results are here.

Simulation

Objective:

To validate the correctness of our architecture on a large scale in a more realistic AS topology setting, and, for specific routing events, to compare convergence behaviour of our architecture with that of the current interdomain routing architecture.

Results:

We seeded a collaboration with Fontas Dimitropoulos and George Riley at Georgia Tech to perform simulations of atomised routing. Georgia Tech has developed BGP++, a BGP simulator based on Zebra 0.92a and implemented in C++. Among available BGP simulators such as SSFNet, BGP++ is unique in that it is able to perform simulations based on a working (Zebra) implementation, rather than requiring a reimplementation specifically for the simulator. However, the difference in version number of the Zebra code base (0.92a vs 0.93b) and implementation language (C++ vs C) meant that we spent a significant amount of time converging the BGP++ and atoms implementations. As part of this collaboration, Fontas spent a week at CAIDA, and Patrick visited Georgia Tech for two weeks. As a result of these visits, we were able to significantly extend the BGP++ simulator functionality, and in the atoms project we are now capable of performing large-scale simulations of our architecture. With the help of Ruomei Gao, also at Georgia Tech, we have generated simulation topologies based on a Route Views AS-level topology. We have performed several simulator test runs and are ready to commence actual simulations. We expect to complete these by April 2004.

Outreach and Feedback

Objective:

Publishing results and obtaining feedback about atomised protocols and implementation.

Presentations

We gave presentations at:

IETF 56 (slides)
RIPE 45 (slides)
Internetworking 2003 Workshop (slides)
CAIDA/WIDE workshop (slides)

Feedback

We consulted and obtained valuable feedback from a number of experts, summarised in the following documents:
- feedback from IETF 55
- feedback from IETF 56
This feedback from the operational community has influenced our thinking about how to progress. The future viability of this work relies on recognition of the following priorities:
- focus on the routing system as a whole, rather than attempting to apply the atoms concept to save on per-prefix processing by a BGP router on a per-packet basis.
- focus on the convergence behaviour and dynamics of the routing system rather than on BGP table size alone. ² Our architecture addresses all three issues.
- explicitly support interdomain routing policy within the architecture. For example, our architecture recognises common business relationships among providers, customers, and peers. Also, we have developed several ideas about how to contain the impact of traffic engineering practices to a limited area, rather than allowing it to spread globally.
- recognise that network operators will see more value in declared atoms than policy atoms as the basis of a routing architecture. Declaring a set of prefixes to be equivalent is a more scalable and manageable technique than empirically determining the equivalence of prefixes.
- recognise the importance of potential misconfiguration of declared atoms. Misconfigurations can lead to blackholing or looping of traffic.
- prove or demonstrate a guarantee of consistency between BGP and the atom membership protocol.
- assess the potential benefits of declaring atoms not at the originator of the prefixes, but further away, e.g. at a provider.
- address the concern that atoms turn out to be smaller, and the number of atoms higher, once our architecture additionally takes into account the presence of non-AS path attributes. We have carried out some preliminary analysis in this area (see the analysis on BGP communities) which indicates that the BGP communities we observe in Route Views do not refine atoms to a great degree. We also address this issue in our technical report [techreport].
The Site Multihoming in IPv6 (multi6) working group of the IETF studies related issues from the perspective of multihoming. We compared our approach with the various approaches described by multi6 drafts, and report our findings here. Our approach is significantly different from those taken by the multi6 working group. In particular, we have tried to avoid departing from the current routing and addressing model too much. While a number of multi6 approaches are architecturally 'cleaner' than our architecture, it remains to be seen whether a clean architecture exists that solves the multihoming problem and that the ISP community finds acceptable.
We set up a mailing list for the purpose of discussing atoms at: http://login.caida.org/mailman/listinfo/atoms.

Miscellaneous

We have provided valuable feedback on Internet drafts such as David Meyer's Internet draft draft-ietf-grow-collection-communities (currently located here), draft-shen-nhop-fastreroute (current location), draft-hsmit-shen-mpls-igp-spf (current location), and draft-agarwal-bgp-proxy-community (current location). Patrick did a review for Transactions on Networking of a paper on interdomain routing architecture.

BGP Bibliography

Objective:

Providing the community with an online annotated list of BGP related research.

Results:

We created a BGP bibliography which has since evolved into a wider networking bibliography located at https://www.caida.org/publications/bib/networking/.

References

[techreport] Patrick Verkaik, Andre Broido, kc claffy, Ruomei Gao, Young Hyun, Ronald van der Pol, "Beyond CIDR Aggregation," CAIDA TR-2004-1, Feb. 2004.

Notes

1 Before arriving at the current version, we also had a working implementation of the previous incarnation of the atoms architecture.

2 Network operators do not consider the increased memory usage of the global BGP table to be of great relevance, these days. However, a large BGP table tends to go hand in hand with an increase in the dynamics of the interdomain routing system, and may affect its stability. These are issues that operators care about more than sheer memory usage.