Building bulletproof internet exchange points with EVPN

In many ways, internet exchange providers (IXPs) are the unsung heroes of the global internet. Among other things, IXPs provide global interconnection services (internet peering) and cloud access to internet service providers (ISPs). IXPs are thus a crucial part of the foundational infrastructure for the internet, handling massive traffic loads efficiently, reliably and securely.

Figure 1: IXPs play a central role in the internet by handling massive traffic loads from ISPs and content providers

IXPs have built large global networks to provide peering connectivity between ISPs. These networks have been constructed as homogenous layer 2 broadcast domains. While a single layer 2 broadcast domain founded on Ethernet virtual local area network (VLAN) switching or Virtual Private LAN Service (VPLS) is conceptually uncomplicated, technical “cracks” began to appear in this infrastructure as large-scale operations stressed network resources in practice.

For example, the venerable Ethernet practice of flooding-and-learning to construct forwarding tables does not scale well for peering applications. In particular, there was an exposure to excessive address resolution protocol/neighbor discovery (ARP/ND) traffic on ISP routers. Resiliency features, such as all-active multi-homing, were not available in VPLS. Advanced security procedures, providing protection against certain denial-of-service attack exposures were also unavailable, but increasingly needed.

These are exactly the kinds of shortcomings that Ethernet Virtual Private Networks (EVPNs) can mitigate as a universal VPN architecture for scalable, secure and resilient networking with highly automated operations. An EVPN can be used to support a number of VPN types ranging from basic pseudowires to fully meshed layer 2 and layer 3 services, including multicast, all on a consistent, efficient and resilient architecture foundation.

Figure 2: EVPN can provide a universal VPN architecture for scalable, secure and resilient networking

An EVPN-based network infrastructure for IXPs retains the conceptual simplicity of the single layer 2 broadcast domain approach while introducing many of the attractive features of EVPN technology. EVPNs have significant advantages over traditional VLAN switching or VPLS in the context of IXP networks.

For example, EVPNs avoid the resource-intensive issues of traditional flood-and-learn networks. With EVPNs, reachability information such as layer 2 and layer 3 addresses is learned and distributed using a Multiprotocol BGP (MP-BGP) control plane, as shown in Figure 3. EVPN supports a strong multi-homing capability that provides rapid resiliency to link and node failure. In the case of all-active mode, dynamic load balancing across provider edge (PE) devices and links enables more efficient usage of network resources.

Figure 3: EVPNs learn and distribute reachability information using MP-BGP

Nokia was privileged to lead a team initiative within the Internet Engineering Task Force (IETF) to address specific IXP operational requirements. In particular, IXPs and attached ISP routers are historically vulnerable to ARP and ND flooding traffic. This traffic is processed at a high priority in ISP routers. In a VLAN switching or VPLS broadcast domain, it can swamp a router’s central control system (which must process these packets) and may cause cascading effects that potentially lead to severe service interruptions.

Causative and excessive ARP/ND traffic can be generated inadvertently, for example, when ISP routers are trying to resolve the IP address of an ISP router that has been removed from service. More seriously, this traffic can be created with malicious intent as a denial-of-service attack.

This IETF work resulted in RFC 9161 “Operational Aspects of Proxy ARP/ND in Ethernet Virtual Private Networks” to reduce the amount of Broadcast, Unknown Unicast and Multicast (BUM) traffic in the broadcast domain and provide protection against ARP/ND or IP spoofing. RFC 9161 makes use of some EVPN extensions specified in RFC 9047 “Propagation of ARP/ND Flags in an Ethernet Virtual Private Network” so that required ARP/ND information can be conveyed in EVPN MAC/IP Advertisement routes, and ARP/ND flooding can be practically removed from the broadcast domain.

We have built a comprehensive EVPN implementation on our routing platforms in support of VPN applications in mobile transport, WAN services and datacenter networks, and in mission-critical verticals.

It has been particularly gratifying to see the smooth migration from traditional VPLS to EVPN leveraging the integration mechanisms specified in RFC 8560 “Seamless Integration of EVPN with VPLS” and the successful deployment of EVPN by leading IXPs, such as DE-CIX. In this role, EVPN is helping to secure the resiliency, reliability and scalability of the infrastructure for the global internet.

For more insights about how Nokia routing platforms support EVPN, visit our EVPN web page and read our EVPN application note.