EVPN for Layer 2 stretch between Data Centers Pt.1

EVPN (Enhanced VPN or Ethernet VPN) is a great technology for stretching Layer 2 between Data Centers (aka Data Center Interconnect or DCI). It uses MP-BGP for control plane exchange of Tenant information and mac-addresses. Data Plane traffic is tunneled inside a tunneling protocol such as MPLS, VXLAN or PBB. EVPN is used in lieu of VPLS because it provides better control over BUM traffic (Broadcast, Unknown Unicast, and Multicast). It also supports the ability to forward traffic over multiple active paths and Multihoming. EVPN used over MPLS provides the benefits of traffic engineering and fast convergence.

In part I, I've created a small single-homed setup to show how this works.

 

The first step is to create the trunk port facing the Leaf switch. The leaf switch is a standard TOR switch with no special config.

set interfaces et-2/2/1 description TO-LEAF1

set interfaces et-2/2/1 flexible-vlan-tagging
set interfaces et-2/2/1 encapsulation flexible-ethernet-services
set interfaces et-2/2/1 unit 100 encapsulation vlan-bridge
set interfaces et-2/2/1 unit 100 vlan-id 100
 

I created a sub-interface and placed it into a routing instance.

set routing-instances evpn100 instance-type evpn

set routing-instances evpn100 vlan-id 100
set routing-instances evpn100 interface et-2/2/1.100
set routing-instances evpn100 route-distinguisher 4.4.4.4:100
set routing-instances evpn100 vrf-target target:65000:100
set routing-instances evpn100 protocols evpn interface et-2/2/1.100
set routing-instances evpn100 protocols evpn label-allocation per-instance

Instance configuration looks like a normal VPLS configuration except for the instance-type and evpn protocol parameters.

Next I configure BGP to exchange control plane info.

set protocols bgp group IBGP type internal

set protocols bgp group IBGP local-address 4.4.4.4
set protocols bgp group IBGP family inet unicast
set protocols bgp group IBGP family evpn signaling
set protocols bgp group IBGP neighbor 5.5.5.5

A new address family is used called evpn.

After that, the normal MPLS, your flavor of MPLS signaling and IGP protocol configuration is used as well as the Core MPLS facing interfaces.

set protocols mpls interface all

set protocols mpls interface fxp0.0 disable
set protocols mpls interface lo0.0
set protocols isis interface all
set protocols isis interface fxp0.0 disable
set protocols isis interface lo0.0 passive
set protocols ldp interface all
set protocols ldp interface fxp0.0 disable
set protocols ldp interface lo0.0

set interfaces et-2/0/0 description TO-CORE1
set interfaces et-2/0/0 unit 0 family inet address 192.168.24.4/24
set interfaces et-2/0/0 unit 0 family iso
set interfaces et-2/0/0 unit 0 family mpls


Once configured, MP-BGP exchanges "control plane" information.

# run show bgp summary
Groups: 1 Peers: 1 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
inet.0              
                       0          0          0          0          0          0
bgp.evpn.0          
                       2          2          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
5.5.5.5               65000        137        136       0       0       57:42 Establ
  inet.0: 0/0/0/0
  bgp.evpn.0: 2/2/2/0
  evpn100.evpn.0: 2/2/2/0
  __default_evpn__.evpn.0: 0/0/0/0


# run show route receive-protocol bgp 5.5.5.5

inet.0: 24 destinations, 24 routes (24 active, 0 holddown, 0 hidden)

inet.3: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)

iso.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)

mpls.0: 15 destinations, 15 routes (15 active, 0 holddown, 0 hidden)

bgp.evpn.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)
  Prefix          Nexthop           MED     Lclpref    AS path
  2:5.5.5.5:100::100::00:00:05:ed:ae:01/304                  
*                         5.5.5.5                      100        I
  3:5.5.5.5:100::100::5.5.5.5/304                  
*                         5.5.5.5                      100        I

evpn100.evpn.0: 4 destinations, 4 routes (4 active, 0 holddown, 0 hidden)
  Prefix          Nexthop           MED     Lclpref    AS path
  2:5.5.5.5:100::100::00:00:05:ed:ae:01/304                  
*                         5.5.5.5                      100        I
  3:5.5.5.5:100::100::5.5.5.5/304                  

You can also check the status of the EVPN and it's mac table


# run show evpn mac-table

MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static, C - Control MAC
           SE - statistics enabled, NM - non configured MAC, R - remote PE MAC)


Ethernet switching table : 2 entries, 2 learned
Routing instance : evpn100
    Vlan                MAC                 MAC         Age    Logical                NH        RTR
    name                address             flags              interface              Index     ID
    __evpn100__         00:00:05:ed:ad:49   D             -   et-2/2/1.100        
    __evpn100__         00:00:05:ed:ae:01   DC            -   pip-13.010010000000    1048577   1048577


This shows you locally learned macs and macs learned over the WAN.

# run show evpn statistics   
Instance: evpn100
   Local interface: et-2/2/1.100, Index: 338
     Broadcast packets:                     1
     Broadcast bytes  :                    60
     Multicast packets:                     0
     Multicast bytes  :                     0
     Flooded packets  :                  4240
     Flooded bytes    :               6341604
     Unicast packets  :               3292539
     Unicast bytes    :            3528822524
     Current MAC count:                     1 (Limit 0)

In Part II I'll go more into configuring Gateway information to prevent the trombone effect.

Use (EVPN) Ethernet Virtual Private Network for Data Center Interconnections (DCI)

As Enterprises build Data Centers at different locations for disaster recovery and traffic distribution, there is a need to interconnect them transparently. Stretching Layer 2 across a WAN poses some challenges.

1) Workload Mobility aka VM migration from one DC to another.

2) Fast convergence in a multi homed environment.

3) Load balancing across multiple active paths between data centers.

The Trombone effect when migrating VMs across a WAN.

When VM1 is moved from one Hypervisor in DC1 to the other Hypervisor in DC2, the default GW for VM1 still resides on DC1. When VM1 sends traffic to VM2, the traffic will traverse the core before tromboning back to DC2.

EVPN solves this.  EVPN is a similar technology to VPLS except that mac addresses are learned and exchanged through the control plane using BGP as the transport protocol.  A new BGP family is introduced called EVPN.

bgp {
    group IBGP {
        local-address 1.1.1.1;        
        family evpn {
            signaling;  
        }
        neighbor 2.2.2.2;
    }
}

First an understanding of how EVPN works.

In a multi-tenant environment, each tenant will correspond to an EVPN instance (EVI). Route Distinguishers are used to distinguish between each EVI and Route Targets are used to share learned mac addresses between EVIs.

For mac learning, each PE router snoops for DHCP and/or ARP(IPv4)/ND(IPv6) packets for a particular EVI. The PE can then advertise the locally learned MAC address to remote PE nodes through MP-iBGP. MAC addresses are aggregated and a MAC prefix is advertised rather than advertising every single MAC address, thus allowing the ability to scale thousands of MAC addresses.  When a remote PE receives this bgp update it will extract the mac address and build a table with the next-hop pointing to the LSP of the advertising PE. Because this is BGP, policies can be created to filter and manipulate forwarding decisions.

When a local PE router sees an ARP request for an IP address and if the PE router has the MAC address binding for that IP address across the wan, the PE router performs a proxy ARP and responds to the ARP Request and can make the forwarding decision locally.  This reduces (BUM) flooding (Broadcast, Unknown Unicast and Multicast) across WAN links.

Gateway IP and MAC addresses syncing in EVPN allows the host to use the nearest gateway to route traffic. You do this by creating IRBs on both PEs using different GW IP addresses. To accomplish this IRBs (IP  + MAC addresses) are advertised using a BGP extended community. When VM1 migrates to DC2, it sends packets to the mac address associated to GW IP address of DC1. The IRB in DC2 notices that the destination mac address for these packets is across the WAN, so it does the routing locally. When the arp entry for the GW in VM1 expires, the VM will arp again and the IRB in DC2 will send a reply to VM1 with it's updated mac address.

Another thing that happens when VM migration is performed in an EVPN network, the MAC address of the VM is now advertised in DC2, the PE in DC2 updates their mac table table while the PE in DC1 withdraws the entry.

To address fast convergence in a multi homed environment, a concept called an Ethernet Segment is introduced. The set of links connecting to two or more local PE routers are called an Ethernet Segment. Each segment has an unique identifier called an ESI. An ethernet tag is also used to identify each broadcast domain such a vlan. When an Ethernet segment fails, the local PE withdraws the corresponding Ethernet "route" from BGP which triggers all remote PE routers to update their forwarding tables to update the corresponding next-hop to the backup PE.

EVPN introduces Split Horizon. BUM flooding aka, Broadcast, Unknown unicast or Multicast traffic are encapsulated in a MPLS packet with the Ethernet Segment Identifier. This allows the Egress PE to make a forwarding decision and prevents loops, because the PEs know where the packet originated from.
This in turn makes it possible to forward traffic over multiple active links through the WAN and allows for the ability to load balance.

With these advantages EVPN makes it a viable choice for interconnecting Data Centers.