[lighty.io] BGP EVPN Route-Reflector (2021)

In our previous blog post, we have introduced you to a Border Gateway Protocol Route-Reflector (BGP-RR) function in SDN controller based on lighty.io. In this article, we’re going to extend the BGP function of an SDN controller with an EVPN extension in the BGP control plane.

Functionality

This article will discuss BGP-EVPN functions in an SDN controller and how the lighty.io BGP function can replace existing legacy route-reflectors running in the service provider’s WAN/DC networks. BGP-EVPN provides:

• Advanced Layer 2 MAC and Layer 3 IP reachability information capabilities in control-plane
• Route-Type 2: advertising MAC/IP address, instead of traditional MAC learning mechanisms
• Route-Type 5: advertising the IP prefix subnet prefix route

We’re going to show you a BGP-EVPN IP subnet routing use-case

A BGP-EVPN control plane can also co-exist with various data-planes, such as MPLS, VXLAN, and PBB.

Use-case: Telecom Data-Center

In this blog, we’re going to show you the BGP-EVPN control plane working together with the VXLAN data plane. The perfect use case for this combination would be a telecom data center.

Virtual Extensible LAN (VXLAN) is an overlay technology for network virtualization. It provides Layer 2 extension over a shared Layer 3 underlay infrastructure network, by using the MAC address in an IP/User Datagram Protocol (MAC in IP/UDP) tunneling encapsulation. The initial IETF VXLAN standards defined a multicast-based flood-and-learn VXLAN without a control plane.

It relies on data-based flood-and-learn behavior for remote VXLAN tunnel endpoint (VTEP) peer-discovery and remote end-host learning. BGP-EVPN, as the control plane for VXLAN, overcomes the limitations of the flood-and-learn mechanism.

Test Bed

In this demo, we will use:

• five Docker containers & three Docker networks.
• Docker auto-generated user-defined bridge networks with mask /16
• Arista’s cEOS software, as we did in our previous demo

Remember, that an Arista cEOS switch creates an EtX port when starting up in the container, which is bridged to the EthX port in Docker.

These auto-generated EtX ports are accessible and configurable from cEOS Cli and on start are in default L2 switching mode. This means they don’t have an IP address assigned.

Well, let’s expand our previous demo topology with few more network elements. Here is a list of Docker containers used in this demo:

• leaf1 & leaf2: WAN switch & access/node
• host1 & host2: Ubuntu VM
• BGP-RR: BGP-EVPN Route Reflector

Here is a list of Docker user-defined networks used in this demo:

• net1 (172.18.0.0/16): connects leaf1 & host1
• net2 (172.19.0.0/16): connects leaf2 & host2
• net3 (172.20.0.0/16: connects leaf1, leaf2 & bgp-rr

Our Goal: Routing

At the end of this blog, we want to be able to reach IP connectivity between virtual machine host1 and host2. For that, we need BGP to advertise loopback networks and VLAN information between nodes.

In this example, we are using one AS-50.

To demonstrate route-reflector EVPN functionality leaf1 & leaf2 doesn’t make an IBGP pair but creates a pair with lighty-BGP instead. This will act as a route reflector. In VxLAN configuration we don’t set up flood vtep. This information should redistribute Route Routing to peers.

The container with lighty-BGP MUST NOT be used as a forwarding node since it doesn’t know the routing table.

Configuration

This demo configuration is prepared and tested on Ubuntu 18.04.2.

Docker Configuration

Before you start, please make sure that you have Docker (download instructions, use version 18.09.6 or higher) & Postman downloaded and installed.

1. Download lighty-BGP Docker image. PANTHEON.tech has its own

https://hub.docker.com/u/pantheontech

sudo docker pull pantheontech/lighty-rr:9.2.0-dev

2. Download the Docker image for Arista cEOS (v. 4.26.1F)

sudo docker import cEOS-lab.tar.xz ceosimage:4.26.1F

3. Download the Ubuntu image from DockerHub

sudo docker pull ubuntu:latest

4. Check the Docker images, successfully installed in the repository

sudo docker images

Preparing the Docker Environment

1. Create Docker networks

sudo docker network create net1
sudo docker network create net2
sudo docker network create net3

2. Check all Docker networks, that have been created

sudo docker network ls

3. Create containers in Docker

sudo docker create --name=bgp-rr --privileged -e INTFTYPE=eth -it pantheontech/lighty-rr:9.2.0-dev
sudo docker create --name=leaf1 --privileged -e INTFTYPE=eth -e ETBA=1 -e SKIP_ZEROTOUCH_BARRIER_IN_SYSDBINIT=1 -e CEOS=1 -e EOS_PLATFORM=ceoslab -e container=docker -i -t ceosimage:4.26.1F /sbin/init systemd.setenv=INTFTYPE=eth systemd.setenv=ETBA=1 systemd.setenv=SKIP_ZEROTOUCH_BARRIER_IN_SYSDBINIT=1 systemd.setenv=CEOS=1 systemd.setenv=EOS_PLATFORM=ceoslab systemd.setenv=container=docker
sudo docker create --name=leaf2 --privileged -e INTFTYPE=eth -e ETBA=1 -e SKIP_ZEROTOUCH_BARRIER_IN_SYSDBINIT=1 -e CEOS=2 -e EOS_PLATFORM=ceoslab -e container=docker -i -t ceosimage:4.26.1F /sbin/init systemd.setenv=INTFTYPE=eth systemd.setenv=ETBA=1 systemd.setenv=SKIP_ZEROTOUCH_BARRIER_IN_SYSDBINIT=1 systemd.setenv=CEOS=2 systemd.setenv=EOS_PLATFORM=ceoslab systemd.setenv=container=docker 
sudo docker create --privileged --name host1 -i -t ubuntu:latest /bin/bash
sudo docker create --privileged --name host2 -i -t ubuntu:latest /bin/bash

4. Connect containers to Docker networks

sudo docker network connect net1 leaf1
sudo docker network connect net1 host1
sudo docker network connect net2 leaf2
sudo docker network connect net2 host2
sudo docker network connect net3 bgp-rr
sudo docker network connect net3 leaf1
sudo docker network connect net3 leaf2

5. Start all containers

sudo docker start leaf1
sudo docker start leaf2
sudo docker start host1
sudo docker start host2
sudo docker start bgp-rr

6. Enable permanent IPv4 forwarding in cEOS containers

sudo docker exec -it leaf1 /sbin/sysctl net.ipv4.conf.all.forwarding=1
sudo docker exec -it leaf2 /sbin/sysctl net.ipv4.conf.all.forwarding=1

7. Check, whether all Docker containers have started successfully

sudo docker container ls

Optional: Use this, if you’re looking for detailed information about running Docker containers (X is replaced by device/host number)

sudo docker container inspect [leaf[X] | bgp-rr | host[X]]

Preparing Ubuntu Environment

1. Get into the machine (X is replaced by device/host number)

sudo docker exec -it host[X] bash

2. Update the machine

apt-get update

3. Install the required packages

apt-get install iproute2
apt-get install iputils-ping

4. Exit the Docker Container (CTRL+D). Repeat steps 2 & 3.

Arista cEOS Switch configuration

Now, we will configure Arista cEOS switches. We will split the configuration of Arista cEOS Switches into several steps.

Click here for full configurations of Arista switches ‘leaf1‘ & ‘leaf2‘.

Ethernet interfaces & connectivity check

1. Go into the Arista switch leaf1

sudo docker exec -it leaf1 Cli

2. Set Privilege, and go to configure-mode

enable
configure terminal

3. Setup the switch’s name

hostname leaf1

4. Setup Ethernet interface. If you use more devices than your devices could be connected to another Ethernet

interface ethernet 2
no switchport
ip address 172.20.0.2/16

5. Check if BGP-RR is reachable from the configured interface.

• When you can’t ping ‘BGP-RR’, check if ‘leaf1′ and ‘BGP-RR’ are located in the same Docker network, or delete the previous step and try it on another Ethernet interface.

ping 172.20.0.4 source ethernet2

6. Repeat Step 5 for ‘leaf2′ & go into the Arista switch leaf2

sudo docker exec -it leaf2 Cli
enable
config t
hostname leaf2
interface ethernet 2 
no switchport
ip address 172.20.0.3/16
ping 172.20.0.4 source ethernet2

Configuring the Border Gateway Protocol

We will have identical configurations for ‘leaf1′ & ‘leaf2′. Exceptions will be highlighted in the instructions below.

1. Enable BGP in Arista switch

• If you are still in the previous settings interface, go to the root of the Arista configuration by repeating the “exit” command.

service routing protocols model multi-agent
ip routing

2. Setup

• For ‘leaf2’, use the Router-ID ‘router-id 172.20.0.3‘

router bgp 50
router-id 172.20.0.2
neighbor 172.20.0.4 remote-as 50
neighbor 172.20.0.4 next-hop-self
neighbor 172.20.0.4 send-community extended
redistribute connected
redistribute attached-host

3. Setup EVPN in BGP

address-family evpn
neighbor 172.20.0.4 activate

Configuring VxLAN Interface & VLAN

We will have identical configurations for leaf1 & leaf2. Exceptions will be highlighted in the instructions below.

1. Enable VLAN with ID 10.

• Make sure that this command is typed in the root of Arista and not in BGP
• If you are still in the BGP configuration, use the command ‘exit’

vlan 10

2. Configure loopback 0, which will be used as a VTEP (VxLAN tunnel endpoint) for VxLAN.

• In ‘leaf2’, use IP ‘10.10.10.2/32’, instead of IP ‘10.10.10.1/32’

interface loopback 0
ip address 10.10.10.1/32

3. Configure VxLAN Interface

• Here we’ll set up loopback 0 as a VTEP and configure VNI (VXLAN Network Identifier) to 3322.

interface Vxlan1
vxlan source-interface Loopback0
vxlan udp-port 4789
vxlan vlan 10 vni 3322
vxlan learn-restrict any

4. Assign Ethernet interface to VLAN

interface Ethernet 1
switchport mode access
switchport access vlan 10

5. Share loopback 0 to BGP-RR

• In ‘leaf2‘, use IP ‘10.10.10.2/32’ instead of ‘10.10.10.1/32’

router bgp 50
address-family ipv4
network 10.10.10.1/32

6. Configure VLAN in BGP

• Here we share the information about VLAN to BGP-RR

router bgp 50
vlan 10
rd 50:3322
route-target both 10:3322
redistribute learned

7. Save your configuration with the ‘wr‘ command in both Arista devices and restart them with the command:

sudo docker restart leaf1 leaf2

lighty.io & BGP Route Reflector

In this part, we will add the Border Gateway Protocol configuration into the lighty.io BGP.

There is a lot to configure, so crucial parts are commented to break it down a little.

If we want to see the logs from lighty.io, we can attach them to the started container:

sudo docker attach bgp-rr

We can start the BGP-RR container with the command:

sudo docker start bgp-rr --attach

to see logs from the beginning. Afterward, send a PUT request to BGP-RR. We should see the following messages in the logs.

More RESTCONF commands can be found here.

Verify device state

Now, we will check if all configurations were set up successfully. We will also check if VxLAN is created and the Virtual PCs can ‘ping’ each other.

1. Check if EVPN BGP peering is established

leaf1(config)#sh bgp evpn summary
BGP summary information for VRF default
Router identifier 172.20.0.2, local AS number 50
Neighbor Status Codes: m - Under maintenance
  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc
  172.20.0.4       4  50                 3         6    0    0 00:00:09 Estab  0      0

leaf2(config)#sh bgp evpn summary
BGP summary information for VRF default
Router identifier 172.20.0.3, local AS number 50
Neighbor Status Codes: m - Under maintenance
  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc
  172.20.0.4       4  50               267       315    0    0 00:01:16 Estab  1      1

If your devices are in the state ‘Connected‘ or ‘Active‘, then you have checked this right after you sent a request to lighty.io. Usually, it takes, at most, one minute to establish a connection.

If you still see this state, then there could be something wrong with the BGP configuration. Please check your configuration in Arista CLI, by typing the command ‘show running-config‘ and compare it with the full Arista configuration above.

After you verify the Arista configuration, then there could be a problem in the BGP-RR container. This can be fixed by restarting the BGP-RR container.

2. Check ip route for available loopbacks from other devices

leaf1(config)#sh ip route
VRF: default
Codes: C - connected, S - static, K - kernel,
       O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type2, B I - iBGP, B E - eBGP,
       R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
       O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
       NG - Nexthop Group Static Route, V - VXLAN Control Service,
       DH - DHCP client installed default route, M - Martian,
       DP - Dynamic Policy Route
 
Gateway of last resort is not set
 
 C      10.10.10.1/32 is directly connected, Loopback0
 B I    10.10.10.2/32 [200/0] via 172.20.0.3, Ethernet2
 C      172.20.0.0/16 is directly connected, Ethernet2

leaf2(config)#sh ip route
VRF: default
Codes: C - connected, S - static, K - kernel,
       O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type2, B I - iBGP, B E - eBGP,
       R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
       O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
       NG - Nexthop Group Static Route, V - VXLAN Control Service,
       DH - DHCP client installed default route, M - Martian,
       DP - Dynamic Policy Route
 
Gateway of last resort is not set
 
 B I    10.10.10.1/32 [200/0] via 172.20.0.2, Ethernet2
 C      10.10.10.2/32 is directly connected, Loopback0
 C      172.20.0.0/16 is directly connected, Ethernet2

3. Check the VxLAN interface, if it creates and contains VTEP

leaf1#sh interfaces vxlan 1
Vxlan1 is up, line protocol is up (connected)
  Hardware is Vxlan
  Source interface is Loopback0 and is active with 10.10.10.1
  Replication/Flood Mode is headend with Flood List Source: EVPN
 Remote MAC learning via EVPN
  VNI mapping to VLANs
  Static VLAN to VNI mapping is
    [10, 3322]      
  Note: All Dynamic VLANs used by VCS are internal VLANs.
        Use 'show vxlan vni' for details.
  Static VRF to VNI mapping is not configured
  Headend replication flood vtep list is:
    10 10.10.10.2    
  VTEP address mask is None

leaf2(config)#sh interfaces vxlan 1
Vxlan1 is up, line protocol is up (connected)
  Hardware is Vxlan
  Source interface is Loopback0 and is active with 10.10.10.2
  Replication/Flood Mode is headend with Flood List Source: EVPN
 Remote MAC learning via EVPN
  VNI mapping to VLANs
  Static VLAN to VNI mapping is
    [10, 3322]      
  Note: All Dynamic VLANs used by VCS are internal VLANs.
        Use 'show vxlan vni' for details.
  Static VRF to VNI mapping is not configured
  Headend replication flood vtep list is:
    10 10.10.10.1    
  VTEP address mask is None

If you don’t see IP in the section ‘Headend replication flood vtep list is:‘, then the BGP-RR container is not started correctly. This problem can be fixed by removing the BGP-RR container and starting it again.

Restarting BGP-RR container

1. Stop the container

sudo docker stop bgp-rr

2. Remove BGP-RR container

sudo docker rm bgp-rr

3. Create a new container

sudo docker create --name=bgp-rr --privileged -e INTFTYPE=eth -it pantheontech/lighty-rr:9.2.0-dev

4. Connect BGP-RR to docker network

sudo docker network connect net3 bgp-rr

5. Start the container again

sudo docker start bgp-rr

Optional: If you want to see logs from light.io, attached them to the container:

sudo docker attach bgp-rr

Testing IP Connectivity

If everything worked out, we can test IP connectivity in a virtual PC.

1. Open Virtual PC host1

sudo docker exec -it host1 bash

2. Setup IP address for this device

ip addr add 31.1.1.1/24 dev eth1

3. Perform the same configuration at host2

sudo docker exec -it host1 bash
ip addr add 31.1.1.2/24 dev eth1

4. Try to ping host2 to host1

ping 31.1.1.1

root@e344ec43c089:/# ip route
default via 172.17.0.1 dev eth0
31.1.1.0/24 dev eth1 proto kernel scope link src 31.1.1.2
172.17.0.0/16 dev eth0 proto kernel scope link src 172.17.0.5
172.19.0.0/16 dev eth1 proto kernel scope link src 172.19.0.3
 
root@e344ec43c089:/# hostname -I
172.17.0.5 172.19.0.3 31.1.1.2
 
root@e344ec43c089:/# ping 31.1.1.1
PING 31.1.1.1 (31.1.1.1) 56(84) bytes of data.
64 bytes from 31.1.1.1: icmp_seq=1 ttl=64 time=114 ms
64 bytes from 31.1.1.1: icmp_seq=2 ttl=64 time=55.5 ms
64 bytes from 31.1.1.1: icmp_seq=3 ttl=64 time=53.0 ms
64 bytes from 31.1.1.1: icmp_seq=4 ttl=64 time=56.1 ms
^C
--- 31.1.1.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 53.082/69.892/114.757/25.929 ms

When we go back to the Arista switch, we can check routed MAC address information.

leaf1#sh mac address-table
          Mac Address Table
------------------------------------------------------------------
 
Vlan    Mac Address       Type        Ports      Moves   Last Move
----    -----------       ----        -----      -----   ---------
  10    0242.211d.8954    DYNAMIC     Et1        1       0:00:54 ago
  10    0242.8b29.b7ea    DYNAMIC     Vx1        1       0:00:40 ago
  10    0242.ac12.0003    DYNAMIC     Et1        1       0:00:14 ago
  10    0242.ac13.0003    DYNAMIC     Vx1        1       0:00:13 ago
  10    ce9a.ca0c.88a1    DYNAMIC     Et1        1       0:00:54 ago
Total Mac Addresses for this criterion: 5
 
          Multicast Mac Address Table
------------------------------------------------------------------
 
Vlan    Mac Address       Type        Ports
----    -----------       ----        -----
Total Mac Addresses for this criterion: 0

leaf2#sh mac address-table
          Mac Address Table
------------------------------------------------------------------
 
Vlan    Mac Address       Type        Ports      Moves   Last Move
----    -----------       ----        -----      -----   ---------
  10    0242.211d.8954    DYNAMIC     Vx1        1       0:00:48 ago
  10    0242.8b29.b7ea    DYNAMIC     Et1        1       0:01:03 ago
  10    0242.ac12.0003    DYNAMIC     Vx1        1       0:00:22 ago
  10    0242.ac13.0003    DYNAMIC     Et1        1       0:00:22 ago
  10    ce9a.ca0c.88a1    DYNAMIC     Vx1        1       0:00:48 ago
Total Mac Addresses for this criterion: 5
 
          Multicast Mac Address Table
------------------------------------------------------------------
 
Vlan    Mac Address       Type        Ports
----    -----------       ----        -----
Total Mac Addresses for this criterion: 0

Conclusion

We have successfully shown the lighty.io BGP functionality, which can replace legacy Route-Reflectors. This situation can be applied to telecom data centers and other use-cases. It demonstrates lighty.io’s versatility and usability. Contact us for more information!

Peter Šuňa & Peter Lučanský

You can contact us at https://pantheon.tech/

Explore our Pantheon GitHub.

Watch our YouTube Channel.