MPLS L3VPN Control and Data Plane

MPLS L3VPN is a type of PE-based L3VPN technology for service provider VPN solutions. It uses BGP to advertise VPN routes and uses MPLS to forward VPN packets on service provider backbones.
MPLS L3VPN provides flexible networking modes, excellent scalability, and convenient support for MPLS QoS and MPLS TE.

• Let consider the below network which have 2 customers at Site A and Site B with same IP address scheme. Both the customers site is connected to the ISP edge device which is called Provider Edge (PE) router in MPLS terminology. And the router between the PE router is called Provider (P) router.

• In order to provide the communication between the 2 customers of different site with same IP address scheme, we have to find a solution to separate both the site customer with each other.
• This mean that at PE router we will have 3 routing domains as:
• Main routing domain
• Blue Routing Domain for Site A customer
• Brown Routing Domain for Site B customer
• For this separation we will use the feature called VRF (VPN Routing and Forwarding).
• In this, a separate Routing Information Base (RIB) and Forwarding Information Base (FIB) will be created for each domain of site customer.
• But with this solution of VRF also not solve a problem, as we have only separated domain at PE router. But as the device in the path between the PE router for Blue and Brown site customer, still have single domain so we have to create a separate domain using VRF on each device in the path.
• But this is not a optimal solution, as for large ISP network, if we have 500-1000 customer, then we have to create 500-1000 vrf on each backbone P router, which will increase load on backbone devices and reduce the performance of network.
• So, we have to find a solution which reduce this complexity of creating vrf in backbone device, and probably a solution where packet is forwarded not based on IP.
• And what we have a technology where packet is been forwarded, not based on IP, and that is MPLS.
• So, with the MPLS network in backbone, packet will be forwarded from one PE router to other PE for the site customer separately, but now at the destination PE router packet arrived as a main route for main routing table instead of individual site customer route, which will be eventually drop as their will be no route for the customer in main routing table, and this is the problem of the data plane.
• Also, how the one PE router will know to which destination PE router to send.

So, we have 2 problems in all which are as below, and we need to solve them.

• MPLS Data Plane issue: Sort the traffic in appropriate Forwarding instance

• When Site A customer send the IP packet to PE1, PE1 will lookup in Blue VRF and forward it to P1 by attaching MPLS label.
• MPLS will forward this labelled packet to PE2.
• Now, PE2 will lookup in Main RIB, instead of Blue VRF for Site A customer. So, PE2 will drop the packet.
• Hence, MPLS will help us to forward the Blue packet from PE1 to PE2, but it will not help us to forward to respective customer site.
• This is data plane issue. So, let first see what is control plane issue?
• MPLS Control Plane issue: Exchange routes between PE devices for different routing instances

• As our data traffic for Site A customer is flowing from PE1 to PE2, so the control plane traffic will need to flow from PE2 to PE1
• Now, we have to communicate the routing information from PE2 to PE1
• There is multiple ways:
• Create separate VRF on Provider router. But this is not scalable.
• Built IBGP session between PE1 and PE2.
• So, when BGP send information from PE2 to PE1, that it has two routes, which are identical.       
• 192.168.2.0/24
• 192.168.2.0/24 
• So, BGP at PE1 will only select one route as a best path, using various attribute

MPLS L3VPN Control Plane Working

In order to solve the control plane issue, we have to run some protocol between the provider and customer edge router, which we referred here as PE-CE Routing protocol, so that customer routes are learned at PE router.
PE-CE Routing protocol which are supported can be as:

• Connected
• Static
• RIPv2
• OSPF
• ISIS
• EIGRP
• EBGP
• For IBGP, cisco doesn’t explicitly support configuring IBGP session between PE-CE router, but we can configure it and it will not give any error messages. But we have to do couple of extra things to make it works.

Now we have PE-CE routing protocol in place, and Site A customer routes has been learned at PE router in Blue vrf domain and Site B customer routes has been learned in Orange vrf domain.
VRF creation on PE router can be done in two ways:

• Old: Only support IPv4

ip vrf ‘vrf_name’

• New: Can support IPv4 and IPv6

vrf definition ‘vrf_name’
                    :
       address-family ipv4
                    :
       address-family ipv6

• Now, we have individual customer routes on PE routers. But now we are interested to get the routes flow from one PE to the other PE router.
• As our data path communication is from PE1 to PE2, so the control plane information will flow from PE2 to PE1.
• We can run the IGP protocol lie OSPF, EIGRP, etc, but we need protocol which can carry large number of routes, and we know we have BGP which we can use.
• We have to build a iBGP session between PE2 to PE1 , so that all customer routes which we will advertised in BGP will be forwarded to other end.
• But here the other problem, at PE1 BGP will get same IP for Site A and B customer.
• So, BGP will only select one route which is best based on various attribute and get it routed.
• But we don’t want these, as each site customer route are best for each site.
• As BGP doesn’t understand any load balancing or any vrf, it just understands the IPv4 address family, which we have configured.
• So instead of IPV4 if we provide a route of greater value and other address family to make the route unique, so that BGP can select both route as best.

Here the concept of Route Distinguisher (RD) come.

• A RD is a 64-bit unique identifier that is prepended to the 32- bit customer prefix or route learned from a CE router, which makes it a unique 96-bit address called VPNv4 address that can be transported between the PE routers in the MPLS domain.

Route Distinguisher can be represented in two ways:

• AS number: RD identifier  64001:100
• IP address: RD identifier  192.168.2.0:100

Depending upon the format use, we have different ranges available for RD identifier.

Encoding of Route Distinguishers and IPv4 addresses

The route distinguisher has only one purpose, to make IPv4 prefixes globally unique in MP-BGP.

• As RD value is configured under VRF, and it is then specified under BGP as a address-family.
• When route will get redistributed from VRF table in to BGP routing table, RD will get prepended to IPv4 address and propagates as 96-bit VPNv4 address.
• So, limitation of using conventional BGP4 to support MPLS L3VPNs is that it was originally designed to carry routing information only for the IPv4 address family.
• Realizing this limitation, the IETF is working to standardize the Multiprotocol Extensions for BGP4.

Here comes the concept for MP-BGP (Multi-Protocol BGP)

• These extensions were originally defined in RFC 2283 (February 1998) and later updated in RFC 2858 (June 2000).
• The extensions allow BGP4 to carry routing information for multiple Network Layer protocols (IPv6, IPX, VPN-IPv4, etc.).
• Therefore, to deploy MPLS L3VPNs and support the distribution of VPN-IPv4 routes, PE routers are required to support the MP-BGP extensions and not just conventional BGP.

As Route Distinguisher is only used to make route unique, and so all RD value for each customer on both sites will be different.
So, RD will not help in sorting each customer route to their individual routing domain, and it will only help that MP-BGP select all customer router as a best route, as RD only purpose is to make route unique.
To sort the route, there should be one-to-one mapping between VPN and VRF.
If there is no one-to-one mapping between VPN and VRF, how does the router know which routes need to be inserted into which VRF?
This problem is solved by introducing Route Target concept in the MPLS L3VPN architecture.

• Whereas route distinguishers are used to maintain uniqueness among identical routes in different VRFs, route targets can be used to share routes among them.
• The route target’s job is to tell the PE routers what VPN a route actually belongs to.
• A single globally unique route target is configured for each VRF as both the import target and the export target.
• The route target is a 64-bit field which is a BGP extended Communities Attribute defined which prefixes are exported and imported on the PE routers.
• Route-Target can be specified under the vrf definition which will be applied to all address-family or we can configured separately under vrf address-family or both.

Route-Target Export mean routes from the VRF will have this RT attached when injected into MP-BGP VPNv4.
Route-Target Import mean routes with this RT in MP-BGP VPNv4 address-family will be installed in the RIB of the VRF.

Flow of Route-Target

• When a VPN route learned from a CE router is injected into VPNv4 BGP, a list of VPN route target extended community attributes is associated with it.
• The export route target is appended to a customer prefix when it is converted to a VPNv4 prefix by the PE router and propagated in MP-BGP updates.
• The import route target is associated with each VRF and identifies the VPNv4 routes to be imported into the VRF for the specific customer.

So, this now solve the control plane issue, as each customer routes has been propagated from one PE to other PE as a best routes with the help of RD and it also get sorted out in their individual routing vrf domain at destination PE with the help of RT.

Overview Control Plane Working

• IGP or iBGP advertises IPv4 route from the customer end to PE1 router.
• IPv4 route is inserted into VRF routing table.
• IPv4 route is redistributed into MP-BGP. RD is added to IPv4 route to make it a VPNv4 unique route. RTs are exported and attached.
• iBGP advertises VPNv4 route with VPN label and RTs.
• RTs indicate to which VRF the route is imported. RD is removed from VPNv4 route.
• IPv4 route is inserted into specific VRF routing table.
• IGP or iBGP advertises IPv4 route from PE2 router to customer end.

Now, let look at the Data plane working.

MPLS L3VPN Data Plane Working

• When the data traffic from PE1 is initiated to PE2, the packet from PE1 will get the LDP label attached which will help the packet to reach the PE2.
• But here the main issue, the packet arrived in the main routing table instead of specific customer vrf.
• We have solved the control plane issue, but what about the data plane.
• How packet will be get sorted at the PE end, to which customer site need to be forwarded.
• So, here we need an extra label which will help to get sorted in data plane, in addition to route distinguisher and route-target in MP-BGP updates.

So, here the concept for VPN Label come:

• The VPN label is to determine what VPN a packet belongs to. But hang on, surely that’s what the RT is for?  No. The RT is for the control plane, while the VPN label is for the data plane
• VPN Label will define a specific VRF to transfer an IP packet.
• There is two ways to assign VPN label:
• Per VRF
• Assigning VPN label as per VRF:
• It only indicates the packet to which VRF table it belongs. But this impact the forwarding decision, as the path to customer is not defined.
• Per Route
• Assigning VPN label as per route:
• It not only indicates the packet, to which VRF table it belongs, it’s also indicated the respective customer site.
• Disadvantage: Memory requirement, as a greater number of route lead to assign large number of VPN label.

So, if memory is concerned per VRF is good, and if forwarding decision is concerned, per route is good.

Overview Data Plane Working

• CE1 will originates a data packet with the source address of 192.168.1.1 and destination of 192.168.2.1
• PE receives the data packet and appends the VPN label V1 and LDP label L1 and forwards the packet to P1.
• P1 receives the data packet destined to PE2 and swaps LDP label L1 with L2.
• P2 receives the data packet destined to PE2 and pops the top label. The resulting labeled packet with VPN Label V1 is forwarded to PE2.
• PE2 pops the VPN label and forwards the data packet to CE2 where the 192.168.2.1 network is located.