Cisco SDWAN Secure Control Plane

Why do we need the overlay?

Traditional routing protocols build per-prefix routing tables with each routing entry pointing to a next-hop IP address. This typically means that each packet is forwarded hop-by-hop across the network according to the routing table of each individual router along the path to the destination. This hop-by-hop routing behavior has many inefficiencies such as:

• Network segmentation and network slicing are hard to achieve at a large scale:

Transporting segmentation tags hop-by-hop across the network requires complex control plane interactions between VRFs, MPLS, and MP-BGP.

Network slicing and multi-tenancy are practically unachievable.

• Scaling is hard. Horizontal scaling is even harder to achieve. Equal cost multipathing (ECMP) over multiple types of WAN transports and multiple different routing protocols at a large scale is practically impossible.
• Service-chaining does not efficiently scale because it requires manual configuration on multiple devices.
• Multicast does not natively traverse public transports such as the Internet.

The Cisco SD-WAN overlay fabric solves most of these inefficiencies by changing the traditional routing concept of a next-hop IP address with a next-hop TLOC. As vEdges build overlay tunnels between their WAN tunnel endpoints (TLOCs), they advertise the site-local networks as reachable via their local TLOCs. Using this technique, each destination prefix in the routing tables points to a remote tunnel endpoint (TLOC).

The Overlay

• Cisco's SD-WAN Overlay network is made of IPsec tunnels that traverse from site to site using the underlay network forming the so-called SD-WAN Fabric. 
• Each overlay tunnel is formed between two TLOCs. The routing within the overlay is governed by the Overlay Management Protocol (OMP), a control-plane protocol very similar to BGP. 
• The OMP protocol runs over secure DTLS or TLS connections between the WAN edge routers and the vSmart controllers. 
• The process is very similar to the BGP operation, the vSmart controller acts as a BGP route reflector (RR), it receives, modifies, and re-advertises routes from the vEdge routers, but never participate in the data-plane (in the packet forwarding).

Control Connection

We have seen multiple times by now that the Cisco SD-WAN architecture decouples the control and management-plane functions away from WAN edge routers. The solution implements these functions into centralized software-based controllers that oversee the entire network environment. Therefore, from the perspective of a vEdge router, the control and management plane functions are remote services that are communicated to the device. That is why each vEdge router must establish and maintain secure control connections to each Cisco SD-WAN controller.

• The SD-WAN controllers are software entities that run as virtual machines or containers, hosted either on-prem or in a public cloud provider.
• Each controller plays a different role in the solution and interacts with the vEdge routers through various protocols.
• For example, vSmart uses the Cisco Overlay Management Protocol (OMP) to communicate routing, TLOC, and service information among vEdges. vManage uses NETCONF to push configurations to devices, SNMP to collect data, and ICMP echo/reply to detect liveliness.
• However, when multiple control protocols are running in a network, ensuring that each one is compliant with the latest security policies is a complex task that quickly gets out of hand.

How do we make sure that each protocol is secured?

• Cisco SD-WAN has taken a different, more scalable approach when it comes to securing the control plane protocols. 
• Every WAN edge router establishes one or more secured control tunnels to each SD-WAN controller. The tunnels are secured using a standard transport security protocol (DTLS or TLS).
• Then each control plane protocol, such as OMP, NETCONF, SNMP, etc., goes through one of the DTLS/TLS secured control tunnels.
• Cisco SD-WAN supports two transport layer security protocols that vEdges use for control-plane tunnels: 

DTLS (Datagram Transport Layer Security) defined in RFC 6347

TLS (Transport Layer Security) defined in RFC 5246.

• By default, Cisco SD-WAN utilizes DTLS for all control plane communications because it is faster than TLS, and latency is essential when devices are managed remotely. 
• However, the solution allows us to change the protocol to TLS easily.

Control Connections to vBond

Once we power on a vEdge router for the first time in an unconfigured state, it tries to find the IP address of vBond. An unconfigured vEdge router needs to contact vBond to receive the necessary information required to join the SD-WAN domain. However, how could a brand new unconfigured router know the IP address of the vBond controller for a particular organization? Well, there are a few different deployment options that routers can utilize to discover the orchestrator’s IP:

• ZTP/PnP - A vEdge router can automatically discover the vBond IP address over the Internet using a technique called Cisco PnP (Plug-and-Play) for IOS-XE devices and ZTP (Zero-touch provisioning) for Viptela SD-WAN devices. At a very high level, the router simply makes a query to devicehelper.cisco.com or ztp.viptela.com over the Internet, presents its serial/chassis number and certificate, and asks for the vBond IP address associated with its serial number.
• Bootstrap configuration - As an alternative option, a vEdge router can load a configuration file from a USB stick. The config file includes the vBond IP address, organization name, and everything else required so that the vEdge can successfully join the overlay fabric.
• Manually - And of course, the time-proven method that always works - a network administrator can configure everything through the CLI of the router, including the vBond IP address and the organization name.

The vEdge router tries to establish a DTLS control-plane tunnel to the vBond IP address over each available transport interface.

• vEdge will try to form a DTLS control connection to the orchestrator over the Internet and over the MPLS cloud. 
• It will authenticate itself over each control tunnel and find whether it sits behind a NAT device along any of the paths. 
• In the end, vBond will send back a list with the IP addresses and ports of all other SD-WAN controllers (vSmart and vManage). 
• DTLS control connections are then terminated because the vEdge router does not need to keep permanent control connections to vBond once it receives all necessary information.
• At this time, the vBond orchestrator will inform vManage and vSmart to expect a control connection request from that authenticated WAN Edge router.

Control Connections to vManage

• Once a vEdge router receives the list with the IP addresses of all SD-WAN controllers from vBond, it initiates a single, persistent DTLS control connection to vManage over only one transport. 
• The vEdge tries every transport network starting with the interface having the lowest port number, and it keeps the first successfully established connection permanently.
• When a vEdge router establishes a control connection to vManage, the controller provisions its configuration based on the device template attached to that vEdge.

Control Connections to vSmart

• Following the successful DTLS tunnel to vManage, the vEdge router initiates a persistent control connection to the vSmart controller over each transport interface.
• Technically, a single control connection to the controller is sufficient for a vEdge to receive control plane information. 
• Still, vEdges keep one permanent DTLS tunnel per transport interface for control-plane resiliency.
• When a vEdge router establishes at least one DTLS control connection to vSmart, it starts a single OMP peering session. 
• The router then advertises its local networks, TLOCs, and service routes to the controller via OMP. 
• Upon receiving the information, the controller recalculates the centralized RIB (routing information base) based on the defined policies and redistributes the routes to all other WAN Edge devices using OMP updates.

Control-connections overview

• When a vEdge router establishes the control plane connections to every SD-WAN controller and receives the route information from vSmart, it joins the overlay fabric by establishing IPsec/GRE tunnels to all remote TLOCs across all local WAN transports.
• The data plane IPsec tunnels should not get confused with the DTLS control plane tunnels to the SD-WAN controllers. 
• It is also important to note that the concept of Transport Locators (TLOCs) does not apply to controllers. 
• All SD-WAN controllers have a single ordinary L3 interface that terminates DTLS/TLS connections, and this interface does not have TLOC configuration (color, encapsulation type, tunnel-group, etc.).

Sequence of Events of the Bring-Up Process

There are two majors bring up Sequence for Control plane which uses Cisco SD-WAN Viptela devices:

• In the first part, you design the network, create virtual machine (VM) instances for cloud routers, and install and boot hardware routers. Then, in Cisco vManage, you add the routers to the network and create configurations for each router. This process is the User Portion of the Bring-Up Sequence.
• The second part of the bring-up process occurs automatically, orchestrated by the Cisco SD-WAN software. As routers join the overlay network, they validate and authenticate themselves automatically, and they establish secure communication channels between each other. For Cisco vBond Orchestrators and Cisco vSmart Controllers, a network administrator must download the necessary authentication-related files from Cisco vManage, and then these Cisco vSmart Controllers and Cisco vBond Orchestrators automatically receive their configurations from Cisco vManage. For vEdge Cloud routers, you must generate a certificate signing request (CSR), install the received certificate, and then upload the serial number that is included in the certificate to Cisco vManage. After Cisco hardware routers start, they are authenticated on the network and receive their configurations automatically from Cisco vManage through a process called zero-touch provisioning (ZTP). This process is the Automatic Portions of the Bring-Up Sequence.

From a functional point of view, the task of bringing up the routers in the control plane (overlay network) occurs in the following sequence:

1. The Cisco vManage software starts on a server in the data center.
2. The Cisco vBond Orchestrator starts on a server in the DMZ.
3. The Cisco vSmart Controller starts on a server in the data center.
4. Cisco vManage and the Cisco vBond Orchestrator authenticate each other, Cisco vManage and the Cisco vSmart Controller authenticate each other, and the Cisco vSmart Controller and the Cisco vBond Orchestrator securely authenticate each other.
5. Cisco vManage sends configurations to the Cisco vSmart Controller and the Cisco vBond Orchestrator.
6. The routers start in the network.
7. The routers authenticate themselves with the Cisco vBond Orchestrator.
8. The routers authenticate themselves with Cisco vManage.
9. The routers authenticate themselves with the Cisco vSmart Controller.
10. Cisco vManage sends configurations to the routers.

Before you start the bring-up process, note the following:

• To provide the highest level of security, only authenticated and authorized routers can access and participation in the Cisco SD-WAN overlay network. To this end, the Cisco vSmart Controller performs automatic authentication on all the routers before they can send data traffic over the network.
• After the routers are authenticated, data traffic flows, regardless of whether the routers are in a private address space (behind a NAT gateway) or in a public address space.

Authorized List Model

All WAN Edge devices and controllers mutually authenticate each other using an authorized list model, where the devices must be authorized before establishing connections and being allowed access onto the network.
There are two authorized lists that are distributed by vManage, one for the controllers and one for WAN Edge devices.

• Authorized controller list: The authorized controller list is a result of the administrator adding the controllers manually into the vManage user interface. This list can be distributed from the vManage to the controllers and subsequently, from the vBond to the vSmart controllers.
• Authorized serial number list for WAN Edge devices: The digitally signed authorized serial number list for the WAN Edge devices can be modified and retrieved from the Plug and Play Connect portal at http://software.cisco.com. The list can be retrieved manually or synced automatically from vManage by a user with a valid Cisco CCO account with access to the proper Smart Account and Virtual Account for the SD-WAN overlay. After the file is uploaded or synced to vManage, it is distributed by vManage to all the controllers.

With the WAN Edge authorized serial number list, the administrator can decide and configure the identity trust of each individual WAN Edge router. The options are:

• Valid: The router is authorized to be fully operational in the SD-WAN network.
• Invalid: The router is not authorized in the SD-WAN network, so no control connections form with the controllers.
• Staging: The router can authenticate and form control connections with the controllers, but OMP does not send any routes, data policies, or TLOCs to the WAN Edge router, so traffic is not forwarded. This state allows you to provision and test a router before allowing it to join the production SD-WAN network.

When the WAN Edge authorized serial number list is loaded or synced to vManage, there is an option to validate devices. If you select the checkbox to validate the devices before the list is imported, all devices are Valid by default. If you do not select the checkbox to validate, all devices are Invalid by default, and you must configure each to Valid before a router can form control connections with the controllers and join the SD-WAN network.

Identity

Authentication between devices involves validating device identity via certificates.
How device certificate validation works:

• The client device presents a CA-signed device certificate to the server.
• The server validates the certificate signature by

Running a hash algorithm on the certificate data to get a value, and

Decrypting the certificate signature with the public key obtained from the CA Root certificate to get a second value

If both values are equal, then the signature is valid.

• The client device is now trusted and the client public key can be trusted for use in encryption.

Note that the corresponding root certificate is required to validate device certificates.

Controller Identity

• Controller identity is provided by a Symantec/Digicert or Cisco-signed certificate, or alternatively, an Enterprise CA certificate. 
• Each controller in the network must have a certificate signed and installed. 
• In addition, the root certificate chain for the corresponding CA must also be installed for each controller before the controller certificates can be installed.

WAN Edge Router Identity

• Identity for vEdge hardware routers is provided by a device certificate signed by Avnet, generated during the manufacturing process and burned into the Trusted Platform Module (TPM) chip. 
• The Symantec/Digicert and Cisco root certificates are pre-loaded in software for trust for the controllers’ certificates. 
• Additional root certificates may either be loaded manually, distributed automatically by vManage, or installed during the ZTP automatic provisioning process.

The figure below shows:

1. The vManage acting as a Certificate Authority (CA) for WAN Edge cloud routers and the ASR 1002-X.
2. vManage distributes the Viptela root certificate to vBond and vSmart in order for them to validate the WAN Edge cloud identity.
3. Once the WAN Edge routers are authenticated via OTP, the vManage CA issues them Viptela-signed certificates that are used from then on for authentication.

Note that if there is a vManage cluster, each vManage signs a certificate for the device and distributes the corresponding root certificate.

Authentication/Authorization of SD-WAN Devices

When the controllers authenticate each other and WAN Edge devices, they are generally:

• Validate the trust for the certificate root Certificate Authority (CA).
• Compare the organization name of the received certificate OU against the locally configured one (except when authenticating against WAN Edge hardware devices)
• Compare the certificate serial numbers against the authorized serial number list distributed from vManage (except when authenticating against vBond)

When WAN Edge devices authenticate to the controllers, they:

• Validate the trust for the certificate root Certificate Authority (CA).
• Compare the organization name of the received certificate OU against the locally configured one.

After authentication and authorization succeeds, a DTLS/TLS connection is established.

The following diagrams illustrate how different devices authenticate with each other using Symantec/Digicert or Cisco certificates. Enterprise CA certificates operate in the same manner. Note that typically:

• Controllers and WAN Edge devices act as clients to initiate connections with the vBond, which acts as a server
• vManage controllers act as clients to initiate connections with the vSmart, which acts as a server
• vSmart controllers act as clients to initiate connections with other vSmart controllers and the one with the highest public IP address acts as a server
• WAN Edge devices act as clients to initiate connections with vManage and vSmart controllers, which act as servers