IPsec Protocol

There are two main IPsec Protocols. These protocols are:

• AH (Authentication Header)
• ESP (Encapsulation Security Payload)

AH and/or ESP are the two protocols that we use to protect user data. Both can be used in transport or tunnel mode

AH (Authentication Header) is the first protocol of IPsec. It provides mainly source Authentication and data integrity. With this mechanism, various attacks are removed. It provides strong hashing algorithms to provide data integrity. But there is no encryption and no data confidentiality mechanisms. This is provided by ESP (Encapsulation Security Protocol).

IPsec Authentication Header

• The IPsec Authentication Header provides connectionless support for data integrity and authentication along with protection against a replay attack.
• The Authentication Header authenticates as much of the IP header as possible along with any upper-layer protocols.
• However, IPsec cannot authenticate any field values that change.
• The Authentication Header can be used alone or with the Encapsulating Security Payload protocol and may be used in either transport or tunnel mode.
• Prior to exchanging data, IPsec creates a security association between the two communicating entities. This provides the attributes necessary for the Authentication Header process.

The AH format is described in RFC 2402. The below shows the position of the Authentication Header fields in the IP packet.

• Next Header: It is an 8-bit field which identifies the type of what follows. This identifies the next protocol.
• Length: It is an 8 bits long field and contains the length of the AH header.
• Reserved: It is reserved for future use. Its length is 16 bits, and it is set to zero.
• SPI (Security Parameters Index): this is a 32-bit identifier, so the receiver knows to which flow this packet belongs.
• The Security Parameter Index is an essential part of IPsec as it distinguishes between the traffic streams that use different encryption rules and algorithms.
• The security association defines the Security Parameter Index and therefore only has local significance between the two entities.
• Sequence: The Sequence Number is a mandatory 32-bit field containing an incrementing counter value that supports anti-replay.
• The sender must always transmit this field, but the receiver doesn’t have to act on it.
• Both sender and receiver will initialize the counter to zero.
• When IPsec establishes a new security association, this will trigger the counter to reinitialize at zero.
• The Sequence Number must never recycle.
• Some vendors support the extended 64-bit Sequence Number, which is an extension to the 32-bit Sequence Number..
• ICV (Integrity Check Value): The Integrity Check Value is a variable field that contains the Integrity Check Value for the packet.
• The field must be a multiple of 32 bits in length and it may include some padding.
• The security association will specify the Integrity Check Value authentication algorithms, which include message authentication code using symmetric encryption, such as AES, or one-way hash function, such as SHA-256 or SHA-384.

The Authentication Header is an appropriate protocol to use when confidentiality is not required, such as providing assurance that a neighbor advertisement comes from an authorized router.
If encryption is required, the network administrator should use the Encapsulating Security Payload protocol. 

Encapsulating Security Payload

• The Encapsulating Security Payload provides confidentiality, authentication, integrity, and anti-replay service for IPv4 and IPv6.
• We can provide security services between a pair of hosts, between a pair of security gateways, or between a security gateway and a host.
• During IPsec conversations, IPsec creates a security association that provides the necessary attributes for the Encapsulating Security Payload.
• The device adds the Security Parameter to the IP header, and that differentiates between the traffic streams using different encryption rules and algorithms.
• You can use the encryption-only feature to provide for confidentiality with the Encapsulating Security Payload. However, using encryption without integrity may leave the communication stream vulnerable to attacks.
• Best practice is to use an integrity method when using Encapsulating Security Payload protocol by using an integrity check value in the Encapsulating Security Payload header or providing authentication separately by adding an authentication header.

Top level format of an Encapsulating Security Payload packet:

The Encapsulating Security Payload has most of the same fields as the authentication header, but the fields are in three sections.

• ESP Header
• The header contains two fields, the Security Parameters Index, and a Sequence number, and comes before the encrypted data.
• Security Parameters Index (32 bits) - Identifies a security association. This field is mandatory. The value of zero is reserved for local, implementation- specific use and MUST NOT be sent on the wire. IPsec adds the security parameter index to the header as this distinguishes between traffic streams that use different encryption rules and algorithms.
• Sequence Number (32 bits) - The sequence number is a mandatory 32-bit field value that contains an incrementing counter value. When possible, use the extended 64-bit sequence number, which is an extension to the current 32-bit sequence number. A monotonically increasing counter value; this provides an anti-replay function, as discussed for AH. The first packet sent using a given SA will have a Sequence number of 1.
• Payload Data (variable) - The Payload data is a variable length field containing data from the original IP packet. This is a transport-level segment (transport mode) or IP packet (tunnel mode) that is protected by encryption. The type of content that was protected is indicated by the Next Header field.

The trailer goes after the encrypted data and may contain padding.

• Padding (0-255 bytes) − Padding for encryption, to extend the payload data to a size that fits the encryptions' cipher block size, and to align the next field.
• Pad Length (8 bits) − Indicates the number of pad bytes immediately preceding this field.
• Next Header (8 bits) − Identifies the type of data contained in the payload data field by identifying the first header in that payload.
• Authentication Data (variable) − A variable-length field (must be an integral number of 32-bit words) that contains the Integrity Check Value computed over the ESP packet minus the Authentication Data field. Authenticated data contains an integrity check value when using the optional authentication feature.

The Encapsulating Security Payload header can provide a mix of security services in IPv4 and IPv6.
The network administrator may choose to apply the Encapsulating Security Payload alone or in combination with the authentication header in a nested fashion.
Below is the Wireshark snap, where we see both the authentication header and the Encapsulating Security Payload.

• We use both because the authentication header ensures authentication and integrity but not confidentiality. 
• The Encapsulating Security Payload provides confidentiality, authentication, and integrity services. However, the field values that IPsec hashes are different.
• The authentication header hashes both the payload and header of a packet.
• The Encapsulating Security Payload uses a hash algorithm. 
• However, it doesn’t include the IP header of the packet in that, that IP header is a mutable field, meaning that it changes as it passes through a NAT device.
• We can use the Encapsulating Security Payload in either transport or tunnel mode.
• The set of services offered will depend on what options the network administrator selected and where on the network IPsec must travel.

Operating Modes
In IPsec, both authentication header and the encapsulating security payload support two modes, transport, and tunnel.

IP packet transformation when using the Authentication Header

• We see here first the original IP packet, and then the authentication header transport mode, which has the IP header first, and then the authentication header.
• In tunnel mode, you’ll see the authentication header and the IP header, but IPsec places a brand-new header around the entire packet and essentially treats the original packet as data.

IP packet transformation when using the encapsulating security payload.

• We see here the original IP packet and then the encapsulating security payload transport mode, where we see the encapsulating security payload header after the IP header.
• In tunnel mode, you’ll see the encapsulating security header and then the IP header, but IPsec places a brand-new header around the entire packet and essentially treats the original IP packet as data.
• Many times, when using encapsulating security payload, a data communication stream will use both transport and tunnel modes when travelling from point A to point B.

Transport mode encrypts only the data portion of each packet, yet leaves the header unencrypted.
We use transport mode when a device, such as a firewall, must see the source and destination address to route the packet and NAT must take place. After passing through the firewall and the NAT device, the packet then changes to a tunnel mode before been sent out onto a internet.

Tunnel mode protects the entire original IP packet by encrypting both the header and the data portion. Tunnel mode will add a new header to the IP packet. IPsec treats the original packet as the data portion. We use tunnel mode between the two gateways, where the gateway will most likely act as a proxy for the host behind the router.

So, imagine the packet is starting in the private network.

• The IP packet might start out in transport mode.
• And then, after passing through the firewall and NAT device, IPsec can add a header to be in tunnel mode as the packet travels across an insecure WAN.
• Coming in from the WAN, and then again through the firewall and NAT device, it’ll then again be safely in the private network on the other side.
• The device can then remove the tunnel mode header and deliver the payload.