IPSec packet types include the authentication header (AH) for data integrity and the encapsulating security payload (ESP) for data confidentiality and integrity.

The authentication header (AH) protocol creates an envelope that provides integrity, data origin identification and protection against replay attacks. It authenticates every packet as a defense against session-stealing attacks. Although the IP header itself is outside the AH header, AH also provides limited verification of it by not allowing changes to the IP header after packet creation (note that this usually precludes the use of AH in NAT environments, which modify packet headers at the point of NAT). AH packets use IP protocol 51.

The encapsulating security payload (ESP) protocol provides the features of AH (except for IP header authentication), plus encryption. It can also be used in a null encryption mode that provides the AH protection against replay attacks and other such attacks, without encryption or IP header authentication. This can allow for achieving some of the benefits of IPSec in a NAT environment that would not ordinarily work well with IPSec. ESP packets use IP protocol 50.

· Transport mode In transport mode, only the IP payload is encrypted, and the original IP headers are left intact. This mode has the advantage of adding only a few bytes to each packet. It also allows devices on the public network to see the final source and destination of the packet. This capability allows you to enable special processing (for example, quality of service) in the intermediate network based on the information in the IP header. However, the Layer 4 header will be encrypted, limiting the examination of the packet.

· Tunnel mode In tunnel mode, the entire original IP datagram is encrypted, and it becomes the payload in a new IP packet. The major advantage of the tunnel mode is that the end systems do not need to be modified to enjoy the benefits of IP Security (e.g. edge2edge VPN between two private IP networks interconnected via public Internet). Tunnel mode also protects against traffic analysis. With tunnel mode, an attacker can only determine the tunnel endpoints and not the true source and destination of the tunnelled packets, even if they are the same as the tunnel endpoints.

Security Association (SA)

Before using IPSec services the two communicating devices must determine exactly which algorithms to use (for example, DES or AES for encryption; MD5 or SHA for integrity). After deciding on the algorithms, the two devices must share session keys. The method that IPSec uses to exchange all that relevant information is the Security Association (SA). A Security Association is a relationship between two (or more) entities that describes how the entities will use IPSec to communicate securely. IPSec does not have a mechanism for creating a Security Association. The standard method of performing Security Associations for IPSec is Internet Key Exchange (IKE). IKE creates an authenticated, secure tunnel between two entities and then negotiates the security association for IPSec. This process requires that the two entities authenticate themselves to each other and establish shared keys. IKE usually uses UDP source port 500 (or any other arbitrary high port) and destination port 500.