IPsec (Internet Protocol Security) is a suite of protocols that authenticates and encrypts traffic at the IP layer, protecting every packet between two endpoints regardless of which application generated it. In networking and network security, it is the long-established standard for site-to-site VPN tunnels between offices and data centers, and it remains common for remote-access connections and links to cloud networks.
IPsec is not a single protocol but a framework: ESP protects packet contents, AH offers integrity without encryption, and IKE negotiates keys and security parameters. Because it operates below TCP and UDP, applications need no changes, but configuration is more demanding than with newer VPN designs.
How IPsec protects packets
IPsec protects packets by applying a security association (SA), a one-way agreement that defines the algorithms, keys and lifetime for traffic between two peers. Each SA is identified by a Security Parameter Index (SPI) carried in the packet header, so the receiver knows which keys to use.
Setting up a tunnel happens in two phases. In the first, IKE establishes an authenticated, encrypted control channel using a Diffie-Hellman exchange and either pre-shared keys or certificates. In the second, the peers use that channel to create the child SAs that protect actual data traffic, usually one pair per traffic selector, such as a pair of subnets.
The following table summarizes the main components and what each contributes.
| Component | Role | Key detail |
|---|---|---|
| ESP | Encryption and integrity of the payload | IP protocol 50 |
| AH | Integrity and authentication without encryption | IP protocol 51, breaks with NAT |
| IKEv2 | Key exchange and SA negotiation | UDP port 500 |
| NAT traversal | Wraps ESP in UDP behind NAT devices | UDP port 4500 |
| Security association database | Holds active SAs and keys | One SA per direction |
| Security policy database | Decides which traffic must be protected | Matched by addresses, ports, protocol |
Tunnel mode and transport mode
IPsec has two encapsulation modes, and the choice depends on whether the protected traffic starts and ends at the IPsec peers themselves. Most VPN deployments use tunnel mode.
| Property | Tunnel mode | Transport mode |
|---|---|---|
| What is protected | The entire original IP packet | Only the payload |
| New IP header | Added by the gateway | Original header is kept |
| Typical use | Site-to-site and remote-access VPN | Host-to-host, often with GRE or L2TP |
| Overhead | About 50 to 80 bytes per packet | About 30 to 60 bytes per packet |
| Hides internal addresses | Yes | No |
The added headers reduce the room available for payload, so tunnels typically need an adjusted MTU of about 1400 bytes or MSS clamping on the gateways. Ignoring this overhead is one of the most frequent causes of connections that start but then stall.
IPsec compared with other VPN technologies
IPsec competes mainly with WireGuard and TLS-based VPNs, and each makes different trade-offs between interoperability, simplicity and flexibility. The VPN as a concept gets its own page; this comparison focuses on protocol characteristics.
| Aspect | IPsec with IKEv2 | WireGuard | TLS-based VPN |
|---|---|---|---|
| Layer | Network layer | Network layer | Transport or application layer |
| Interoperability | Supported by almost all firewalls and routers | Growing, fewer enterprise devices | Usually vendor-specific clients |
| Cryptographic agility | Many negotiable suites | One fixed modern suite | Many suites through TLS |
| Configuration complexity | High | Low | Medium |
| Firewall friendliness | Needs UDP 500, 4500 or ESP | Single UDP port | Often runs over TCP 443 |
The decisive advantage of IPsec is interoperability: a branch firewall from one vendor can build a tunnel to a cloud gateway from another, because both implement the same standards. Its main weakness is the large number of parameters that must match exactly on both sides.
Recommended parameters and configuration choices
A sound IPsec configuration uses IKEv2, modern authenticated encryption and strong Diffie-Hellman groups, and avoids legacy options kept only for compatibility. Both peers must agree on every value, so documenting the proposal on each side is essential.
| Parameter | Recommended | Avoid |
|---|---|---|
| Key exchange version | IKEv2 | IKEv1 aggressive mode |
| Encryption | AES-GCM with 256-bit keys | DES, 3DES |
| Integrity | Built into GCM, or SHA-256 and above | MD5, SHA-1 |
| Diffie-Hellman group | Group 19, 20 or 31 | Groups 1, 2 and 5 |
| Authentication | Certificates or long random pre-shared keys | Short shared passwords |
| Lifetimes | IKE SA about 24 hours, child SA about 1 hour | Mismatched values per side |
Perfect forward secrecy should be enabled for child SAs, so that each rekey performs a new Diffie-Hellman exchange and a compromised key does not expose earlier traffic. Dead peer detection helps both sides notice a failed tunnel and rebuild it quickly.
Typical IPsec problems
Most IPsec failures are negotiation mismatches rather than defects, and the gateway logs usually state which phase failed. Reading them carefully saves much guesswork.
- Phase one fails: different authentication methods, pre-shared keys or IKE proposals on the two peers.
- Phase two fails: traffic selectors do not match, for example one side defines a /24 and the other a /23.
- Tunnel is up but no traffic passes: missing routes, firewall rules, or NAT applied to traffic that should enter the tunnel.
- Tunnel drops periodically: mismatched lifetimes or rekey behavior between vendors.
- Large transfers stall: tunnel overhead exceeds the path MTU.
Put simply
IPsec works like an armored courier service between two buildings. Every letter that leaves one office is sealed in a locked box, sent across the open city and unlocked only at the other office. Anyone who intercepts the box on the way sees only that a box is traveling, not what is inside or who inside the building wrote it.
Before the service starts, both offices agree on the locks and exchange the keys in a secure way. The people writing the letters do not need to do anything special, because the protection is applied automatically to everything that passes between the buildings.
In practice
A manufacturing company connects its headquarters network, 10.10.0.0/16, to a cloud virtual network, 10.50.0.0/16, so that its planning software can query a database hosted in the cloud. The network team configures a route-based IPsec tunnel with IKEv2, AES-256-GCM, Diffie-Hellman group 20 and certificate authentication between the headquarters firewall and the cloud VPN gateway.
The tunnel comes up at once, but database exports larger than a few kilobytes hang. A packet capture shows full-size segments disappearing inside the tunnel. The team sets MSS clamping to 1350 bytes on the tunnel interface, after which exports complete normally. A second tunnel to a redundant cloud gateway, with dynamic routing over both, gives the link failover within seconds when one gateway is maintained.