Download als PDF File

When IPsec Suddenly Becomes One-Way Behind CGNAT

We recently ran into an interesting IPsec issue across three branch sites. Each branch used the same basic design:

• FortiGate firewall at the branch
• Teltonika 5G router in front of the FortiGate
• NAT on the Teltonika
• Swisscom mobile network using CGNAT
• Dial-up IPsec tunnel from the branch to a central FortiGate HQ

This setup had been working reliably for roughly one to two years. Then, from one day to the next, all three tunnels developed the same symptom:

• IKE and IPsec SAs were up
• HQ could send traffic into the tunnel
• HQ received 0 bytes from the branches
• NAT-T was disabled on all tunnels

Enabling or forcing NAT-T immediately fixed all three sites.

Simplified Topology

flowchart LR
    LAN[Branch LAN]
    BF[Branch FortiGate]
    TEL[Teltonika 5G Router]
    CGNAT[Swisscom CGNAT]
    INET[Internet]
    HQ[HQ FortiGate]

    LAN --> BF
    BF --> TEL
    TEL --> CGNAT
    CGNAT --> INET
    INET --> HQ

The FortiGate itself had no exclusive public IP address. It was effectively behind two stateful translation layers:

Branch FortiGate
    |
    | private transport network
    v
Teltonika NAT
    |
    v
Swisscom CGNAT
    |
    v
Internet
    |
    v
HQ FortiGate

Minimal FortiGate Configuration

The following is intentionally reduced to the relevant Phase 1 and Phase 2 settings.

HQ — Dial-up IPsec

config vpn ipsec phase1-interface
    edit "s2s-branches"
        set type dynamic
        set interface "wan1"
        set ike-version 2
        set peertype any
        set proposal aes256gcm-prfsha256
        set dhgrp 19
        set psksecret 
<psk>
        set nattraversal disable
    next
end

config vpn ipsec phase2-interface
    edit "s2s-branches-p2"
        set phase1name "s2s-branches"
        set proposal aes256gcm
        set src-subnet 0.0.0.0 0.0.0.0
        set dst-subnet 0.0.0.0 0.0.0.0
    next
end

Branch — Dial-up Client

config vpn ipsec phase1-interface
    edit "s2s-hq"
        set interface "wan1"
        set ike-version 2
        set remote-gw 192.0.2.99
        set proposal aes256gcm-prfsha256
        set dhgrp 19
        set psksecret </psk>
<psk>
        set nattraversal disable
    next
end

config vpn ipsec phase2-interface
    edit "s2s-hq-p2"
        set phase1name "s2s-hq"
        set proposal aes256gcm
        set src-subnet 0.0.0.0 0.0.0.0
        set dst-subnet 0.0.0.0 0.0.0.0
    next
end

The important part is this:

set nattraversal disable

Why It Worked for Years

Without NAT-T, IKE and IPsec data traffic use different transport methods.

IKE uses UDP:

IKEv2
UDP/500

The actual IPsec payload uses native ESP:

ESP
IP Protocol 50

A simplified flow looked like this:

flowchart LR
    BF[Branch FortiGate]
    TEL[Teltonika NAT]
    CG[Swisscom CGNAT]
    HQ[HQ FortiGate]

    BF -->|IKE UDP/500| TEL
    TEL -->|PAT| CG
    CG -->|UDP mapping| HQ

    BF -->|ESP / IP Proto 50| TEL
    TEL -->|ESP passthrough| CG
    CG -->|ESP-aware state| HQ

UDP is easy for NAT devices because they can build a state table around source and destination ports.

For example:

Branch FortiGate
192.168.10.2:500
        |
        v
Teltonika NAT
10.x.x.x:32784
        |
        v
Swisscom CGNAT
178.197.208.40:16193
        |
        v
HQ
192.0.2.99:500

ESP is different. It has no TCP or UDP ports.

A NAT or CGNAT platform therefore needs special IPsec-aware handling, for example based on the ESP SPI and the associated IKE session.

For roughly one to two years, this apparently worked.

The most likely explanation is that the provider infrastructure was sufficiently ESP-aware to track native ESP through CGNAT.

What Changed

All three branches failed at almost the same time.

The tunnels still showed:

IKE SA:     established
IPsec SA:   established
TX:         increasing
RX:         0 bytes

At HQ, the situation looked similar to:

enc:pkts/bytes=57/5324
dec:pkts/bytes=0/0

This is the key distinction:

Control Plane:
IKE over UDP/500      OK

Data Plane:
ESP Protocol 50       Broken

Because all three sites were affected simultaneously, and all three used the same provider and CGNAT design, the most plausible explanation is a provider-side change.

For example:

• new CGNAT platform
• firmware or software upgrade
• changed ESP ALG behaviour
• removed ESP passthrough
• traffic moved to another CGNAT cluster
• changed state handling for native ESP

The provider may not have intentionally “blocked IPsec”. It may simply have stopped handling native ESP through CGNAT in the same way as before.

Without NAT-T vs With NAT-T

Packet Structure

Without NAT-T

IP Header
└── ESP
    ├── SPI
    ├── Sequence Number
    └── Encrypted Payload

With NAT-T

IP Header
└── UDP/4500
    └── ESP
        ├── SPI
        ├── Sequence Number
        └── Encrypted Payload

The Fix: Enable NAT-T

We changed the tunnels to use NAT-T.

A typical configuration is:

config vpn ipsec phase1-interface
    edit "s2s-hq"
        set nattraversal forced
    next
end

And on HQ:

config vpn ipsec phase1-interface
    edit "s2s-branches"
        set nattraversal forced
    next
end

After rebuilding the IKE/IPsec SAs, all three tunnels immediately passed traffic again.

What NAT-T Changes

With NAT-T, ESP is transported inside UDP/4500.

flowchart LR
    BF[Branch FortiGate]
    TEL[Teltonika NAT]
    CG[Swisscom CGNAT]
    HQ[HQ FortiGate]

    BF -->|UDP/4500 + ESP| TEL
    TEL -->|UDP NAT/PAT| CG
    CG -->|UDP NAT/PAT| HQ

A concrete example could look like this:

Branch FortiGate
192.168.10.2:4500
        |
        v
Teltonika NAT
10.x.x.x:32784
        |
        v
Swisscom CGNAT
178.197.208.40:52831
        |
        v
HQ FortiGate
192.0.2.99:4500

The provider no longer needs to understand ESP.

It only needs to maintain a normal UDP mapping:

178.197.208.40:52831
        ↕
Branch UDP/4500 session

Side-by-Side Traffic Flow

WITHOUT NAT-T                      WITH NAT-T
-------------                      ----------

Branch FortiGate                   Branch FortiGate
      |                                  |
      | ESP Proto 50                     | UDP/4500
      v                                  v
Teltonika NAT                       Teltonika NAT
      |                                  |
      | ESP special handling             | normal UDP NAT
      v                                  v
Swisscom CGNAT                       Swisscom CGNAT
      |                                  |
      | ESP-aware state                  | normal UDP PAT
      |                                  |
      X after provider change            ✓
      |                                  |
      v                                  v
HQ FortiGate                        HQ FortiGate
RX = 0 bytes                        RX increases

Conclusion

The interesting part of this incident was not that NAT-T fixed the problem. That was expected.

The interesting part was that the design had worked for years without NAT-T, even though the branch FortiGates were behind a Teltonika NAT device and Swisscom CGNAT.

The most likely explanation is that the provider’s previous CGNAT implementation was able to track and forward native ESP. After a provider-side infrastructure or software change, that behaviour disappeared or became unreliable.

The VPN control plane continued to work because IKE used UDP/500, while the IPsec data plane failed because native ESP could no longer traverse the NAT path correctly.

For IPsec tunnels behind mobile networks, carrier-grade NAT or multiple NAT layers, the safer design is therefore:

ESP inside UDP/4500
        =
NAT-T

In this type of topology, forcing NAT-T removes the dependency on ESP-aware NAT behaviour and makes the tunnel far more predictable.

Samuel Heinrich

Senior Network Engineer at Selution AG (Switzerland)

Arbeitet in Raum Basel (Switzerland) als Senior Network Engineer mit über 15 Jahren Erfahrung im Bereich Netzwerk