Overview
DMVPN troubleshooting is already complex on its own. Add VRF-awareness โ where the GRE tunnel front-end sits in a transport VRF while payload traffic is in a separate customer VRF โ and failures become much harder to isolate. This guide walks through a systematic approach for diagnosing DMVPN issues in a VRF-aware deployment.
Topology Reference
Step 1 โ Verify Tunnel Interface and VRF Binding
The most common mistake is the tunnel interface bound to the wrong VRF, or tunnel vrf not set to match the physical WAN interface VRF.
! On HUB โ verify tunnel configHUB# show run interface Tunnel0! Correct config should look like this:interface Tunnel0 ip vrf forwarding VRF-CUST ! payload VRF โ customer traffic rides here ip address 172.16.0.1 255.255.255.0 no ip redirects ip nhrp map multicast dynamic ip nhrp network-id 100 ip nhrp holdtime 300 tunnel source GigabitEthernet0/0 tunnel mode gre multipoint tunnel key 12345 tunnel vrf VRF-WAN ! underlay VRF โ must match WAN interface VRF! On SPOKE-ASPOKE-A# show run interface Tunnel0interface Tunnel0 ip vrf forwarding VRF-CUST ip address 172.16.0.2 255.255.255.0 ip nhrp map 172.16.0.1 203.0.113.1 ! hub tunnel IP -> hub WAN IP ip nhrp map multicast 203.0.113.1 ip nhrp network-id 100 ip nhrp nhs 172.16.0.1 ip nhrp holdtime 300 tunnel source GigabitEthernet0/0 tunnel mode gre multipoint tunnel key 12345 tunnel vrf VRF-WANKey rule:
ip vrf forwarding= the VRF that OSPF/EIGRP runs in (customer payload).tunnel vrf= the VRF containing the physical WAN interface (transport underlay). These are often different.
Step 2 โ Check NHRP Registration (Spoke to Hub)
NHRP registration is the heartbeat of DMVPN. If spokes aren't registering with the hub, no spoke-to-spoke tunnels will ever form.
! On HUB โ view all registered spokesHUB# show ip nhrp detail172.16.0.2/32 via 172.16.0.2 Tunnel0 created 00:12:34, expire 00:04:26 Type: dynamic, Flags: registered used nhop NBMA address: 203.0.113.2 ! SPOKE-A WAN IP172.16.0.3/32 via 172.16.0.3 Tunnel0 created 00:08:10, expire 00:04:55 Type: dynamic, Flags: registered used nhop NBMA address: 198.51.100.3 ! SPOKE-B WAN IP! If a spoke is missing โ debug on the spokeSPOKE-A# debug nhrp registration! Look for: "Registration Request sent" then "Registration Reply received"! If request sent but no reply โ underlay routing is brokenTest underlay reachability in the transport VRF
! Can SPOKE-A reach the HUB WAN IP through VRF-WAN?SPOKE-A# ping vrf VRF-WAN 203.0.113.1 source GigabitEthernet0/0! Check routing table in transport VRFSPOKE-A# show ip route vrf VRF-WAN 203.0.113.1! If no route โ add static or fix IGP in VRF-WANStep 3 โ Verify Spoke-to-Spoke NHRP Resolution (Phase 2/3)
In DMVPN Phase 2/3, spokes resolve each other's NBMA addresses via NHRP. If this fails, all traffic hairpins through the hub indefinitely.
! On SPOKE-A โ trigger traffic toward SPOKE-BSPOKE-A# ping vrf VRF-CUST 10.20.20.1 source 10.10.10.1! Check if NHRP resolved SPOKE-B tunnel IPSPOKE-A# show ip nhrp 172.16.0.3! Should show: Type: dynamic, NBMA address: 198.51.100.3! If missing โ resolution failed! Watch NHRP in real timeSPOKE-A# debug ip nhrp! Good output:! "NHRP: Sending Resolution Request for 172.16.0.3"! "NHRP: Resolution Reply from 172.16.0.3 -> NBMA: 198.51.100.3"! Bad output (only request, no reply):! "NHRP: Sending Resolution Request for 172.16.0.3" <-- hub not forwarding! On HUB โ confirm it forwards NHRP requests between spokesHUB# debug ip nhrp! Should show: "NHRP: Forwarding Resolution Request to 172.16.0.3"! Fix split-horizon if hub is blocking multicast/NHRP forwardingHUB(config-if)# no ip split-horizon eigrp 1HUB(config-if)# no ip split-horizon nhrpStep 4 โ Check Routing in the Customer VRF
Even with NHRP working, traffic won't flow if the routing protocol isn't running in the correct VRF across the tunnel.
! Check routing table in customer VRF on hubHUB# show ip route vrf VRF-CUST! Check OSPF neighbors across the tunnelHUB# show ip ospf neighbor! Spokes should appear as neighbors via Tunnel0! Check EIGRP neighbors in the customer VRFHUB# show ip eigrp vrf VRF-CUST neighbors! If no neighbors โ verify routing protocol config includes tunnel networkrouter ospf 1 vrf VRF-CUST network 172.16.0.0 0.0.0.255 area 0 ! tunnel network network 10.0.0.0 0.255.255.255 area 0 ! LAN networks! Verify multicast works across tunnel (needed for OSPF/EIGRP hellos)HUB# show ip nhrp multicast! All spokes should appear in this listRoute leaking between VRFs if needed
! If customer routes need to cross from VRF-CUST to VRF-WAN or globalip vrf VRF-WAN rd 65001:100 route-target export 65001:100 route-target import 65001:200ip vrf VRF-CUST rd 65001:200 route-target export 65001:200 route-target import 65001:100! Verify leaked routesHUB# show ip route vrf VRF-CUST | include VRF-WANStep 5 โ Check IPSec (if DMVPN with IPSec Protection)
! Check IPSec profile bound to the tunnelHUB# show run interface Tunnel0 | include profile tunnel protection ipsec profile DMVPN-IPSEC-PROFILE! Verify the profile and transform setHUB# show crypto ipsec profile DMVPN-IPSEC-PROFILE! Check active SAs per spokeHUB# show crypto ipsec sa peer 203.0.113.2! encaps/decaps should both be incrementing! One side zero = one-way encryption problem! Check IKE SAsHUB# show crypto isakmp sa! Should show: AM_ACTIVE or IKEv2 ESTABLISHED per spoke! Common VRF issue โ crypto keyring must be bound to transport VRFcrypto keyring DMVPN-KEYRING vrf VRF-WAN pre-shared-key address 0.0.0.0 0.0.0.0 key YourDMVPNKey!crypto isakmp profile DMVPN-PROFILE vrf VRF-WAN ! must match transport VRF keyring DMVPN-KEYRING match identity address 0.0.0.0 VRF-WANStep 6 โ End-to-End Traffic Verification
! Extended ping spoke-A LAN to spoke-B LANSPOKE-A# ping vrf VRF-CUST 10.20.20.1 source 10.10.10.1 repeat 100! Traceroute โ confirms spoke-to-spoke or hub hairpinSPOKE-A# traceroute vrf VRF-CUST 10.20.20.1 source 10.10.10.1! Phase 2/3 working: SPOKE-A -> 172.16.0.3 -> SPOKE-B LAN (direct)! Phase 1 / broken: SPOKE-A -> 172.16.0.1 -> 172.16.0.3 (via hub)! MTU check โ GRE adds 24 bytes overhead, IPSec adds moreSPOKE-A# ping vrf VRF-CUST 10.20.20.1 source 10.10.10.1 size 1472 df-bit! If this fails but smaller ping works โ MTU issue! Fix MTU on tunnel interfacesinterface Tunnel0 ip mtu 1400 ip tcp adjust-mss 1360Quick Reference โ Common Failures
Final Verification Commands
! 1. Underlay reachabilityping vrf VRF-WAN <hub-wan-ip> source <spoke-wan-interface>! 2. NHRP registrationshow ip nhrp detail! 3. Spoke-to-spoke NHRP resolutionshow ip nhrp <remote-tunnel-ip>! 4. Routing in customer VRFshow ip route vrf VRF-CUST! 5. Routing protocol neighbors across tunnelshow ip ospf neighborshow ip eigrp vrf VRF-CUST neighbors! 6. IPSec SAsshow crypto ipsec sashow crypto isakmp sa! 7. MTUshow interface Tunnel0 | include MTU! 8. End-to-endping vrf VRF-CUST <remote-lan-ip> source <local-lan-ip> repeat 100