troubleshooting Question

Watchguard BOVPN Up, but no data flowing

Avatar of Ubertam
UbertamFlag for United States of America asked on
VPNInternet Protocol SecurityNetwork Architecture
9 Comments1 Solution8767 ViewsLast Modified:
I've been banging my head against a wall on this.  Our internet went down yesterday (Megapath outage) and when it finally came back up, the VPN was no longer working.  I checked Firebox System Manager and see the VPN is up on both sides, and Tacoma is sending and receiving (few packets), but Seattle is Sending only (small packets).  I can't ping across the VPN (usually not an issue) either direction.

I've rebooted both Fireboxes (Core x550e in Tacoma, Edge x55e in Seattle).
I've rebooted both Edgemarc's (phone system/router)
I've rebooted Tacoma's Cisco Bonded T1 router.
I've contacted Megapath who checked that nothing in the configuration changed, no firewall, and that our T1 lines are good.
I've submitted a ticket with Watchguard...that was about 12 hours ago.
Now I'm contacting the Experts.

It seems like it's done this before and come back up on its own, but after 16 hours of downtime, it's not coming back on its own.

I've tried rekeying the VPN, which is recognized on both sides, and the VPN stays up, but still no traffic flows.  


Recent logs:
Type	Date-Time	Detailed Message	
Debug	2011-08-10 07:42:36	 Sending DPD R_U_THERE_ACK message to 216.254.1.146:500, priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:36	******** RECV message on fd_server(7) ********, priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:36	RECV cmd:7, xpath:/ping (status:1, action:2, notification:5, wgcmd:7), priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:36	******** RECV an IKE packet at 64.81.14.238:500(socket:11 ifIndex:2) from Peer 216.254.1.146:500 ********, priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:36	IkeFindIsakmpPolicy: -->, priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:36	ike_match_if_name: Match pcy [Tacoma] dev:eth0, pkt if[2]:eth0, priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:36	Found IKE Policy [Tacoma, dev:eth0] for peer IP:216.254.1.146, numXform:1, pkt ifIndex:2, priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:36	IKE Policy details: 1th xform: grp:1 auth:1 encrypt:5 hash:1 lifeTime:86400 lifeKB:0, priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:36	IkeFindIsakmpPolicy: <--, priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:36	Use IKE Policy[Tacoma], priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:36	SetCipherAlg[DES-EDE3-CBC], priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:36	IkeNotifyPayloadNtoH : SPI Size 16 first4(0xf83a9e45), priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:36	Process Notify Payload : NOTIFY-TYPE : 36136 , priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:36	Process ISAKMP Notify : from peer 0xd8fe0192 protocol 1 SPI 459e3af8, priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:36	IkeInNotifyProcess: Recieved a DPD R_U_THERE message from 216.254.1.146:500 Seq:165337989 DataSz:4, priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:36	Received a DPD R_U_THERE message from 216.254.1.146:500, priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:36	IkeInNotifyProcess: Notify kernel - peer gateway is UP (peerIp:216.254.1.146, ifIdx:2, pcyName:Tacoma), priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:36	ike_p1_status_chg: ikePcyName:Tacoma, status:UP, priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:36	ikeMultiWanVpnFailBack: -->, priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:36	ikeMultiWanVpnFailBack: MWAN notify ikePcy:0x8287b94(Tacoma), p1said:0x2219c43 UP, priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:36	ikeMultiWanMarkIkePcyObj: MWAN using ikePcyGrp(Tacoma) and ipsecPcy(PCVKVPNTunnel), priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:36	ikeMultiWanMarkIkePcyObj: MWAN ikePcyGroup(Tacoma)'s ikePcy(Tacoma) ifStatus:0x80000002, mark it as UP, priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:36	ikeMultiWanVpnFailBack: MWAN ikePcyGrp(Tacoma)'s numofMbrs is eqaul to 1, do nothing, priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:36	IkeNotifyPayloadHtoN : net order spi(0x45 0x9e 0x3a 0xf8)  , priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:36	SetCipherAlg[DES-EDE3-CBC], priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:36	 Sending DPD R_U_THERE_ACK message to 216.254.1.146:500, priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:56	******** RECV message on fd_server(7) ********, priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:56	RECV cmd:7, xpath:/ping (status:1, action:2, notification:5, wgcmd:7), priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:56	******** RECV an IKE packet at 64.81.14.238:500(socket:11 ifIndex:2) from Peer 216.254.1.146:500 ********, priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:56	IkeFindIsakmpPolicy: -->, priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:56	ike_match_if_name: Match pcy [Tacoma] dev:eth0, pkt if[2]:eth0, priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:56	Found IKE Policy [Tacoma, dev:eth0] for peer IP:216.254.1.146, numXform:1, pkt ifIndex:2, priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:56	IKE Policy details: 1th xform: grp:1 auth:1 encrypt:5 hash:1 lifeTime:86400 lifeKB:0, priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:56	IkeFindIsakmpPolicy: <--, priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:56	Use IKE Policy[Tacoma], priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:56	SetCipherAlg[DES-EDE3-CBC], priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:56	IkeNotifyPayloadNtoH : SPI Size 16 first4(0xf83a9e45), priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:56	Process Notify Payload : NOTIFY-TYPE : 36136 , priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:56	Process ISAKMP Notify : from peer 0xd8fe0192 protocol 1 SPI 459e3af8, priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:56	IkeInNotifyProcess: Recieved a DPD R_U_THERE message from 216.254.1.146:500 Seq:165337990 DataSz:4, priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:56	Received a DPD R_U_THERE message from 216.254.1.146:500, priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:56	IkeInNotifyProcess: Notify kernel - peer gateway is UP (peerIp:216.254.1.146, ifIdx:2, pcyName:Tacoma), priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:56	ike_p1_status_chg: ikePcyName:Tacoma, status:UP, priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:56	ikeMultiWanVpnFailBack: -->, priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:56	ikeMultiWanVpnFailBack: MWAN notify ikePcy:0x8287b94(Tacoma), p1said:0x2219c43 UP, priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:56	ikeMultiWanMarkIkePcyObj: MWAN using ikePcyGrp(Tacoma) and ipsecPcy(PCVKVPNTunnel), priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:56	ikeMultiWanMarkIkePcyObj: MWAN ikePcyGroup(Tacoma)'s ikePcy(Tacoma) ifStatus:0x80000002, mark it as UP, priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:56	ikeMultiWanVpnFailBack: MWAN ikePcyGrp(Tacoma)'s numofMbrs is eqaul to 1, do nothing, priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:56	IkeNotifyPayloadHtoN : net order spi(0x45 0x9e 0x3a 0xf8)  , priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:42:56	SetCipherAlg[DES-EDE3-CBC], priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:43:19	 Sending DPD R_U_THERE_ACK message to 216.254.1.146:500, priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:43:19	******** RECV message on fd_server(7) ********, priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:43:19	RECV cmd:7, xpath:/ping (status:1, action:2, notification:5, wgcmd:7), priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:43:19	******** RECV an IKE packet at 64.81.14.238:500(socket:11 ifIndex:2) from Peer 216.254.1.146:500 ********, priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:43:19	IkeFindIsakmpPolicy: -->, priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:43:19	ike_match_if_name: Match pcy [Tacoma] dev:eth0, pkt if[2]:eth0, priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:43:19	Found IKE Policy [Tacoma, dev:eth0] for peer IP:216.254.1.146, numXform:1, pkt ifIndex:2, priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:43:19	IKE Policy details: 1th xform: grp:1 auth:1 encrypt:5 hash:1 lifeTime:86400 lifeKB:0, priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:43:19	IkeFindIsakmpPolicy: <--, priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:43:19	Use IKE Policy[Tacoma], priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:43:19	SetCipherAlg[DES-EDE3-CBC], priority=7, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:43:19	IkeNotifyPayloadNtoH : SPI Size 16 first4(0xf83a9e45), priority=6, proc_id=iked, msg_id=, tag=1002
Debug	2011-08-10 07:43:19	Process Notify Payload : NOTIFY-TYPE : 36136 , priority=7, proc_id=iked, msg_id=, tag=1002
One side is sending and receiving, the other is only sending.
Join the community to see this answer!
Join our exclusive community to see this answer & millions of others.
Unlock 1 Answer and 9 Comments.
Join the Community
Learn from the best

Network and collaborate with thousands of CTOs, CISOs, and IT Pros rooting for you and your success.

Andrew Hancock - VMware vExpert
See if this solution works for you by signing up for a 7 day free trial.
Unlock 1 Answer and 9 Comments.
Try for 7 days

”The time we save is the biggest benefit of E-E to our team. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange.

-Mike Kapnisakis, Warner Bros