We help IT Professionals succeed at work.

Watchguard T35 site-to-site issues

We are having loads of trouble configuring a Site2Site VPN with a pair of Watchguard T35 firewalls.
Neither is configured pretty much outside of the initial setup wizard.
The current site 2 site vpn is stock from the vpn configuration guide from Watchguard.

We tried a number of different configs, but have currently deleted them to restart fresh.
Also we are trying to set the connection to initiate from SiteB to SiteA just to limit randomness, but can set bidirection or SiteA to SiteB as initiator.  Doesn't really matter to us

My theories may be off, so I'll just throw out the logs from each to see what you may think is happening.

Thank you in advance.


Site A
*** WG Diagnostic Report for Gateway "AA-to-TC-Gateway" ***
Created On: Tue Oct 29 09:22:49 2019

[Conclusion]
	Error Messages for Gateway Endpoint #1(name "AA-to-TC-Gateway")
		        Oct 29 09:22:35 2019 ERROR  0x02030015 Message retry timeout. Check the connection between local and remote gateway endpoints.


[Gateway Summary]
	Gateway "AA-to-TC-Gateway" contains "1" gateway endpoint(s). IKE Version is IKEv1.
	  Gateway Endpoint #1 (name "AA-to-TC-Gateway") Enabled
		Mode: Main
		PFS: Disabled 	AlwaysUp: Disabled
		DPD: Enabled 	Keepalive: Disabled
		Local ID<->Remote ID: {IP_ADDR(A.A.A.A) <-> IP_ADDR(B.B.B.B)}
		Local GW_IP<->Remote GW_IP: {A.A.A.A <-> B.B.B.B}
		Outgoing Interface: eth0 (ifIndex=4)
			ifMark=0x10000
			linkStatus=0 (0:unknown, 1:down, 2:up)
		Stored user messages:
		        Oct 29 09:22:35 2019 ERROR  0x02030015 Message retry timeout. Check the connection between local and remote gateway endpoints.


[Tunnel Summary]
	"1" tunnel(s) are found using the previous gateway

	  Name: "AA-to-TC-Tunnel" Enabled
		PFS: "Enabled" DH-Group: "14"
		Number of Proposals: "1"
		  Proposal "ESP-AES256-SHA256"
			ESP:
			  EncryptAlgo: "AES" KeyLen: "32(bytes)"
			  AuthAlgo: "SHA2-256" 
			  LifeTime: "28800(seconds)" LifeByte: "0(kbytes)"
		Number of Tunnel Routes: "1"
			#1
			  Direction: "BOTH"
			  "10.1.0.0/22<->10.1.4.0/22"


[Run-time Info (gateway IKE_SA)]


[Run-time Info (tunnel IPSEC_SA)]
	"0" IPSEC SA(s) are found under tunnel "AA-to-TC-Tunnel"

[Run-time Info (tunnel IPSEC_SP)]
	"1" IPSEC SP(s) are found under tunnel "AA-to-TC-Tunnel"
	  #1
		Tunnel Endpoint: "A.A.A.A->B.B.B.B"
		Tunnel Selector: 10.1.0.0/22 -> 10.1.4.0/22	Proto: ANY
		Created On: Tue Oct 29 09:20:15 2019
		Gateway Name: "AA-to-TC-Gateway"
		Tunnel Name: "AA-to-TC-Tunnel"

[Address Pairs in Firewalld]
	Address Pairs for tunnel "AA-to-TC-Tunnel"
		Direction: BOTH
		10.1.0.0/22 <-> 10.1.4.0/22

[Policy checker result]
	Tunnel name: AA-to-TC-Tunnel
		#1 tunnel route 10.1.0.0/22<->10.1.4.0/22
		 No policy checker results for this tunnel(no P2SA found or some other error)

[Related Logs]
<158>Oct 29 13:22:30 iked[2058]: (A.A.A.A<->B.B.B.B)******** RECV an IKE packet at A.A.A.A:500(socket=14 ifIndex=4) from Peer B.B.B.B:500 ********
<158>Oct 29 13:22:31 iked[2058]: (A.A.A.A<->B.B.B.B)Resending phase-1 message to B.B.B.B:500. Gateway-Endpoint:AA-to-TC-Gateway p1saId:0x0
<155>Oct 29 13:22:34 iked[2058]: msg_id="0203-0015" (A.A.A.A<->B.B.B.B)IKE phase-1 negotiation from A.A.A.A:500 to B.B.B.B failed. Gateway-Endpoint='AA-to-TC-Gateway' Reason=Message retry timeout. Check the connection between local and remote gateway endpoints.
<158>Oct 29 13:22:34 iked[2058]: (A.A.A.A<->B.B.B.B)ike_p1_status_chg: ikePcyName=AA-to-TC-Gateway, status=DOWN
<158>Oct 29 13:22:34 iked[2058]: (A.A.A.A<->B.B.B.B)MWAN-Failover notify ikePcy=0x105cc648(AA-to-TC-Gateway ver#1), mwanFlags:0x00000000 p1said=0x0 DOWN continuous-fails:5
<158>Oct 29 13:22:34 iked[2058]: (A.A.A.A<->B.B.B.B)WAN-Failover: start "AlwaysUp" timer(expires in 20s) for ikePcy(AA-to-TC-Gateway)
<158>Oct 29 13:22:34 iked[2058]: (A.A.A.A<->B.B.B.B)IkeDeleteIsakmpSA: try to delete Isakmp SA 0x10294d90 for Gateway AA-to-TC-Gateway. State:3
<158>Oct 29 13:22:34 iked[2058]: (A.A.A.A<->B.B.B.B)IkeDeleteIsakmpSA: try to delete QMState SA 0x102d5628 for Gateway AA-to-TC-Gateway
<158>Oct 29 13:22:34 iked[2058]: (A.A.A.A<->B.B.B.B)IkeDeleteQMState: try to delete QMState 0x102d5628 (ID 0) with IsakmpSA(0x10294d90) Gateway(AA-to-TC-Gateway)
<158>Oct 29 13:22:34 iked[2058]: (A.A.A.A<->B.B.B.B)SA Nego Fail: saHandle 0x0x105de408 InitMode 1, reason 2
<158>Oct 29 13:22:34 iked[2058]: (A.A.A.A<->B.B.B.B)SA Nego Fail: free saHandle, ipsecPcy("AA-to-TC-Tunnel")
<158>Oct 29 13:22:34 iked[2058]: (A.A.A.A<->B.B.B.B)Totally 1 Pending P2 SA Requests Got Dropped.
<158>Oct 29 13:22:34 iked[2058]: (A.A.A.A<->B.B.B.B)IkeDeleteIsakmpSA: Stop Phase One Retry and Life Timer
<158>Oct 29 13:22:34 iked[2058]: (A.A.A.A<->B.B.B.B)IkeDeleteIsakmpSA: Stop Phase One DPD Retry timer
<158>Oct 29 13:22:34 iked[2058]: (A.A.A.A<->B.B.B.B)ikeSADeleteFromCookieHashTable: IKE SA event: Delete IsakmpSA(0x10294d90) in IkeIsakmpSATable[131],pPrev((nil)) pNext((nil)) ikePcy(AA-to-TC-Gateway) Cookies(i=61045020c68d78e1 r=0000000000000000)
<158>Oct 29 13:22:34 iked[2058]: (A.A.A.A<->B.B.B.B)IkeDeleteIsakmpSA: reclaim isakmpSA(0x10294d90)'s memory and mark it as "FREED"
<155>Oct 29 13:22:35 iked[2058]: msg_id="0203-0015" (A.A.A.A<->B.B.B.B)IKE phase-1 negotiation from A.A.A.A:500 to B.B.B.B:500 failed. Gateway-Endpoint='AA-to-TC-Gateway' Reason=Message retry timeout. Check the connection between local and remote gateway endpoints.
<158>Oct 29 13:22:35 iked[2058]: (A.A.A.A<->B.B.B.B)ike_p1_status_chg: ikePcyName=AA-to-TC-Gateway, status=DOWN
<158>Oct 29 13:22:35 iked[2058]: (A.A.A.A<->B.B.B.B)MWAN-Failover notify ikePcy=0x105cc648(AA-to-TC-Gateway ver#1), mwanFlags:0x00000000 p1said=0x0 DOWN continuous-fails:6
<158>Oct 29 13:22:35 iked[2058]: (A.A.A.A<->B.B.B.B)IkeDeleteIsakmpSA: try to delete Isakmp SA 0x10293c18 for Gateway AA-to-TC-Gateway. State:5
<158>Oct 29 13:22:35 iked[2058]: (A.A.A.A<->B.B.B.B)IkeDeleteIsakmpSA: try to delete QMState SA 0x102d4c58 for Gateway AA-to-TC-Gateway
<158>Oct 29 13:22:35 iked[2058]: (A.A.A.A<->B.B.B.B)IkeDeleteQMState: try to delete QMState 0x102d4c58 (ID 0) with IsakmpSA(0x10293c18) Gateway(AA-to-TC-Gateway)
<158>Oct 29 13:22:35 iked[2058]: (A.A.A.A<->B.B.B.B)SA Nego Fail: saHandle 0x0x105e0628 InitMode 1, reason 2
<158>Oct 29 13:22:35 iked[2058]: (A.A.A.A<->B.B.B.B)SA Nego Fail: free saHandle, ipsecPcy("AA-to-TC-Tunnel")
<158>Oct 29 13:22:35 iked[2058]: (A.A.A.A<->B.B.B.B)Totally 1 Pending P2 SA Requests Got Dropped.
<158>Oct 29 13:22:35 iked[2058]: (A.A.A.A<->B.B.B.B)IkeDeleteIsakmpSA: Stop Phase One Retry and Life Timer
<158>Oct 29 13:22:35 iked[2058]: (A.A.A.A<->B.B.B.B)IkeDeleteIsakmpSA: Stop Phase One DPD Retry timer
<158>Oct 29 13:22:35 iked[2058]: (A.A.A.A<->B.B.B.B)ikeSADeleteFromCookieHashTable: IKE SA event: Delete IsakmpSA(0x10293c18) in IkeIsakmpSATable[18],pPrev((nil)) pNext((nil)) ikePcy(AA-to-TC-Gateway) Cookies(i=ad3d9bc27efa084f r=facd386c3a2c8e98)
<158>Oct 29 13:22:35 iked[2058]: (A.A.A.A<->B.B.B.B)IkeDeleteIsakmpSA: reclaim isakmpSA(0x10293c18)'s memory and mark it as "FREED"




Site B log
*** WG Diagnostic Report for Gateway "TC-to-AA" ***
Created On: Tue Oct 29 17:23:41 2019

[Conclusion]
	Error Messages for Gateway Endpoint #1(name "TC-to-AA")
		        Oct 29 17:23:37 2019 ERROR  0x02030015 Message retry timeout. Check the connection between local and remote gateway endpoints.


[Gateway Summary]
	Gateway "TC-to-AA" contains "1" gateway endpoint(s). IKE Version is IKEv1.
	  Gateway Endpoint #1 (name "TC-to-AA") Enabled
		Mode: Main
		PFS: Disabled 	AlwaysUp: Disabled
		DPD: Enabled 	Keepalive: Disabled
		Local ID<->Remote ID: {IP_ADDR(B.B.B.B) <-> IP_ADDR(A.A.A.A)}
		Local GW_IP<->Remote GW_IP: {B.B.B.B <-> A.A.A.A}
		Outgoing Interface: eth0 (ifIndex=4)
			ifMark=0x10000
			linkStatus=0 (0:unknown, 1:down, 2:up)
		Stored user messages:
		        Oct 29 17:23:37 2019 ERROR  0x02030015 Message retry timeout. Check the connection between local and remote gateway endpoints.


[Tunnel Summary]
	"1" tunnel(s) are found using the previous gateway

	  Name: "TC-to-AA-Tunnel" Enabled
		PFS: "Enabled" DH-Group: "14"
		Number of Proposals: "1"
		  Proposal "ESP-AES256-SHA256"
			ESP:
			  EncryptAlgo: "AES" KeyLen: "32(bytes)"
			  AuthAlgo: "SHA2-256" 
			  LifeTime: "28800(seconds)" LifeByte: "0(kbytes)"
		Number of Tunnel Routes: "1"
			#1
			  Direction: "BOTH"
			  "10.1.4.0/22<->10.1.0.0/22"


[Run-time Info (gateway IKE_SA)]


[Run-time Info (tunnel IPSEC_SA)]
	"0" IPSEC SA(s) are found under tunnel "TC-to-AA-Tunnel"

[Run-time Info (tunnel IPSEC_SP)]
	"1" IPSEC SP(s) are found under tunnel "TC-to-AA-Tunnel"
	  #1
		Tunnel Endpoint: "B.B.B.B->A.A.A.A"
		Tunnel Selector: 10.1.4.0/22 -> 10.1.0.0/22	Proto: ANY
		Created On: Tue Oct 29 17:17:17 2019
		Gateway Name: "TC-to-AA"
		Tunnel Name: "TC-to-AA-Tunnel"

[Address Pairs in Firewalld]
	Address Pairs for tunnel "TC-to-AA-Tunnel"
		Direction: BOTH
		10.1.4.0/22 <-> 10.1.0.0/22

[Policy checker result]
	Tunnel name: TC-to-AA-Tunnel
		#1 tunnel route 10.1.4.0/22<->10.1.0.0/22
		 No policy checker results for this tunnel(no P2SA found or some other error)

[Related Logs]
<158>Oct 29 17:23:25 iked[2046]: (B.B.B.B<->A.A.A.A)Resending phase-1 message to A.A.A.A. Gateway-Endpoint:TC-to-AA p1saId:0x0
<158>Oct 29 17:23:29 iked[2046]: (B.B.B.B<->A.A.A.A)Resending phase-1 message to A.A.A.A. Gateway-Endpoint:TC-to-AA p1saId:0x0
<158>Oct 29 17:23:31 iked[2046]: alwaysUpTimerCb trigger autoStart for ikePcy(TC-to-AA) ipsecPcy(TC-to-AA-Tunnel)
<158>Oct 29 17:23:31 iked[2046]: AUTOSTART: RECV ipecPcy(TC-to-AA-Tunnel), ikePcy(TC-to-AA), ifIndex(4), tunnel_src=B.B.B.B, tunnel_dst=A.A.A.A
<158>Oct 29 17:23:31 iked[2046]: (B.B.B.B<->A.A.A.A)do the ACQUIRE action for the tunnel route [src:10.1.4.0/22 <-> dst:10.1.0.0/22], ike_ver=1, peer_udp_port=0
<158>Oct 29 17:23:31 iked[2046]: (B.B.B.B<->A.A.A.A)(NATT)IkeFindIsakmpSABySPD: Matched IP and peer_udp_port=0 p1saId=0 : pIsakmpSA p1saID=0 DestPort=0
<158>Oct 29 17:23:31 iked[2046]: (B.B.B.B<->A.A.A.A)(NATT)IkeFindIsakmpSABySPD: Matched IP and peer_udp_port=0 p1saId=0 : pIsakmpSA p1saID=0 DestPort=0
<158>Oct 29 17:23:31 iked[2046]: (B.B.B.B<->A.A.A.A)StartNegotiation: P1 negotiation is still going on... Increment Pending P2SA counter 1 (Gateway-Endpoint TC-to-AA)
<158>Oct 29 17:23:31 iked[2046]: (B.B.B.B<->A.A.A.A)(StartNego) maxPendingP2SARequest 128 current 1
<158>Oct 29 17:23:33 iked[2046]: (B.B.B.B<->A.A.A.A)Resending phase-1 message to A.A.A.A. Gateway-Endpoint:TC-to-AA p1saId:0x0
<155>Oct 29 17:23:37 iked[2046]: msg_id="0203-0015" (B.B.B.B<->A.A.A.A)IKE phase-1 negotiation from B.B.B.B:500 to A.A.A.A failed. Gateway-Endpoint='TC-to-AA' Reason=Message retry timeout. Check the connection between local and remote gateway endpoints.
<158>Oct 29 17:23:37 iked[2046]: (B.B.B.B<->A.A.A.A)ike_p1_status_chg: ikePcyName=TC-to-AA, status=DOWN
<158>Oct 29 17:23:37 iked[2046]: (B.B.B.B<->A.A.A.A)MWAN-Failover notify ikePcy=0x103ad808(TC-to-AA ver#1), mwanFlags:0x00000000 p1said=0x0 DOWN continuous-fails:2
<158>Oct 29 17:23:37 iked[2046]: (B.B.B.B<->A.A.A.A)WAN-Failover: start "AlwaysUp" timer(expires in 20s) for ikePcy(TC-to-AA)
<158>Oct 29 17:23:37 iked[2046]: (B.B.B.B<->A.A.A.A)IkeDeleteIsakmpSA: try to delete Isakmp SA 0x102c7120 for Gateway TC-to-AA. State:3
<158>Oct 29 17:23:37 iked[2046]: (B.B.B.B<->A.A.A.A)IkeDeleteIsakmpSA: try to delete QMState SA 0x102e7c88 for Gateway TC-to-AA
<158>Oct 29 17:23:37 iked[2046]: (B.B.B.B<->A.A.A.A)IkeDeleteQMState: try to delete QMState 0x102e7c88 (ID 0) with IsakmpSA(0x102c7120) Gateway(TC-to-AA)
<158>Oct 29 17:23:37 iked[2046]: (B.B.B.B<->A.A.A.A)SA Nego Fail: saHandle 0x0x103a8878 InitMode 1, reason 2
<158>Oct 29 17:23:37 iked[2046]: (B.B.B.B<->A.A.A.A)SA Nego Fail: free saHandle, ipsecPcy("TC-to-AA-Tunnel")
<158>Oct 29 17:23:37 iked[2046]: (B.B.B.B<->A.A.A.A)Totally 1 Pending P2 SA Requests Got Dropped.
<158>Oct 29 17:23:37 iked[2046]: (B.B.B.B<->A.A.A.A)IkeDeleteIsakmpSA: Stop Phase One Retry and Life Timer
<158>Oct 29 17:23:37 iked[2046]: (B.B.B.B<->A.A.A.A)IkeDeleteIsakmpSA: Stop Phase One DPD Retry timer
<158>Oct 29 17:23:37 iked[2046]: (B.B.B.B<->A.A.A.A)ikeSADeleteFromCookieHashTable: IKE SA event: Delete IsakmpSA(0x102c7120) in IkeIsakmpSATable[211],pPrev((nil)) pNext((nil)) ikePcy(TC-to-AA) Cookies(i=aa7f6c847d5a15cc r=0000000000000000)
<158>Oct 29 17:23:37 iked[2046]: (B.B.B.B<->A.A.A.A)IkeDeleteIsakmpSA: reclaim isakmpSA(0x102c

Open in new window

7120)'s memory and mark it as "FREED"
Comment
Watch Question

Commented:
Post screenshots of  the site to site VPN configuration of each T35 from the policy manager.  Redact the public IP info.
Thank you for reaching out.  

Pretty sure it was ISP related as it started working right after they "confirmed" nothing was blocked.  

We moved the devices to their new homes with different ISP's and worked like a charm

Thank you!