We help IT Professionals succeed at work.
Get Started

Functioning BOVPN tunnels to Dubai just stop working - URGENT

EddieWr
EddieWr asked
on
1,765 Views
Last Modified: 2013-11-16
We have an office in the UK with a  Watchguard Firebox X1000 connected to two other UK offices with Firebox X700 and an office in the Isle of Man with a Firebox II. All of these offices are connected to the other three using BOVPN tunnels and this configuration has been up and running for several years and has been very stable.

We have now opened an office in Dubai and are using a Firebox X20e to connect it to the rest of the offices using BOVPN tunnels. The local ISP, Etisalat, has provided a 2Mb DSL line with dynamic IP addressing.

Whilst I was in Dubai I set up the Firebox locally and used remote access tools to set up the BOVPN tunnel to London  this all connected OK once I had got my head around the new interface. I stuck with the default settings for Phase 1 and Phase 2 and provided the same shared key at both ends. I then proceeded to install the local Windows server and join it to the company domain.

On my return to London I made a trip to Manchester (not just for the BOVPN) and configured their local Firebox to connect to Dubai. This went through with no problems and connected OK.

The following week the connection between London and Dubai just stopped working for no apparent reason. As I was travelling to the third office (Oxford) at the time I was unable to address the issue until the following morning. All the configurations at both ends looked to be OK and the Manchester to Dubai link was working with no problems. After trying various issues I used the connectivity between Manchester and Dubai to delete the BOVPN tunnel between London and Dubai on both boxes and then recreated them from scratch. This seemed to fix the problem  although it didnt indicate the reason for the problem in the first place. I also configured a BOVPN tunnel between Oxford and Dubai with no problems at all.

The following day (Friday) I set up some jobs to transfer a reasonably large amount of data (>5Gb) to Dubai from London as that is the start of the weekend in Dubai. On Sunday morning I got a text from Dubai (06:15 local time!!) telling me that the BOVPN link between London and Dubai was down again. I checked the system and this time all three links to Dubai were down  all of the UK and Isle of Man links were working fine.

I have tried various changes on the London end and got someone in Dubai to make changes on their Firebox but nothing would bring the link back up. I even tried deleting and recreating the link between London and Dubai, but again no joy. I have checked and the IP address in Dubai has not changed.

As it stands, the links are still down so this is urgent  can anyone shed any light?

On the London Firebox where it has been rebooted several times yesterday, there is no error message showing against the BOVPN tunnel but there is no connectivity either.

On the other two Fireboxes which have not be rebooted, it shows Key has expired: Renegotiating [SHA1-HMAC Authentication 3DES-CBC Encryption] followed by Key has expired: Renegotiation Failed [SHA1-HMAC Authentication 3DES-CBC Encryption]

The Firebox System Monitor, Traffic Monitor shows various error messages. Starting with the box that has been rebooted I get the following sequence of messages (86.xx.xx.xx is the IP addresso fthe Dubai end):

Iked(165) RE-TO 86.xx.xx.xx MM-HDR  ISA_SA ISA_VENDORID ISA_VENDORID ISA_VENDORID ISA_VENDORID
Iked(165) FROM 86.xx.xx.xx IF-HDR -535Axxxx ISA_NOTIFY
Received a packet for an unknown SA

iked[165]:  Deleting SA: peer        86.xx.xx.xx
iked[165]:               my_cookie   F49A54E1xxxxxxxx
iked[165]:               peer_cookie 0000000000000000

However, I do see the following which seems to indicate some traffic is going throuhg
tunneld[161]:  recv echo-request from 86.xx.xx.xx
tunneld[161]:  sent echo-reply

I am not sure whether this message is relevant but there are quite a lot of them showing on this box.
kernel:  GRE: out of order: as:2615 seq:2614 from:0x3b1axxxx

On the two boxes which have not been rebooted, I am getting the following sequence of messages:

iked[145]:  RE-TO 86.98.26.59 AG-HDR   ISA_SA ISA_KE ISA_NONCE ISA_ID ISA_VENDORID ISA_VENDORID ISA_VENDORID ISA_VENDORID
iked[145]:  Deleting SA: peer        86.xx.xx.xx
iked[145]:               my_cookie   2316CBEAxxxxxxxx
iked[145]:               peer_cookie 0000000000000000

I am totally at a loss as to what to try next  I cannot understand why a BOVPN tunnel that is working fine just suddenly drops out for no apparent reason.

At the Dubai office they are able to access the Internet with no problems and make a client VPN connection into London. I can also access Outlook Web Access on their server. The major problem is that all incoming and outgoing mail goes via London so I have got the Dubai office jumping up and down on me  bear in mind that yesterday (Sunday) is a working day for them, so they have now been down for a day and a half.

Any assistance or information that anyone can provide will be most appreciated!

Many thanks, Eddie
Comment
Watch Question
CERTIFIED EXPERT
Top Expert 2007
Commented:
This problem has been solved!
Unlock 1 Answer and 4 Comments.
See Answer
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE