No traffic over IPSEC VPN (Cisco RV042, Netgear FVS318)

My Cisco RV042 now RV042G stays up for months at a time.

You say your RV042 works and for up to (say) a week. But it works and passes traffic. <-- Is this true?

When you have it going, go the Router management page, VPN tab down the side. Your VPN tunnels show up on the right and you should see a Disconnect Button (meaning it is connected). Then look again when it is not passing traffic. It now should say Connect.  <--- is this true.

Assuming the above, you would appear (first) to have a flaky modem and (second) to have a fault in the RV042.

From your description it sounds like you have a bad modem, flaky service, or both.

Down at the bottom of the tunnel configuration, I have:

Aggressive mode: off
Compress: off
Keep Alive: on
AH Hash: off
NAT Traversal: on (do not use unless you need to)
Dead Peer Detect: on 10 seconds
Tunnel backup: off
Split DNS: off

The parameters before that must be correct or you would never get a connection.

No clue what's going wrong here. But it's worth it to do another test: physically disconnect router and modem from each other for more than 10 minutes when the VPN is failing.

Am I getting it correct that the Internet connection is still fully functional all the time? (It has to for the software VPN, so the answer should be "yes")

To answer the first questions:
- Traffic may or may not go through when it says "disconnect" (meaning that the VPN is connected). That goes for both the Netgear and the Cisco. As a matter of fact, the Netgear is on right now, it says "disconnect", yet the traffic died around midnight, JUST like yesterday. Coincidence? Or is something going on the modem/AT&T? Note that this is happening with the new Arris modem. My settings are fairly similar to yours (I just turned to "Main" rather than Aggressive yesterday morning).
- I didn't physically disconnect the two devices. Note however that the Internet doesn't go down for long when it happens. It is what happens after that may or may not allow the VPN to route traffic.

Here's another weird tidbit: Connection went down at 11:20PM on Friday night, and at 12:20AM on Saturday night. Unknown whether the Internet connection went down at the same time (sadly the site B PRTG Monitor tool decided to quit working two days ago and I didn't notice - I restarted it now.  There were no Internet interruptions at site A.)

At this point, I'm not in a position to reboot the modem for longer than 10 minutes, so I don't know that I can bring the VPN back up for another round of monitoring. I'll see if I get to it on Monday (tomorrow) but more likely on Tuesday.

Thank you so much for your replies. I think it'd be very useful if we could get to the bottom of this.

At this point, based on the above, I think you have a modem / connection issue. Let us know tomorrow or Tuesday what you find.

Do you have static public IP addresses from both ISPs?
Or, using DynDNS?
NAT in the modems?

It sounds to me like one of your public IP addresses is changing outside of your control and you have to do a "reset" of sorts to get things back on line.  That's why I ask....
The infrequent lapses timing suggest that sort of thing.

Site to site VPNs need static IP addresses or a DynDNS setup if you don't have static IP addresses.

I have static addresses at both ends. Rebooted router 3 times so far this morning. The VPN reconnects every time, but no traffic being routed. I know it's pointless at the moment, but I'm trying to definitively rule out a router configuration problem.

Which leaves me with the modem itself (or as suggested a modem - router issue) at site B. Since I have a similar setup at a different location where I have zero problems, I'll try to compare the settings and see if I notice any difference (Comcast Cable and AT&T U-Verse).

As for NAT-ing... I don't know about the AT&T modem. There's a "Cascaded router" option which is currently OFF, and I will not turn it on remotely for obvious reasons (I'd hate a trip to the office on this lovely Sunday). That's an interesting option as it requires a router address, network address and a mask.

I'm not sure where I'd find NAT on this modem. There's a NAT/Gaming section, and it's not configured. There's an IP Passthrough which is off. There's that Cascading router option (OFF), under Subnets and DHCP. The MTU is set at 1500, and the only reason I bring this up is that the AT&T technician whom I talked to on Friday said that the MTU at AT&T (not on my modem strangely enough) should be set to 1472 (lower) when you have a router behind the modem.

Let me point out though, that none of this can explain why does it all work just fine after a longer reboot of the modem?

I kept on saying this is an Arris modem, but I just noticed that although that's what is says on the box, it says Motorola NVG510 in the software interface itself.

I'll risk a regular software reboot of the modem now. Thanks everybody!

Ah.... One more comment this Sunday...

So, I compared the two modem settings. There are no differences, other than one setup works ALL the time.

I also did the software reboot. Contrary to the 10 minutes power down that I promoted earlier, this time rebooting the modem did fix the problem. For how long? We will see - hopefully the monitoring tool will not go down tonight.

To recap: it's working now (I can ping across the VPN, access resources, etc.). Rebooting the Netgear router (3-4 times this morning) didn't accomplish anything (connected, but no traffic). Simply rebooting the U-Verse (DSL) modem did fix the problem.

Puzzled? I am...

You might try replacing the VPN box at the problem end. While this is not first on my list, it may be worth a try.

Here is the manual:
http://www.ron-berman.com/wp-content/uploads/2011/11/nvg510manual.pdf

The modem definitely has a NAT mode providing private IP addresses behind it.
Look at page 58 for IPsec passthrough.
In order to have IPsec behind NAT, you need to have the passthrough and maybe port forwarding set up.
You will know if NAT is alive if there are private-range IP addresses on the LAN side of the router.

There is no reason to switch the modem to router mode. Using NAT at the modem complicates things. A modem just forwards the traffic, using a different protocol on the physical layer of network, and is to prefer if there is a router doing all the real network stuff.

And this statement is misleading: "You will know if NAT is alive if there are private-range IP addresses on the LAN side of the router." Of course the LAN side uses NAT in almost all cases for small networks, that is what the router is used for. Anyway, NAT on LAN doesn't matter here at all, only NAT between WAN interface of the router and the modem is of importance.

MTU issues can break connections at strange times, so there might be an issue with that. If you have a chance, perform some tests with mturoute -t (http://www.elifulkerson.com/projects/mturoute.php):
1. from LAN to remote LAN using VPN
2. from LAN to remote gateway (public IP)
Mturoute will display whether and where MTUs are reduced and hence require TCP/IP fragmentation. In particular as part of VPN traffic fragmentation can be a big issue. And to work best, the endpoints (= PCs) should have a suitable MTU not requiring fragmentation on transfer.

I don't know where there might be any suggestions to change the mode of the modem since the VPN works most of the time.  The issue is what might be breaking the connection.

Note that all my comments were about the modem and not the router.  The modem has a NAT mode and I believe it's to your best advantage to know which mode it's operating in.  If it's providing a public address then there's no need for IPSEC passthrough.  If it's providing private addresses then there is,  Even so, I don't know why that would break.

An update, which doesn't really shed any more light on this issue. Suffice to say that for the past two weeks I haven't had a single drop of the tunnel, and no, I have not changed any settings on the modem or router. The only thing is that I went back to the Cisco RV042 simply because it takes less time to reboot. But, I didn't have to do it.

The one thing that changed is that we haven't had any drops in our Internet connection over AT&T's U-Verse. Remember that it appeared that the trigger were the drops in the Internet connection, although not generally when I rebooted the modem.

After looking for the cause of the DSL drops in the wiring and the AT&T infrastructure, I lucked out by having the AT&T tech on the premises as I was experiencing the drops - and he was able to test the circuit right there and then. He found nothing wrong, and so the only thing that I was left with was electrical or Radio Interference (RFI). I installed a couple of ferrite coils on the DSL line, and haven't had a problem since. Time will tell whether or not this fix will hold, but it's been good for about 10 days now.

So to summarize, I still can't think of a reason why the VPN would not come back up when the errors subsided on the AT&T circuits (whether an AT&T or RFI issue), and I don't know or think that the problem I was reporting has gotten resolved. However, the trigger is gone at least for now, and I'm OK with that. I will try to reboot the modem a couple of times tomorrow and see what happens, but aside for that, I can't think of anything logical I could try. And yes, RFI has apparently been known to cause problems with DSL circuits.

And one more thing - I did replace the modem in the process, prior to installing the ferrite coils. I still got disconnects and the VPN tunnel not routing (although connected) after the Internet connectivity came back up.

Update, in case anyone still finds this interesting...  After about 30 days of uninterrupted VPN (with the exception of a few drops in the Internet connectivity when the VPN reconnected normally), it went down again over the weekend. As always (it seems) it coincided with a drop in Internet connectivity, with the caveat that other drops did not force the VPN to go down permanently.

Ever since, the VPN showed as connected, yet no traffic is being routed. Pressing the "Disconnect" button does not help, nor does disabling and re-enabling the VPN policy. It reconnects, but will not route traffic.

For the past 30+ days, I've been using the Cisco RV042 at point B, the Netgear FVS318N at point A.

Here's the moment the VPN dropped:                                                down       coverage
8/8/2014 6:00:00 PM − 7:00:00 PM                                                  100 %      100 %
8/8/2014 5:00:00 PM − 6:00:00 PM                                                  100 %      100 %
8/8/2014 4:00:00 PM − 5:00:00 PM      54 msec      49 msec      58 msec      0 %      2 %      100 %
8/8/2014 3:00:00 PM − 4:00:00 PM      52 msec      46 msec      58 msec      0 %      0 %      100 %
8/8/2014 2:00:00 PM − 3:00:00 PM      52 msec      46 msec      58 msec      0 %      0 %      100 %

And there was an incident with the Internet connectivity around that same time:
8/8/2014 5:00:00 PM − 6:00:00 PM      62 msec      72 %      100 %
8/8/2014 4:00:00 PM − 5:00:00 PM      70 msec      3 %      100 %
8/8/2014 3:00:00 PM − 4:00:00 PM      75 msec      0 %      100 %

After about 30 days of uninterrupted VPN ....  it went down again over the weekend. As always (it seems) it coincided with a drop in Internet connectivity

It would seem to be your outside network somehow. The only other thing I can think of is to go back to the advanced settings and change 1 (at most 2) and see how those changes impact things.

I got it back online after I changed the IPSec Preshared key to no longer match (which disconnected the VPN), and leaving it this way for a few hours... The first "attempt" left it misconfigured and disconnected for about 30 minutes, the second lasted about 2 hours and was successful. Hard to tell whether this was just a coincidence, or something else that might have happened in the interim on the AT&T network.

Any more ideas???

Thanks.

Right - and thank you... Yes, it appears that way, and I was hoping that I'm not the only one impacted by this AT&T "fluke" if indeed that's what it is. They *obviously* don't know anything about it, and their techies don't have the ability to troubleshoot outside protocols (VPN), so I have this feeling that I'm really stuck.

I have one more thing left to try - which is a third squarish modem from AT&T - forgot which brand (I'll get to it tomorrow). When I first got it though, it never let the VPN go through (again - although the tunnel was "connected"), so I abandoned it pretty quickly. I'll revisit it tomorrow, when I have a new troubleshooting appointment setup with - this time - the AT&T field manager for my area.

A side note: since I've lost VPN mostly when the internet went down (in about 10% of the mini outages), I'm also focusing on reducing the incidence of such drops in my Internet connectivity. Obviously, I'm trying to mitigate the circumstances, not to implement an actual solution. This is why I've complained to AT&T multiple times (for good reason), and they've finally escalated the case to their field manager. We will see...

THANKS!

BTW, John, thanks for your suggestion. I will try that - but it's a shot in the dark, and it may take days, if not weeks to determine whether or not it is working. 30 days with no incidents actually gave me hope only to see it dashed this morning.

At this late date, perhaps this isn't very helpful but anyway:
When building VPNs with RV042s (all in the same room when building), I have seen one side connected and the other side NOT connected.  Not often but I've seen it.  Perhaps it was just a delay in the displayed information but I don't remember it that way.  Just a tidbit so one is not confused more than necessary by disagreeing indications.  When in that state, that there's no communication should be obvious.

One presumes that *both* of the VPN  interfaces are your internet gateway interfaces for the sites but that's not been said definitively has it?  It makes a difference.  e.g. one could have their internet gateway with one public IP and another VPN terminating device with another public IP at the same site connected to the same LAN.  In the latter case, internal LAN routing for the remote site subnet is necessary.

Also, these are both terminated on the *internet* with public IPs and not on an MPLS for example?  Just to be sure.

I don't think these are great avenues of investigation but completeness in cases like this seems important.

Yes, I've seen that discrepancy in indications, and thank you for clarifying that. I found it a bit odd and confusing. As it stands right now, it does say connected at both ends.

Yes, both VPN interfaces are my internet routers, both preceded by an AT&T modem at site B and Comcast at site A. Both have static public IPs.

I'd not expect the ISPs to be at all helpful in cases like this as the VPN "tunnels" right through their systems from end to end.  They shouldn't be able to do much except see that there's a secure packet going through.  I suppose they might be able to see fragmented packets but it's generally not their job to do that methinks.

If I had a choice, I'd always use the same model of router at both ends.  You can call that a very wise approach or pure superstition.  :-)  I guess the point is "it can't hurt and it might help".  
Beyond that, the RV042 Advanced settings for a VPN tunnel has a "Keep Alive" check box.  The Help section says:

Keep-Alive: This  mechanism helps to keep up the connection of IPSec tunnels. Whenever a connection is dropped and detected, it will be re-established immediately.

If this isn't checked then the behavior you've been seeing would be expected.
I imagine the Netgear has the same sort of thing.
This too may be far-fetched but we *are* grasping at straws here.  If the internet connection is flaky (or not) then the frequency of needing this setting to be as you'd like it to be could be very low and appear mysterious.  Ditto if that function is somehow intermittently broken or disabled otherwise.
Sorry if I'm preaching to the choir.

Thank you so much for the reply. Frankly, any idea, no matter how far fetched may be helpful in this matter. AND, there have been new developments, which I will try to summarize:

1. It turns out that our AT&T connection has been under par at site B. We knew that, to the extent that the internet connectivity fluctuates at times pretty wildly, and usually during business hours. Frankly, in the end we suspected RFI (radio interference). AND, as I mentioned, the trigger to a lost VPN appeared to have always been one of these drops in connectivity (if I were to guess I'd say that in 10% of the drops I'd also lose the VPN). So, our first approach was to mitigate this obvious trigger (if it doesn't go down, the VPN will stay up as if on cruise control).

2. This Tuesday (2 days ago) AT&T worked on our circuit, and this tech (one of the bright ones) did see a strange dip in some graph (a tap he says), that he is now working to fix (see point #1). While he was searching for a better circuit, the VPN went down multiple times and came back up, until the last time when it didn't. And it didn't since, no matter for how long I rebooted the AT&T modem. So it's been down for 2 days now. In desperation, I tried a few other tests, and here's what I found. I do work with another office belonging to another business. Let's call that site C (they use Comcast).

So I created VPN connections using the same parameters as from A to B (the one where we are suspecting errors), but this time connected C to A (Comcast) and C to B (AT&T). The result was a bit baffling, actually a lot, although it may in fact highlight an actual problem:

A to B - not good (that's the tunnel we've been fooling with)
B to C - working, which came as a surprise (B is the site with all of the Internet woes) and begs the question, what if the problem has been at site A all along... So...
A to C - ALSO working!!!!

Rebooted all routers (except for C because it wouldn't be nice to mess with someone else's device) to confirm that this holds, and it did... for a while....

The only thing that explains this IMHO is some form of VPN software compatibility in between these devices, which indeed was just suggested (or alluded to) by FMarshall above.

What makes this particularly troublesome and annoying is that at some point during my reboots A to C stopped routing traffic while saying "connected". And site A-B no longer connected at all (remember, before I only had the symptom I reported initially - connected but no routing). I won't mess with C, and rebooting A didn't fix the problem. The difference is that both A and C have the Netgear FVS318N router, firmware 4.3.0-19 at C (newer), and 4.2.0 at A (older). So routers by the same manufacturer do experience this problem.

Still with me? :-)

Today, I made real progress. I rebooted the Comcast modem at A, and the Netgear. This was a clean reboot, by the book, and NO tunnel came back up (A-B disconnected, A-C connected but no routing).

So I upgraded the firmware at A (Netgear). I didn't go to the newest version (4.3.1-22), since A-C is working (Cisco - Netgear 4.3.0-19), but rather to the second newest (4.3.0) just like at site C.

After the upgrade, and without a secondary reboot, both tunnels from A said connected, AND my main worry - A - B also routed traffic. Victory (of sorts)!!!!!! Surprisingly, now A - C (both Netgear and IDENTICAL since the firmware upgrade) is not routing traffic.

So, this still points to buggy VPN software in potentially both the Cisco RV042 and the Netgear FVS318N routers. Remember, the premise was excluding this possibility because (A) the connection stayed up for long stretches of time, and (B), most reboots would NOT affect the tunnel traffic, and (C) the symptoms were similar no matter which combination I used (remember, I actually started with 2 Cisco routers). The caveat was that I kept on focusing on site B, AND I could only do software reboots at the other site (since I was not physically there to unplug the power).

Finally, let me quote from a tech support reply in response to a similar (somewhat) problem I had with my Mac VPN Tracker 7 software trying to connect to the Cisco RV042 (tunnel established, no traffic):
// begin quote
It looks like you have an RV042 with a well-known bug: The device does not accept
VPN connection that originate (!) from network ports other than 500.

The originating network port is determined by two things:

1. The port VPN Tracker uses.

2. What your local NAT router does to the port.

VPN Tracker uses port 500 if available, and silently falls back to another
available port if 500 is in use.

Most NAT routers try to preserve the port, but as soon as you have 2 computers
connecting to VPN from behind the same router (or even one computer connecting in
short succession), you'll end up with at least one getting a different port.

This is perfectly normal behavior for a NAT router, but unfortunately, the RV042
is unable to deal with it.
// end of quote

That may be not what we're dealing with here, but it highlights "well known" software bugs in these low-end consumer VPN routers, which BTW - don't appear to get fixed very often (a firmware upgrade didn't do anything in that other case, but switching to the Netgear fixed the problem).

Sooooo, one more reboot at A (with the new but not newest Netgear firmware) - and we have a status quo - A - B working, A to C not routing (but I don't care).

Can anyone suggest another VPN router model that is not too professional (and expensive)? (I tried a Zyxel for about $110, but didn't like it at all).

THANKS so much for sticking to this - in case your eyes have not already glazed over :-)

The good ones I know are all in a non- consumer pricing area, but I didn't hear bad news about DrayTek Vigor.

VPN Tracker uses port 500 if available, and silently falls back to another
 available port if 500 is in use.

 Most NAT routers try to preserve the port, but as soon as you have 2 computers
 connecting to VPN from behind the same router (or even one computer connecting in
 short succession), you'll end up with at least one getting a different port.

Hmmm... I'm a little confused by the context here.  

What has been described, as I understood it at least, was that you were running site-to-site VPNs with both terminating devices using public IP addresses.  I really didn't pay any attention to the VPN Tracker part.  

Nonetheless, it's not clear to me why port 500 should be a problem as long as there's no NAT traversal - which it appears there is not.  That is, none of the VPN devices is situated *behind* a NAT device which carries the public IP address.
IF that were the case then you'd only be able to port forward 500 to a single internal IP.  That's what the description sounds like and that doesn't sound like what you're doing.

(A common arrangement is to have a VPN router behind a NATting router.  In that case, you have to use IPSEC passthrough and/or port forward port 500 to the WAN side of the VPN router, etc. And, doing this rather obviates having two VPN routers behind the NAT router.  But, as far as I know, it doesn't prevent multiple tunnels - haven't tried that).

"Ports" are really just address extensions like Apartment B added to 101 Main ST.  If port 500 were a problem for multiple tunnels then I have a hard time imagining how a VPN terminating device could advertise it will handle 50 tunnels as in the case of the RV0xx series.

There are settings called IPSEC passthrough and they often don't tell you what these do exactly.  But, it's obviously about forwarding port 500, etc.

So I'd want to understand this port 500 issue a bit better if it's a likely source of trouble.  I just stupidly don't get it yet.

I didn't mean to confuse you. My point, of which I have a record is that the firmware for these routers has been known to have well known bugs - not that this specific bug applies to my case. Sorry I didn't make that clearer.

Interesting but stupid bug ... Some devices will indeed try to keep the source and destination port to udp/500 for IPSec, but it is more than common that the source port is (and needs to be) dynamic.

Right - I may have some updates later today... Still working with AT&T, though for now just to mitigate the number of instances when the internet goes down.

There was no solution. In fact, there may be no solution for this issue without inside support from AT&T.