• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 2910
  • Last Modified:

WatchGuard HIgh Latency Ping Request Following Arp Cache Entry

I have just installed a pair of WatchGuard X Peak 5500e Firewall appliances in Drop-In Mode (meaning all interfaces share the same IP).

The external interface is a router with the first IP in my public range.

The WatchGuard is

When I directly attach a server and configure it with an IP (let's say and use the WatchGuard IP as a Gateway. The Arp Cache (which isn't manipulatable) logs the MAC and IP as it should, however, when pinging to that IP ( the initial ping is 350-450 ms. The remaining pings are fine after that.

This is reproducible every time I clear the Arp Cache I get the same result (i've configured it in both Routed and Drop-in Mode with the same outcome).

Now add to this some additional complexity:

Router 1 (ISP1) ------------------------------------------ (ISP2) Router 2
       \                                                                                           /
WatchGuard1                                                               WatchGuard2
          \_______BigIP Link Controller Redundant Pair_______/
                                      Internal Network

Some notes: The ISP1 node on the left is isolated from the ISP2 Node on the right. The only reason they are connecting is due to the Redundant Pair. There is no physical connection between ISP1 and ISP2. The BigIP's act as a load balancer directing traffic to one or both ISP's based on varying requirements.

Not to confuse the issue to much here, but when you attempt to ping in to a BigIP Virtualized IP (meaning the shared IP of the redundant pair or any NAT/Virtual Server created within BigIP) you get connectivity, however  the 350-450ms delay on ping #1 sting exists. Even worse the Arp Cache on the WatchGuard doesn't hold onto any Arp entries for any BigIP virtualized resources.

So in summary - if you ping the floating ip (shared ip between the BigIP Redundant BigIP appliances) in this case your ping would look like:

Pinging with 32 bytes of data:

Reply from bytes=32 time=400ms TTL=243
Reply from bytes=32 time=5ms TTL=243
Reply from bytes=32 time=5ms TTL=243
Reply from bytes=32 time=6ms TTL=243

Ping statistics for
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 5ms, Maximum = 400ms, Average = 104ms

The WatchGuard then has the IP/MAC for this resource in its Arp Cache, and will maintain it for 5 Minutes. If connectivity continues, and additional requests are being made of the resource it will keep the entry in its Arp Cache beyond 5 minutes for the duration of the connectivity, however remove it within 30 seconds of that activity ceasing.

The next request will then result in a new high ping and we see the condition loop over and over. If I clear the Arp cache I can reproduce the result everytime.

Does anyone have any ideas why the WatchGuard is taking so long to process that initial ping. The Ethereal packet capture shows the arp exchange happening in Micro-Seconds. Then for whatever reason the initial ping is delayed for that 350-450ms.

The arp cache issue could be lived with if the latency wasn't so bad.

I am looking for real insight into how to resolve the high latency ping in the WatchGuard.

Thanks in advance - this is a tough one.

1 Solution

Not that i know much about WG's but this might be useful -

"When automatic mode is enabled, the Hosts list is useful to lock a host to the specified interface.
To add specific hosts that the Firebox should use proxy ARP for, enter the IP address and the
interface they reside on in the Hosts section of the Drop-In Configuration tab."

Taken from : http://w3.watchgaurd.com/help/docs/v461UserGuide.pdf     -  page 47.

Might help to reduce the time the WG proxy arp's ..
When a Firebox is configured for drop-in mode, as part of the proxy ARP process the Firebox will attempt to discover all the hosts with IP addresses in the drop-in range, and the Ethernet interface on which each publicly addressed host resides. So, when the entry is not listed in the ARP table of WG, it takes time, once the entry is added it then knows where to route the packet and hence the time taken is normal.

As suggested by trinak96, adding manual entry in the WG would help.

If you are using WSM/WFS version 8.1 or lower, I would recommend to upgrade to 8.3.1 or higher; there is a known issue with 8.1 where WG when configured in drop-in mode can cause ARP storm.

Thank you.

Featured Post

When ransomware hits your clients, what do you do?

MSPs: Endpoint security isn’t enough to prevent ransomware.
As the impact and severity of crypto ransomware attacks has grown, Webroot fought back, not just by building a next-gen endpoint solution capable of preventing ransomware attacks but also by being a thought leader.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now