?
Solved

Cisco 3845 / Arp issue

Posted on 2006-11-27
9
Medium Priority
?
932 Views
Last Modified: 2008-01-09
Okay so this is a tough one, and as such I'm going to reward the highest amount of points for it. On the same token, I probably will not disclose all of the information the first time around required to help debug this issue. It will require multiple posts which is fine.

This is potentially an arp issue or a VLAN misconfiguration issue. I'm dealing with a 3845 router which sits at our datacenter and serves as a ds3 cross connect to our office. We have two VLANs:

VLAN 50 - web servers (10.50/16)
VLAN 51 - databases (10.51/16)

When trying to access VLAN 51 interfaces from our office subnet,  it can take a considerable amount of time for arp to properly cache on the destination box for 10.50.0.5. It is unreachable and arp -n shows the following for it:

[root@cc70-5 ~]# arp -n |grep 10.50.0.5
10.50.0.5                        (incomplete)                              eth1

You can see the initial ping latency below also (and traceroutes are broken until the arp cache is set after a ping)

tag1349:~ sfinkelstein$ ping 10.51.5.70
PING 10.51.5.70 (10.51.5.70): 56 data bytes
64 bytes from 10.51.5.70: icmp_seq=248 ttl=62 time=1002.861 ms <--- bad
64 bytes from 10.51.5.70: icmp_seq=249 ttl=62 time=2.827 ms
64 bytes from 10.51.5.70: icmp_seq=250 ttl=62 time=1.930 ms
64 bytes from 10.51.5.70: icmp_seq=251 ttl=62 time=1.979 ms
64 bytes from 10.51.5.70: icmp_seq=252 ttl=62 time=1.824 ms

If I set the arp entry manually then I never see this issue. I do something like the following as a temporary work around:

arp -s 10.50.0.5 00:15:F9:0C:65:A1 dev eth0

Just another note. If I ping a 10.51 interface from a 10.50 interface, this creates the arp entry right away alleviating the issue from the office subnet not being able to ping it. It'll create the arp cache for an office ping/tcp socket request after 30 seconds to 5 minutes after the initial try.

Thanks again for any assistance and please let me know if there's any other information I can provide with my network topology, router versions/configs etc to help fix this problem.
0
Comment
Question by:stevefNYC
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 3
9 Comments
 
LVL 50

Expert Comment

by:Don Johnston
ID: 18021889
So you're saying that if you try to ping a device on VLAN 50 from a device on VLAN 51, you experience this problem, but if you ping the router itself you don't have the delay?

How is the 3845 connected to the VLAN's? Are you trunking to a switch or are you using two seperate interfaces on the 3845?

Do the workstations have a default gateway set or are you using Proxy ARP?

0
 
LVL 3

Author Comment

by:stevefNYC
ID: 18021993
I'm trying to ping a device on VLAN 51 from the office subnet. If I ping from VLAN 50, it'll properly create the the arp cache on the linux box and allows me to ping. Also for boxes which have both a VLAN 50 and VLAN 51 interface, if I ping the VLAN 50 interface first, I can then make subsequent requests to VLAN 51 without a problem.

Here's some results directly from the router:

nap2gbxds3#ping 10.51.5.66

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.51.5.66, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/1 ms
nap2gbxds3#ping 10.51.5.70

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.51.5.70, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
nap2gbxds3#ping 10.51.5.61

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.51.5.61, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/1 ms
nap2gbxds3#

Some are 80 percent, some are 100.

The 3845 is connected to the VLANs with two seperate interfaces. One RJ45 twisted copper line into a VLAN 50 interface on one of our two edge 6509s and the same for VLAN 51.

I hope this answers your questions, donjohnston.  Thanks a bunch.
0
 
LVL 50

Expert Comment

by:Don Johnston
ID: 18022116
It is typical for the first ping to fail while the ARP entries are populated at end and intermediate devices. If that's the only problem you're experiencing, I wouldn't worry about it.
0
Supports up to 4K resolution!

The VS192 2-Port 4K DisplayPort Splitter is perfect for anyone who needs to send one source of DisplayPort high definition video to two or four DisplayPort displays. The VS192 can split and also expand DisplayPort audio/video signal on two or four DisplayPort monitors.

 
LVL 3

Author Comment

by:stevefNYC
ID: 18022290
Only the first ping from the router is failing. There is a much bigger issue in which boxes behind the office subnet cannot access hosts on 10.51/16 (VLAN 51) for what can be five minutes unless a prior TCP connection has been made or a static arp cache has been placed onto the destination box for the ds3 router.

tag1349:~ sfinkelstein$ time ping 10.51.5.70
PING 10.51.5.70 (10.51.5.70): 56 data bytes
^C
--- 10.51.5.70 ping statistics ---
423 packets transmitted, 0 packets received, 100% packet loss

real    7m2.833s
user    0m0.004s
sys     0m0.022s

I had to ^C out of there after 7 minutes of waiting. It can take up to 30 minutes sometimes for the arp lookup to take place.
0
 
LVL 50

Accepted Solution

by:
Don Johnston earned 2000 total points
ID: 18022858
>If I set the arp entry manually then I never see this issue. I do something like the following as a temporary work >around:
>
>arp -s 10.50.0.5 00:15:F9:0C:65:A1 dev eth0

When you did this, what device (IP address) did you do this from?

Also, are you setting a default gateway on your end stations?

And just so we know what is what, can you list the IP addresses of the router and some of the end stations you're connunicating between?
0
 
LVL 3

Author Comment

by:stevefNYC
ID: 18023327
I invoked that arp command on any of my 200 boxes which have a VLAN 51 segment subnet aliased to one of the interfaces. ie: 10.51.3.250 as an example. Yes I am setting the default gateway through each end station through the following:

[root@cc60-5 ~]# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
192.168.100.0   10.50.0.5       255.255.254.0   UG        0 0          0 eth1

You know, now that you mention it .. I removed the static route from the routing table on one linux box. We have a static route set on our Netscalers for this network. There is now no timeout at all. It also never caches the arp for the ds3 router for which packets traverse, but I *think* I'm able to ping VLAN 51 interfaces without any issues now that I removed the static route from the local linux boxes.

That is totally weird!  Any idea why? let me confirm and I'll reward you the points for your generous help, donjohnston.

Steve
0
 
LVL 3

Author Comment

by:stevefNYC
ID: 18196228
Feel free to put this in the clean up area, keith.

Thank you.

S.
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

New Server 172.16.200.2  was moved from behind Router R2 f0/1 to behind router R1 int f/01 and has now address 172.16.100.2. But we want users still to be able to connected to it by old IP. How to do it ? We can used destination NAT (DNAT).  In DNAT…
It happens many times that access list (ACL) have to be applied to outgoing router interface in order to limit some traffic.This article is about how to test ACL from the router which is not very intuitive for everyone. Below scenario shows simple s…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
Suggested Courses

764 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question