Solved

Statically build the ARP cache.

Posted on 2004-10-11
41
535 Views
Last Modified: 2013-12-27
Hi,

This is related to a post that I posted on:
http://www.experts-exchange.com/Networking/Q_21041593.html

Basically, I have two machines, one of them is a Solaris 8 / E450 and the other is Solaris 7 / x86, on the same DMZ. They work perfectly as independant machines, but when I tried to communicated from one to the other, nothing worked.

I asked my network admin to open the wall between them (because that is what he told me - there was a wall), and after he did, I still got the same problem.

After worrying for two months just trying to find a solution, I finally realized that the ARP cache is empty, therefore ping is not working, and anything else afterwards.

So when I run arp -a I get nothing....

There seems to be a problem in this DMZ that neither the network admin nor I can solve...

Anyway to get around it?

0
Comment
Question by:kalmen
  • 20
  • 13
  • 6
  • +1
41 Comments
 
LVL 38

Assisted Solution

by:yuzh
yuzh earned 50 total points
ID: 12274570
What type of communication are you trying to do?

Are the 2 machine configured in the same subnet? also check /etc/defaultrouter file
to see if they have the correct router infor (the IP of the router!)

Both machine knows each other, DNS, /etc/hosts, NIS/NIS+, LDAP etc.

do a "nslookup" to find out.


0
 
LVL 1

Author Comment

by:kalmen
ID: 12274627
I'm using the hosts file for identification and both machines are added to the host list of the other.

For example:

Server 1
has
xxx.xxx.xxx.xxx Server2

and Server 2
has
xxx.xxx.xxx.xxx Server1

I'm not using anything else.
0
 
LVL 38

Expert Comment

by:yuzh
ID: 12274682
use files are ok, how about my other questions, eg, are they sitting in the same subnet ... etc.

Sorry, I have to go now, see if any one can give you a hand, I'll get back to you tomorrow, if
the question still open.

0
 
LVL 1

Author Comment

by:kalmen
ID: 12275327
No problems.
My default router is the same, my I'm in the same subnet.

Everything looks normal.... but itsn't.
0
 
LVL 20

Expert Comment

by:tfewster
ID: 12278205
You can add entries to the arp cache manually with:
arp -s [hostname|ip_address] et:he:ra:dr:es:s  [temp]

Use the temp keyword while testing; `arp -d` can be used to remove the entry.

However, if the network is routed, the only arp table entry would be the one for the gateway.
0
 
LVL 38

Expert Comment

by:yuzh
ID: 12283332
Please have a look at the following doc:

"Need to find the IP address of a network node? Let ARP be your matchmaker "
http://sunsite.uakom.sk/sunworldonline/swol-03-1997/swol-03-sysadmin.html
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 12311944
are both IPs in the same subnet and do they both have the same netmask?
check with
  ipconfig -a
0
 
LVL 1

Author Comment

by:kalmen
ID: 12355040
Hi all,

Sorry i'm late. I've added them using the arp -s command. The two computers could see each other. They are both webservers. As far as web is concerned, there is not problem of access from the public. However, I can't see DNS or mail even though I've added the arp address of the gateway.

Very strange.

Mohammad
0
 
LVL 38

Expert Comment

by:yuzh
ID: 12355197
Could you please check your /etc/resolv.conf to see if you have the correct DNS IP in the
file, eg:
nameserver x.x.x.x

and check your /etc/nsswitch.conf  file, and make sure that it has:

hosts:     files dns [NOTFOUND=continue]

# you can just modify the record to make it use files and DNS
0
 
LVL 38

Expert Comment

by:yuzh
ID: 12355245
Have you applied the latest Recommended Solaris Patch Clusters to boxes?
0
 
LVL 1

Author Comment

by:kalmen
ID: 12356230
You know, this is the strangest problem. Since we installed this new CISCO network, I've become literally crippled with my SUN machines. They all used to work so perfectly. I mean, yes, I have applied the patches, and have the correct configuration on the network config files.

I have installed a Solaris 9 box as a test (not in DMZ, but local network), when I tried to install networking at installation, it told me it couldn't find a network, so I installed it without networking, and then manually configured the network on that box later. For a while, I thought something was wrong with this box, because nothing was working, I couldn't ping, telnet, I couldn't see anything, just by accident, I ran telnet from my PC (Windows XP) to the Solaris box... it worked... I could FTP, web, everything... but from the box, I was literally blind to the network.

I could be a CISCO issue with SUN machines... but no one can seem to figure out the problem.
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 12356733
did you read my comment?
0
 
LVL 38

Expert Comment

by:yuzh
ID: 12358362
check the port configuration to see if you can set the ports for the
Solaris boxes to use "full duplex" and also diasble auto-negotiation
for the 2 Solaris boxes.

see: http:Q_20996552.html
0
 
LVL 1

Author Comment

by:kalmen
ID: 12369244
I'll have a look at that.
Yes ahoffmann I have seen your comment. Sorry I didn't reply to that. But yes, they are in the same subnet and have the same subnet masks.
I'll double check.
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 12369863
what does snoop tell you when you try a ping?
you should see "arp who has" and arp replys, followed by "ICMP echo request"
0
 
LVL 1

Author Comment

by:kalmen
ID: 12418709
Hi,

I used:
snoop | grep arp
snoop | grep ICMP

When I tried to ping another host in my subnet, I just got:
ICMP: me ->  other-host ICMP Echo request
arp:  me -> dns-ip DNS C [reverse-dns-ip].in-addr.arpa. Internet PTR ?

And nothing reverse... I don't get it...
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 12418759
you should look at snoop unfiltered, or like:
   snoop|egrep -i 'arp|icmp'

You should see somthing like
   me -> (broadcast)  ARP C Who is ip.ip.ip.ip remotehost ?
0
 
LVL 1

Author Comment

by:kalmen
ID: 12420385
I guess i didn't know how to use snoop.

Here are my results (i'm host1, trying host2 (.99))

        host1 -> (broadcast)  ARP C Who is 111.111.111.99, host2 ?
        host2 -> host1         ARP R 111.111.111.99, host2 is 0:e0:b6:5:8a:af
        host1 -> host2      ICMP Echo request (ID: 18819 Sequence number: 0)
        host2 -> host1         ARP R 111.111.111.99, host2 is 0:a0:c9:e0:4e:e8
        host1 -> host2      ICMP Echo request (ID: 18819 Sequence number: 1)
        host1 -> host2      ICMP Echo request (ID: 18819 Sequence number: 2)
        host1 -> host2      ICMP Echo request (ID: 18819 Sequence number: 3)
        host1 -> host2      ICMP Echo request (ID: 18819 Sequence number: 4)

When I ping it gives me no answer. They are on the same switch too.
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 12424052
arp works
Why are you asking about arp?
That you get no answer for your ping may have several problems:
  - wrong netmask
  - missing routing
  - ICMP disabled on remote site
  - firewall on either site or inbetween
  - probably some more ...
0
 
LVL 1

Author Comment

by:kalmen
ID: 12426302
You see, i think that too, but what confuses me is the following.

In host1, and do:
arp -s host2 mac-adress

The my ping works and the snoop displays:
        host1 -> host2      ICMP Echo request (ID: 18943 Sequence number: 0)
     host2 -> host1         ICMP Echo reply (ID: 18943 Sequence number: 0)

What are your thoughts on that?

0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 51

Expert Comment

by:ahoffmann
ID: 12431216
ok, that's strange somehow ...
Did you check /var/adm/messages and/or /var/log/syslog?
0
 
LVL 1

Author Comment

by:kalmen
ID: 12435528
I get this from var/adm/messages:
Oct 27 23:44:53 host2 unix: WARNING: IP: Hardware address '00:a0:c9:e0:4e:e8' trying to be our address 111.111.111.099!
Oct 27 23:44:53 host2 last message repeated 1 time
But the above doesn't happen when I add the mac addresses manually.
and for /var/log/syslog just sendmail errors.
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 12438943
do you have the same MAC on both NICs?
Please verify with ifconfig on both boxes.
0
 
LVL 1

Author Comment

by:kalmen
ID: 12451180
Host1:
bash-2.03# ifconfig hme0
hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 111.111.111.100 netmask ffffff00 broadcast 111.111.111.255
        ether 8:0:20:b6:5b:58
Host2 (yes its an intel):
bash-3.00# ifconfig iprb0
iprb0: flags=863<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST> mtu 1500
        inet 111.111.111.99 netmask ffffff00 broadcast 111.111.111.255
        ether 0:a0:c9:e0:4e:e8
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 12451521
that's just one NIC, please use ifconfig -a
or is there another system in the network having this MAC?
0
 
LVL 1

Author Comment

by:kalmen
ID: 12452290
No that's two... but i just went directly to the interface. Here's a sample:
bash-3.00# ifconfig -a
lo0: flags=849<UP,LOOPBACK,RUNNING,MULTICAST> mtu 8232
        inet 127.0.0.1 netmask ff000000
iprb0: flags=863<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST> mtu 1500
        inet 111.111.111.99 netmask ffffff00 broadcast 111.111.111.255
        ether 0:a0:c9:e0:4e:e8
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 12453540
and on host1?
0
 
LVL 1

Author Comment

by:kalmen
ID: 12454048
Here's host1:

bash-2.03# ifconfig -a
lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 111.111.111.100 netmask ffffff00 broadcast 111.111.111..255
        ether 8:0:20:b6:5b:58
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 12456175
still strange ...
can you please unplug host2 from network,
then start snoop and ping on host1 and lets seen what snoop reports
0
 
LVL 1

Author Comment

by:kalmen
ID: 12468892
Hi,

I downed host2... Do you want me to uplug it phyiscally? I'm not really on the premises...
Also, I got output from snoop, but what exactly would you want me to ping? Host2? If so, here is the ouput:

        host1 -> host2      ICMP Echo request (ID: 20778 Sequence number: 0)
        host1 -> host2      ICMP Echo request (ID: 20778 Sequence number: 1)
......
        host1 -> host2      ICMP Echo request (ID: 20778 Sequence number: 19)

I have a feeling i misunderstood you... can you tell me exactly what to do?

Thanks and sorry for the headache.
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 12471038
> I have a feeling i misunderstood you ..
I've a duplicate MAC or IP in mind, which confuses your arp

with "unplug" I just meant the network cable, no reboot or shutdown necessary
If you did so and then tried to ping host2, is the above output from
  snoop|egrep -i 'arp|icmp'

If so, then   arp -a   should tell you at least one entry for host2
0
 
LVL 1

Author Comment

by:kalmen
ID: 12471614
Hi,

I did an ifconfig iprb0 down for host 2. If that's okay (i'm not physically there - but I can access the console.)

I did a:  snoop|egrep -i 'arp|icmp' but there were mixed arp requests for clients accessing the webserver so I did a snoop|egrep -i 'host2'

I pinged host2 from host1 and I got:
bash-2.03# snoop | egrep -i 'host2'
Using device /dev/hme (promiscuous mode)
        host1 -> host2      ICMP Echo request (ID: 20933 Sequence number: 0)
        host1 -> host2      ICMP Echo request (ID: 20933 Sequence number: 1)
        host1 -> host2      ICMP Echo request (ID: 20933 Sequence number: 2)
        host1 -> host2      ICMP Echo request (ID: 20933 Sequence number: 3)
        host1 -> host2      ICMP Echo request (ID: 20933 Sequence number: 4)
        host1 -> host2      ICMP Echo request (ID: 20933 Sequence number: 5)
        host1 -> host2      ICMP Echo request (ID: 20933 Sequence number: 6)
        host1 -> host2      ICMP Echo request (ID: 20933 Sequence number: 7)
        host1 -> host2      ICMP Echo request (ID: 20933 Sequence number: 8)
        host1 -> host2      ICMP Echo request (ID: 20933 Sequence number: 9)
        host1 -> host2      ICMP Echo request (ID: 20933 Sequence number: 10)
        host1 -> host2      ICMP Echo request (ID: 20933 Sequence number: 11)
        host1 -> host2      ICMP Echo request (ID: 20933 Sequence number: 12)
        host1 -> host2      ICMP Echo request (ID: 20933 Sequence number: 13)
        host1 -> host2      ICMP Echo request (ID: 20933 Sequence number: 14)
        host1 -> host2      ICMP Echo request (ID: 20933 Sequence number: 15)
        host1 -> host2      ICMP Echo request (ID: 20933 Sequence number: 16)
        host1 -> host2      ICMP Echo request (ID: 20933 Sequence number: 17)
        host1 -> host2      ICMP Echo request (ID: 20933 Sequence number: 18)
        host1 -> host2      ICMP Echo request (ID: 20933 Sequence number: 19)


0
 
LVL 1

Author Comment

by:kalmen
ID: 12471721
Also, when I do an arp -a on host1, the list comes up immediately, and for the first time, my DNS shows up properly. Like Below:

bash-2.03# arp -a

Net to Media Table: IPv4
Device   IP Address               Mask      Flags   Phys Addr
------ -------------------- --------------- ----- ---------------
hme0   DNS-Server        255.255.255.255       00:07:e9:1a:aa:58
hme0   Host2             255.255.255.255 SP    00:a0:c9:e0:4e:e8
hme0   Gateway       255.255.255.255       00:e0:b6:05:8a:af
hme0   Cisco-css      255.255.255.255       00:0d:65:e2:4e:c4
hme0   host1                255.255.255.255 SP    08:00:20:b6:5b:58

Host1, host2, DNS, Cisco-css are in the same gateway, they all have 111.111.111. prefix

Now I can connect to DNS, and everything works.
I'm thinking that there is something on host2 that is conflicting with host1...

Here's some background about these two servers... Host2 had 111.111.111.100 as an IP before host1 was created (better server). While host1 was being prepared to replace host2 it had 111.111.111.173 as an IP.

so it was like:
Host1: 111.111.111.173
Host2: 111.111.111.100

Now its like:
Host1: 111.111.111.100
Host2: 111.111.111.99

Do you think this has anything to do with it?
0
 
LVL 1

Author Comment

by:kalmen
ID: 12471802
I upped host2 again, and tried to ping from host1, it gave no answer... I went to host2 and added host1's mac address to the arp cache and then it worked.

0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 12472184
did you reset the arp cache on your router (Cosco?)
0
 
LVL 1

Author Comment

by:kalmen
ID: 12476560
I don't have access to it... But I can certainly arrange for that...
What shall I say? Just ask the Cisco guy to reset it for me?

Thanks.
0
 
LVL 51

Accepted Solution

by:
ahoffmann earned 450 total points
ID: 12482390
yes
0
 
LVL 1

Author Comment

by:kalmen
ID: 12802615
Sorry guys,

I was away. I sincerely appologise. I'm back.
0
 
LVL 1

Author Comment

by:kalmen
ID: 12802617
I'll tell you what... I'll give you the points and then worry about my problem in another post. This has gotten too long.
0
 
LVL 1

Author Comment

by:kalmen
ID: 13379197
Hi All,

It turns out that the problem was in the firewall. As I understood; my gateway was the firewall and everytime an arp request would be initiated, the firewall will try to replicate the record on its redundant counterpart (which has a problematic connection) and stall.... basically timing out without being able to do anything.

A lot headache... not your standard problem... I'm just glad its over.

Sorry I made you lives hard.
0

Featured Post

Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

Join & Write a Comment

Let's say you need to move the data of a file system from one partition to another. This generally involves dismounting the file system, backing it up to tapes, and restoring it to a new partition. You may also copy the file system from one place to…
I have been running these systems for a few years now and I am just very happy with them.   I just wanted to share the manual that I have created for upgrades and other things.  Oooh yes! FreeBSD makes me happy (as a server), no maintenance and I al…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now