IBM System x Server with Broadcom NICs

Posted on 2009-12-27
Medium Priority
Last Modified: 2013-12-27
Hi I have a couple of IBM system x 3550 servers. They have onboard broadcom nic's.
I have an issue where they keep dropping packets to a SUN Solaris server. .
Further investigation determined that it seemed that arp poisoning  was occuring.  I have found some articles that suggest upgrade to the  latest version if the nic drivers, delete the nic teaming, which i have done. I have also disabled the 2nd nic. But the problem still exists. I thought I got rid of it over 10 months ago, but has not come back.
the model nic is bcm5708c.
Any suggestions would be appreciated.
Server is running windows 2003 r2

Here is an article about it
Question by:Cdewar
  • 5
  • 4
LVL 81

Expert Comment

ID: 26129108
I've seen times that the NIC looses its own HW encoded MAC address.
I.e. when this issue occurs, if you log into the console of the server and run ipconfig /all
you may see that the MAC address reflected for the interface is 00:00:00:00:00:00
See whether setting the MAC address within the configuration setting of the network interface under advanced will solve the issue.

The other issue is that you may have a firewall IP lock to a specific MAC address that result in lack of access when the wrong MAC is associated with the NIC because of teaming switch.
Where does the MAC address you mark as being incorrect comes from?

Author Comment

ID: 26129160
yeah I tried manually setting the mac address. still not luck.. There are no firewalls involved.

Author Comment

ID: 26129206
Just found that the other mac address is from one of our other servers ????how does the happen
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

LVL 81

Expert Comment

ID: 26129550
Do you have clustered servers?  Do you have DHCP servers that assigned the IPs based on MAC?  Do you have a pair of DHCP server one of which might be misconfigured i.e. associated the wrong MAC with the IP.
Are you servers configured with static IPs or are they using DHCP with reserved IPs based on the MAC?

Author Comment

ID: 26129744
Hi arnold,
We only have one dhcp server. All servers that are affected have static IP's, rather than have them setup as a reservation in DHCP.
We do have clustered servers, they are also having the intermitant issue........
Upon checking the other servers, I found that one of them is not using an update driver, is still on an old driver that we were having issues with... Funny enough this server has the mac address that keeps being populated in the arp table on the solaris box against the other server that I was having issues with....
I am an now thinking that this is casuing the issue ....  what do you think ??
LVL 81

Accepted Solution

arnold earned 1000 total points
ID: 26131055
Its possible.  The question is if it is a cluster server, is the cluster IP configured with a shared MAC.
Besides clustering do you also use NLB?
You could reduce the amount of time an ARP entry remains on the Solaris box.

The issue might be that this IP fails over or is used as a load-balance and once the arp is cached on the Sun box, it will reject packets with different MACs.
See what the current arp expiration period is by using ndd -get /dev/arp arp_cleanup_interval
But until you resolve the issue with the IP switching to a different system or not using a shared/common MAC address, the time it takes for the ARP entry to expire on the Solaris box is the amount of time this type of issue will continue.  Lowering the ARp caching too low on the Solaris could alter its performance given that it will need to query more frequently for the ARP entry.

Author Comment

ID: 26135031
the nic teaming has been set for failover only.
Just checked the arp expiration and its set to 300000.  What would you suggest that the timeout value be ??
LVL 81

Expert Comment

ID: 26135589
300000 I think is in milliseconds, so five minutes is fine and that should be the amount of time those systems can not access the sun box.

In failover mode, it might actually failover under certain circumstances that are short of the switch to which the primary is connected fails.  i.e. it test whether the primary is still there.  You may have a condition under which the check fails and initiates the failover.
I'm not sure, but I though the teaming setup should use a common MAC address to avoid these types of issues.
i.e. ethernet adapter1 and ethernet adapter2 have the same MAC address associated with the IP that fails over.
IP1 has macaddress1 without regard on which Network Adapter it resides.


Author Comment

ID: 26135630
yeah thats exactly how the teaming work. One common MAC address. Thanks for all your ideas. Even though my issue is still ongoing your have provide me with some useful information, ao I will award the points to you

Featured Post

Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Data center, now-a-days, is referred as the home of all the advanced technologies. In-fact, most of the businesses are now establishing their entire organizational structure around the IT capabilities.
New style of hardware planning for Microsoft Exchange server.
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
With the advent of Windows 10, Microsoft is pushing a Get Windows 10 icon into the notification area (system tray) of qualifying computers. There are many reasons for wanting to remove this icon. This two-part Experts Exchange video Micro Tutorial s…
Suggested Courses
Course of the Month14 days, 11 hours left to enroll

839 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question