?
Solved

Network or BigIP Load Balancer problem

Posted on 2006-10-19
8
Medium Priority
?
571 Views
Last Modified: 2008-01-09
Hi,

We are having serious network problems and we're basically trying to pinpoint the problem.

Our LAN consist of :

2 failover PIX firewalls
2 unmanaged switches
15 Windows 2000 servers
1 BigIP load balancer
2 Load balanced web servers

The problem is that we're getting intermittent network connectivity between the servers.
I have analyzed traffic on a few servers and i'm getting a lot of TCP packets that has to be reassembled. (TCP segment of a reassembled PDU)

I'm also getting the following log entries in our load balancer that sits between the web servers and the other servers.
arp info overwritten for 10.1.88.38 by .....

We're at a point where we really don't know what to do and we have had 7 people working on this for a looooong time now. If anyone knows anything that could help me I would be reaaaly greatful!

Thanks!
0
Comment
Question by:entronet
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
8 Comments
 
LVL 57

Expert Comment

by:giltjr
ID: 17771248
---> arp info overwritten for 10.1.88.38 by .....

Some other box either believes that it is 10.1.88.38 or is configured to do proxy arp for that address.  Do you happen to have two boxes that have this same IP address.

--> I have analyzed traffic on a few servers and i'm getting a lot of TCP packets that has to be reassembled. (TCP segment of a reassembled PDU)

Sounds like you have a server that has a NIC that supports packet segmentation and has that option enabled.   Generally this is an option on Gigabit NIC's.  Do your servers have Gigabit NIC's?
0
 

Author Comment

by:entronet
ID: 17775410
I have found out that the ARP issue is a serious problem. What happens is that my servers ARP tables get overwritten with the wrong MAC address. The wrong MAC address is the MAC address of the PIX firewall. I have no idea how this is happening.

Some servers have Gigabit NIC's. So packet segmentation should be disabled on all the Gigabit cards?
0
 
LVL 57

Accepted Solution

by:
giltjr earned 2000 total points
ID: 17775963
Does the PIX have this IP address coded on it, the 10.1.88.38?  Do you know what this IP address is supposed to be?

Well packet segmentation is a funny thing.  It reduces CPU on the sending box, but increases CPU on the receiving box.  If you have a sever that is talking to 100 clients.  You will reduce the CPU utilizaton on the sever, say by 5%, while increasing the CPU utilization on each client by say 0.0001 %.  these are made up percentages just to try and make a point.

Say you have 14,600 bytes to send out to each of the 100 clients.  

Disabling segmentation offload means the server's CPU will be used to break down the 14,600 bytes into 10 packets each in its own Ethernet frame (so 10 frames).  It must do this 100 times because there are 100 clients.  Now each client receives 10 Ethernet frames with one packet in each.

With segmentation offload enabled, the servers CPU just sends 14,600 bytes for each client.  The NIC will break these down into 10 Ethernet frames/IP segments of  1,460 bytes each, thus save CPU on the server.  Each client will still receive 10 ethernet frames that contain the segmented IP packet and will use a bit of more of their CPU to re-assemable the 10 Ethernet frames into 1 IP packet.

So, disabling segmentation could increase CPU utilization on the server.

You may want to read:

     http://en.wikipedia.org/wiki/TCP_segmentation_offloading

Also

     http://en.wikipedia.org/wiki/TCP_Offload_Engine

In case your NIC's have TOE.
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 

Author Comment

by:entronet
ID: 19289740
Let me just add to this: If i didn't necessarily had to upgrade the app to be a Visual Studio 2005 program, could i more easily upgrade it to be compatible and use 2000, XP, Vista features?
0
 
LVL 57

Expert Comment

by:giltjr
ID: 19292099
What?  How does Visual Studio fit into this?  Maybe you updated the wrong question?
0
 

Author Comment

by:entronet
ID: 19292378
hehe, yes sorry :)
0

Featured Post

Free Backup Tool for VMware and Hyper-V

Restore full virtual machine or individual guest files from 19 common file systems directly from the backup file. Schedule VM backups with PowerShell scripts. Set desired time, lean back and let the script to notify you via email upon completion.  

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Sometimes you might need to configure routing based not only on destination IP address, but also on a combination of destination IP address (or hostname) and destination port number. I will describe a method how to accomplish this with free tools. …
Resolve DNS query failed errors for Exchange
Michael from AdRem Software explains how to view the most utilized and worst performing nodes in your network, by accessing the Top Charts view in NetCrunch network monitor (https://www.adremsoft.com/). Top Charts is a view in which you can set seve…
In this video, Percona Solution Engineer Dimitri Vanoverbeke discusses why you want to use at least three nodes in a database cluster. To discuss how Percona Consulting can help with your design and architecture needs for your database and infras…

649 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question