Solved

Network or BigIP Load Balancer problem

Posted on 2006-10-19
8
548 Views
Last Modified: 2008-01-09
Hi,

We are having serious network problems and we're basically trying to pinpoint the problem.

Our LAN consist of :

2 failover PIX firewalls
2 unmanaged switches
15 Windows 2000 servers
1 BigIP load balancer
2 Load balanced web servers

The problem is that we're getting intermittent network connectivity between the servers.
I have analyzed traffic on a few servers and i'm getting a lot of TCP packets that has to be reassembled. (TCP segment of a reassembled PDU)

I'm also getting the following log entries in our load balancer that sits between the web servers and the other servers.
arp info overwritten for 10.1.88.38 by .....

We're at a point where we really don't know what to do and we have had 7 people working on this for a looooong time now. If anyone knows anything that could help me I would be reaaaly greatful!

Thanks!
0
Comment
Question by:entronet
  • 3
  • 3
8 Comments
 
LVL 57

Expert Comment

by:giltjr
Comment Utility
---> arp info overwritten for 10.1.88.38 by .....

Some other box either believes that it is 10.1.88.38 or is configured to do proxy arp for that address.  Do you happen to have two boxes that have this same IP address.

--> I have analyzed traffic on a few servers and i'm getting a lot of TCP packets that has to be reassembled. (TCP segment of a reassembled PDU)

Sounds like you have a server that has a NIC that supports packet segmentation and has that option enabled.   Generally this is an option on Gigabit NIC's.  Do your servers have Gigabit NIC's?
0
 

Author Comment

by:entronet
Comment Utility
I have found out that the ARP issue is a serious problem. What happens is that my servers ARP tables get overwritten with the wrong MAC address. The wrong MAC address is the MAC address of the PIX firewall. I have no idea how this is happening.

Some servers have Gigabit NIC's. So packet segmentation should be disabled on all the Gigabit cards?
0
 
LVL 57

Accepted Solution

by:
giltjr earned 500 total points
Comment Utility
Does the PIX have this IP address coded on it, the 10.1.88.38?  Do you know what this IP address is supposed to be?

Well packet segmentation is a funny thing.  It reduces CPU on the sending box, but increases CPU on the receiving box.  If you have a sever that is talking to 100 clients.  You will reduce the CPU utilizaton on the sever, say by 5%, while increasing the CPU utilization on each client by say 0.0001 %.  these are made up percentages just to try and make a point.

Say you have 14,600 bytes to send out to each of the 100 clients.  

Disabling segmentation offload means the server's CPU will be used to break down the 14,600 bytes into 10 packets each in its own Ethernet frame (so 10 frames).  It must do this 100 times because there are 100 clients.  Now each client receives 10 Ethernet frames with one packet in each.

With segmentation offload enabled, the servers CPU just sends 14,600 bytes for each client.  The NIC will break these down into 10 Ethernet frames/IP segments of  1,460 bytes each, thus save CPU on the server.  Each client will still receive 10 ethernet frames that contain the segmented IP packet and will use a bit of more of their CPU to re-assemable the 10 Ethernet frames into 1 IP packet.

So, disabling segmentation could increase CPU utilization on the server.

You may want to read:

     http://en.wikipedia.org/wiki/TCP_segmentation_offloading

Also

     http://en.wikipedia.org/wiki/TCP_Offload_Engine

In case your NIC's have TOE.
0
Network it in WD Red

There's an industry-leading WD Red drive for every compatible NAS system to help fulfill your data storage needs. With drives up to 8TB, WD Red offers a wide array of solutions for customers looking to build the biggest, best-performing NAS storage solution.  

 

Author Comment

by:entronet
Comment Utility
Let me just add to this: If i didn't necessarily had to upgrade the app to be a Visual Studio 2005 program, could i more easily upgrade it to be compatible and use 2000, XP, Vista features?
0
 
LVL 57

Expert Comment

by:giltjr
Comment Utility
What?  How does Visual Studio fit into this?  Maybe you updated the wrong question?
0
 

Author Comment

by:entronet
Comment Utility
hehe, yes sorry :)
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

Downtime reduced, data recovered by utilizing an Experts Exchange Business Account Challenge The United States Marine Corps employs more than 200,000 active-duty Marines with operations in four continents, all requiring complex networking system…
Greetings, Experts! First let me state that this website is top notch. I thoroughly enjoy the community that is shared here; those seeking help and those willing to sacrifice their time to help. It is fantastic. I am writing this article at th…
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…
This tutorial demonstrates a quick way of adding group price to multiple Magento products.

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now