Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 970
  • Last Modified:

Network tools for Networking Issues

What tools could I use to troubleshoot networking Issues?

We are seeing intermittent network problems between two servers.

One is a virtual and the other is a physical.

they last 20 seconds or more before pinging starts again
0
MECIT
Asked:
MECIT
  • 33
  • 27
  • 2
  • +2
1 Solution
 
dmarinenkoCommented:
Wireshark works good.
It is a free packet analyzer http://www.wireshark.org/

Also try doing a "tracert ip-of-other-comp" at a dos prompt.  This will ping all the gateways in between.
0
 
andeporterCommented:
What sort of network hardware do you have between the two servers? You could use the logging feature of any Cisco equipment to identify collisions and such.
0
 
MECITAuthor Commented:
i am using (3) Dell Powerconnect 6248 gigabit switchs.

I did the tracert on both computers
They had 1 hop at 1ms to the other server.
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 
dmarinenkoCommented:
What kind of network adapters do you have on the machines?  
I have seen this exact issue with Atheros NIC's.
Have you tried connecting different computers to the virtual to figure out whether it's an issue with the virtual or the other computer?
Have you updated the NIC drivers to the latest versions?
0
 
schmitty007Commented:
If this is a VMware VM I would look at all your VMs resources and make sure your not ballooning.

Also is it only between this on VM on this Host and this one Physical box or can you run a constant ping to the VM from any device on your next work and have the ping fail or visa verse with the physical box.
It seems odd that you only have issues with just these to specific devices talking to each other. Also if possible you could try moving to different switch ports.
IF it is these two boxes are you pinging by IP or host name?

Has there been any network or hardware changes recently which may have triggered this or are one of these servers new?
0
 
MECITAuthor Commented:
on the vritual it is using broadcom NetXtreme II BCM5708 and Intel 82575GB.

on the physical it is using broadcom Netxtreme II BCM5708

Im looking to see if there are new drivers for the physical.
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
At first check by ping:

From Server A, ping server A and B both at a time
From server B, ping server B and A both at  a time.

What do you see?IS there any PKT loss to it's own interface?
0
 
MECITAuthor Commented:
I have updated the firmware on the switches and I am updating the drivers on the VM Hosts.
Once completed I will run the tests again.

On the Powerconnect 6248, is there an option to monitor the ports that are connected to the physical server.

How can i troubleshoot from the switches?
0
 
MECITAuthor Commented:
Server A and Server B are pinging at 1ms but after a fairly few 1ms they both increase up 13ms one and 10 ms on the other.
Then they both return back to 1ms.
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
You have 3 switches between 2 servers.Are those managable and IP assigned?
So, ping every switch step by step to find high latency.

You also need to check traffic usage of each switch.Does this switch support SNMP ?
If this switch supports SNMP then you could configure Cacti/MRTG and check each port utilization.This could give you idea.
0
 
MECITAuthor Commented:
They are managed ad stacked with one ip

this is what i get when pinging the switch

1ms
2ms
2ms
1ms
5ms
1ms
1ms
1ms
2ms
3ms
2ms
1ms
1ms
1ms
2ms
6ms
1ms
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
Latency become higher for several reasons:
High usage, Memory/CPU issue or firewall,routing issue etc.

However, only 6ms shouldn't be problem for network for communicating or data transferring between two servers.
You indicated that you are facing network issue.What exact network problem are you facing? Slow data transfer? PKT loss? or high latency?
0
 
MECITAuthor Commented:
The results were from my laptop to the switch.

We are having slow data tranfers and some packet loss
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
Please check from your server to switch.At first ping test then you need to check throughput.
To check throughput between two servers you may use 'Iperf'
http://openmaniak.com/iperf.php

Download for linux version:
http://sourceforge.net/projects/iperf/files/latest/download
For windows version:
http://www.mayoxide.com/iperf/iperf-2.0.5-cygwin.zip
0
 
MECITAuthor Commented:
do i install the application on both servers?

could i install it on my laptop or does it need to be a server.
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
Iperf is a tool to check throughput between two points.You may install iperf into your both server to check throughput between two servers.Or if you want to check throughput between laptop to server then install a copy into laptop and another copy into server.

But throughput might vary based on computer performance and configuration or even Operating System.So better, try to test between two servers.
0
 
MECITAuthor Commented:
Do i just place a copy on the c: of each server?

How do I get it to install?

Does this open its own application or am i going to be running it in the command prompt?
0
 
MECITAuthor Commented:
This is the server end results
Interval - 10.2 secs
Transfer - 6.38 MBytes
Bandwidth - 5.23 Mbits/sec

These are the client side results
Interval - 10.2 secs
Transfer - 6.38 MBytes
Bandwidth - 5.27 Mbits/sec

What do the results mean?
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
Too poor performance.

Keep each copy of iperf into both server.Then go to command prompt into server and go to iperf directory by 'cd' command.
Then run "iperf -s"                 //iperf -s inicates server mode
From another server use
iperf -c "Iperf Server IP"                  //This is client mode.

If you get the same results it means there is some problem in network.
For that case I'd suggest you to check throughput step by step:

1. Server to laptop by using a cable
2.Server to laptop through switch.

Then you could become to understand which is the cause for this less throughput.
0
 
MECITAuthor Commented:
Called Dell tech support , about the dropped packets.

They stated it might have to do with the spanning tree settings. But they are doing a best effort since I dont have support only hardware support.

i have no idea what the spanning tree does or doesnt do. Would you think that could be the issue?
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
Of course Spanning tree could be an issue for throughput.But there are other configurations also could impact for slow transfer.
To know about Spanning tree:
http://www.enterprisenetworkingplanet.com/netsp/article.php/3580966/Networking-101-Understanding-Spanning-Tree.htm
0
 
MECITAuthor Commented:
What other configurrations could affect it?

I enabled some ports to port fast. I set the root id to 4096 to allow the switch to become the root under the spanning tree. for some reason , the root id was pointing to a WAP.
This was the best effort advice from Dell.

I am still seeing packets dropped and i also ran th iperf test again and had the same results.

What else can I do?
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
Leave the switch and connect  your server and laptop using a cable.Then make iperf test between laptop and server.Be sure that server and laptop performance is ok.Then you could go for switch.
0
 
MECITAuthor Commented:
The majority of our servers are virtual. The physical servers are our critical servers and at this tie I can not unplg the nic to do the testing.

Can I do from desktop to laptop? Will this work?
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
Ok..you could check between desktop and latop using cable at first.Then connect both laptop and desktop into same switch and test again.
0
 
MECITAuthor Commented:
here are my results

Desktop to Laptop

Interval- 10 sec
Transfer- 68 MBytes
Bandwidth-57 Mbits/sec
 
desktop to switch to laptop

Interval- 10 sec
Transfer- 90.4 MBytes
Bandwidth-75.8 Mbits/sec

Server to laptop

Interval- 10.2 sec
Transfer- 6.25MBytes
Bandwidth-5.16Mbits/sec
0
 
MECITAuthor Commented:
for the latop to switch to desktop it is connected a differnt switch
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
It could be identified if you could check by the same switch that you used for server.Also be sure about your servers NIC speed settings.
0
 
MECITAuthor Commented:
I did not understand the first part of your comment.


Both servers - Nic speed is set to auto
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
Use the same switch that is connected to your server.Connect your laptop and desktop into 2 free ports of that switch and make test between laptop and desktop.Then you can find out if there is any difference.
0
 
MECITAuthor Commented:
there was
 the server to laptop results

Server to laptop

Interval- 10.2 sec
Transfer- 6.25MBytes
Bandwidth-5.16Mbits/sec
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
You didn't get my point.

You already made test between laptop to desktop which looks fine.But you used different switch.
So now use the switch which you used to test between server to laptop.

Please check if there are 2 free ports into that switch.Then connect both laptop and desktop into that switch and test between laptop to desktop.Basically, you are going to test that switch.
0
 
MECITAuthor Commented:
Interval- 10 sec
Transfer- 6.12MBytes
Bandwidth-5.14Mbits/sec

here are the results
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
So, it's the configuration of your switch.

Could you please disable Spanning Tree for a while? Then you could test again if it's the reason for Spanning tree.One by one you can check other configurations too.
0
 
MECITAuthor Commented:
Would it affect te network more if i disable the spannng tree?
Will the switch need to reboot?
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
You can disable Spanning Tree for temporarily. After test you can enable it again.I don't think it requires reboot.However, based on Switch model if it requires reboot it'll alert you to reboot.

Before changing any configuration better you take snapshot of each page/configuration backup.It'll help you to reconfigure again.
0
 
MECITAuthor Commented:
I have disabled spanning tree and flow control.

Still getting the same results
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
So, it's not for the spanning tree but other configuration of your switch.Are you using VLAN ? Check by connecting to other ports.
0
 
MECITAuthor Commented:
no VLANs
I connected two laptops to each dell switch
1st test 223Mbits/Sec , laptop 1 server;laptop2 client

2nd test 141Mbits/Sec  , laptop 2 server;laptop1 client
and kept same switch and ports on switch


laptop and physical server
3rd Test 84.3 Mbits/Sec  , Server was the server;laptop was client

4th Test 4.98 Mbits/Sec , Server was client; laptop was server and same ports and same switch
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
Which switch are you using?? simply
iperf -c "server iP"        //for client
iperf  -s                          //for server

If it is same port and same switch for both cases you should get same throughput.However, it seems there are some port configuration which is the reason for different throughput for different traffic direction.

Finally, to be sure use laptop2 and physical server:
Physical server as server and laptop2 as client
Laptop2 as server and Physical server as client.

If it is the same result it must be your switch.Might be some switchport configuration.
0
 
MECITAuthor Commented:
Dell support recommended to break the stack.
Sw 1 prt 1 --> Sw2 prt 1 ; Sw2 prt 2 --> Sw3 prt 1


If I do the iperf test from laptop to laptop I am getting

iperf -c x.x.x.x -d

transfer                    bandwidth
244MB                         205Mb
235MB                         195Mb
 
If I do the iperf test from laptop to server I get
 
iperf -c x.x.x.x -d

transfer                    bandwidth
271MB                         227Mb
6MB                               5.03Mb
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
iperf - x.x.x.x -d indicates bi-directional throughput.From your test we can decide that server upload throughput is too low.So, there are only two possible reason:

1. Server upload capacity
2. Switchport configuration connected to that server.

If it possible you can connect your server to another tested switchport.If the throughput result shows same then it's the problem of your server.Otherwise, it's your specific that switchport
0
 
MECITAuthor Commented:
We did plug it into another switch and port same results.

if its the server , what would cause that because we have tested iperf on 4 or 5 severs and com back with the same results.
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
Are those physical server?Have the samw NIC model ??
0
 
MECITAuthor Commented:
Yes it is a physical server and we purchased a new NIC for the server.

we used a different switch, new cables that were tested.

we get the same results.
How do we get to improve server throughput to increase?
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
So, it's sure that your server is the reason for this.

However, you need to check server performance first.Based on Server OS there are many server performance tool available.

1. Be sure all server performance is ok.
2. Check memory and CPU usage when you are downloading/iperf test
3. Try by disabling antivirus(If there is any antivirus)
4. Stop other unnecessary applications/program/services then try again.
0
 
MECITAuthor Commented:
I and get this on a few servers

xception: STATUS_ACCESS_VIOLATION at eip=6110D923
eax=00000014 ebx=00000000 ecx=FFFFFFFF edx=00000014 esi=00000001 edi=00000014
ebp=1A22C858 esp=1A22C854 program=C:\iperf\iperf.exe, pid 7892, thread unknown (0x2F54)
cs=001B ds=0023 es=0023 fs=003B gs=0000 ss=0023
Stack trace:
Frame     Function  Args
1A22C858  6110D923  (00000014, 1A22CB1A, 00410B8E, 00000001)
1A22CB38  61142926  (1A22D000, 1A22CB58, 00410B88, 1A22CBF8)
1A22CBD8  61118839  (004161F8, 00000020, 00410B88, 9F63D75A)
1A22CC18  610C01A5  (004161F8, 00000020, 30ED8C00, 42C27300)
1A22CC68  00403F9C  (00EC7B20, 00000000, 1A22CCB8, 610E1469)
1A22CC88  004056A0  (00EC7AC0, 00000001, 00000001, 05A1DB2E)
1A22CCC8  004059F5  (00EC7AC0, 00000000, 00000001, 610713B0)
1A22CCF8  00405D19  (00EC7AB8, 00000000, 00000000, 00000000)
1A22CD18  00405DCD  (00EC7AB8, 00412088, 00000000, 00000000)
1A22CD38  00405FBE  (00E31088, 00000000, 00000000, 00E512C8)
1A22CD58  0040980D  (00E31088, 00000000, 00000000, 00000000)
1A22CD98  610E38C5  (00E512C8, 1A22CDD4, 610E3810, 00E512C8)
End of stack trace
0
 
MECITAuthor Commented:
I am also getting some of these

connect failed: Connection timed out
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
Might be corrupted file or antivirus/firewall is blocking
0
 
MECITAuthor Commented:
We tried all 4 steps
We are getting the same results.

I am going through all our servers, virtual and physical, running the iperf test. So far all have the same results.

should the results be the same as laptop to latop. around 100 to 200 Mbps
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
Could you please check server to laptop directly (Without using switch) by a cross cable?
0
 
MECITAuthor Commented:
Same results.
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
Which OS do you use for your servers?
0
 
MECITAuthor Commented:
Windows 2008 and 2003
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
Download Microsoft Baseline Security Analyzer and check that which updates are missing:
http://www.microsoft.com/download/en/details.aspx?id=7558

Also Check windows Server 2003 performance Advisor:
http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=15506

For Windows 2008:
http://msdn.microsoft.com/en-us/library/windows/hardware/gg463394.aspx

Also check Event log for any kinds of error.
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
Also post your server network card settings.Go to Device Manager>NIC Properties>Advanced.Then check all the options one by one and if possible post here.
0
 
MECITAuthor Commented:
Intel Pro/1000 MT

Gigabit Master Slave Mode  - Auto Detect
Jumbo Frames - Disabled
Locally Administred Address - empty
Log Link State Event - Enabled
Performance Options -
Adaptive Inter-frame spacing -  Enabled
Flow Control - Generate & Respond
Interrupt Moderation Rate - Adaptive
Receive Descriptors - 256
Transmit Descriptors - 256

Qos Packet Tagging - Disabled
TCP/IP Offloading options - Everything is checked off
Wait for Link - Auto Detect
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
Change receive and Transmit Descriptors and test with following values and test with iperf

1. RxD:128  TxD: 128
2. RxD:256  TxD: 128
3. RxD:256  TxD:  64
4. RxD:256  TxD:  32
5. RxD:256  TxD:  16
6. RxD:128  TxD:  64
7. RxD:64    TxD:  64
8. RxD:16    TxD:256
9. RxD:16    TxD:128

Collect the iperf result and see if any differences.

NB: each time you change enable/disable NIC
0
 
MECITAuthor Commented:
i tried a couple but same results. This are from the exchange server so it is hard to test out.

I looked at the other servers and they do not have all the options the same on the nic.
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
So, it might be some other options.But it's not the switch but your server is causing problem.
I don't know if you have updated your server or not.Using Baseline Security Analyzer you could check missing updates as I suggested before.I don't have no more idea.Might be some other experts could look on this issue.
0
 
MECITAuthor Commented:
Dell is going to replace one of the switches.
0
 
Khandakar Ashfaqur RahmanExpert/ConsultantCommented:
But you checked throughput between laptop and server directly using a cross cable.So, still there's doubt by changing switch the problem would resolve or not.
0
 
MECITAuthor Commented:
I know i will keep reasearching but for now the switch needs to be replaced anyways.
0
 
MECITAuthor Commented:
Thanks for all the help. They are replacing the switch because they believe it has other hardware symptoms that are failing.
0

Featured Post

Get your problem seen by more experts

Be seen. Boost your question’s priority for more expert views and faster solutions

  • 33
  • 27
  • 2
  • +2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now