asked on

Networking Issue / Web server Timeout / Cisco Routing Issue?

The problem is that routing fails. This might not be in the router, but maybe related to another component on the internal network.

Here is the problem:
Connections to the web server timeout. There seems to be no pattern to when these timeouts will occur. I need to get rid of these timeouts.

I have been able to setup a continuous ping to the web servers and the timeouts occur at the same point when the ping fails.

Additional Network Information:
Continuous ping from the internet to the external port on the router does not fail.
Ping from the router to the web server does not fail.
Ping from the same internet location to the web server eventually fails.

The topology is Internet < -- > Router <--> Switch <--> Web Server

I am not sure what is causing this, it might be the router, the web server or a third piece of networking equipment. I have done some testing, but will do additional testing as requested.

asmo

ASKER

Additionally once the ping fails restarting it will not work. The ping only seems to start working again after I have established a TCP connection. I use lynx to connect to the web server. Once this works (usually takes 2-3 attempts to get a page served) the ping will start working.

Blaz

Is there a chance that the default route on the web server is changed when the connections do not work? This would explain why you can ping from router but not from the internet location.

asmo

ASKER

The web server has four network interfaces, two are not used. The other two, one is a public IP the other is a private IP. The public IP's have been changed (in a consistent fashion).

As far as I know the default routes are set correctly...the box is running solaris 10 (ultrasparc).

The assigned IP range is 29.23.15.0 / 28 (netmask of 255.255.255.240)

Here is the output from netstat and ipconfig in case it helps:

# netstat -rn
29.23.15.176 29.23.15.181 U 1 95 bge0
29.23.15.176 29.23.15.182 U 1 0 bge0:1
192.168.0.0 192.168.0.22 U 1 103 bge1
224.0.0.0 29.23.15.181 U 1 0 bge0
default 29.23.15.17 UG 1 66058
default 192.168.0.1 UG 1 22285
127.0.0.1 127.0.0.1 UH 2 3871 lo0

# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
bge0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 29.23.15.181 netmask fffffff0 broadcast 29.23.15.191
bge0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 29.23.15.182 netmask fffffff0 broadcast 29.23.15.191
bge1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 192.168.0.22 netmask ffffff00 broadcast 192.168.0.255

asmo

ASKER

Some additional information:
It appears the web server can not ping out. It can ping hosts on the LAN but nothing on the internet (IE google.com).

I can telnet out of the webserver, however, and DNS is configured and properly resolving:

# nslookup cnn.com
Server: 127.0.0.1
Address: 127.0.0.1#53

Non-authoritative answer:
Name: cnn.com
Address: 64.236.29.120
Name: cnn.com
Address: 64.236.16.20
Name: cnn.com
Address: 64.236.16.52
Name: cnn.com
Address: 64.236.24.12

The fact that I can not ping out seems significant, however...

Blaz

Hm. In the "netstat -rn" command output it is written that you have two default gateways. Why? I dont know what the last number in the output is - could you explain?

ASKER CERTIFIED SOLUTION

omarfarid

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

asmo

ASKER

The last entry in the netstat -rn command is the loopback address, every machine should have this entry.

I will try removing the default route for the internal network and then add static routing for that network.

asmo

ASKER

A couple of other things that I have done/been able to get to work:

I now have a constant ping from the server to the internet working, I'm not sure this was ever broken, the problem could have been an incorrect command line.

I have increased the size of the queues on the router. I do not believe this to have any effect since the queues were not filling, but perhaps under load something was happening.

I have removed the default route for the private network.

I will continue to monitor the pings to see if there is a failure.

Blaz

I didn't mean the last line - I mean the last number column 66058 vs. 22285 for the default gateways

Let us know how removing one default gateway works.

asmo

ASKER

Oh... my fault... that's the problem with English :-D That column is a usage statistic, how much traffic has gone through the given route.

asmo

ASKER

So it turns out the operating system does in fact allow you to do morbidly silly things. After removing the second default route all seems to be working. I am still seeing some timeouts, but these appear to be a bug in the 64 bit APR on Solaris.

Thanks all!

Blaz

Hm. If removing one of the default routes was the solution I would wish to receive some points on this question.

I was the one who pointed to a probable trouble with the default route in my first comment (default route changing). And the first one who pointed to the probable trouble with two default routes in my second comment. Both comments were made before the "accepted answer" repeated my findings...

asmo

ASKER

I'll see what we can do to get you some points, you did steer the discussion towards the default routes though did not explicitly suggest removing one of them.

I agree with you that you deserve points for this. I'm not sure of the procedure, it used to be I'd post a question titled "Points for Blaz" is that still what is done?