Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.
Networking Issue / Web server Timeout / Cisco Routing Issue?
The problem is that routing fails. This might not be in the router, but maybe related to another component on the internal network.
Here is the problem:
Connections to the web server timeout. There seems to be no pattern to when these timeouts will occur. I need to get rid of these timeouts.
I have been able to setup a continuous ping to the web servers and the timeouts occur at the same point when the ping fails.
Additional Network Information:
Continuous ping from the internet to the external port on the router does not fail.
Ping from the router to the web server does not fail.
Ping from the same internet location to the web server eventually fails.
The topology is Internet < -- > Router <--> Switch <--> Web Server
I am not sure what is causing this, it might be the router, the web server or a third piece of networking equipment. I have done some testing, but will do additional testing as requested.
Here is the problem:
Connections to the web server timeout. There seems to be no pattern to when these timeouts will occur. I need to get rid of these timeouts.
I have been able to setup a continuous ping to the web servers and the timeouts occur at the same point when the ping fails.
Additional Network Information:
Continuous ping from the internet to the external port on the router does not fail.
Ping from the router to the web server does not fail.
Ping from the same internet location to the web server eventually fails.
The topology is Internet < -- > Router <--> Switch <--> Web Server
I am not sure what is causing this, it might be the router, the web server or a third piece of networking equipment. I have done some testing, but will do additional testing as requested.
Who is Participating?
Additionally once the ping fails restarting it will not work. The ping only seems to start working again after I have established a TCP connection. I use lynx to connect to the web server. Once this works (usually takes 2-3 attempts to get a page served) the ping will start working.
Is there a chance that the default route on the web server is changed when the connections do not work? This would explain why you can ping from router but not from the internet location.
The web server has four network interfaces, two are not used. The other two, one is a public IP the other is a private IP. The public IP's have been changed (in a consistent fashion).
As far as I know the default routes are set correctly...the box is running solaris 10 (ultrasparc).
The assigned IP range is 29.23.15.0 / 28 (netmask of 255.255.255.240)
Here is the output from netstat and ipconfig in case it helps:
# netstat -rn
29.23.15.176 29.23.15.181 U 1 95 bge0
29.23.15.176 29.23.15.182 U 1 0 bge0:1
192.168.0.0 192.168.0.22 U 1 103 bge1
224.0.0.0 29.23.15.181 U 1 0 bge0
default 29.23.15.17 UG 1 66058
default 192.168.0.1 UG 1 22285
127.0.0.1 127.0.0.1 UH 2 3871 lo0
# ifconfig -a
lo0: flags=2001000849<UP,LOOPBA
inet 127.0.0.1 netmask ff000000
bge0: flags=1000843<UP,BROADCAST
inet 29.23.15.181 netmask fffffff0 broadcast 29.23.15.191
bge0:1: flags=1000843<UP,BROADCAST
inet 29.23.15.182 netmask fffffff0 broadcast 29.23.15.191
bge1: flags=1000843<UP,BROADCAST
inet 192.168.0.22 netmask ffffff00 broadcast 192.168.0.255
As far as I know the default routes are set correctly...the box is running solaris 10 (ultrasparc).
The assigned IP range is 29.23.15.0 / 28 (netmask of 255.255.255.240)
Here is the output from netstat and ipconfig in case it helps:
# netstat -rn
29.23.15.176 29.23.15.181 U 1 95 bge0
29.23.15.176 29.23.15.182 U 1 0 bge0:1
192.168.0.0 192.168.0.22 U 1 103 bge1
224.0.0.0 29.23.15.181 U 1 0 bge0
default 29.23.15.17 UG 1 66058
default 192.168.0.1 UG 1 22285
127.0.0.1 127.0.0.1 UH 2 3871 lo0
# ifconfig -a
lo0: flags=2001000849<UP,LOOPBA
inet 127.0.0.1 netmask ff000000
bge0: flags=1000843<UP,BROADCAST
inet 29.23.15.181 netmask fffffff0 broadcast 29.23.15.191
bge0:1: flags=1000843<UP,BROADCAST
inet 29.23.15.182 netmask fffffff0 broadcast 29.23.15.191
bge1: flags=1000843<UP,BROADCAST
inet 192.168.0.22 netmask ffffff00 broadcast 192.168.0.255
Some additional information:
It appears the web server can not ping out. It can ping hosts on the LAN but nothing on the internet (IE google.com).
I can telnet out of the webserver, however, and DNS is configured and properly resolving:
# nslookup cnn.com
Server: 127.0.0.1
Address: 127.0.0.1#53
Non-authoritative answer:
Name: cnn.com
Address: 64.236.29.120
Name: cnn.com
Address: 64.236.16.20
Name: cnn.com
Address: 64.236.16.52
Name: cnn.com
Address: 64.236.24.12
The fact that I can not ping out seems significant, however...
It appears the web server can not ping out. It can ping hosts on the LAN but nothing on the internet (IE google.com).
I can telnet out of the webserver, however, and DNS is configured and properly resolving:
# nslookup cnn.com
Server: 127.0.0.1
Address: 127.0.0.1#53
Non-authoritative answer:
Name: cnn.com
Address: 64.236.29.120
Name: cnn.com
Address: 64.236.16.20
Name: cnn.com
Address: 64.236.16.52
Name: cnn.com
Address: 64.236.24.12
The fact that I can not ping out seems significant, however...
Hm. In the "netstat -rn" command output it is written that you have two default gateways. Why? I dont know what the last number in the output is - could you explain?
The last entry in the netstat -rn command is the loopback address, every machine should have this entry.
I will try removing the default route for the internal network and then add static routing for that network.
I will try removing the default route for the internal network and then add static routing for that network.
A couple of other things that I have done/been able to get to work:
I now have a constant ping from the server to the internet working, I'm not sure this was ever broken, the problem could have been an incorrect command line.
I have increased the size of the queues on the router. I do not believe this to have any effect since the queues were not filling, but perhaps under load something was happening.
I have removed the default route for the private network.
I will continue to monitor the pings to see if there is a failure.
I now have a constant ping from the server to the internet working, I'm not sure this was ever broken, the problem could have been an incorrect command line.
I have increased the size of the queues on the router. I do not believe this to have any effect since the queues were not filling, but perhaps under load something was happening.
I have removed the default route for the private network.
I will continue to monitor the pings to see if there is a failure.
I didn't mean the last line - I mean the last number column 66058 vs. 22285 for the default gateways
Let us know how removing one default gateway works.
Let us know how removing one default gateway works.
Oh... my fault... that's the problem with English :-D That column is a usage statistic, how much traffic has gone through the given route.
So it turns out the operating system does in fact allow you to do morbidly silly things. After removing the second default route all seems to be working. I am still seeing some timeouts, but these appear to be a bug in the 64 bit APR on Solaris.
Thanks all!
Thanks all!
Hm. If removing one of the default routes was the solution I would wish to receive some points on this question.
I was the one who pointed to a probable trouble with the default route in my first comment (default route changing). And the first one who pointed to the probable trouble with two default routes in my second comment. Both comments were made before the "accepted answer" repeated my findings...
I was the one who pointed to a probable trouble with the default route in my first comment (default route changing). And the first one who pointed to the probable trouble with two default routes in my second comment. Both comments were made before the "accepted answer" repeated my findings...
I'll see what we can do to get you some points, you did steer the discussion towards the default routes though did not explicitly suggest removing one of them.
I agree with you that you deserve points for this. I'm not sure of the procedure, it used to be I'd post a question titled "Points for Blaz" is that still what is done?
I agree with you that you deserve points for this. I'm not sure of the procedure, it used to be I'd post a question titled "Points for Blaz" is that still what is done?
Question has a verified solution.
Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.
Have a better answer? Share it in a comment.
All Courses
From novice to tech pro — start learning today.
Remove this routing entry and put static entries for your internal networks (use route add command) and put it part of your startup scripts (in /etc/rc2.d)