How can IT departments tackle the challenges of a Big Data world? This white paper provides a roadmap to success and helps companies ensure that all their data is safe and secure, no matter if it resides on-premise with physical or virtual machines or in the cloud.
Solved
Networking Issue / Web server Timeout / Cisco Routing Issue?
Posted on 2007-11-29
The problem is that routing fails. This might not be in the router, but maybe related to another component on the internal network.
Here is the problem:
Connections to the web server timeout. There seems to be no pattern to when these timeouts will occur. I need to get rid of these timeouts.
I have been able to setup a continuous ping to the web servers and the timeouts occur at the same point when the ping fails.
Additional Network Information:
Continuous ping from the internet to the external port on the router does not fail.
Ping from the router to the web server does not fail.
Ping from the same internet location to the web server eventually fails.
The topology is Internet < -- > Router <--> Switch <--> Web Server
I am not sure what is causing this, it might be the router, the web server or a third piece of networking equipment. I have done some testing, but will do additional testing as requested.
Here is the problem:
Connections to the web server timeout. There seems to be no pattern to when these timeouts will occur. I need to get rid of these timeouts.
I have been able to setup a continuous ping to the web servers and the timeouts occur at the same point when the ping fails.
Additional Network Information:
Continuous ping from the internet to the external port on the router does not fail.
Ping from the router to the web server does not fail.
Ping from the same internet location to the web server eventually fails.
The topology is Internet < -- > Router <--> Switch <--> Web Server
I am not sure what is causing this, it might be the router, the web server or a third piece of networking equipment. I have done some testing, but will do additional testing as requested.
[X]
Welcome to Experts Exchange
Add your voice to the tech community where 5M+ people just like you are talking about what matters.
- Help others & share knowledge
- Earn cash & points
- Learn & ask questions
-
8
-
4
13 Comments
Additionally once the ping fails restarting it will not work. The ping only seems to start working again after I have established a TCP connection. I use lynx to connect to the web server. Once this works (usually takes 2-3 attempts to get a page served) the ping will start working.
Is there a chance that the default route on the web server is changed when the connections do not work? This would explain why you can ping from router but not from the internet location.
The web server has four network interfaces, two are not used. The other two, one is a public IP the other is a private IP. The public IP's have been changed (in a consistent fashion).
As far as I know the default routes are set correctly...the box is running solaris 10 (ultrasparc).
The assigned IP range is 29.23.15.0 / 28 (netmask of 255.255.255.240)
Here is the output from netstat and ipconfig in case it helps:
# netstat -rn
29.23.15.176 29.23.15.181 U 1 95 bge0
29.23.15.176 29.23.15.182 U 1 0 bge0:1
192.168.0.0 192.168.0.22 U 1 103 bge1
224.0.0.0 29.23.15.181 U 1 0 bge0
default 29.23.15.17 UG 1 66058
default 192.168.0.1 UG 1 22285
127.0.0.1 127.0.0.1 UH 2 3871 lo0
# ifconfig -a
lo0: flags=2001000849<UP,LOOPBA
inet 127.0.0.1 netmask ff000000
bge0: flags=1000843<UP,BROADCAST
inet 29.23.15.181 netmask fffffff0 broadcast 29.23.15.191
bge0:1: flags=1000843<UP,BROADCAST
inet 29.23.15.182 netmask fffffff0 broadcast 29.23.15.191
bge1: flags=1000843<UP,BROADCAST
inet 192.168.0.22 netmask ffffff00 broadcast 192.168.0.255
As far as I know the default routes are set correctly...the box is running solaris 10 (ultrasparc).
The assigned IP range is 29.23.15.0 / 28 (netmask of 255.255.255.240)
Here is the output from netstat and ipconfig in case it helps:
# netstat -rn
29.23.15.176 29.23.15.181 U 1 95 bge0
29.23.15.176 29.23.15.182 U 1 0 bge0:1
192.168.0.0 192.168.0.22 U 1 103 bge1
224.0.0.0 29.23.15.181 U 1 0 bge0
default 29.23.15.17 UG 1 66058
default 192.168.0.1 UG 1 22285
127.0.0.1 127.0.0.1 UH 2 3871 lo0
# ifconfig -a
lo0: flags=2001000849<UP,LOOPBA
inet 127.0.0.1 netmask ff000000
bge0: flags=1000843<UP,BROADCAST
inet 29.23.15.181 netmask fffffff0 broadcast 29.23.15.191
bge0:1: flags=1000843<UP,BROADCAST
inet 29.23.15.182 netmask fffffff0 broadcast 29.23.15.191
bge1: flags=1000843<UP,BROADCAST
inet 192.168.0.22 netmask ffffff00 broadcast 192.168.0.255
Some additional information:
It appears the web server can not ping out. It can ping hosts on the LAN but nothing on the internet (IE google.com).
I can telnet out of the webserver, however, and DNS is configured and properly resolving:
# nslookup cnn.com
Server: 127.0.0.1
Address: 127.0.0.1#53
Non-authoritative answer:
Name: cnn.com
Address: 64.236.29.120
Name: cnn.com
Address: 64.236.16.20
Name: cnn.com
Address: 64.236.16.52
Name: cnn.com
Address: 64.236.24.12
The fact that I can not ping out seems significant, however...
It appears the web server can not ping out. It can ping hosts on the LAN but nothing on the internet (IE google.com).
I can telnet out of the webserver, however, and DNS is configured and properly resolving:
# nslookup cnn.com
Server: 127.0.0.1
Address: 127.0.0.1#53
Non-authoritative answer:
Name: cnn.com
Address: 64.236.29.120
Name: cnn.com
Address: 64.236.16.20
Name: cnn.com
Address: 64.236.16.52
Name: cnn.com
Address: 64.236.24.12
The fact that I can not ping out seems significant, however...
Hm. In the "netstat -rn" command output it is written that you have two default gateways. Why? I dont know what the last number in the output is - could you explain?
The problem could be (mostly) due to the 2nd default routing since it is going to private network (192.168.1.0).
Remove this routing entry and put static entries for your internal networks (use route add command) and put it part of your startup scripts (in /etc/rc2.d)
Remove this routing entry and put static entries for your internal networks (use route add command) and put it part of your startup scripts (in /etc/rc2.d)
The last entry in the netstat -rn command is the loopback address, every machine should have this entry.
I will try removing the default route for the internal network and then add static routing for that network.
I will try removing the default route for the internal network and then add static routing for that network.
A couple of other things that I have done/been able to get to work:
I now have a constant ping from the server to the internet working, I'm not sure this was ever broken, the problem could have been an incorrect command line.
I have increased the size of the queues on the router. I do not believe this to have any effect since the queues were not filling, but perhaps under load something was happening.
I have removed the default route for the private network.
I will continue to monitor the pings to see if there is a failure.
I now have a constant ping from the server to the internet working, I'm not sure this was ever broken, the problem could have been an incorrect command line.
I have increased the size of the queues on the router. I do not believe this to have any effect since the queues were not filling, but perhaps under load something was happening.
I have removed the default route for the private network.
I will continue to monitor the pings to see if there is a failure.
I didn't mean the last line - I mean the last number column 66058 vs. 22285 for the default gateways
Let us know how removing one default gateway works.
Let us know how removing one default gateway works.
Oh... my fault... that's the problem with English :-D That column is a usage statistic, how much traffic has gone through the given route.
So it turns out the operating system does in fact allow you to do morbidly silly things. After removing the second default route all seems to be working. I am still seeing some timeouts, but these appear to be a bug in the 64 bit APR on Solaris.
Thanks all!
Thanks all!
Hm. If removing one of the default routes was the solution I would wish to receive some points on this question.
I was the one who pointed to a probable trouble with the default route in my first comment (default route changing). And the first one who pointed to the probable trouble with two default routes in my second comment. Both comments were made before the "accepted answer" repeated my findings...
I was the one who pointed to a probable trouble with the default route in my first comment (default route changing). And the first one who pointed to the probable trouble with two default routes in my second comment. Both comments were made before the "accepted answer" repeated my findings...
I'll see what we can do to get you some points, you did steer the discussion towards the default routes though did not explicitly suggest removing one of them.
I agree with you that you deserve points for this. I'm not sure of the procedure, it used to be I'd post a question titled "Points for Blaz" is that still what is done?
I agree with you that you deserve points for this. I'm not sure of the procedure, it used to be I'd post a question titled "Points for Blaz" is that still what is done?
Featured Post
TCP/IP is a set of network protocols which is best known for connecting the machines that make up the Internet. The truth is that TCP/IP is one of the oldest network protocols and its survival is mainly based on its simplicity and universality.
Question has a verified solution.
If you are experiencing a similar issue, please ask a related question
If your site has a few sections that need to be secure when data is transmitted between the server and local computer, such as a /order/ section for ordering or /customer/ which contains customer data, etc it would of course be recommended to secure…
It is possible to boost certain documents at query time in Solr. Query time boosting can be a powerful resource for finding the most relevant and "best" content. Of course the more information you index, the more fields you will be able to use for y…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
If you're a developer or IT admin, you’re probably tasked with managing multiple websites, servers, applications, and levels of security on a daily basis. While this can be extremely time consuming, it can also be frustrating when systems aren't wor…
Suggested Courses
Course of the Month9 days, 1 hour left to enroll
721 members asked questions and received personalized solutions in the past 7 days.
Join the community of 500,000 technology professionals and ask your questions.