[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

Apache sends partial data to browser and hangs indefinitely on random pages

Posted on 2006-04-12
32
Medium Priority
?
334 Views
Last Modified: 2010-03-04
I am running Apache/2.0.55 (Unix) mod_ssl/2.0.55 OpenSSL/0.9.7i PHP/4.4.2

Here is the configure command:
'./configure' '--with-apxs2=/usr/local/apache20/bin/apxs' '--with-mysql' '--with-gd' '--with-gd2' '--with-mbstring' '--with-mcrypt' '--with-zlib' '--with-curl' '--enable-gd' '--enable-zlib' '--enable-mysql' '--enable-gd2' '--enable-mbstring' '--enable-curl' '--enable-mcrypt'

Registered PHP streams are: php, http, ftp, compress.zlib

Problem Description:

The server will hang on random pages. I monitor which pages are hanging by using the server-status extension that comes with apache. When the server hangs, it usually sends partial HTML data to the browser and then stops. The connection is never closed, and the page remains in a "sending data" status indefinately. Whatever page hung eventually times out, but no information is given on what the problem was. I have been around in circles for days on this issue and have worked closely with my server administrator on this issue and have not been able to find a work-around. I have customers who are very frustrated right now because of this problem, so the matter is urgent. I have placed the highest point value on this issue. Please help.

Jon Watson
Sherlock Watson Software
0
Comment
Question by:jcwatson11
  • 19
  • 6
  • 5
  • +2
32 Comments
 

Author Comment

by:jcwatson11
ID: 16441611
One more thing. This does not only occur on PHP pages. It also occurs on the server-status page, with hte same random frequency. About 1 in every 100 page requests sends partial data and then never completes sending data, and never closes the connection to the browser.
0
 

Author Comment

by:jcwatson11
ID: 16441616
For more PHP configuration information about the server, visit http://www.sherlockwatson.com/env.php
0
 
LVL 10

Expert Comment

by:sleep_furiously
ID: 16441878
This may not really be a solution, but might help alleviate the problem while you work on the root cause.

I notice from your PHP config page that you have MaxRequestsPerChild 0 (meaning child process will not be recycled).

You may want to assign a non-zero number (default is 10000).

MaxRequestsPerChild 10000

When a child process has handled 10000 requests, it will be retired (terminated).  Fresh child processes will be spawned as needed.  This helps in case of memory/handle leaks, etc.
0
Veeam and MySQL: How to Perform Backup & Recovery

MySQL and the MariaDB variant are among the most used databases in Linux environments, and many critical applications support their data on them. Watch this recorded webinar to find out how Veeam Backup & Replication allows you to get consistent backups of MySQL databases.

 

Author Comment

by:jcwatson11
ID: 16442600
The default is zero (0) which means unlimited. However, I changed this value to a "large number" (10000) as recommended by talks.php.net for PHP performance tuning. The number desplayed in the env.php is the number I changed it to. Normally it would be a zero, meaning unlimited.
0
 

Author Comment

by:jcwatson11
ID: 16442602
I just changed it to a smaller number (100) in hopes that respawning might alieviate the problem. Lets see if it helps.
0
 

Author Comment

by:jcwatson11
ID: 16442607
The problem persists, this time after 4 page hits. I have changed the number back to 10000.
0
 
LVL 27

Accepted Solution

by:
Nopius earned 2000 total points
ID: 16443290
Set 'ExtendedStatus On' and look to status what hapens when someone 'hangs'.
Also look to your error logs.
Your problem is only with dynamic pages (/server-status is also dymanic).

Also I've noticed blocked 'icmp' and I'm unable to ping your host - this may lead to problem like you have described (you may read more about TCP path MTU discovery if you like)

Try to open your firewall for all icmp and see what happens.
0
 

Author Comment

by:jcwatson11
ID: 16443916
I will try to get my admin people to open up ICMP traffic on the servers. But the part about that that doesn't make sense is that it would be affecting all traffic on the webserver, including images, and other static content because TCP doesn't care whether PHP generated the file, or if it is being read directly from a hard-disk.

I suppose the best way to test this is to get a page with a bunch of images on it and see how it loads...

Well, there you go. I just tried a page with a bunch of images, and found it hanging immediately on one of them. So since the image is static, this proves that the problem is not just with dynamic pages.

Okay. If turning on ICMP traffic fixes this problem, I'm going to send you a gift-certificate to your favorite restaurant. If so, I'll make it enough for two if you have someone in mind.

Jon
0
 
LVL 27

Expert Comment

by:Nopius
ID: 16444013
'because TCP doesn't care whether PHP generated the file' ,,,
on my expereince, problems begin when data transfer for single TCP session becomes > 40K,
When keepalive is enabled (and you have keepalives) , every request-response will go through 1 TCP session and problem may appear suddenly.
When you have requests like open-get-close, there is no problem.
0
 

Author Comment

by:jcwatson11
ID: 16444019
So is it possible to end the problem by turning off KeepAlive?
0
 
LVL 15

Expert Comment

by:periwinkle
ID: 16444937
What information (if any) is logged in the error logs?
0
 

Author Comment

by:jcwatson11
ID: 16448231
Good question, however, nothing gets logged in the error logs. Even the access log does not show the request for the hanging page. This would be consistent with the fact that Apache doesn't actually log the access request until the page was either successfully delivered, or failed and the connection is closed (as far as I understand it anyway).
0
 

Author Comment

by:jcwatson11
ID: 16449560
I finally got my admin people to re-enable ICMP traffic to my server. Initial testing is very promising. So far I am unable to get the pages to hang like they were before. Before, if the page hung at all (longer than a split second) it would hang forever. Now if it hangs, I only have to wait a half-second longer and the page continues to load. The answer that makes the most sense in this case would be the MTU discovery answer from Nopius.

I am going to give this 1 more day to pass ratification with my customers. But if it does, I would like to award Nopius the entire points and send him a gift certificate to his favorite restaurant. In the mean-time while we wait for customer feedback, you can email me your address at 'jon at sherlockwatson dot com' where the appropraite words are replaced with the appropriate symbols. I do this only to avoid spam robots finding my email address on this thread.
0
 
LVL 27

Expert Comment

by:Nopius
ID: 16450241
jcwatson11 thank you, but test this solution first :-)
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 16451676
does this happen with a specific browser only (in particular IE)?
does it happen with SSL only?
0
 

Author Comment

by:jcwatson11
ID: 16451992
When the bug was occurring, I tested in MS IE and Mozilla Firefox. Both showed the same results.

Now also, after ICMP traffic has been enabled, so far I have not been able to get the pages to hang, and my customers' complaints have magically disappeared. I will give this a full-day tomorrow with my customers because the solution (of turning on ICMP traffic) was activated late in the work-day.
0
 

Author Comment

by:jcwatson11
ID: 16451997
Also, before I asked this question, we had tried with and without SSL, and the results were the same.
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 16452069
ICMP has nothing to do with this problem
It's either a low level network problem (MTU, frame size, packet size, etc.) or your apache does not handle the requests proper (which is unusal).
Some browsers cause problems if they use HTTP/1.1 and Connection:Keep-Alive.
0
 

Author Comment

by:jcwatson11
ID: 16452209
I must defer to your decades of experience. However, may I humbly submit an article I found at Nobius' prompting to search for the key-words TCP path MTU discovery. The link is here: http://www.netheaven.com/pmtu.html

The article discusses how "modern servers" use an MTU negotiation process that communicates entirely using ICMP. I believe that the original purpose of ICMP was to insure packet integrity anyway, right?
0
 

Author Comment

by:jcwatson11
ID: 16452227
Just out of curiosity, ahoffmann, what makes you so sure that ICMP has nothing to do with the problem? Isn't ICMP a "low level network" thing? It's even under the TCP/IP layer if I understand correctly.
0
 

Author Comment

by:jcwatson11
ID: 16452239
Oops! I was wrong about that. ICMP is inside the TCP/IP layer as shown at http://www.google.com/search?hl=en&q=define%3A+ICMP

0
 
LVL 27

Expert Comment

by:Nopius
ID: 16458766
ahoffman, if you don't beleave me, beleave someone else: http://alive.znep.com/~marcs/mtu/
Here is exact description of that problem, conditions what it occures.
Usually very restrictive  firewalls on routers are the real problem for webhosts behind them.
Try to traceroute to www.sharlockwatson.com from outside.
19  vl-1.ar3.phx.cybertrails.com (162.42.145.163)  323.794 ms * *
20  * * *
21  * * *
from now to the end, no replies are come from any router, I suppose that vl-1.ar3.phx.cybertrails.com have some kind of 'smart defence' . This may lead to unpredictable results. I've seen problem that was very hard to resolve (forwarded dns requests where not replied only when DNS server asked for resolution, but was OK from clients) because there was 'smart defense' on some firewall between two servers.

 jcwatson11, you may check if this problem is 100% in MTU discovery or not, by changing your local machine settings (disable MTU discovery) and turning on firewall rules back.
On Windows it may be done by fixing regestry (set to 0): HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters\EnablePMTUDiscovery

on Linux you should write 1 to some file: 'echo 1 > /proc/sys/net/ipv4/ip_no_pmtu_disc'
But use it only for test purposes. Normally PMTU-D should be enabled.

Also here is a link from Cisco site: http://www.cisco.com/warp/public/105/38.shtml#prob_desc
 
The problem may arise when client is connected via some kind of VPN, ppp or DSL (pppoe), where MTU is lower then 1500. (1492 for PPPOE)
0
 

Author Comment

by:jcwatson11
ID: 16459333
Okay, I have checked with many of my clients today and they all seem to agree that they are no longer experiencing this problem. Since the only thing I have changed in order to help this is enabling ICMP traffic on the sherlockwatson.com network, Nopius' answer seems to make the most sense. Unfortunately, I don't have the leisure of turning ICMP traffic off again to do further testing. If I had direct access to the firewall settings, I might tinker with it in order to be more certain. Perhaps turning off PMTU-D would help someone who has no way of changing the firewall settings? But in my case, I would have to submit another request to my server administrator and wait. This would put my customers in another awkward situation because the pages would most likely hang again. And in that case, I don't want to risk the chance of this alternative action (turning off PMTU-D) not working.

Regardless, turning on ICMP traffic has seemed to be "the proof in the pudding". So points go to Nopius.

Nopius, so far I have not received an email from you about where to send a gift certificate for your favorite restaurant. If you are interested, send it to jon at sherlockwatson dot com.

-- Jon
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 16475501
> ahoffman, if you don't beleave me, ..
I don't disagree with the MTU thing at all, but if there is a need to open ICMP in the firewall, I'd get rid of the either the web server or the browser. ICMP is obsolete for working with TCP/IP (even if it is the a low level part of IP, we don't need it). traceroute (tracert) is a nice thing for testing connections, if ICMP (or SMB) is enabled, but all good firewalls have ICMP diabled for security reason.
That's why I asked about ICMP. Hope this explains it a bit more.
0
 

Author Comment

by:jcwatson11
ID: 16475527
ahoffman,

I believe you have a great deal of experience in this and I value your input. Can you explain how modern TCP/IP can solve the MTU negotiation described in the thread above without ICMP? I am running MS Internet Explorer 6 and Apache 2.0, which are the most current stable versions of both browser and server. And yet, they seem to miscommunicate unless ICMP traffic is enabled. Is there a better solution?

-- Jon
0
 

Author Comment

by:jcwatson11
ID: 16475546
I found an article on google that gives partial validity to your claim. According to:

http://www.aarnet.edu.au/engineering/networkdesign/qos/icmp.html

SOME ICMP traffic is obsolete, but not all. The article admits for example that ICMP traffic that used to be used to perform certain DHCP functions is obsolete, etc (obviously because DHCP handles it now). However, the article still gives validity to the claim that ICMP traffic is still used for "enabling hosts to discover the path MTU.
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 16477000
> .. which are the most current stable versions of both browser and server. And yet, they seem to miscommunicate unless ICMP traffic is enabled. Is there a better solution?
the application (either browser or web server) is not responsible for setting up basic network configuration (like MTU)
If there's someone claiming that ICMP is reuired (in some cases) for MTU negotiation, (s)he's barking up the wrong tree. TCP/IP is independent of ICMP, it does not need ICMP, and a good firewall always blocks ICMP.
Or do we talk about different things (ICMP) here?

If it helped in your case anyway, bee happy (in this case:).
0
 

Author Comment

by:jcwatson11
ID: 16480497
>... Or do we talk about different things (ICMP) here?

As far as I know, ICMP is used for many different things, but it is all on the same level, and ICMP is actually an integral part of TCP/IP as far as I understand. But I really am not the expert. I was hoping the article posted above from www.aarnet.edu.au would answer that question. Did you review it? Section 4.1.1 is especially interesting to this discussion because it warns never to block this type of ICMP traffic. And this is not the only source online that warns against blocking this type of ICMP traffic.

I was also hoping you might be able to explain in more depth <b>why</b> and <b>how</b> ICMP is not the right answer, perhaps responding to the articles quoted above by explaining what the alternatives are? So far I hear you saying <b>that</b> it is the wrong answer, but I do encourage you to help me understand why and how to get around its necessity for PMTU-D. Is there an alternate router configuration we should use? Is there a different router we should get? etc. Do you have any outside sources that can validate your claim that <b>all</b> ICMP traffic is archaic, including PMTU-D? I would imagine those sources would offer some alternative way around this problem.

Again, here is another article (the internet seems rife with them) that explains why PMTU-D is important, and warns against blindly filtering all ICMP traffic because of the problems it causes: http://alive.znep.com/~marcs/mtu/. And the more I keep looking online, the more all the sources seem to agree that it is a bad idea to block ICMP traffic that has to do with PMTU-D. Please note that the article does admit that there are many very good reasons to block other types of ICMP traffic, but there seems to be a consensus online that blocking PMTU-D seems to cause problems.

-- Jon
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 16482320
> I was also hoping you might be able to explain in more depth <b>why</b> and <b>how</b> ICMP is not the right answer,
see http:#16477000 : in other words 'cause HTTP is an TCP/IP protocol and does *not* need ICMP anyhow

> ..  would answer that question.
no that article does not answer the question.
If you read carefully you'll realize that most parts talk about subnets and router-to-router messages. Just 4.1.1 is about end-to-end messages, and that's what I mean by obsolete: no good firewall will allow such messages 'cause it indicates that the requested resources is known or not, while the absence of this message indicates nothing.
I guess that no application programmer relies on such things.

> .. and warns against blindly filtering all ICMP traffic ..
again: if someone relies on ICMP when doing TCP/IP (s)he did not understand how it works. ICMP is obsolte for modern applications.

> .. that it is a bad idea to block ICMP traffic ..
depends on the point of view: ICMP crossing boundaries of trusted networks should be blocked if security counts.

> ..  there are many very good reasons ..
there're good reason having the whole ICMP working, but not outside well-known networks (like your own one). It's nice to have, but not a must, and you better block it if security counts.
0
 
LVL 27

Expert Comment

by:Nopius
ID: 16483059
'I guess that no application programmer relies on such things' - of course.
ICMP is used on another layer of your favorit OS (I don't know which one is yours). NOT on application layer.
Look inside network protocol drivers (IP protocol).

'ICMP is obsolte for modern applications.' - also true
as I said application knows nothing of ICMP, but drivers

'ICMP crossing boundaries of trusted networks should be blocked if security counts.' - partitially true, not ALL ICMP traffic.
jcwatson11 has already provided link to which ICMP messages SHOULD be passed. I've suggested to open all icmp just for simplicity of firewall configuration. They may close unnessessary icmp types.
0
 

Author Comment

by:jcwatson11
ID: 16484728
Nopius and ahoffman,

I believe we have a consensus that at least Path MTU Discovery is necessary ICMP traffic. I am willing to concede any other points about ICMP traffic because it appears that the only necessary one is PMTU-D. If we all agree on that, I think we are all saying about the same thing.

Agreed?

Jon
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 16484793
think we still agree in most parts, except that I insist on my paranoid security settings: ICMP is obsolete (except see my comments above ;-)
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

As Wikipedia explains 'robots.txt' as -- the robot exclusion standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a websit…
Over the last year I have answered a couple of basic URL rewriting questions several times so I thought I might as well have a stab at: explaining the basics, providing a few useful links and consolidating some of the most common queries into a sing…
In a question here at Experts Exchange (https://www.experts-exchange.com/questions/29062564/Adobe-acrobat-reader-DC.html), a member asked how to create a signature in Adobe Acrobat Reader DC (the free Reader product, not the paid, full Acrobat produ…
When cloud platforms entered the scene, users and companies jumped on board to take advantage of the many benefits, like the ability to work and connect with company information from various locations. What many didn't foresee was the increased risk…
Suggested Courses
Course of the Month19 days, 5 hours left to enroll

834 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question