Apache sends partial data to browser and hangs indefinitely on random pages

I am running Apache/2.0.55 (Unix) mod_ssl/2.0.55 OpenSSL/0.9.7i PHP/4.4.2

Here is the configure command:
'./configure' '--with-apxs2=/usr/local/apache20/bin/apxs' '--with-mysql' '--with-gd' '--with-gd2' '--with-mbstring' '--with-mcrypt' '--with-zlib' '--with-curl' '--enable-gd' '--enable-zlib' '--enable-mysql' '--enable-gd2' '--enable-mbstring' '--enable-curl' '--enable-mcrypt'

Registered PHP streams are: php, http, ftp, compress.zlib

Problem Description:

The server will hang on random pages. I monitor which pages are hanging by using the server-status extension that comes with apache. When the server hangs, it usually sends partial HTML data to the browser and then stops. The connection is never closed, and the page remains in a "sending data" status indefinately. Whatever page hung eventually times out, but no information is given on what the problem was. I have been around in circles for days on this issue and have worked closely with my server administrator on this issue and have not been able to find a work-around. I have customers who are very frustrated right now because of this problem, so the matter is urgent. I have placed the highest point value on this issue. Please help.

Jon Watson
Sherlock Watson Software
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

jcwatson11Author Commented:
One more thing. This does not only occur on PHP pages. It also occurs on the server-status page, with hte same random frequency. About 1 in every 100 page requests sends partial data and then never completes sending data, and never closes the connection to the browser.
jcwatson11Author Commented:
For more PHP configuration information about the server, visit http://www.sherlockwatson.com/env.php
This may not really be a solution, but might help alleviate the problem while you work on the root cause.

I notice from your PHP config page that you have MaxRequestsPerChild 0 (meaning child process will not be recycled).

You may want to assign a non-zero number (default is 10000).

MaxRequestsPerChild 10000

When a child process has handled 10000 requests, it will be retired (terminated).  Fresh child processes will be spawned as needed.  This helps in case of memory/handle leaks, etc.
Acronis True Image 2019 just released!

Create a reliable backup. Make sure you always have dependable copies of your data so you can restore your entire system or individual files.

jcwatson11Author Commented:
The default is zero (0) which means unlimited. However, I changed this value to a "large number" (10000) as recommended by talks.php.net for PHP performance tuning. The number desplayed in the env.php is the number I changed it to. Normally it would be a zero, meaning unlimited.
jcwatson11Author Commented:
I just changed it to a smaller number (100) in hopes that respawning might alieviate the problem. Lets see if it helps.
jcwatson11Author Commented:
The problem persists, this time after 4 page hits. I have changed the number back to 10000.
Artysystem administratorCommented:
Set 'ExtendedStatus On' and look to status what hapens when someone 'hangs'.
Also look to your error logs.
Your problem is only with dynamic pages (/server-status is also dymanic).

Also I've noticed blocked 'icmp' and I'm unable to ping your host - this may lead to problem like you have described (you may read more about TCP path MTU discovery if you like)

Try to open your firewall for all icmp and see what happens.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
jcwatson11Author Commented:
I will try to get my admin people to open up ICMP traffic on the servers. But the part about that that doesn't make sense is that it would be affecting all traffic on the webserver, including images, and other static content because TCP doesn't care whether PHP generated the file, or if it is being read directly from a hard-disk.

I suppose the best way to test this is to get a page with a bunch of images on it and see how it loads...

Well, there you go. I just tried a page with a bunch of images, and found it hanging immediately on one of them. So since the image is static, this proves that the problem is not just with dynamic pages.

Okay. If turning on ICMP traffic fixes this problem, I'm going to send you a gift-certificate to your favorite restaurant. If so, I'll make it enough for two if you have someone in mind.

Artysystem administratorCommented:
'because TCP doesn't care whether PHP generated the file' ,,,
on my expereince, problems begin when data transfer for single TCP session becomes > 40K,
When keepalive is enabled (and you have keepalives) , every request-response will go through 1 TCP session and problem may appear suddenly.
When you have requests like open-get-close, there is no problem.
jcwatson11Author Commented:
So is it possible to end the problem by turning off KeepAlive?
What information (if any) is logged in the error logs?
jcwatson11Author Commented:
Good question, however, nothing gets logged in the error logs. Even the access log does not show the request for the hanging page. This would be consistent with the fact that Apache doesn't actually log the access request until the page was either successfully delivered, or failed and the connection is closed (as far as I understand it anyway).
jcwatson11Author Commented:
I finally got my admin people to re-enable ICMP traffic to my server. Initial testing is very promising. So far I am unable to get the pages to hang like they were before. Before, if the page hung at all (longer than a split second) it would hang forever. Now if it hangs, I only have to wait a half-second longer and the page continues to load. The answer that makes the most sense in this case would be the MTU discovery answer from Nopius.

I am going to give this 1 more day to pass ratification with my customers. But if it does, I would like to award Nopius the entire points and send him a gift certificate to his favorite restaurant. In the mean-time while we wait for customer feedback, you can email me your address at 'jon at sherlockwatson dot com' where the appropraite words are replaced with the appropriate symbols. I do this only to avoid spam robots finding my email address on this thread.
Artysystem administratorCommented:
jcwatson11 thank you, but test this solution first :-)
does this happen with a specific browser only (in particular IE)?
does it happen with SSL only?
jcwatson11Author Commented:
When the bug was occurring, I tested in MS IE and Mozilla Firefox. Both showed the same results.

Now also, after ICMP traffic has been enabled, so far I have not been able to get the pages to hang, and my customers' complaints have magically disappeared. I will give this a full-day tomorrow with my customers because the solution (of turning on ICMP traffic) was activated late in the work-day.
jcwatson11Author Commented:
Also, before I asked this question, we had tried with and without SSL, and the results were the same.
ICMP has nothing to do with this problem
It's either a low level network problem (MTU, frame size, packet size, etc.) or your apache does not handle the requests proper (which is unusal).
Some browsers cause problems if they use HTTP/1.1 and Connection:Keep-Alive.
jcwatson11Author Commented:
I must defer to your decades of experience. However, may I humbly submit an article I found at Nobius' prompting to search for the key-words TCP path MTU discovery. The link is here: http://www.netheaven.com/pmtu.html

The article discusses how "modern servers" use an MTU negotiation process that communicates entirely using ICMP. I believe that the original purpose of ICMP was to insure packet integrity anyway, right?
jcwatson11Author Commented:
Just out of curiosity, ahoffmann, what makes you so sure that ICMP has nothing to do with the problem? Isn't ICMP a "low level network" thing? It's even under the TCP/IP layer if I understand correctly.
jcwatson11Author Commented:
Oops! I was wrong about that. ICMP is inside the TCP/IP layer as shown at http://www.google.com/search?hl=en&q=define%3A+ICMP

Artysystem administratorCommented:
ahoffman, if you don't beleave me, beleave someone else: http://alive.znep.com/~marcs/mtu/
Here is exact description of that problem, conditions what it occures.
Usually very restrictive  firewalls on routers are the real problem for webhosts behind them.
Try to traceroute to www.sharlockwatson.com from outside.
19  vl-1.ar3.phx.cybertrails.com (  323.794 ms * *
20  * * *
21  * * *
from now to the end, no replies are come from any router, I suppose that vl-1.ar3.phx.cybertrails.com have some kind of 'smart defence' . This may lead to unpredictable results. I've seen problem that was very hard to resolve (forwarded dns requests where not replied only when DNS server asked for resolution, but was OK from clients) because there was 'smart defense' on some firewall between two servers.

 jcwatson11, you may check if this problem is 100% in MTU discovery or not, by changing your local machine settings (disable MTU discovery) and turning on firewall rules back.
On Windows it may be done by fixing regestry (set to 0): HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters\EnablePMTUDiscovery

on Linux you should write 1 to some file: 'echo 1 > /proc/sys/net/ipv4/ip_no_pmtu_disc'
But use it only for test purposes. Normally PMTU-D should be enabled.

Also here is a link from Cisco site: http://www.cisco.com/warp/public/105/38.shtml#prob_desc
The problem may arise when client is connected via some kind of VPN, ppp or DSL (pppoe), where MTU is lower then 1500. (1492 for PPPOE)
jcwatson11Author Commented:
Okay, I have checked with many of my clients today and they all seem to agree that they are no longer experiencing this problem. Since the only thing I have changed in order to help this is enabling ICMP traffic on the sherlockwatson.com network, Nopius' answer seems to make the most sense. Unfortunately, I don't have the leisure of turning ICMP traffic off again to do further testing. If I had direct access to the firewall settings, I might tinker with it in order to be more certain. Perhaps turning off PMTU-D would help someone who has no way of changing the firewall settings? But in my case, I would have to submit another request to my server administrator and wait. This would put my customers in another awkward situation because the pages would most likely hang again. And in that case, I don't want to risk the chance of this alternative action (turning off PMTU-D) not working.

Regardless, turning on ICMP traffic has seemed to be "the proof in the pudding". So points go to Nopius.

Nopius, so far I have not received an email from you about where to send a gift certificate for your favorite restaurant. If you are interested, send it to jon at sherlockwatson dot com.

-- Jon
> ahoffman, if you don't beleave me, ..
I don't disagree with the MTU thing at all, but if there is a need to open ICMP in the firewall, I'd get rid of the either the web server or the browser. ICMP is obsolete for working with TCP/IP (even if it is the a low level part of IP, we don't need it). traceroute (tracert) is a nice thing for testing connections, if ICMP (or SMB) is enabled, but all good firewalls have ICMP diabled for security reason.
That's why I asked about ICMP. Hope this explains it a bit more.
jcwatson11Author Commented:

I believe you have a great deal of experience in this and I value your input. Can you explain how modern TCP/IP can solve the MTU negotiation described in the thread above without ICMP? I am running MS Internet Explorer 6 and Apache 2.0, which are the most current stable versions of both browser and server. And yet, they seem to miscommunicate unless ICMP traffic is enabled. Is there a better solution?

-- Jon
jcwatson11Author Commented:
I found an article on google that gives partial validity to your claim. According to:


SOME ICMP traffic is obsolete, but not all. The article admits for example that ICMP traffic that used to be used to perform certain DHCP functions is obsolete, etc (obviously because DHCP handles it now). However, the article still gives validity to the claim that ICMP traffic is still used for "enabling hosts to discover the path MTU.
> .. which are the most current stable versions of both browser and server. And yet, they seem to miscommunicate unless ICMP traffic is enabled. Is there a better solution?
the application (either browser or web server) is not responsible for setting up basic network configuration (like MTU)
If there's someone claiming that ICMP is reuired (in some cases) for MTU negotiation, (s)he's barking up the wrong tree. TCP/IP is independent of ICMP, it does not need ICMP, and a good firewall always blocks ICMP.
Or do we talk about different things (ICMP) here?

If it helped in your case anyway, bee happy (in this case:).
jcwatson11Author Commented:
>... Or do we talk about different things (ICMP) here?

As far as I know, ICMP is used for many different things, but it is all on the same level, and ICMP is actually an integral part of TCP/IP as far as I understand. But I really am not the expert. I was hoping the article posted above from www.aarnet.edu.au would answer that question. Did you review it? Section 4.1.1 is especially interesting to this discussion because it warns never to block this type of ICMP traffic. And this is not the only source online that warns against blocking this type of ICMP traffic.

I was also hoping you might be able to explain in more depth <b>why</b> and <b>how</b> ICMP is not the right answer, perhaps responding to the articles quoted above by explaining what the alternatives are? So far I hear you saying <b>that</b> it is the wrong answer, but I do encourage you to help me understand why and how to get around its necessity for PMTU-D. Is there an alternate router configuration we should use? Is there a different router we should get? etc. Do you have any outside sources that can validate your claim that <b>all</b> ICMP traffic is archaic, including PMTU-D? I would imagine those sources would offer some alternative way around this problem.

Again, here is another article (the internet seems rife with them) that explains why PMTU-D is important, and warns against blindly filtering all ICMP traffic because of the problems it causes: http://alive.znep.com/~marcs/mtu/. And the more I keep looking online, the more all the sources seem to agree that it is a bad idea to block ICMP traffic that has to do with PMTU-D. Please note that the article does admit that there are many very good reasons to block other types of ICMP traffic, but there seems to be a consensus online that blocking PMTU-D seems to cause problems.

-- Jon
> I was also hoping you might be able to explain in more depth <b>why</b> and <b>how</b> ICMP is not the right answer,
see http:#16477000 : in other words 'cause HTTP is an TCP/IP protocol and does *not* need ICMP anyhow

> ..  would answer that question.
no that article does not answer the question.
If you read carefully you'll realize that most parts talk about subnets and router-to-router messages. Just 4.1.1 is about end-to-end messages, and that's what I mean by obsolete: no good firewall will allow such messages 'cause it indicates that the requested resources is known or not, while the absence of this message indicates nothing.
I guess that no application programmer relies on such things.

> .. and warns against blindly filtering all ICMP traffic ..
again: if someone relies on ICMP when doing TCP/IP (s)he did not understand how it works. ICMP is obsolte for modern applications.

> .. that it is a bad idea to block ICMP traffic ..
depends on the point of view: ICMP crossing boundaries of trusted networks should be blocked if security counts.

> ..  there are many very good reasons ..
there're good reason having the whole ICMP working, but not outside well-known networks (like your own one). It's nice to have, but not a must, and you better block it if security counts.
Artysystem administratorCommented:
'I guess that no application programmer relies on such things' - of course.
ICMP is used on another layer of your favorit OS (I don't know which one is yours). NOT on application layer.
Look inside network protocol drivers (IP protocol).

'ICMP is obsolte for modern applications.' - also true
as I said application knows nothing of ICMP, but drivers

'ICMP crossing boundaries of trusted networks should be blocked if security counts.' - partitially true, not ALL ICMP traffic.
jcwatson11 has already provided link to which ICMP messages SHOULD be passed. I've suggested to open all icmp just for simplicity of firewall configuration. They may close unnessessary icmp types.
jcwatson11Author Commented:
Nopius and ahoffman,

I believe we have a consensus that at least Path MTU Discovery is necessary ICMP traffic. I am willing to concede any other points about ICMP traffic because it appears that the only necessary one is PMTU-D. If we all agree on that, I think we are all saying about the same thing.


think we still agree in most parts, except that I insist on my paranoid security settings: ICMP is obsolete (except see my comments above ;-)
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Apache Web Server

From novice to tech pro — start learning today.