What is causing my 5 second latency?

Hosting servers have very little extra running on them.  They don't use a GUI so they don't have any overhead associated with maintaining the screen.  Or a mouse.  Their versions of Apache, PHP, and MySQL are optimized for speed where WAMP is set up to be 'easy' to use.  Mainly the bottleneck is that WAMP is running on top of Windows and your hosting server isn't.

Let's fairly compare.... on your hosting service, did you choose a Windows or Linux server?

But it is true that servers on hosting services are designed in a way to be optimized for speed and reliability. You're running everything on one server... chances are that the databases for a hosting provider are run on separate servers (take a look at the MVC model sometime.. MVC = Model, View, Controller). In hosting database servers will generally be getting a pretty heavy workload.

There are other factors also, such as other software on the machines that will be running, such as antivirus software or other server programs. Even internet connectivity is a factor.

Thanks for all the responses!

>> What happens when you turn off virus protection software?
No difference (running MSE - disabled real-time protection)


>> Have you gone through the steps of creating a simple .html page and surfing to it via localhost?
Waiting: 1ms; Receiving: 8ms (see Timeline/Network graph 01_WhereIsBottleneck_SimpleHTMLpage.png)

>>  the same but renaming it .php
Waiting: 1s; Receiving: 8ms (see 02_WhereIsBottleneck_SimpleHTMLpage.php.png)
Anything I can do about that 1s php processing module overhead?

and adding display of 5000 random numbers:
Waiting: 1s; Receiving: 17ms (see 03_WhereIsBottleneck_5000randomNumbers.php)

>> time with php connecting to the db (without random number generation)
Waiting: 1s; Receiving: 8ms (see 04_WhereIsBottleneck_ConnectToDB.php.png)

>> time with php connecting to the db and select 77430 rows
Waiting: 1.2s; Receiving: 7ms (see 05_WhereIsBottleneck_ConnectToDB&Select77430rows.php.png)

Everything seems screaming on my end except for the 1s php processing overhead - am I correct?

Whatever joomla is doing seems to be taking a lot of time...
01-WhereIsBottleneck-SimpleHTMLp.png
02-WhereIsBottleneck-SimpleHTMLp.png
03-WhereIsBottleneck-5000randomN.png
04-WhereIsBottleneck-ConnectToDB.png
05-WhereIsBottleneck-ConnectToDB.png

A status of '304 Not Modified' means that your browser is fetching the data from it's cache and not the server.  On a real server, the PHP interpreter is kept in memory.  It looks like on your computer that it has to be loaded from disk each time which is normal for a home or regular desktop computer.  And on Joomla pages, the MySQL server adds a certain amount of time to each page load because that's where a lot of the page code is kept.

Thanks for taking a look at that--

>> A status of '304 Not Modified' means that your browser is fetching the data from it's cache and not the server.
I assume this is regarding the simple html page?  If so, note that the remote server is also showing status 304.

>> On a real server, the PHP interpreter is kept in memory.  It looks like on your computer that it has to be loaded from disk each time which is normal for a home or regular desktop computer.
How can you tell this?

The remote server is where the '304' comes from.  If the browser has the file in it's cache, it sends a GET request to the server with an " If-Modified-Since field".   The server sends a '304' code back in the HTTP response to tell the browser that the file hasn't changed since the  If-Modified-Since time.  http://www.w3.org/Protocols/HTTP/HTRESP.html

Actually I should just say that is normal operation for a desktop, not that I can tell.  The one second delay is an indication though.  In simpler times we used to use 'RAMDISKs' to speed up things but I don't know how to do with modern versions of Windows.

I uploaded the test files to the remote server and compared the Timings (attached).  Most events ran faster locally except for send request which seems primarily responsible for the latency:

I am going to do the same comparison tests for the joomla home page (i.e. new empty joomla installs).

I am already way in over my head and sure would appreciate more feedback from some of you experts out there.  padas - I liked the direction you sent me... any comments?

Thanks--
01s-WhereIsBottleneck-SimpleHTML.png
02s-WhereIsBottleneck-SimpleHTML.png
03s-WhereIsBottleneck-5000random.png
04s-WhereIsBottleneck-ConnectToD.png
05s-WhereIsBottleneck-ConnectToD.png

The author has taken his question over here: https://www.experts-exchange.com/questions/27812437/What-is-causing-4-2s-latency-on-a-4-2s-Send-Request.html

I read through a lot of others with the same problem and outside of the virus software issue (are you 100% sure MSE is all you have?  Double check)   Running WAMP and IIS together is a problem.  Try turning off  IIS>

http://www.youtube.com/watch?v=64fNc7AqOMA&feature=related

http://www.youtube.com/watch?v=LGhyYmbJS7A&feature=related

Joomla debug profile information shows it takes less than 500ms to render the page, so I do not believe it is a joomla problem.  I have reposted with updated information to Apache/webserver topic.

And your other php pages also took about that time. Nothing was 6 seconds.

Thanks for your post.  Clarification: The send request for a joomla page takes almost 6 seconds with 5 second latency per my original post (it was the tests I was subsequently asked to run which took less).  

I have reduced joomla empty home page response to about 4.5 seconds with APC where the Time and Latency of GET are both 4.14s (see Network attached)
jlocal3-timings-2012-08-01.png
jlocal3-network-2012-08-01.png

Hmmm....

For local PC when in joomla debug mode, all the events (loading css/js/parsing/layout/paint...) are delayed for 3 seconds (4160ms - 1120ms).
If I have joomla debug disabled, the Receive Response happens after about 1.2s.

Very different for remote server.
So when not in joomla debug mode, my local server can turn around a request so that it is displayed by my browser in 1.12s - but after that the the browser spends 3s receiving?!  After the 3s, the page is updated with the "Font sizeBigger.Reset.Smaller" at the top (default beez20 template).

He He!  The 1s latency was because xdebug was enabled.  
That shaves 1s off the 4s latency.  I suspect the other 3s is somewhere in the joomla code base.

Still digging...

having xdebug enabled was the culprit.  commented out the "zend_extension=..." line in my php.ini and now everything is screaming fast (well, afaic!)

Latency dropped further from 4.14s to 404ms (sometimes only 200ms = faster than remote server)

Thanks to everyone for your help