Solved

Dedicated server should be fast fast, but images very slow to load

Posted on 2013-06-05
7
313 Views
Last Modified: 2013-06-20
Hi, everyone,

The description of this problem is a little long, but it's really just one problem.

I have a dedicated server running Apache 2.2.3 on Red Hat. I noticed that on the main site that the server runs, FacialSurgery.com, some images are very late to load on many pages -- sometimes taking up to ten seconds to load, sometimes not loading at all.

For example, if you go to a before-and-after page, such as this:
http://www.facialsurgery.com/BAPg1qnd2jqh1_psfd_nl08_hmc3.html 

there are a bunch of little thumbnails toward the bottom of the page. Usually, some of those thumbnails might take up to 10 seconds to load, or they don't load at all. You can browse to other patients to see the same thing happen.

Then, on most of the intra-op pages, such as:
http://www.facialsurgery.com/ClkoffTPgt3_2011_01_02mh.html 
(caution: there may be a tiny bit of blood pictured on some of the intra-op pages)
...usually a couple of the pictures just won't load, or they take *many seconds to load.

I never remember that being a problem from when I designed the site.  I called some friends in other cities to see if they were seeing the same thing, and they were.

The httpd.conf that is *currently running the site is now stripped-down.  I took out the other VirtualHost's, made FacialSurgery.com NOT a virtual host, cut down the log levels, etc.  Here is a link to the current httpd.conf:
http://www.facialsurgery.com/httpd.txt

I know I must be loading a bunch of modules that I don't need, but as I mentioned, I haven't had this problem forever. Modules that I don't need are being loaded probably because I was too lazy to find out just what they do and remove them if I didn't need them.  I'm no expert in Apache -- I just read a lot.

The site needs to be updated, of course -- the html is ponderous and doesn't completely validate, though it's not bad. Although the pages look like static .html files, the URLs get rewritten and call perl programs with parameters taken from the static-appearing URLs, and the html is then generated on the fly. But that should still all happen very quickly.   I don't know why it should take forever to get some .jpgs to the user's browser.

Here's a test page I made:
http://www.facialsurgery.com/images.html

In most browsers, some of the little pics on the test page don't load, or they take forever.  When I load this page and run "tail -f" on rewrite.log, nothing bad happens. Just: not all of the .jpgs are rewritten at first, and then, ten seconds after trying to load the page I see the last three images appear in the browser window, and there they are in rewrite.log, too.  No indication of why they were so late to load.

Okay, so things are clunky.  Still, there's a Red Hat physical (non-virtual) server doing nothing at all but serving this one site, and seems to me it should be waaaaaay faster.

I've browsed around to all the "10 ways to speed up your site" pages. I haven't yet contacted tech support at my server farm to see if there might be a problem with the server's connection to the Internet pipes.

Running "free -m" shows that I've got 4 gig, and almost all of it is free.
When I run "top", my load average is 0.15, 0.23, 0.25, and cpu is 99.1%id.
If I try to download a large file to my server, it runs at about 11M/s.

I'm dying to get some suggestions of how to make all the images on these pages blit immediately into place.  Seems like it should be possible!

Thanks for any assistance.

--Steve D.
0
Comment
Question by:StevenMiles
  • 5
  • 2
7 Comments
 
LVL 28

Assisted Solution

by:FishMonger
FishMonger earned 500 total points
ID: 39224558
All of the pages loaded fine for me on their initial load but when I refreshed the pages (Ctrl-F5), they loaded very slowly.  Since you haven't posted the perl code that generates the pages, I can't say if the script(s) are at fault.

Have you tried profiling the scripts?
0
 

Author Comment

by:StevenMiles
ID: 39225827
Hi, FishMonger,

Thanks for helping. You don't want to see the perl code -- they're spaghetti.  But they take less than 0.1 second to complete.

And, at least on my systems, even images.html, where no perl is exectued, fails to deliver all the images without a big delay.

--Steve
0
 

Author Comment

by:StevenMiles
ID: 39229946
More: let's say I try to load:
http://www.facialsurgery.com/images.html

when I "tail -f" the access log, I see that, when a couple images show up very late, around 15 seconds after the rest of the page loads, those images only appear in the access.log exactly at the time they appear, with a date stamp 15 seconds after the date stamp of the other 30 images, which show up in the access.log instantaneously.

So does my server only receive the request from the browser late? Or does the server receive all of the image requests immediately, and only "processes" the late photos later? Is there a way to tell?

And why should the problem be the same for many different browsers? And for the same few photos each time?  There doesn't seem to be any problem with the photos as they exist in their directories.

More yet: I created another test file:
http://www.facialsurgery.com/images_b.html

In images.html, it's often jpgs number 3, 4, and 5 that show up late.  So in images_b.html, I moved those jpgs to the bottom of the page. And now its the 4th, 5th, and 6th images that load very late, and the images at the bottom of the page load fine. And it happens the same in Chrome and Safari.

Something very fishy is happening.

/SD
0
Microsoft Certification Exam 74-409

Veeam® is happy to provide the Microsoft community with a study guide prepared by MVP and MCT, Orin Thomas. This guide will take you through each of the exam objectives, helping you to prepare for and pass the examination.

 
LVL 28

Assisted Solution

by:FishMonger
FishMonger earned 500 total points
ID: 39230109
Can you run a packet capture from both the server side and the client?  Comparing the 2 captures should give a clue to the problem.
0
 

Author Comment

by:StevenMiles
ID: 39250550
Hi,

Sorry for the delay.  Still having no luck. The server farm doesn't see a problem, maybe because they are there!

Here are a couple of traceroutes to the site in question.  Can anyone help me interpret them? "peer1" is the name of the server farm company. They are in Fremont, CA. (by the way, I have always been verrrry happy with their tech support)


---------------------
This from Des Moines, IA:
s-s-computer:~ paul$ traceroute facialsurgery.com
traceroute to facialsurgery.com (216.65.63.184), 64 hops max, 52 byte packets
 1  192.168.1.1 (192.168.1.1)  1.159 ms  0.525 ms  0.526 ms
 2  * 172.30.71.81 (172.30.71.81)  6.437 ms  8.064 ms
 3  172.30.1.145 (172.30.1.145)  8.188 ms  7.409 ms  7.792 ms
 4  12.249.52.13 (12.249.52.13)  7.695 ms  8.517 ms  6.515 ms
 5  cr2.kc9mo.ip.att.net (12.122.150.206)  24.985 ms  26.253 ms  23.747 ms
 6  cr1.dlstx.ip.att.net (12.122.28.85)  71.548 ms  54.260 ms  45.439 ms
 7  12.122.212.9 (12.122.212.9)  23.925 ms  23.748 ms  23.857 ms
 8  * * *
 9  if-2-2.tcore2.dt8-dallas.as6453.net (66.110.56.6)  124.514 ms  84.020 ms  88.691 ms
10  if-8-2.tcore1.lvw-losangeles.as6453.net (66.110.57.82)  88.844 ms * *
11  if-2-2.tcore2.lvw-losangeles.as6453.net (66.110.59.2)  86.565 ms  86.015 ms  85.256 ms
12  if-3-2.tcore1.sqn-sanjose.as6453.net (63.243.205.13)  94.202 ms  90.016 ms  91.331 ms
13  66.110.8.50 (66.110.8.50)  88.993 ms  88.037 ms  90.889 ms
14  * * *
15  * * *
16  vlan3.fmt6509-1.fmt.peer1.net (66.40.24.106)  93.280 ms  93.263 ms  88.082 ms
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *^C
---------------------


---------------------
This from Northern California:
 1  192.168.168.1 (192.168.168.1)  0.583 ms  0.420 ms  0.362 ms
 2  10.22.7.1 (10.22.7.1)  2.228 ms  2.239 ms  2.723 ms
 3  10.254.6.254 (10.254.6.254)  9.395 ms  7.633 ms  6.955 ms
 4  10.252.70.1 (10.252.70.1)  12.120 ms  9.502 ms  6.159 ms
 5  172.20.7.153 (172.20.7.153)  11.607 ms  8.417 ms  7.274 ms
 6  172.20.8.225 (172.20.8.225)  21.584 ms  24.140 ms  18.152 ms
 7  172.20.8.217 (172.20.8.217)  23.514 ms  18.353 ms  16.420 ms
 8  172.20.105.1 (172.20.105.1)  18.346 ms  16.329 ms  20.980 ms
 9  172.20.200.254 (172.20.200.254)  18.262 ms  22.326 ms  24.385 ms
10  172.24.0.4 (172.24.0.4)  32.965 ms  19.416 ms  26.196 ms
11  xo-edge.calwisp.com (64.55.111.65)  20.028 ms  25.480 ms  25.022 ms
12  216.55.44.129 (216.55.44.129)  46.137 ms  29.933 ms  50.207 ms
13  vb1510.rar3.sanjose-ca.us.xo.net (216.156.0.153)  40.399 ms  41.599 ms  38.945 ms
14  * * 207.88.14.226.ptr.us.xo.net (207.88.14.226)  70.454 ms
15  xe-1-1-0.sjc12.ip4.tinet.net (77.67.79.217)  46.273 ms  39.983 ms  23.498 ms
16  peer1-gw.ip4.tinet.net (77.67.70.202)  41.664 ms  36.517 ms  37.546 ms
17  10ge.ten1-1.sj-mkp2-dis-1.peer1.net (216.187.89.101)  27.446 ms  29.735 ms  35.474 ms
18  216.187.88.174 (216.187.88.174)  32.243 ms  33.022 ms  44.545 ms
19  vlan3.fmt6509-1.fmt.peer1.net (66.40.24.106)  48.426 ms  43.981 ms  36.753 ms
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *^C
---------------------



---------------------
This from Omaha, where I am:
traceroute to facialsurgery.com (216.65.63.184), 64 hops max, 52 byte packets
 1  router.belkin (192.168.2.1)  1.452 ms  1.292 ms  1.195 ms
 2  10.108.128.1 (10.108.128.1)  9.741 ms  10.326 ms  8.639 ms
 3  68.13.10.197 (68.13.10.197)  10.648 ms  9.812 ms  12.028 ms
 4  68.13.9.245 (68.13.9.245)  10.998 ms  23.045 ms  10.879 ms
 5  mtc1dsrj02-ae3.0.rd.om.cox.net (68.13.14.13)  10.100 ms  10.499 ms  9.410 ms
 6  68.1.2.109 (68.1.2.109)  29.788 ms * *
 7  10ge.ten1-3.dal-eqx-cor-1.peer1.net (206.223.118.30)  35.892 ms  35.999 ms  33.721 ms
 8  10ge.ten1-2.la-600w-cor-2.peer1.net (216.187.124.121)  63.966 ms  82.162 ms  62.104 ms
 9  10ge-ten1-1.la-600w-cor-1.peer1.net (216.187.88.145)  60.989 ms  62.035 ms  62.398 ms
10  * * *
11  * * *
12  vlan3.fmt6509-1.fmt.peer1.net (66.40.24.106)  74.749 ms  71.574 ms  70.665 ms
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *^C
---------------------



---------------------
If anyone is familiar with WinMTR, here is the output of that program from here in Omaha. WinMTR runs many traces and compiles the results:
|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                             192.168.1.1 -    0 |  785 |  785 |    1 |    7 |  202 |    9 |
|                            10.102.128.1 -   76 |  179 |   43 |    9 |   21 |   74 |   11 |
|                            68.13.10.125 -   79 |  174 |   37 |    9 |  102 |  476 |  202 |
|                              68.13.9.93 -   76 |  179 |   43 |    9 |   20 |   77 |   15 |
|          mtc1dsrj02-ae3.0.rd.om.cox.net -   74 |  184 |   49 |   10 |   27 |   87 |   47 |
|                              68.1.2.109 -   89 |  159 |   18 |   26 |   38 |   66 |   66 |
|     10ge.ten1-3.dal-eqx-cor-1.peer1.net -   85 |  165 |   26 |   31 |   46 |  116 |   54 |
|     10ge.ten1-2.la-600w-cor-2.peer1.net -   76 |  179 |   43 |   62 |   74 |  143 |   63 |
|     10ge-ten1-1.la-600w-cor-1.peer1.net -   76 |  180 |   44 |   60 |   80 |  284 |   62 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|           vlan3.fmt6509-1.fmt.peer1.net -   76 |  180 |   44 |   70 |   82 |  135 |   76 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|________________________________________________|______|______|______|______|______|______|

My server farm people interpreted the above and said that my ISP, cox.net, is the problem.  I guess I don't see it, but I'm not that savvy about this stuff.  I'm about to send them all of the other traceroutes.
---------------------


---------------------
For comparison, I did a traceroute to a friend's server.  He put my images on a page on his server, and everything loaded instantaneously:
MacbookPro:PaiGow stevendenenberg$ traceroute jjd.com
traceroute to jjd.com (24.234.165.109), 64 hops max, 52 byte packets
 1  router.belkin (192.168.2.1)  1.951 ms  1.006 ms  2.681 ms
 2  10.108.128.1 (10.108.128.1)  10.584 ms  9.230 ms  9.570 ms
 3  68.13.10.193 (68.13.10.193)  10.102 ms *  13.581 ms
 4  68.13.8.221 (68.13.8.221)  14.968 ms  12.499 ms  11.471 ms
 5  mtc1dsrj01-ae4.0.rd.om.cox.net (68.13.14.5)  12.151 ms  15.519 ms  9.184 ms
 6  sestdsrj01-ae1.0.rd.lv.cox.net (68.1.0.89)  40.630 ms  59.590 ms  41.160 ms
 7  24-234-6-253.ptp.lvcm.net (24.234.6.253)  40.559 ms  48.540 ms  46.360 ms
 8  24-234-6-246.ptp.lvcm.net (24.234.6.246)  45.552 ms
    24-234-6-238.ptp.lvcm.net (24.234.6.238)  41.562 ms  42.978 ms
 9  24-234-16-246.ptp.lvcm.net (24.234.16.246)  71.831 ms
    24-234-16-242.ptp.lvcm.net (24.234.16.242)  41.360 ms
    24-234-16-246.ptp.lvcm.net (24.234.16.246)  57.460 ms
10  jjd.com (24.234.165.109)  49.173 ms  69.560 ms  49.904 ms
---------------------


So.  Question: do these traceroutes seem to indicate that the problem is at peer1, or before, or do they not indicate anything in particular?

Thanks again,
--Steve
0
 

Accepted Solution

by:
StevenMiles earned 0 total points
ID: 39250715
Problem solved.  The server farm people discovered that there was a problem with my iptables configuration.  I had never heard of iptables before.

They removed iptables, since the server has a hardware firewall, and the problem is **gone**.
0
 

Author Closing Comment

by:StevenMiles
ID: 39261880
I'm delivering points for the attempt. My own comment is the solution because the solution was actually found by a tech support team unrelated to posters here at experts-exchange.
0

Featured Post

Comprehensive Backup Solutions for Microsoft

Acronis protects the complete Microsoft technology stack: Windows Server, Windows PC, laptop and Surface data; Microsoft business applications; Microsoft Hyper-V; Azure VMs; Microsoft Windows Server 2016; Microsoft Exchange 2016 and SQL Server 2016.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Have you tried to learn about Unicode, UTF-8, and multibyte text encoding and all the articles are just too "academic" or too technical? This article aims to make the whole topic easy for just about anyone to understand.
Building a website can seem like a daunting task to the uninitiated but it really only requires knowledge of two basic languages: HTML and CSS.
In this tutorial viewers will learn how to style elements, such a divs, with a "drop shadow" effect using the CSS box-shadow property Start with a normal styled element, such as a div.: In the element's style, type the box shadow property: "box-shad…
HTML5 has deprecated a few of the older ways of showing media as well as offering up a new way to create games and animations. Audio, video, and canvas are just a few of the adjustments made between XHTML and HTML5. As we learned in our last micr…

803 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question