Solved

Dedicated server should be fast fast, but images very slow to load

Posted on 2013-06-05
7
302 Views
Last Modified: 2013-06-20
Hi, everyone,

The description of this problem is a little long, but it's really just one problem.

I have a dedicated server running Apache 2.2.3 on Red Hat. I noticed that on the main site that the server runs, FacialSurgery.com, some images are very late to load on many pages -- sometimes taking up to ten seconds to load, sometimes not loading at all.

For example, if you go to a before-and-after page, such as this:
http://www.facialsurgery.com/BAPg1qnd2jqh1_psfd_nl08_hmc3.html

there are a bunch of little thumbnails toward the bottom of the page. Usually, some of those thumbnails might take up to 10 seconds to load, or they don't load at all. You can browse to other patients to see the same thing happen.

Then, on most of the intra-op pages, such as:
http://www.facialsurgery.com/ClkoffTPgt3_2011_01_02mh.html
(caution: there may be a tiny bit of blood pictured on some of the intra-op pages)
...usually a couple of the pictures just won't load, or they take *many seconds to load.

I never remember that being a problem from when I designed the site.  I called some friends in other cities to see if they were seeing the same thing, and they were.

The httpd.conf that is *currently running the site is now stripped-down.  I took out the other VirtualHost's, made FacialSurgery.com NOT a virtual host, cut down the log levels, etc.  Here is a link to the current httpd.conf:
http://www.facialsurgery.com/httpd.txt

I know I must be loading a bunch of modules that I don't need, but as I mentioned, I haven't had this problem forever. Modules that I don't need are being loaded probably because I was too lazy to find out just what they do and remove them if I didn't need them.  I'm no expert in Apache -- I just read a lot.

The site needs to be updated, of course -- the html is ponderous and doesn't completely validate, though it's not bad. Although the pages look like static .html files, the URLs get rewritten and call perl programs with parameters taken from the static-appearing URLs, and the html is then generated on the fly. But that should still all happen very quickly.   I don't know why it should take forever to get some .jpgs to the user's browser.

Here's a test page I made:
http://www.facialsurgery.com/images.html

In most browsers, some of the little pics on the test page don't load, or they take forever.  When I load this page and run "tail -f" on rewrite.log, nothing bad happens. Just: not all of the .jpgs are rewritten at first, and then, ten seconds after trying to load the page I see the last three images appear in the browser window, and there they are in rewrite.log, too.  No indication of why they were so late to load.

Okay, so things are clunky.  Still, there's a Red Hat physical (non-virtual) server doing nothing at all but serving this one site, and seems to me it should be waaaaaay faster.

I've browsed around to all the "10 ways to speed up your site" pages. I haven't yet contacted tech support at my server farm to see if there might be a problem with the server's connection to the Internet pipes.

Running "free -m" shows that I've got 4 gig, and almost all of it is free.
When I run "top", my load average is 0.15, 0.23, 0.25, and cpu is 99.1%id.
If I try to download a large file to my server, it runs at about 11M/s.

I'm dying to get some suggestions of how to make all the images on these pages blit immediately into place.  Seems like it should be possible!

Thanks for any assistance.

--Steve D.
0
Comment
Question by:StevenMiles
  • 5
  • 2
7 Comments
 
LVL 28

Assisted Solution

by:FishMonger
FishMonger earned 500 total points
Comment Utility
All of the pages loaded fine for me on their initial load but when I refreshed the pages (Ctrl-F5), they loaded very slowly.  Since you haven't posted the perl code that generates the pages, I can't say if the script(s) are at fault.

Have you tried profiling the scripts?
0
 

Author Comment

by:StevenMiles
Comment Utility
Hi, FishMonger,

Thanks for helping. You don't want to see the perl code -- they're spaghetti.  But they take less than 0.1 second to complete.

And, at least on my systems, even images.html, where no perl is exectued, fails to deliver all the images without a big delay.

--Steve
0
 

Author Comment

by:StevenMiles
Comment Utility
More: let's say I try to load:
http://www.facialsurgery.com/images.html

when I "tail -f" the access log, I see that, when a couple images show up very late, around 15 seconds after the rest of the page loads, those images only appear in the access.log exactly at the time they appear, with a date stamp 15 seconds after the date stamp of the other 30 images, which show up in the access.log instantaneously.

So does my server only receive the request from the browser late? Or does the server receive all of the image requests immediately, and only "processes" the late photos later? Is there a way to tell?

And why should the problem be the same for many different browsers? And for the same few photos each time?  There doesn't seem to be any problem with the photos as they exist in their directories.

More yet: I created another test file:
http://www.facialsurgery.com/images_b.html

In images.html, it's often jpgs number 3, 4, and 5 that show up late.  So in images_b.html, I moved those jpgs to the bottom of the page. And now its the 4th, 5th, and 6th images that load very late, and the images at the bottom of the page load fine. And it happens the same in Chrome and Safari.

Something very fishy is happening.

/SD
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 28

Assisted Solution

by:FishMonger
FishMonger earned 500 total points
Comment Utility
Can you run a packet capture from both the server side and the client?  Comparing the 2 captures should give a clue to the problem.
0
 

Author Comment

by:StevenMiles
Comment Utility
Hi,

Sorry for the delay.  Still having no luck. The server farm doesn't see a problem, maybe because they are there!

Here are a couple of traceroutes to the site in question.  Can anyone help me interpret them? "peer1" is the name of the server farm company. They are in Fremont, CA. (by the way, I have always been verrrry happy with their tech support)


---------------------
This from Des Moines, IA:
s-s-computer:~ paul$ traceroute facialsurgery.com
traceroute to facialsurgery.com (216.65.63.184), 64 hops max, 52 byte packets
 1  192.168.1.1 (192.168.1.1)  1.159 ms  0.525 ms  0.526 ms
 2  * 172.30.71.81 (172.30.71.81)  6.437 ms  8.064 ms
 3  172.30.1.145 (172.30.1.145)  8.188 ms  7.409 ms  7.792 ms
 4  12.249.52.13 (12.249.52.13)  7.695 ms  8.517 ms  6.515 ms
 5  cr2.kc9mo.ip.att.net (12.122.150.206)  24.985 ms  26.253 ms  23.747 ms
 6  cr1.dlstx.ip.att.net (12.122.28.85)  71.548 ms  54.260 ms  45.439 ms
 7  12.122.212.9 (12.122.212.9)  23.925 ms  23.748 ms  23.857 ms
 8  * * *
 9  if-2-2.tcore2.dt8-dallas.as6453.net (66.110.56.6)  124.514 ms  84.020 ms  88.691 ms
10  if-8-2.tcore1.lvw-losangeles.as6453.net (66.110.57.82)  88.844 ms * *
11  if-2-2.tcore2.lvw-losangeles.as6453.net (66.110.59.2)  86.565 ms  86.015 ms  85.256 ms
12  if-3-2.tcore1.sqn-sanjose.as6453.net (63.243.205.13)  94.202 ms  90.016 ms  91.331 ms
13  66.110.8.50 (66.110.8.50)  88.993 ms  88.037 ms  90.889 ms
14  * * *
15  * * *
16  vlan3.fmt6509-1.fmt.peer1.net (66.40.24.106)  93.280 ms  93.263 ms  88.082 ms
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *^C
---------------------


---------------------
This from Northern California:
 1  192.168.168.1 (192.168.168.1)  0.583 ms  0.420 ms  0.362 ms
 2  10.22.7.1 (10.22.7.1)  2.228 ms  2.239 ms  2.723 ms
 3  10.254.6.254 (10.254.6.254)  9.395 ms  7.633 ms  6.955 ms
 4  10.252.70.1 (10.252.70.1)  12.120 ms  9.502 ms  6.159 ms
 5  172.20.7.153 (172.20.7.153)  11.607 ms  8.417 ms  7.274 ms
 6  172.20.8.225 (172.20.8.225)  21.584 ms  24.140 ms  18.152 ms
 7  172.20.8.217 (172.20.8.217)  23.514 ms  18.353 ms  16.420 ms
 8  172.20.105.1 (172.20.105.1)  18.346 ms  16.329 ms  20.980 ms
 9  172.20.200.254 (172.20.200.254)  18.262 ms  22.326 ms  24.385 ms
10  172.24.0.4 (172.24.0.4)  32.965 ms  19.416 ms  26.196 ms
11  xo-edge.calwisp.com (64.55.111.65)  20.028 ms  25.480 ms  25.022 ms
12  216.55.44.129 (216.55.44.129)  46.137 ms  29.933 ms  50.207 ms
13  vb1510.rar3.sanjose-ca.us.xo.net (216.156.0.153)  40.399 ms  41.599 ms  38.945 ms
14  * * 207.88.14.226.ptr.us.xo.net (207.88.14.226)  70.454 ms
15  xe-1-1-0.sjc12.ip4.tinet.net (77.67.79.217)  46.273 ms  39.983 ms  23.498 ms
16  peer1-gw.ip4.tinet.net (77.67.70.202)  41.664 ms  36.517 ms  37.546 ms
17  10ge.ten1-1.sj-mkp2-dis-1.peer1.net (216.187.89.101)  27.446 ms  29.735 ms  35.474 ms
18  216.187.88.174 (216.187.88.174)  32.243 ms  33.022 ms  44.545 ms
19  vlan3.fmt6509-1.fmt.peer1.net (66.40.24.106)  48.426 ms  43.981 ms  36.753 ms
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *^C
---------------------



---------------------
This from Omaha, where I am:
traceroute to facialsurgery.com (216.65.63.184), 64 hops max, 52 byte packets
 1  router.belkin (192.168.2.1)  1.452 ms  1.292 ms  1.195 ms
 2  10.108.128.1 (10.108.128.1)  9.741 ms  10.326 ms  8.639 ms
 3  68.13.10.197 (68.13.10.197)  10.648 ms  9.812 ms  12.028 ms
 4  68.13.9.245 (68.13.9.245)  10.998 ms  23.045 ms  10.879 ms
 5  mtc1dsrj02-ae3.0.rd.om.cox.net (68.13.14.13)  10.100 ms  10.499 ms  9.410 ms
 6  68.1.2.109 (68.1.2.109)  29.788 ms * *
 7  10ge.ten1-3.dal-eqx-cor-1.peer1.net (206.223.118.30)  35.892 ms  35.999 ms  33.721 ms
 8  10ge.ten1-2.la-600w-cor-2.peer1.net (216.187.124.121)  63.966 ms  82.162 ms  62.104 ms
 9  10ge-ten1-1.la-600w-cor-1.peer1.net (216.187.88.145)  60.989 ms  62.035 ms  62.398 ms
10  * * *
11  * * *
12  vlan3.fmt6509-1.fmt.peer1.net (66.40.24.106)  74.749 ms  71.574 ms  70.665 ms
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *^C
---------------------



---------------------
If anyone is familiar with WinMTR, here is the output of that program from here in Omaha. WinMTR runs many traces and compiles the results:
|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                             192.168.1.1 -    0 |  785 |  785 |    1 |    7 |  202 |    9 |
|                            10.102.128.1 -   76 |  179 |   43 |    9 |   21 |   74 |   11 |
|                            68.13.10.125 -   79 |  174 |   37 |    9 |  102 |  476 |  202 |
|                              68.13.9.93 -   76 |  179 |   43 |    9 |   20 |   77 |   15 |
|          mtc1dsrj02-ae3.0.rd.om.cox.net -   74 |  184 |   49 |   10 |   27 |   87 |   47 |
|                              68.1.2.109 -   89 |  159 |   18 |   26 |   38 |   66 |   66 |
|     10ge.ten1-3.dal-eqx-cor-1.peer1.net -   85 |  165 |   26 |   31 |   46 |  116 |   54 |
|     10ge.ten1-2.la-600w-cor-2.peer1.net -   76 |  179 |   43 |   62 |   74 |  143 |   63 |
|     10ge-ten1-1.la-600w-cor-1.peer1.net -   76 |  180 |   44 |   60 |   80 |  284 |   62 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|           vlan3.fmt6509-1.fmt.peer1.net -   76 |  180 |   44 |   70 |   82 |  135 |   76 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  144 |    0 |    0 |    0 |    0 |    0 |
|________________________________________________|______|______|______|______|______|______|

My server farm people interpreted the above and said that my ISP, cox.net, is the problem.  I guess I don't see it, but I'm not that savvy about this stuff.  I'm about to send them all of the other traceroutes.
---------------------


---------------------
For comparison, I did a traceroute to a friend's server.  He put my images on a page on his server, and everything loaded instantaneously:
MacbookPro:PaiGow stevendenenberg$ traceroute jjd.com
traceroute to jjd.com (24.234.165.109), 64 hops max, 52 byte packets
 1  router.belkin (192.168.2.1)  1.951 ms  1.006 ms  2.681 ms
 2  10.108.128.1 (10.108.128.1)  10.584 ms  9.230 ms  9.570 ms
 3  68.13.10.193 (68.13.10.193)  10.102 ms *  13.581 ms
 4  68.13.8.221 (68.13.8.221)  14.968 ms  12.499 ms  11.471 ms
 5  mtc1dsrj01-ae4.0.rd.om.cox.net (68.13.14.5)  12.151 ms  15.519 ms  9.184 ms
 6  sestdsrj01-ae1.0.rd.lv.cox.net (68.1.0.89)  40.630 ms  59.590 ms  41.160 ms
 7  24-234-6-253.ptp.lvcm.net (24.234.6.253)  40.559 ms  48.540 ms  46.360 ms
 8  24-234-6-246.ptp.lvcm.net (24.234.6.246)  45.552 ms
    24-234-6-238.ptp.lvcm.net (24.234.6.238)  41.562 ms  42.978 ms
 9  24-234-16-246.ptp.lvcm.net (24.234.16.246)  71.831 ms
    24-234-16-242.ptp.lvcm.net (24.234.16.242)  41.360 ms
    24-234-16-246.ptp.lvcm.net (24.234.16.246)  57.460 ms
10  jjd.com (24.234.165.109)  49.173 ms  69.560 ms  49.904 ms
---------------------


So.  Question: do these traceroutes seem to indicate that the problem is at peer1, or before, or do they not indicate anything in particular?

Thanks again,
--Steve
0
 

Accepted Solution

by:
StevenMiles earned 0 total points
Comment Utility
Problem solved.  The server farm people discovered that there was a problem with my iptables configuration.  I had never heard of iptables before.

They removed iptables, since the server has a hardware firewall, and the problem is **gone**.
0
 

Author Closing Comment

by:StevenMiles
Comment Utility
I'm delivering points for the attempt. My own comment is the solution because the solution was actually found by a tech support team unrelated to posters here at experts-exchange.
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

I have been pestered over the years to produce and distribute regular data extracts, and often the request have explicitly requested the data be emailed as an Excel attachement; specifically Excel, as it appears: CSV files confuse (no Red or Green h…
In the distant past (last year) I hacked together a little toy that would allow a couple of Manager types to query, preview, and extract data from a number of MongoDB instances, to their tool of choice: Excel (http://dilbert.com/strips/comic/2007-08…
In this tutorial viewers will learn how to style elements, such a divs, with a "drop shadow" effect using the CSS box-shadow property Start with a normal styled element, such as a div.: In the element's style, type the box shadow property: "box-shad…
The viewer will receive an overview of the basics of CSS showing inline styles. In the head tags set up your style tags: (CODE) Reference the nav tag and set your properties.: (CODE) Set the reference for the UL element and styles for it to ensu…

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

17 Experts available now in Live!

Get 1:1 Help Now