web server performance
I have an Ubuntu Web Server (Karmic Koala v9.1) housed on a cloud server from rackspace server from here http://www.rackspacecloud.com/cloud_hosting_products/servers/pricing . I have the lowest option and am having speed issues. I don't think the speed issues are due to memory but rather other factors. I know there are a number of things I can do myself to improve the speed from a management standpoint.
I would like to set up some kind of monitoring to make sure that we are as optimized as we can be given our current level of 256mb of RAM. What kind of monitoring tool would you guys recommend having?
I would like to set up some kind of monitoring to make sure that we are as optimized as we can be given our current level of 256mb of RAM. What kind of monitoring tool would you guys recommend having?
What sort of speed issues? Scripts taking a long time to complete? (Which scripting language? Database server being used for queries?) If it's slow for heavy scripted pages, but fast for static pages, that would rule out network connectivity being the culprit. But if static pages are just as slow, then possibly the network connection is the limiting factor.
Oh, and what does a cloud server do that a standard shared hosting solution doesn't do? (Cloud is one of those terms I keep hearing, but no one seems to be able to explain what's new or useful about the idea.)
ASKER
HackneyCab,
With a server in the cloud, it's like having a dedicated box without the usual $100+ monthly cost of having a hosting co set one up for you. This means you have full root access and can reboot as needed. Backups are loads easier, and the list goes on. For more info about how the cloud is revolutionizing the industry of web services, see this http://www.rackspacecloud.com/cloud_hosting_products/servers
About monitoring, I would like a monitoring package for web server that is low on resource usage.
With a server in the cloud, it's like having a dedicated box without the usual $100+ monthly cost of having a hosting co set one up for you. This means you have full root access and can reboot as needed. Backups are loads easier, and the list goes on. For more info about how the cloud is revolutionizing the industry of web services, see this http://www.rackspacecloud.com/cloud_hosting_products/servers
About monitoring, I would like a monitoring package for web server that is low on resource usage.
Hi!
I think nbandan's suggestion is good. Use 'sudo top' to see what's taking resources. Maybe you don't get the promised performance from your cloud provider? They don't say how much CPU performance you'll get, only a vauge "x virtual cores". Running in what speed? Making which performance compared to what? What's the latency on the network?
Regards, Tobias
I think nbandan's suggestion is good. Use 'sudo top' to see what's taking resources. Maybe you don't get the promised performance from your cloud provider? They don't say how much CPU performance you'll get, only a vauge "x virtual cores". Running in what speed? Making which performance compared to what? What's the latency on the network?
Regards, Tobias
ASKER
Tobias. That's outstanding!
I now know what to ask the chat guy. Perfect!
I now know what to ask the chat guy. Perfect!
ASKER
If you were to ask a specific list of questions about the rackspacecloud 256mb service (regarding those "other" variables), what you would ask is:
What speed of CPU? measurable
What other measurable items could they give me?
As it turns out, my developer has identified that swap is too small but that is something that should have been done in the beginning. What else can I push rackspace for?
What speed of CPU? measurable
What other measurable items could they give me?
As it turns out, my developer has identified that swap is too small but that is something that should have been done in the beginning. What else can I push rackspace for?
A start can be to monitor the CPU, disk and network traffic. You can use 'dstat' for this:
Monitor Your System With Dstat
dstat is a sort of general-purpose system profiling tool that's CLI-based. It's usable to log data or view it in real time; it can show CPU load, disk I/O, network receive/send, etc. It also produces colorized output at the console, which makes it somewhat easier to scan results quickly.
If you want to analyze your data later, you can use the --output option to write the information to a CSV file suitable for import into OpenOffice.org, Gnumeric, or whatever your favorite spreadsheet might be.
If you want to see a "full" output that includes CPU usage by CPU (for SMP systems), disk read and write, network traffic, and paging, use dstat -f.
The most recent version of dstat ships with quite a few plugins that are suitable for servers and personal systems. This includes plugins to examine MySQL performance, Postfix, Sendmail, Qmail, and much more. You can combine the plugins with built-in functions of dstat as well. So, for instance, if you want to watch MySQL I/O along with the -a (all) option, you can use dstat --mysql-io -a. Some experimentation may be required and not all options can be combined with other options.
To install: sudo apt-get install dstat
But maybe it could be nice to measure some physical servers too and then compare the performance with your cloud server. I've been using virtual servers for some time now, and I know it's difficult to guarantee the performance to a single virtual machine. I suppose the smallest 256MB machines at RackSpaceCloud get the "leftovers" in performance and that the bigger virtual machines are prioritized in their systems.
Maybe the best you can do is to check your server for running processes that you might be able to shut down to optimize the speed of the server. The 'top' command is a great tool, check the 'man top' for a lot of different ways to measure processes.
Regards, Tobias
Monitor Your System With Dstat
dstat is a sort of general-purpose system profiling tool that's CLI-based. It's usable to log data or view it in real time; it can show CPU load, disk I/O, network receive/send, etc. It also produces colorized output at the console, which makes it somewhat easier to scan results quickly.
If you want to analyze your data later, you can use the --output option to write the information to a CSV file suitable for import into OpenOffice.org, Gnumeric, or whatever your favorite spreadsheet might be.
If you want to see a "full" output that includes CPU usage by CPU (for SMP systems), disk read and write, network traffic, and paging, use dstat -f.
The most recent version of dstat ships with quite a few plugins that are suitable for servers and personal systems. This includes plugins to examine MySQL performance, Postfix, Sendmail, Qmail, and much more. You can combine the plugins with built-in functions of dstat as well. So, for instance, if you want to watch MySQL I/O along with the -a (all) option, you can use dstat --mysql-io -a. Some experimentation may be required and not all options can be combined with other options.
To install: sudo apt-get install dstat
But maybe it could be nice to measure some physical servers too and then compare the performance with your cloud server. I've been using virtual servers for some time now, and I know it's difficult to guarantee the performance to a single virtual machine. I suppose the smallest 256MB machines at RackSpaceCloud get the "leftovers" in performance and that the bigger virtual machines are prioritized in their systems.
Maybe the best you can do is to check your server for running processes that you might be able to shut down to optimize the speed of the server. The 'top' command is a great tool, check the 'man top' for a lot of different ways to measure processes.
Regards, Tobias
ASKER
This comment / clarification is from rtod2's developer....
I've increased swap space on the server but there still seem to be performance issues. Running top is sometimes not even possible when the server gets tied up; when it is possible it's usually apache taking the most time. That seems reasonable given that this is a web server...
I'd like some help on identifying performance bottlenecks in the LAMP stack, rather than just being advised to monitor cpu / memory / disk usage. If apache is taking a lot of time, what exactly is it doing? Could there be be some sort of double-lookup going on, i.e. a problem with the virtual server or DNS setup? What about the mysql layer--I have a feeling that a lot of the time is spent there. I want to understand what is happening in the existing system before adding a cache layer to improve performance. I can only think that something is not right in the basic configuration of apache-mysql-php and I want to straighten that out first.
Thanks to all for any suggestions here.
I've increased swap space on the server but there still seem to be performance issues. Running top is sometimes not even possible when the server gets tied up; when it is possible it's usually apache taking the most time. That seems reasonable given that this is a web server...
I'd like some help on identifying performance bottlenecks in the LAMP stack, rather than just being advised to monitor cpu / memory / disk usage. If apache is taking a lot of time, what exactly is it doing? Could there be be some sort of double-lookup going on, i.e. a problem with the virtual server or DNS setup? What about the mysql layer--I have a feeling that a lot of the time is spent there. I want to understand what is happening in the existing system before adding a cache layer to improve performance. I can only think that something is not right in the basic configuration of apache-mysql-php and I want to straighten that out first.
Thanks to all for any suggestions here.
If you are running a LAMP stack on 256MB you will need to strip it down to bare bones...
If you really don't need them, get rid of email stuff and just use Google Apps etc.
That size server you will also not be able to use Cpanel, it just eats up too many resources.
Apache fires up tons of child processes that need to be limited
Many people who run small images decide to run alternative webservers such as Nginx or Lighttpd as they don't gobble up as many resources.
If you really don't need them, get rid of email stuff and just use Google Apps etc.
That size server you will also not be able to use Cpanel, it just eats up too many resources.
Apache fires up tons of child processes that need to be limited
Many people who run small images decide to run alternative webservers such as Nginx or Lighttpd as they don't gobble up as many resources.
ASKER
from the developer, again:
we are using webmin rather than cpanel. we are trying to run a worpress site with a craftysyntax subsite, plus joomla sites. These things don't like to run without email.
Thanks. Looking forward to some more ideas, especially on how to analyze the performance of the server tasks.
we are using webmin rather than cpanel. we are trying to run a worpress site with a craftysyntax subsite, plus joomla sites. These things don't like to run without email.
Thanks. Looking forward to some more ideas, especially on how to analyze the performance of the server tasks.
ASKER
I agree Chuck,
I think the processes are the main issue here which is why I wanted you to list them. In understanding what they all are, we can start eliminating them.
I think the processes are the main issue here which is why I wanted you to list them. In understanding what they all are, we can start eliminating them.
ASKER
Chuck, it may be too much to run rackspace if we don't really have the expertise to run it on the limited resources they give us. We may want to consider other options. One option might be the company you sent me some time back. I think I remember seeing separate charges (and pretty low). I think they started at a little more than rackspace but not much. You could buy more memory only if that's what you needed, same for CPU, and other things.
You might think this suggestion is a bit out of the park, but this is what I went through.
Most of the cloud server offerings are unmanaged, and most of the default images are pretty raw, intended for people who love tinkering or employ full time server geeks.
I was running 3 nodes quite happily on VPS.net that I configured myself, reverse proxy Nginx/Apache until I got hacked twice in 3 months.
I spent $400 on a good server admin to fix stuff, rebuild server etc, but that was taking forever. Then I was going to move over to Amazon, or maybe Linode where he could provision quicker but it dragged out forever and the downtime I had with the jury rigged setup after the hacking was costing me money.
I bit the bullet, Storm from liquid web launched and I sacked the server geek and moved it myself in 1 hour.
https://www.stormondemand.com/cloud-hosting/
It is a full managed Cpanel solution, here is my verbatim server monitor log
# [ 2010-03-21 17:08:44 ] Server is down: http did not respond.
#
2010-03-21 17:08:44 Server is back up: http is responding.
2010-03-21 17:07:54 Server is down: http did not respond.
2010-03-21 17:02:16 Server is down: http did not respond.
# [ 2010-02-03 01:29:37 ] Server is down: http did not respond.
#
2010-02-03 01:29:37 Server is back up: http is responding.
2010-02-03 01:28:54 Server is down: http did not respond.
# [ 2010-01-22 14:06:22 ] Clean Reboot
#
2010-01-22 14:06:10 Server has been started
2010-01-22 14:05:56 Server has been shutdown
2010-01-22 14:05:08 Clean Reboot
# [ 2010-01-19 04:40:20 ] Create Xen Server
See those 2 "Clean reboots" ?
That was me that caused the downtime - I live in Poland on a variable IP and occasionally my ISP switches me to an IP that is blacklisted by my server, though I have access to the server control panel.
The server wasn't actually offline, I just couldn't see it.
I know how to fix that myself now if it happens in the future, but I could just raise a support ticket.
You are claiming you will put all of those sites on the Rackspace server, but it is never going to cope, and there will always be extensive maintenance costs to keep it running.
You will most likely end up running at least 2 but possibly 3 nodes.
My current monthly bill on Storm is $50 +$20 (cpanel fully managed) + bandwidth + backup storage, about $85/month total.
I spent possibly 6 months not being able to sleep at night and now I know my server will be online and can relax. I haven't messed around with optimization as I don't use anywhere close to the capacity any more.
You might posibly beat the price, but it would be a stretch if you allow $10 for Cpanel and $30 for someone like Platinun Server Management.
The only negative about Storm is there is currently no affiliate program
Most of the cloud server offerings are unmanaged, and most of the default images are pretty raw, intended for people who love tinkering or employ full time server geeks.
I was running 3 nodes quite happily on VPS.net that I configured myself, reverse proxy Nginx/Apache until I got hacked twice in 3 months.
I spent $400 on a good server admin to fix stuff, rebuild server etc, but that was taking forever. Then I was going to move over to Amazon, or maybe Linode where he could provision quicker but it dragged out forever and the downtime I had with the jury rigged setup after the hacking was costing me money.
I bit the bullet, Storm from liquid web launched and I sacked the server geek and moved it myself in 1 hour.
https://www.stormondemand.com/cloud-hosting/
It is a full managed Cpanel solution, here is my verbatim server monitor log
# [ 2010-03-21 17:08:44 ] Server is down: http did not respond.
#
2010-03-21 17:08:44 Server is back up: http is responding.
2010-03-21 17:07:54 Server is down: http did not respond.
2010-03-21 17:02:16 Server is down: http did not respond.
# [ 2010-02-03 01:29:37 ] Server is down: http did not respond.
#
2010-02-03 01:29:37 Server is back up: http is responding.
2010-02-03 01:28:54 Server is down: http did not respond.
# [ 2010-01-22 14:06:22 ] Clean Reboot
#
2010-01-22 14:06:10 Server has been started
2010-01-22 14:05:56 Server has been shutdown
2010-01-22 14:05:08 Clean Reboot
# [ 2010-01-19 04:40:20 ] Create Xen Server
See those 2 "Clean reboots" ?
That was me that caused the downtime - I live in Poland on a variable IP and occasionally my ISP switches me to an IP that is blacklisted by my server, though I have access to the server control panel.
The server wasn't actually offline, I just couldn't see it.
I know how to fix that myself now if it happens in the future, but I could just raise a support ticket.
You are claiming you will put all of those sites on the Rackspace server, but it is never going to cope, and there will always be extensive maintenance costs to keep it running.
You will most likely end up running at least 2 but possibly 3 nodes.
My current monthly bill on Storm is $50 +$20 (cpanel fully managed) + bandwidth + backup storage, about $85/month total.
I spent possibly 6 months not being able to sleep at night and now I know my server will be online and can relax. I haven't messed around with optimization as I don't use anywhere close to the capacity any more.
You might posibly beat the price, but it would be a stretch if you allow $10 for Cpanel and $30 for someone like Platinun Server Management.
The only negative about Storm is there is currently no affiliate program
ASKER
Chuck says:
Storm sounds interesting, Andy, but Ted's costs at rackspace are currently a whole lot lower than that. I have never seen the cpu usage go over about 5% so that's not a problem. Disk space is okay even if we keep the three virtual nodes I'm talking about. What we just don't have enough of is memory. Now that we increased swap to 2Gb we don't have the total freezes anymore but we definitely have swapping going on if anyone does much on any of the sites. Swapping always slows the system down significantly.
Ted, the list of processes is not so important as the list of the processes that are using up resources especially memory resources. Everything I've seen indicates that Apache is using the most, so the system is basically doing what it is supposed to do but it doesn't have enough memory to do it efficiently.
Andy, I was interested in your mention of linode. I looked at them and their xen-based solution looks pretty good. I noticed they install webmin by default. I think webmin is a better solution than cpanel for management, and not just because it is free.
Storm sounds interesting, Andy, but Ted's costs at rackspace are currently a whole lot lower than that. I have never seen the cpu usage go over about 5% so that's not a problem. Disk space is okay even if we keep the three virtual nodes I'm talking about. What we just don't have enough of is memory. Now that we increased swap to 2Gb we don't have the total freezes anymore but we definitely have swapping going on if anyone does much on any of the sites. Swapping always slows the system down significantly.
Ted, the list of processes is not so important as the list of the processes that are using up resources especially memory resources. Everything I've seen indicates that Apache is using the most, so the system is basically doing what it is supposed to do but it doesn't have enough memory to do it efficiently.
Andy, I was interested in your mention of linode. I looked at them and their xen-based solution looks pretty good. I noticed they install webmin by default. I think webmin is a better solution than cpanel for management, and not just because it is free.
ASKER
That's very interesting Chuck. I'm very pleased that we aren't getting total freezes and that CPU is low. The memory is the kicker apparently so... Is linode the one you mentioned some time back? I might pay for more memory at Rackspace but I'd sure rather make sure it's perfectly configured first (i.e. every process clearly identified and run by the good folks here as to what to remove).
Chuck, you are doing a fantastic job! I'm sorry we can't get more memory though at a reasonable price.
Chuck, you are doing a fantastic job! I'm sorry we can't get more memory though at a reasonable price.
ASKER
Is 2GB too much swap. That's what I told you to use but is it really possible to have too much if you have the disk space?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
[from Chuck]
Thanks for that, Tobias. Ram * 2 is the conventional guideline. I went ahead with Ted's recommendation of 2Gb total after reading up on it a bit. It seems that if the disk space is available there is no reason not to use it for swap--the worst that will happen is that it doesn't get used.
Thanks for that, Tobias. Ram * 2 is the conventional guideline. I went ahead with Ted's recommendation of 2Gb total after reading up on it a bit. It seems that if the disk space is available there is no reason not to use it for swap--the worst that will happen is that it doesn't get used.
>It seems that if the disk space is available there is no reason not to use it for swap--the worst that will happen is that it doesn't get used.
Yes, I agree!
Regards, Tobias
Yes, I agree!
Regards, Tobias
All about Linux swap space : http://www.linux.com/news/software/applications/8208-all-about-linux-swap-space
Regards, Tobias
Regards, Tobias
Tool for measuring HDD performance:
http://www.linuxinsight.com/how_fast_is_your_disk.html
Regards, Tobias
1MeasureDiskPerf.gif
http://www.linuxinsight.com/how_fast_is_your_disk.html
Regards, Tobias
1MeasureDiskPerf.gif
ASKER
The rackspace server came back with 274 MB in 3.01 seconds = 91.03 MB/sec
ok. That's ok performance. Shouldn't be the disk that's slows your server down then!
Regards, Tobias
Regards, Tobias
ASKER
This Message is for AndyBeard:
Your solution is looking more and more like the way to go. I know my developer is trying but from a performance stand-point, even if he were to finish the Joomla template, I feel certain the server would die. If not from speed issues, then from something else. I'd like to talk with you about the option you chose. I am tedpenner on gmail. I would greatly appreciate your time. I think I am going to allow Chuck to keep after it for a couple days but it's just getting to be too much. I'm loosing money every day that I sit on this.
My goal was to put up an adsense page that I could add to in Joomla. We can't do that. It seems like I should be able to. I never should have gotten off of shared hosting.
Your solution is looking more and more like the way to go. I know my developer is trying but from a performance stand-point, even if he were to finish the Joomla template, I feel certain the server would die. If not from speed issues, then from something else. I'd like to talk with you about the option you chose. I am tedpenner on gmail. I would greatly appreciate your time. I think I am going to allow Chuck to keep after it for a couple days but it's just getting to be too much. I'm loosing money every day that I sit on this.
My goal was to put up an adsense page that I could add to in Joomla. We can't do that. It seems like I should be able to. I never should have gotten off of shared hosting.
Will drop you an email... I honestly felt the same and whilst I am technically inclined, and even had stuff set up relatively well on VPS.net, I still had downtime and being hacked twice was the final straw.
Then I went the route of paying someone to do it, as I had plans to set up clusters, but things just dragged out.
For geeks webmin is great, but if you are used to shared hosting, you are used to Cpanel, and managed hosting with Storm costs that extra $20 whether you have cpanel or not.
I am a geeky marketer, but at the end of the day the marketer side of me knows when I should change priorities.
Then I went the route of paying someone to do it, as I had plans to set up clusters, but things just dragged out.
For geeks webmin is great, but if you are used to shared hosting, you are used to Cpanel, and managed hosting with Storm costs that extra $20 whether you have cpanel or not.
I am a geeky marketer, but at the end of the day the marketer side of me knows when I should change priorities.
ASKER
Ted, I think it would make more sense to revert to shared hosting than to go to the fully managed cloud server solution that Andy suggests. --Chuck
ASKER
good suggestion
Before you install any monitoring tool I would suggest using the many command line tools to figuring out what is slowing your computer down. I would start with executing 'sudo top'.
The top command displays quite a bit of information about your system. Processes can be sorted by PID, age, CPU or memory usage, time, or user. This command also provides process management, and system administrators can use its k and r keypress commands to kill and reschedule running tasks, respectively.
Keep in mind if you install additional monitoring tools on your system, it will just slow it down even more.
Good Luck!