How to expand my server capacity from 2 servers - app + db servers?

I'm now running CentOS on 2 Dual Core Xeon at 2GB RAM on 2 servers - Linux/Apache/PHP app server and MySQL server. I want to expand the capacity as I'm expecting heavier web traffic for my PHP/MySQL app soon.  What is the next step to expand this?
melwongAsked:
Who is Participating?
 
idumitruConnect With a Mentor Commented:
hi melwong

your question queries a huge topic. it's almost impossible to give you the right answer by ping-ponging some Q&A here.

expanding a server system must be done based on requirements and then based reasons. for example, small_student suggested optimization, which should be the first step before considering more hardware. you said you had some one who already did that. maybe it was good maybe it wasn't , we can't know for sure. but for example optimization doesn't mean just fixing mysql queries, there are many levels where you can make things better:
- sql optimizations (saves mysql cpu power)
- html optimizations - just a small example. if you have many hits on a web page which includes a lot of js or css scripts, it means that number of extra requests to apache for those scripts. a simple solution would be combining them into one js and one css where possible to minimize apache requests - the impact is memory and cpu consumption of apache requests.
- serving strategy optimization. example: there are other web server alternatives to apache. although apache is extremely good with pnp/mysql sripts, it is pretty awfull when it comes to serving static files, such as images js files etc. a simple image request uses an apache process to solve which in your case was about 26mb of ram ... other web servers such as lighttpd or nginx have a very small memory footprint, something like 4mb and that's the entire process, which can handle thousands of requests. so you caould try having such an alternative web server to serve static files and just proxy to apache the relevant php script requests.

there are many examples like this which are not very complicated but can save you $$ in extra hardware.

i said that you need to expand based on requirements and then reasons. if the optimizations i mentioned above could solve the issues, then your requirements dissapear and you don't need to upgrade. but if for some reason it doesn't help or you can't touch all the points, you need to think about the reasons. as fosiul01 has pointed out, you need to figure out what needs more hardware. basically you need to understand which process needs more power, is it apache consuming resources? is it mysql running too many hard queries?

if it's apache then you get an extra box with apache and do round robin dns on them soo you don't spend on hardware load balancer, it does the trick very well.

if it mysql then you have 2 alternatives to standard config
- mysql replication
- mysql cluster

both of them require special planning and have pros and cons. replication is good for heavy reads on relatively normal tables. cluster is good for heavy writes as well, but can overcome replication on reads if you have extremely large tables.

configuration for all those things can be a pain so you must study and know what you want to do.

if you need my help with all this, let me know and i'll give you my contact. this is exactly the business we're doing and we may be able to help you save $$ even in hardware. we also are developing a management panel which does all the things i mentioned above, is very scalable and makes it really easy to manage.
0
 
Monis MontherSystem ArchitectCommented:
First of all before expanding anything , how are the 2 servers set up , are they load balanced or they are made as fail over, or they each have a different WebApp.


Second thing we do before expanding is tuning, this can sometimes boost what you can handle to double if the server was not well tuned.

Please provide info on the above and we will start to help you straight ahead
0
 
melwongAuthor Commented:
There's no load balance or fail over. One is hosting the app, the other server is hosting the database. On tuning, what kinda info you need? Well, i think it has been "tuned" (optimized) by my outsourced server administrator.
0
Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

 
Monis MontherSystem ArchitectCommented:
Ok lets take the Web server first


We will do the following
1- Check how tunned the server is
2- Check how much resource is used to estimate what to do next
3- Run a test to measure response time for your server

What is the output of the following commands


grep Include conf.d /etc/httpd/conf/httpd.conf

free

0
 
melwongAuthor Commented:
hi small_student, here're the info
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
#   Direct modifications to the Apache configuration file may be lost upon subsequent regeneration of the       #
#   configuration file. To have modifications retained, all modifications must be checked into the              #
#   configuration system by running:                                                                            #
#       /usr/local/cpanel/bin/apache_conf_distiller --update                                                    #
#   To see if your changes will be conserved, regenerate the Apache configuration file by running:              #
#       /usr/local/cpanel/bin/build_apache_conf                                                                 #
#   and check the configuration file for your alterations. If your changes have been ignored, then they will    #
#   need to be added directly to their respective template files.                                               #
#                                                                                                               #
#   It is also possible to add custom directives to the various "Include" files loaded by this httpd.conf       #
#   For detailed instructions on using Include files and the apache_conf_distiller with the new configuration   #
#   system refer to the documentation at: http://www.cpanel.net/support/docs/ea/ea3/customdirectives.html       #
#                                                                                                               #
#   This configuration file was built from the following templates:                                             #
#     /var/cpanel/templates/apache2/main.default                                                                #
#     /var/cpanel/templates/apache2/main.local                                                                  #
#     /var/cpanel/templates/apache2/vhost.default                                                               #
#     /var/cpanel/templates/apache2/vhost.local                                                                 #
#     /var/cpanel/templates/apache2/ssl_vhost.default                                                           #
#     /var/cpanel/templates/apache2/ssl_vhost.local                                                             #
#                                                                                                               #
#  Templates with the '.local' extension will be preferred over templates with the '.default' extension.        #
#  The only template updated by the apache_conf_distiller is main.default.                                      #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #


Include "/usr/local/apache/conf/includes/pre_main_global.conf"
Include "/usr/local/apache/conf/includes/pre_main_2.conf"

 

LoadModule auth_passthrough_module modules/mod_auth_passthrough.so
LoadModule bwlimited_module modules/mod_bwlimited.so
LoadModule frontpage_module modules/mod_frontpage.so
LoadModule bw_module modules/mod_bw.so
LoadModule jk_module modules/mod_jk.so




Include "/usr/local/apache/conf/includes/errordocument.conf"
Include "/usr/local/apache/conf/php.conf"
Include "/usr/local/apache/conf/jk.conf"
Include "/usr/local/apache/conf/mod_bandwidth.conf"
Include "/usr/local/apache/conf/modsec2.conf"


RLimitMEM 408766805 
RLimitCPU 240 
ErrorLog "logs/error_log"
DefaultType text/plain
ScriptAliasMatch ^/?controlpanel/?$ /usr/local/cpanel/cgi-sys/redirect.cgi
ScriptAliasMatch ^/?cpanel/?$ /usr/local/cpanel/cgi-sys/redirect.cgi
ScriptAliasMatch ^/?kpanel/?$ /usr/local/cpanel/cgi-sys/redirect.cgi
ScriptAliasMatch ^/?securecontrolpanel/?$ /usr/local/cpanel/cgi-sys/sredirect.cgi
ScriptAliasMatch ^/?securecpanel/?$ /usr/local/cpanel/cgi-sys/sredirect.cgi
ScriptAliasMatch ^/?securewhm/?$ /usr/local/cpanel/cgi-sys/swhmredirect.cgi
ScriptAliasMatch ^/?webmail/?$ /usr/local/cpanel/cgi-sys/wredirect.cgi
ScriptAliasMatch ^/?whm/?$ /usr/local/cpanel/cgi-sys/whmredirect.cgi

AddType text/html .shtml

RewriteEngine on
Alias /akopia /usr/local/cpanel/3rdparty/interchange/share/akopia/
Alias /bandwidth /usr/local/bandmin/htdocs/
Alias /img-sys /usr/local/cpanel/img-sys/
Alias /interchange /usr/local/cpanel/3rdparty/interchange/share/interchange/
Alias /interchange-5 /usr/local/cpanel/3rdparty/interchange/share/interchange-5/
Alias /java-sys /usr/local/cpanel/java-sys/
Alias /mailman/archives /usr/local/cpanel/3rdparty/mailman/archives/public/
Alias /pipermail /usr/local/cpanel/3rdparty/mailman/archives/public/
Alias /sys_cpanel /usr/local/cpanel/sys_cpanel/


ScriptAlias /cgi-sys /usr/local/cpanel/cgi-sys/
ScriptAlias /mailman /usr/local/cpanel/3rdparty/mailman/cgi-bin/
ScriptAlias /scgi-bin /usr/local/cpanel/cgi-sys/scgiwrap


<Directory "/">
    Options ExecCGI FollowSymLinks Includes IncludesNOEXEC Indexes -MultiViews SymLinksIfOwnerMatch
    AllowOverride All
</Directory>

<Directory "/usr/local/apache/htdocs">
    Options Includes Indexes FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all

</Directory>

<Files ~ "^error_log$">
    Order allow,deny
    Deny from all

    Satisfy All
</Files>

<FilesMatch "^\.ht">
    Order allow,deny
    Deny from all

    Satisfy All
</FilesMatch>

<IfModule log_config_module>
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common

    CustomLog logs/access_log common

    <IfModule logio_module>
        LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio

    </IfModule>

</IfModule>

<IfModule alias_module>
    ScriptAlias /cgi-bin/ "/usr/local/apache/cgi-bin/"

</IfModule>

<Directory "/usr/local/apache/cgi-bin">
    AllowOverride None
    Options None
    Order allow,deny
    Allow from all

</Directory>

<IfModule mime_module>
    TypesConfig conf/mime.types
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz
    AddType text/vnd.sun.j2me.app-descriptor .jad
    AddType application/java-archive .jar

</IfModule>

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
#   Direct modifications to the Apache configuration file may be lost upon subsequent regeneration of the       #
#   configuration file. To have modifications retained, all modifications must be checked into the              #
#   configuration system by running:                                                                            #
#       /usr/local/cpanel/bin/apache_conf_distiller --update                                                    #
#   To see if your changes will be conserved, regenerate the Apache configuration file by running:              #
#       /usr/local/cpanel/bin/build_apache_conf                                                                 #
#   and check the configuration file for your alterations. If your changes have been ignored, then they will    #
#   need to be added directly to their respective template files.                                               #
#                                                                                                               #
#   It is also possible to add custom directives to the various "Include" files loaded by this httpd.conf       #
#   For detailed instructions on using Include files and the apache_conf_distiller with the new configuration   #
#   system refer to the documentation at: http://www.cpanel.net/support/docs/ea/ea3/customdirectives.html       #
#                                                                                                               #
#   This configuration file was built from the following templates:                                             #
#     /var/cpanel/templates/apache2/main.default                                                                #
#     /var/cpanel/templates/apache2/main.local                                                                  #
#     /var/cpanel/templates/apache2/vhost.default                                                               #
#     /var/cpanel/templates/apache2/vhost.local                                                                 #
#     /var/cpanel/templates/apache2/ssl_vhost.default                                                           #
#     /var/cpanel/templates/apache2/ssl_vhost.local                                                             #
#                                                                                                               #
#  Templates with the '.local' extension will be preferred over templates with the '.default' extension.        #
#  The only template updated by the apache_conf_distiller is main.default.                                      #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #


PidFile logs/httpd.pid
LockFile logs/accept.lock
# Defined in /var/cpanel/cpanel.config: apache_port
Listen 0.0.0.0:80
Timeout 300
User nobody
Group nobody
ExtendedStatus On
LogLevel warn

# These can be set in WHM under 'Apache Global Configuration'
TraceEnable On
ServerSignature Off
ServerTokens Full
FileETag All
StartServers 5
<IfModule prefork.c>
MinSpareServers 5 
MaxSpareServers 10
</IfModule>
MaxClients 250
MaxRequestsPerChild 3000

RewriteEngine on
RewriteMap LeechProtect prg:/usr/local/cpanel/bin/leechprotect
RewriteLock /usr/local/apache/logs/rewrite_lock

UserDir public_html

# DirectoryIndex is set via the WHM -> Service Configuration -> Apache Setup -> DirectoryIndex Priority
DirectoryIndex index.html.var index.htm index.html index.shtml index.xhtml index.wml index.perl index.pl index.plx index.ppl index.cgi index.jsp index.js index.jp index.php4 index.php3 index.php index.phtml default.htm default.html home.htm index.php5 Default.html Default.htm home.html

# SSLCipherSuite can be set in WHM under 'Apache Global Configuration'
SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP
SSLPassPhraseDialog  builtin
SSLSessionCache         dbm:/usr/local/apache/logs/ssl_scache
SSLSessionCacheTimeout  300
SSLMutex  file:/usr/local/apache/logs/ssl_mutex
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin

<IfDefine SSL>
    # Defined in /var/cpanel/cpanel.config: apache_ssl_port
    Listen 0.0.0.0:443
    AddType application/x-x509-ca-cert .crt
    AddType application/x-pkcs7-crl .crl
</IfDefine>


AddHandler cgi-script .cgi .pl .plx .ppl .perl
AddHandler server-parsed .shtml
AddType text/html .shtml
AddType application/x-tar .tgz
AddType text/vnd.wap.wml .wml
AddType image/vnd.wap.wbmp .wbmp
AddType text/vnd.wap.wmlscript .wmls
AddType application/vnd.wap.wmlc .wmlc
AddType application/vnd.wap.wmlscriptc .wmlsc

<Location /whm-server-status>
    SetHandler server-status
    Order deny,allow
    Deny from all
    Allow from 127.0.0.1
</Location>



# SUEXEC is supported

Include "/usr/local/apache/conf/includes/pre_virtualhost_global.conf"
Include "/usr/local/apache/conf/includes/pre_virtualhost_2.conf"



Include "/usr/local/apache/conf/includes/post_virtualhost_global.conf"
Include "/usr/local/apache/conf/includes/post_virtualhost_2.conf"


# DO NOT EDIT. AUTOMATICALLY GENERATED.  IF YOU NEED TO MAKE A CHANGE PLEASE USE THE INCLUDE FILES.  




             total       used       free     shared    buffers     cached
Mem:       2059216    1941212     118004          0     262352     861924
-/+ buffers/cache:     816936    1242280
Swap:      4096532     409464    3687068

Open in new window

0
 
fosiul01Commented:
hi

I would not go into tuning server as Small_student is helping you on that.

just some Extra add :

you said, you have database server and web server

which is a good design

Now which server is getting busy ?? database or web server ??


if you want to share the load then you need to think of :

replication for database ( replication more then 2 or 3 server)

and load balancing between more then 2 server


if all your servers are in side the same network, then you can use hardware base load balancing where everything would be control by hardware from ssl to http

read this how it works : http://www.hardwareloadbalancer.com/

there are different type of load balancing , mainly round robin where load will be shared between more then 2 server

but you will have to modify your website script example :
if you use session either you will have to save session into database or you need to implement something layer 3 type network where all your session will be save in central location .


then you need to think, how frequently your data ( not database, content of the page or files)  would be changed dependes on that you might have to think of implementing something real time replication of data , i.e ( DRBD)


its just a guide lines.
0
 
melwongAuthor Commented:
I need a low-cost solution and load balancing is too expensive. I'd say my db is more loaded, but i think both shall be highly loaded too.
0
 
fosiul01Commented:
if Everything is in your same network then you can go with software base free solution.

example :

DRBD ( for file system replication)

and pfsense ( it works as firewall but it has load balancing capacity)

there is some other free software base solution you can use to cut-down cost , but then you wil have know details about those

ohhh, Centos clustering + centos load balancer

its free and good products


0
 
fosiul01Commented:
0
 
fosiul01Commented:
Or this one might give you better idea

http://www.centos.org/docs/2/rh-cm-en-1.0/s1-swinfo-lvs.html


what happening here

(1st layer) you have phirana gateway load balancer
(2nd layer) all your apache server

(3rd layer) all your mysql server
(4th layer) you can use it or you dont have to. the purose is, to create a central location for your files and session. so every files would be saved from here




0
 
melwongAuthor Commented:
yes there are on same network
0
 
Monis MontherSystem ArchitectCommented:
From your free command output it seems that you are not much loaded, Apache cares about memory

You have 816936 Used memory
You have 1.2GB free memory

So I think you are fine with that.. just a final thing on that server

post the output of

ps ax -yl --sort :rss | grep httpd

This will show how much each httpd process will consume of memory.

Last thing we need to check is if your App server is getting an I/O bottle neck or not

Run the top command and check the wi parameter in the CPU area, what are the numbers you get , although this is not the best way to check this , its better checked by using the sysstat package and measured on long period of time to make sure if you have an I/O bottle neck.

When this is done, we can move on to your DB server
0
 
fosiul01Commented:
then you can consider any  of those. just google for more documentation, there are load of nice howto about how to implement those.

i would of prefer redhat cluster suite + phirana


but currently i am using

pfsense + (drbd + openfiler) + redhat cluster

0
 
melwongAuthor Commented:
small_student, i got "ERROR: Unknown sort specifier" when running "ps ax -yl --sort :rss | grep httpd". My server is not loaded now, but I want to be prepared for it.
0
 
Monis MontherSystem ArchitectCommented:
I would recommend Fosiul01 Solution when you reach over 70% utilization of your servers before you reach that point you dont need to spend and expand, but its also a good idea to start the planning process for this.

At the end you might never need to expand, or maybe by simply adding RAM or maybe you actually do need to get new servers and start Load balancing , but before you do anything you must measure your Load.

You should look for monitoring solutions first



0
 
Monis MontherSystem ArchitectCommented:
There is no space between the --sort and the :rss
0
 
melwongAuthor Commented:
i'm researching about virtualization, virtual private server, Amazon EC2. I'm just wondering, do I even need to consider those options to expand my server capacity?
0
 
melwongAuthor Commented:
ps ax -yl --sort :rss | grep httpd
S     0  9476  7702  0  75   0   728 15286 pipe_w pts/1      0:00 grep httpd
S     0  3198 19736  0  81   4 13236 43594 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  5332 19736  0  81   4 22788 54001 semtim ?          0:04 /usr/local/apache/bin/httpd -k start -DSSL
S     0 19736     1  0  81   4 23764 52564 -      ?          0:07 /usr/local/apache/bin/httpd -k start -DSSL
S    99  5327 19736  0  79   4 25192 54633 semtim ?          0:09 /usr/local/apache/bin/httpd -k start -DSSL
S    99  6847 19736  1  79   4 25228 54630 -      ?          0:22 /usr/local/apache/bin/httpd -k start -DSSL
S    99  6852 19736  0  79   4 25304 54690 semtim ?          0:07 /usr/local/apache/bin/httpd -k start -DSSL
S    99  3954 19736  1  79   4 25328 54639 semtim ?          0:25 /usr/local/apache/bin/httpd -k start -DSSL
S    99  5328 19736  1  82   4 25356 54690 semtim ?          0:23 /usr/local/apache/bin/httpd -k start -DSSL
S    99  4802 19736  1  79   4 25420 54706 semtim ?          0:18 /usr/local/apache/bin/httpd -k start -DSSL
S    99  6851 19736  0  79   4 26720 55035 semtim ?          0:06 /usr/local/apache/bin/httpd -k start -DSSL
S    99  3206 19736  3  79   4 26812 55503 semtim ?          1:24 /usr/local/apache/bin/httpd -k start -DSSL
S    99  3204 19736  1  79   4 57080 62376 -      ?          0:24 /usr/local/apache/bin/httpd -k start -DSSL
0
 
fosiul01Commented:
i'm researching about virtualization, virtual private server, Amazon  EC2. I'm just wondering, do I even need to consider those options to  expand my server capacity?  :


if you have everything inside same network, then NO, you dont need , but if you have enough money to buy those server then yes, you can go ahead but those are optional depends on your business needs


if you have replication type of setup then i would not even bother to buy highly specification server as load will be shared among rest so there is not any point of buying high spec server . if any problem you can remove the server , fix it, put it back without even stoping services, may be there would be a 2/3 sec ping drop while changing the services






0
 
Monis MontherSystem ArchitectCommented:
Your Apache processes seem to be fine and not exploiting much memory and they don't even seem to be many of them, but your memory usage indicates that you have something else .

What runlevel are you running your server, it should be runlevel 3 to make sure run the following command

runlevel


0
 
melwongAuthor Commented:
oh, so i guess what I'm doing now (dedicated servers) are the cheapest option there is.
0
 
fosiul01Commented:
for better performance Dedicated server is best

virtual server cant be alternative to dedicated server.

about the out put @small_student ask

make sure that out put is when your server is in pick time, when you think you are getting excessive amount of request into your Apache server , take a snapshot of that time

also

your free out put :
-/+ buffers/cache:     816936    1242280


you have 1.2Gb of Free ram and 800 mb of used ram

i would not really worry about this right now. as @small_student said , you might be all right just by adding few GB of ram into the system .



0
 
melwongAuthor Commented:
but by adding few GB into the RAM, how much extra capacity can i get. Basicaly my question is how to I measure, project my required server capacity? Is there like a formula or calc for this? Or even a rule of thumb?
0
 
fosiul01Commented:
LOL
yes, there is( its a guessing game)


example

the output you showed of ps aux , you have 12 httpd process running

and each process using arround 26Mb of ram ( except one , 57 mb)

so now you can multiply that by

26*12=312MB
so your apache using arround 312 MB of ram

now if you have example 50 process running
so it will be arround 50*26=1.3GB of ram

so you need to find out, how many process create around a busy time

then get a rough idea about how much ram they using, then calculate as i said before

this is how you calculate , how much ram your system needs


also
look at vmstat command, as long as your sytem does not use swap memory they are fine





0
 
Monis MontherSystem ArchitectCommented:
One simple solution that I use sometimes is creating a script that runs on my server to measure

1- CPU
2- Memory
3- No. of apache process running
4- Bandwidth Used
5- No. of Established HTTP connections on port 80

I run this job every 10 seconds and graph it weekly with Excel

This gives an approximate of system resource used to help plan how to expand
0
 
melwongAuthor Commented:
small_student, would you be kind enough to share that script?
0
 
melwongAuthor Commented:
Someone recommended this,

"Cheapest solution is to purchase an additional server and use round robin DNS to load balance the traffic between both servers, without the use of a load balancer of course."
0
 
melwongAuthor Commented:
small_student, do you mind to share that script? I can award you this question.
0
 
melwongAuthor Commented:
small_student, can we also look into the db server
0
 
melwongAuthor Commented:
Is it ok to have free memory under 200MB?
0
 
melwongAuthor Commented:
Hi, EE just reinstated my acc. Can anyone help me with this question?
0
 
melwongAuthor Commented:
thx a lot!
0
All Courses

From novice to tech pro — start learning today.