Link to home
Start Free TrialLog in
Avatar of Nilesh Havire
Nilesh HavireFlag for India

asked on

Suggesions on large scale web applications Server architecture

Hi All Experts,

This is too big and very IMP Project in my Life :) We are running a website with LAMP that is not big at the moment, 2 web servers with HA Proxy load balancing for PHP Application and only 1 MySQL servers, Now a days it works well because we do not have a large traffic for the moment.

But the users are growing rapidly, we will have to scale our architecture to satisfy the needs.There are some articles on the net that I can go through with. But I really need some practical experiences to prepare for this issue feasibly and efficiently .

Any help is appreciated and thanks in advance
SOLUTION
Avatar of arober11
arober11
Flag of United Kingdom of Great Britain and Northern Ireland image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
ASKER CERTIFIED SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of Nilesh Havire

ASKER

Thanks Bro,

I'm searching on google but didnt get proper solution yet, I hope we will have more than 1bn
users in next one year,

Any other Experts Please suggest

Thanks in Advance
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Realy Sorry for Late reply

We have following configuration on Server
Processor:- Inter(R) Xeon(TM) CPU 3.20GHz (16 CPUs),
Memory :- 32 GB

url:- http://www.mydomain.com

Application Made in PHP and Java

All Static Content js, css, static Images(png) are called from nginx server (http://static.mydomain.com)
IMP:- Apache, Tomcat, Nginx is on same Server

web Server:- Apache for "php" and Tomcat for "Java" and nginx for "Static content"

OS:- 64 bit

users are growing rapidly and as per today status we will have 1 lacs user in next 6 month


Please check my web server Status by Apache AB for 500 users


C:\Program Files (x86)\Apache Software Foundation\Apache2.2\bin>ab -n 500 -c 10
http://www.mydomain.com/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking www.mydomain.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests


Server Software:        Apache/2.2.3
Server Hostname:        www.mydomain.com
Server Port:            80

Document Path:          /
Document Length:        22737 bytes

Concurrency Level:      10
Time taken for tests:   95.850 seconds
Complete requests:      500
Failed requests:        1
   (Connect: 1, Receive: 0, Length: 0, Exceptions: 0)
Write errors:           0
Total transferred:      11569000 bytes
HTML transferred:       11368500 bytes
Requests per second:    5.22 [#/sec] (mean)
Time per request:       1917.010 [ms] (mean)
Time per request:       191.701 [ms] (mean, across all concurrent requests)
Transfer rate:          117.87 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       88  142 265.5    115    3088
Processing:   598 1680 2911.1   1028   21994
Waiting:      119  719 2146.4    440   21543
Total:        736 1822 2916.9   1148   22108

Percentage of the requests served within a certain time (ms)
  50%   1148
  66%   1221
  75%   1311
  80%   1455
  90%   3912
  95%   4156
  98%  10250
  99%  22066
 100%  22108 (longest request)

Below is my Mysql Configuration


max-connections                 = 600
max-connect-errors              = 10000
max_allowed_packet              = 64M
interactive-timeout             = 300
wait-timeout                    = 300
innodb_buffer_pool_size         = 8G
table_cache                     = 768k
thread_cache_size               = 64
max_heap_table_size             = 120M
tmp_table_size                  = 120M
query_cache_size                = 120M
query_cache_limit               = 12M
join_buffer_size                = 2M
key_buffer_size                 = 120M

also please check my Memory Information
[root@localhost ~]# cat /proc/meminfo
MemTotal:     32954572 kB
MemFree:       1219008 kB
Buffers:       1653304 kB
Cached:       26845304 kB
SwapCached:          0 kB
Active:       10696404 kB
Inactive:     19806756 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32954572 kB
LowFree:       1219008 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:             868 kB
Writeback:           0 kB
AnonPages:     2004276 kB
Mapped:          37588 kB
Slab:          1128360 kB
PageTables:      26312 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83588812 kB
Committed_AS:  7924908 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296608 kB
VmallocChunk: 34359440571 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB
[root@localhost ~]#

Please suggest me if I have to change anything in my Server configuration
Your AB test was only sending 10 concurrent requests for you home page (so your 500 requests were fired at the server in 50 sequential batches of 10). I'd also suggest you have a look at jMeter, as it will allow you to stress test the server with a more realistic mix of requests (URL's)

Anyway what's the typical number of concurrent users now, and what's predicted. If you don't have any figures, consider running your access logs through: Awstats.

Once you have an idea of the number of concurrent users you intend to support, check your Apache httpd.conf for any maxClients and serverLimit (default = 256) directives and / or if your using Mod_JK to talk to the Tomcat instance, maxProcessors (worker.properties - default 20)  values, add if not present add, and set to what's required.

Next look for / add a maxThreads (default = 200) attribute to the Tomcat server.xml, with the same number FYI: http://tomcat.apache.org/tomcat-7.0-doc/config/http.html

Don't forget to restart your server(s).

Have a look at using Cacti, nagios or another snmp server monitoring tool, to monitor your servers performance, while being tested, if not already monitored.

Then re-run with your ab test with -c xxxxx  where xxxxx is the required number of concurrent page serves.

As to tuning the server and / or app, your probably best hiring a consultant for a few days, as there's a lot of trial and error.

Anyway and as a first hack, if Cacti / meminfo indicates there's still free memory (Active < MemTotal), up the memory allocated to Tomcat via the JAVA_OPTS  (-Xmx) in your catalina.sh, in steps, say 1GB at a time, and also have a look at the size of your JDBC Database connection pool in the applicable web.xml (maxActive - default 100) , and see if it speeds things up (re-run the test).
Make sure the number is <= to the max connections in your mysql config, same goes for the connection timeout values.  Ditto for the settings in the applicable php.ini.

Also look at the max CPU load while under test and if > your CPU count, bring down the maximum number of concurrent connections, in your server conf, till you've got 90->95% utilisation.

If the average page load times are unacceptable, decide on what's acceptable, and pester the consultant you hired to have a look at the App's code, environment e.g. mod_php vs fastcgi vs  zend, and the Database (Cache, indexes, storage engine, data model,....) and see if their are any obvious tweaks. If nothing obvious ask them to assess whether the app will scale horizontally (onto similar kit running in parallel), and if not to estimate how big a single box you'll need.