Nilesh Havire
asked on
Suggesions on large scale web applications Server architecture
Hi All Experts,
This is too big and very IMP Project in my Life :) We are running a website with LAMP that is not big at the moment, 2 web servers with HA Proxy load balancing for PHP Application and only 1 MySQL servers, Now a days it works well because we do not have a large traffic for the moment.
But the users are growing rapidly, we will have to scale our architecture to satisfy the needs.There are some articles on the net that I can go through with. But I really need some practical experiences to prepare for this issue feasibly and efficiently .
Any help is appreciated and thanks in advance
This is too big and very IMP Project in my Life :) We are running a website with LAMP that is not big at the moment, 2 web servers with HA Proxy load balancing for PHP Application and only 1 MySQL servers, Now a days it works well because we do not have a large traffic for the moment.
But the users are growing rapidly, we will have to scale our architecture to satisfy the needs.There are some articles on the net that I can go through with. But I really need some practical experiences to prepare for this issue feasibly and efficiently .
Any help is appreciated and thanks in advance
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Realy Sorry for Late reply
We have following configuration on Server
Processor:- Inter(R) Xeon(TM) CPU 3.20GHz (16 CPUs),
Memory :- 32 GB
url:- http://www.mydomain.com
Application Made in PHP and Java
All Static Content js, css, static Images(png) are called from nginx server (http://static.mydomain.com)
IMP:- Apache, Tomcat, Nginx is on same Server
web Server:- Apache for "php" and Tomcat for "Java" and nginx for "Static content"
OS:- 64 bit
users are growing rapidly and as per today status we will have 1 lacs user in next 6 month
Please check my web server Status by Apache AB for 500 users
C:\Program Files (x86)\Apache Software Foundation\Apache2.2\bin>a b -n 500 -c 10
http://www.mydomain.com/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking www.mydomain.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests
Server Software: Apache/2.2.3
Server Hostname: www.mydomain.com
Server Port: 80
Document Path: /
Document Length: 22737 bytes
Concurrency Level: 10
Time taken for tests: 95.850 seconds
Complete requests: 500
Failed requests: 1
(Connect: 1, Receive: 0, Length: 0, Exceptions: 0)
Write errors: 0
Total transferred: 11569000 bytes
HTML transferred: 11368500 bytes
Requests per second: 5.22 [#/sec] (mean)
Time per request: 1917.010 [ms] (mean)
Time per request: 191.701 [ms] (mean, across all concurrent requests)
Transfer rate: 117.87 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 88 142 265.5 115 3088
Processing: 598 1680 2911.1 1028 21994
Waiting: 119 719 2146.4 440 21543
Total: 736 1822 2916.9 1148 22108
Percentage of the requests served within a certain time (ms)
50% 1148
66% 1221
75% 1311
80% 1455
90% 3912
95% 4156
98% 10250
99% 22066
100% 22108 (longest request)
Below is my Mysql Configuration
max-connections = 600
max-connect-errors = 10000
max_allowed_packet = 64M
interactive-timeout = 300
wait-timeout = 300
innodb_buffer_pool_size = 8G
table_cache = 768k
thread_cache_size = 64
max_heap_table_size = 120M
tmp_table_size = 120M
query_cache_size = 120M
query_cache_limit = 12M
join_buffer_size = 2M
key_buffer_size = 120M
also please check my Memory Information
[root@localhost ~]# cat /proc/meminfo
MemTotal: 32954572 kB
MemFree: 1219008 kB
Buffers: 1653304 kB
Cached: 26845304 kB
SwapCached: 0 kB
Active: 10696404 kB
Inactive: 19806756 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 32954572 kB
LowFree: 1219008 kB
SwapTotal: 67111528 kB
SwapFree: 67111528 kB
Dirty: 868 kB
Writeback: 0 kB
AnonPages: 2004276 kB
Mapped: 37588 kB
Slab: 1128360 kB
PageTables: 26312 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 83588812 kB
Committed_AS: 7924908 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 296608 kB
VmallocChunk: 34359440571 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB
[root@localhost ~]#
Please suggest me if I have to change anything in my Server configuration
We have following configuration on Server
Processor:- Inter(R) Xeon(TM) CPU 3.20GHz (16 CPUs),
Memory :- 32 GB
url:- http://www.mydomain.com
Application Made in PHP and Java
All Static Content js, css, static Images(png) are called from nginx server (http://static.mydomain.com)
IMP:- Apache, Tomcat, Nginx is on same Server
web Server:- Apache for "php" and Tomcat for "Java" and nginx for "Static content"
OS:- 64 bit
users are growing rapidly and as per today status we will have 1 lacs user in next 6 month
Please check my web server Status by Apache AB for 500 users
C:\Program Files (x86)\Apache Software Foundation\Apache2.2\bin>a
http://www.mydomain.com/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking www.mydomain.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests
Server Software: Apache/2.2.3
Server Hostname: www.mydomain.com
Server Port: 80
Document Path: /
Document Length: 22737 bytes
Concurrency Level: 10
Time taken for tests: 95.850 seconds
Complete requests: 500
Failed requests: 1
(Connect: 1, Receive: 0, Length: 0, Exceptions: 0)
Write errors: 0
Total transferred: 11569000 bytes
HTML transferred: 11368500 bytes
Requests per second: 5.22 [#/sec] (mean)
Time per request: 1917.010 [ms] (mean)
Time per request: 191.701 [ms] (mean, across all concurrent requests)
Transfer rate: 117.87 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 88 142 265.5 115 3088
Processing: 598 1680 2911.1 1028 21994
Waiting: 119 719 2146.4 440 21543
Total: 736 1822 2916.9 1148 22108
Percentage of the requests served within a certain time (ms)
50% 1148
66% 1221
75% 1311
80% 1455
90% 3912
95% 4156
98% 10250
99% 22066
100% 22108 (longest request)
Below is my Mysql Configuration
max-connections = 600
max-connect-errors = 10000
max_allowed_packet = 64M
interactive-timeout = 300
wait-timeout = 300
innodb_buffer_pool_size = 8G
table_cache = 768k
thread_cache_size = 64
max_heap_table_size = 120M
tmp_table_size = 120M
query_cache_size = 120M
query_cache_limit = 12M
join_buffer_size = 2M
key_buffer_size = 120M
also please check my Memory Information
[root@localhost ~]# cat /proc/meminfo
MemTotal: 32954572 kB
MemFree: 1219008 kB
Buffers: 1653304 kB
Cached: 26845304 kB
SwapCached: 0 kB
Active: 10696404 kB
Inactive: 19806756 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 32954572 kB
LowFree: 1219008 kB
SwapTotal: 67111528 kB
SwapFree: 67111528 kB
Dirty: 868 kB
Writeback: 0 kB
AnonPages: 2004276 kB
Mapped: 37588 kB
Slab: 1128360 kB
PageTables: 26312 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 83588812 kB
Committed_AS: 7924908 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 296608 kB
VmallocChunk: 34359440571 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB
[root@localhost ~]#
Please suggest me if I have to change anything in my Server configuration
Your AB test was only sending 10 concurrent requests for you home page (so your 500 requests were fired at the server in 50 sequential batches of 10). I'd also suggest you have a look at jMeter, as it will allow you to stress test the server with a more realistic mix of requests (URL's)
Anyway what's the typical number of concurrent users now, and what's predicted. If you don't have any figures, consider running your access logs through: Awstats.
Once you have an idea of the number of concurrent users you intend to support, check your Apache httpd.conf for any maxClients and serverLimit (default = 256) directives and / or if your using Mod_JK to talk to the Tomcat instance, maxProcessors (worker.properties - default 20) values, add if not present add, and set to what's required.
Next look for / add a maxThreads (default = 200) attribute to the Tomcat server.xml, with the same number FYI: http://tomcat.apache.org/tomcat-7.0-doc/config/http.html
Don't forget to restart your server(s).
Have a look at using Cacti, nagios or another snmp server monitoring tool, to monitor your servers performance, while being tested, if not already monitored.
Then re-run with your ab test with -c xxxxx where xxxxx is the required number of concurrent page serves.
As to tuning the server and / or app, your probably best hiring a consultant for a few days, as there's a lot of trial and error.
Anyway and as a first hack, if Cacti / meminfo indicates there's still free memory (Active < MemTotal), up the memory allocated to Tomcat via the JAVA_OPTS (-Xmx) in your catalina.sh, in steps, say 1GB at a time, and also have a look at the size of your JDBC Database connection pool in the applicable web.xml (maxActive - default 100) , and see if it speeds things up (re-run the test).
Make sure the number is <= to the max connections in your mysql config, same goes for the connection timeout values. Ditto for the settings in the applicable php.ini.
Also look at the max CPU load while under test and if > your CPU count, bring down the maximum number of concurrent connections, in your server conf, till you've got 90->95% utilisation.
If the average page load times are unacceptable, decide on what's acceptable, and pester the consultant you hired to have a look at the App's code, environment e.g. mod_php vs fastcgi vs zend, and the Database (Cache, indexes, storage engine, data model,....) and see if their are any obvious tweaks. If nothing obvious ask them to assess whether the app will scale horizontally (onto similar kit running in parallel), and if not to estimate how big a single box you'll need.
Anyway what's the typical number of concurrent users now, and what's predicted. If you don't have any figures, consider running your access logs through: Awstats.
Once you have an idea of the number of concurrent users you intend to support, check your Apache httpd.conf for any maxClients and serverLimit (default = 256) directives and / or if your using Mod_JK to talk to the Tomcat instance, maxProcessors (worker.properties - default 20) values, add if not present add, and set to what's required.
Next look for / add a maxThreads (default = 200) attribute to the Tomcat server.xml, with the same number FYI: http://tomcat.apache.org/tomcat-7.0-doc/config/http.html
Don't forget to restart your server(s).
Have a look at using Cacti, nagios or another snmp server monitoring tool, to monitor your servers performance, while being tested, if not already monitored.
Then re-run with your ab test with -c xxxxx where xxxxx is the required number of concurrent page serves.
As to tuning the server and / or app, your probably best hiring a consultant for a few days, as there's a lot of trial and error.
Anyway and as a first hack, if Cacti / meminfo indicates there's still free memory (Active < MemTotal), up the memory allocated to Tomcat via the JAVA_OPTS (-Xmx) in your catalina.sh, in steps, say 1GB at a time, and also have a look at the size of your JDBC Database connection pool in the applicable web.xml (maxActive - default 100) , and see if it speeds things up (re-run the test).
Make sure the number is <= to the max connections in your mysql config, same goes for the connection timeout values. Ditto for the settings in the applicable php.ini.
Also look at the max CPU load while under test and if > your CPU count, bring down the maximum number of concurrent connections, in your server conf, till you've got 90->95% utilisation.
If the average page load times are unacceptable, decide on what's acceptable, and pester the consultant you hired to have a look at the App's code, environment e.g. mod_php vs fastcgi vs zend, and the Database (Cache, indexes, storage engine, data model,....) and see if their are any obvious tweaks. If nothing obvious ask them to assess whether the app will scale horizontally (onto similar kit running in parallel), and if not to estimate how big a single box you'll need.
ASKER
I'm searching on google but didnt get proper solution yet, I hope we will have more than 1bn
users in next one year,
Any other Experts Please suggest
Thanks in Advance