Suggesions on large scale web applications  Server architecture

Posted on 2012-08-18
Medium Priority
Last Modified: 2012-10-17
Hi All Experts,

This is too big and very IMP Project in my Life :) We are running a website with LAMP that is not big at the moment, 2 web servers with HA Proxy load balancing for PHP Application and only 1 MySQL servers, Now a days it works well because we do not have a large traffic for the moment.

But the users are growing rapidly, we will have to scale our architecture to satisfy the needs.There are some articles on the net that I can go through with. But I really need some practical experiences to prepare for this issue feasibly and efficiently .

Any help is appreciated and thanks in advance
Question by:Nilesh Havire
  • 3
  • 2
LVL 26

Assisted Solution

arober11 earned 999 total points
ID: 38307847
How far do you want to go? At the extreme you can simply Google for "Facebook Architecture" and you'll get the details of a LAMP set-up that's capable of supporting 1bn users e.g.

LVL 11

Accepted Solution

HuaMinChen earned 501 total points
ID: 38307908
You can consider to use Sql server instead, which is powerful, cost-efficient, in comparison with Oracle.

Furthermore, you may also consider to use Asp.net as the web tool instead of PHP now.

Author Comment

by:Nilesh Havire
ID: 38307909
Thanks Bro,

I'm searching on google but didnt get proper solution yet, I hope we will have more than 1bn
users in next one year,

Any other Experts Please suggest

Thanks in Advance
Configuration Guide and Best Practices

Read the guide to learn how to orchestrate Data ONTAP, create application-consistent backups and enable fast recovery from NetApp storage snapshots. Version 9.5 also contains performance and scalability enhancements to meet the needs of the largest enterprise environments.

LVL 26

Assisted Solution

arober11 earned 999 total points
ID: 38308367
Without any details of:

- What the app does.

- The existing traffic patterns e.g the number of pages / messages, page / object size, nature of the objects (static vs dynamic content and types e.g. css, js, png, mp3....), maximum concurrent requests, average page delivery times, protocols used e.g. http, xhttp, and https

- Existing load on the servers, memory and I/O utilisation, and storage (Play with Cacti or Nagios if you don't have any stat's already)

- Server, network and storage specs, mysql storage engine(s) used

- Nature of the DB accesses (read's vs writes, is the app transactional, what's the query complexity)

- php frameworks used

- web server(s) used e.g. Apache httpd vs nginx vs lighttpd

- OS types and architecture(s) e.g. 32 vs 64bit, and file systems.

- What the maximim through is for the existing hardware (from a stress / load test, Google jmeter or Apache AB).

- What traffic levels you expect / have budget to support.....

- Does the app have to meet any external standards e.g. PCI-DSS


Your unlikely to get any detailed recommendations.

Author Comment

by:Nilesh Havire
ID: 38371635
Realy Sorry for Late reply

We have following configuration on Server
Processor:- Inter(R) Xeon(TM) CPU 3.20GHz (16 CPUs),
Memory :- 32 GB

url:- http://www.mydomain.com

Application Made in PHP and Java

All Static Content js, css, static Images(png) are called from nginx server (http://static.mydomain.com)
IMP:- Apache, Tomcat, Nginx is on same Server

web Server:- Apache for "php" and Tomcat for "Java" and nginx for "Static content"

OS:- 64 bit

users are growing rapidly and as per today status we will have 1 lacs user in next 6 month

Please check my web server Status by Apache AB for 500 users

C:\Program Files (x86)\Apache Software Foundation\Apache2.2\bin>ab -n 500 -c 10
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking www.mydomain.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests

Server Software:        Apache/2.2.3
Server Hostname:        www.mydomain.com
Server Port:            80

Document Path:          /
Document Length:        22737 bytes

Concurrency Level:      10
Time taken for tests:   95.850 seconds
Complete requests:      500
Failed requests:        1
   (Connect: 1, Receive: 0, Length: 0, Exceptions: 0)
Write errors:           0
Total transferred:      11569000 bytes
HTML transferred:       11368500 bytes
Requests per second:    5.22 [#/sec] (mean)
Time per request:       1917.010 [ms] (mean)
Time per request:       191.701 [ms] (mean, across all concurrent requests)
Transfer rate:          117.87 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       88  142 265.5    115    3088
Processing:   598 1680 2911.1   1028   21994
Waiting:      119  719 2146.4    440   21543
Total:        736 1822 2916.9   1148   22108

Percentage of the requests served within a certain time (ms)
  50%   1148
  66%   1221
  75%   1311
  80%   1455
  90%   3912
  95%   4156
  98%  10250
  99%  22066
 100%  22108 (longest request)

Below is my Mysql Configuration

max-connections                 = 600
max-connect-errors              = 10000
max_allowed_packet              = 64M
interactive-timeout             = 300
wait-timeout                    = 300
innodb_buffer_pool_size         = 8G
table_cache                     = 768k
thread_cache_size               = 64
max_heap_table_size             = 120M
tmp_table_size                  = 120M
query_cache_size                = 120M
query_cache_limit               = 12M
join_buffer_size                = 2M
key_buffer_size                 = 120M

also please check my Memory Information
[root@localhost ~]# cat /proc/meminfo
MemTotal:     32954572 kB
MemFree:       1219008 kB
Buffers:       1653304 kB
Cached:       26845304 kB
SwapCached:          0 kB
Active:       10696404 kB
Inactive:     19806756 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32954572 kB
LowFree:       1219008 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:             868 kB
Writeback:           0 kB
AnonPages:     2004276 kB
Mapped:          37588 kB
Slab:          1128360 kB
PageTables:      26312 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83588812 kB
Committed_AS:  7924908 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296608 kB
VmallocChunk: 34359440571 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB
[root@localhost ~]#

Please suggest me if I have to change anything in my Server configuration
LVL 26

Expert Comment

ID: 38378170
Your AB test was only sending 10 concurrent requests for you home page (so your 500 requests were fired at the server in 50 sequential batches of 10). I'd also suggest you have a look at jMeter, as it will allow you to stress test the server with a more realistic mix of requests (URL's)

Anyway what's the typical number of concurrent users now, and what's predicted. If you don't have any figures, consider running your access logs through: Awstats.

Once you have an idea of the number of concurrent users you intend to support, check your Apache httpd.conf for any maxClients and serverLimit (default = 256) directives and / or if your using Mod_JK to talk to the Tomcat instance, maxProcessors (worker.properties - default 20)  values, add if not present add, and set to what's required.

Next look for / add a maxThreads (default = 200) attribute to the Tomcat server.xml, with the same number FYI: http://tomcat.apache.org/tomcat-7.0-doc/config/http.html

Don't forget to restart your server(s).

Have a look at using Cacti, nagios or another snmp server monitoring tool, to monitor your servers performance, while being tested, if not already monitored.

Then re-run with your ab test with -c xxxxx  where xxxxx is the required number of concurrent page serves.

As to tuning the server and / or app, your probably best hiring a consultant for a few days, as there's a lot of trial and error.

Anyway and as a first hack, if Cacti / meminfo indicates there's still free memory (Active < MemTotal), up the memory allocated to Tomcat via the JAVA_OPTS  (-Xmx) in your catalina.sh, in steps, say 1GB at a time, and also have a look at the size of your JDBC Database connection pool in the applicable web.xml (maxActive - default 100) , and see if it speeds things up (re-run the test).
Make sure the number is <= to the max connections in your mysql config, same goes for the connection timeout values.  Ditto for the settings in the applicable php.ini.

Also look at the max CPU load while under test and if > your CPU count, bring down the maximum number of concurrent connections, in your server conf, till you've got 90->95% utilisation.

If the average page load times are unacceptable, decide on what's acceptable, and pester the consultant you hired to have a look at the App's code, environment e.g. mod_php vs fastcgi vs  zend, and the Database (Cache, indexes, storage engine, data model,....) and see if their are any obvious tweaks. If nothing obvious ask them to assess whether the app will scale horizontally (onto similar kit running in parallel), and if not to estimate how big a single box you'll need.

Featured Post


Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

An alternative to the "For XML" way of pivoting and concatenating result sets into strings, and an easy introduction to "common table expressions" (CTEs). Being someone who is always looking for alternatives to "work your data", I came across this …
When trying to connect from SSMS v17.x to a SQL Server Integration Services 2016 instance or previous version, you get the error “Connecting to the Integration Services service on the computer failed with the following error: 'The specified service …
Via a live example, show how to set up a backup for SQL Server using a Maintenance Plan and how to schedule the job into SQL Server Agent.
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
Suggested Courses

850 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question