Suggesions on large scale web applications  Server architecture

Posted on 2012-08-18
Last Modified: 2012-10-17
Hi All Experts,

This is too big and very IMP Project in my Life :) We are running a website with LAMP that is not big at the moment, 2 web servers with HA Proxy load balancing for PHP Application and only 1 MySQL servers, Now a days it works well because we do not have a large traffic for the moment.

But the users are growing rapidly, we will have to scale our architecture to satisfy the needs.There are some articles on the net that I can go through with. But I really need some practical experiences to prepare for this issue feasibly and efficiently .

Any help is appreciated and thanks in advance
Question by:Nilesh Havire
    LVL 26

    Assisted Solution

    How far do you want to go? At the extreme you can simply Google for "Facebook Architecture" and you'll get the details of a LAMP set-up that's capable of supporting 1bn users e.g.
    LVL 10

    Accepted Solution

    You can consider to use Sql server instead, which is powerful, cost-efficient, in comparison with Oracle.

    Furthermore, you may also consider to use as the web tool instead of PHP now.

    Author Comment

    by:Nilesh Havire
    Thanks Bro,

    I'm searching on google but didnt get proper solution yet, I hope we will have more than 1bn
    users in next one year,

    Any other Experts Please suggest

    Thanks in Advance
    LVL 26

    Assisted Solution

    Without any details of:

    - What the app does.

    - The existing traffic patterns e.g the number of pages / messages, page / object size, nature of the objects (static vs dynamic content and types e.g. css, js, png, mp3....), maximum concurrent requests, average page delivery times, protocols used e.g. http, xhttp, and https

    - Existing load on the servers, memory and I/O utilisation, and storage (Play with Cacti or Nagios if you don't have any stat's already)

    - Server, network and storage specs, mysql storage engine(s) used

    - Nature of the DB accesses (read's vs writes, is the app transactional, what's the query complexity)

    - php frameworks used

    - web server(s) used e.g. Apache httpd vs nginx vs lighttpd

    - OS types and architecture(s) e.g. 32 vs 64bit, and file systems.

    - What the maximim through is for the existing hardware (from a stress / load test, Google jmeter or Apache AB).

    - What traffic levels you expect / have budget to support.....

    - Does the app have to meet any external standards e.g. PCI-DSS


    Your unlikely to get any detailed recommendations.

    Author Comment

    by:Nilesh Havire
    Realy Sorry for Late reply

    We have following configuration on Server
    Processor:- Inter(R) Xeon(TM) CPU 3.20GHz (16 CPUs),
    Memory :- 32 GB


    Application Made in PHP and Java

    All Static Content js, css, static Images(png) are called from nginx server (
    IMP:- Apache, Tomcat, Nginx is on same Server

    web Server:- Apache for "php" and Tomcat for "Java" and nginx for "Static content"

    OS:- 64 bit

    users are growing rapidly and as per today status we will have 1 lacs user in next 6 month

    Please check my web server Status by Apache AB for 500 users

    C:\Program Files (x86)\Apache Software Foundation\Apache2.2\bin>ab -n 500 -c 10
    This is ApacheBench, Version 2.3 <$Revision: 655654 $>
    Copyright 1996 Adam Twiss, Zeus Technology Ltd,
    Licensed to The Apache Software Foundation,

    Benchmarking (be patient)
    Completed 100 requests
    Completed 200 requests
    Completed 300 requests
    Completed 400 requests
    Completed 500 requests
    Finished 500 requests

    Server Software:        Apache/2.2.3
    Server Hostname:
    Server Port:            80

    Document Path:          /
    Document Length:        22737 bytes

    Concurrency Level:      10
    Time taken for tests:   95.850 seconds
    Complete requests:      500
    Failed requests:        1
       (Connect: 1, Receive: 0, Length: 0, Exceptions: 0)
    Write errors:           0
    Total transferred:      11569000 bytes
    HTML transferred:       11368500 bytes
    Requests per second:    5.22 [#/sec] (mean)
    Time per request:       1917.010 [ms] (mean)
    Time per request:       191.701 [ms] (mean, across all concurrent requests)
    Transfer rate:          117.87 [Kbytes/sec] received

    Connection Times (ms)
                  min  mean[+/-sd] median   max
    Connect:       88  142 265.5    115    3088
    Processing:   598 1680 2911.1   1028   21994
    Waiting:      119  719 2146.4    440   21543
    Total:        736 1822 2916.9   1148   22108

    Percentage of the requests served within a certain time (ms)
      50%   1148
      66%   1221
      75%   1311
      80%   1455
      90%   3912
      95%   4156
      98%  10250
      99%  22066
     100%  22108 (longest request)

    Below is my Mysql Configuration

    max-connections                 = 600
    max-connect-errors              = 10000
    max_allowed_packet              = 64M
    interactive-timeout             = 300
    wait-timeout                    = 300
    innodb_buffer_pool_size         = 8G
    table_cache                     = 768k
    thread_cache_size               = 64
    max_heap_table_size             = 120M
    tmp_table_size                  = 120M
    query_cache_size                = 120M
    query_cache_limit               = 12M
    join_buffer_size                = 2M
    key_buffer_size                 = 120M

    also please check my Memory Information
    [root@localhost ~]# cat /proc/meminfo
    MemTotal:     32954572 kB
    MemFree:       1219008 kB
    Buffers:       1653304 kB
    Cached:       26845304 kB
    SwapCached:          0 kB
    Active:       10696404 kB
    Inactive:     19806756 kB
    HighTotal:           0 kB
    HighFree:            0 kB
    LowTotal:     32954572 kB
    LowFree:       1219008 kB
    SwapTotal:    67111528 kB
    SwapFree:     67111528 kB
    Dirty:             868 kB
    Writeback:           0 kB
    AnonPages:     2004276 kB
    Mapped:          37588 kB
    Slab:          1128360 kB
    PageTables:      26312 kB
    NFS_Unstable:        0 kB
    Bounce:              0 kB
    CommitLimit:  83588812 kB
    Committed_AS:  7924908 kB
    VmallocTotal: 34359738367 kB
    VmallocUsed:    296608 kB
    VmallocChunk: 34359440571 kB
    HugePages_Total:     0
    HugePages_Free:      0
    HugePages_Rsvd:      0
    Hugepagesize:     2048 kB
    [root@localhost ~]#

    Please suggest me if I have to change anything in my Server configuration
    LVL 26

    Expert Comment

    Your AB test was only sending 10 concurrent requests for you home page (so your 500 requests were fired at the server in 50 sequential batches of 10). I'd also suggest you have a look at jMeter, as it will allow you to stress test the server with a more realistic mix of requests (URL's)

    Anyway what's the typical number of concurrent users now, and what's predicted. If you don't have any figures, consider running your access logs through: Awstats.

    Once you have an idea of the number of concurrent users you intend to support, check your Apache httpd.conf for any maxClients and serverLimit (default = 256) directives and / or if your using Mod_JK to talk to the Tomcat instance, maxProcessors ( - default 20)  values, add if not present add, and set to what's required.

    Next look for / add a maxThreads (default = 200) attribute to the Tomcat server.xml, with the same number FYI:

    Don't forget to restart your server(s).

    Have a look at using Cacti, nagios or another snmp server monitoring tool, to monitor your servers performance, while being tested, if not already monitored.

    Then re-run with your ab test with -c xxxxx  where xxxxx is the required number of concurrent page serves.

    As to tuning the server and / or app, your probably best hiring a consultant for a few days, as there's a lot of trial and error.

    Anyway and as a first hack, if Cacti / meminfo indicates there's still free memory (Active < MemTotal), up the memory allocated to Tomcat via the JAVA_OPTS  (-Xmx) in your, in steps, say 1GB at a time, and also have a look at the size of your JDBC Database connection pool in the applicable web.xml (maxActive - default 100) , and see if it speeds things up (re-run the test).
    Make sure the number is <= to the max connections in your mysql config, same goes for the connection timeout values.  Ditto for the settings in the applicable php.ini.

    Also look at the max CPU load while under test and if > your CPU count, bring down the maximum number of concurrent connections, in your server conf, till you've got 90->95% utilisation.

    If the average page load times are unacceptable, decide on what's acceptable, and pester the consultant you hired to have a look at the App's code, environment e.g. mod_php vs fastcgi vs  zend, and the Database (Cache, indexes, storage engine, data model,....) and see if their are any obvious tweaks. If nothing obvious ask them to assess whether the app will scale horizontally (onto similar kit running in parallel), and if not to estimate how big a single box you'll need.

    Featured Post

    Highfive Gives IT Their Time Back

    Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

    Join & Write a Comment

    Everyone has problem when going to load data into Data warehouse (EDW). They all need to confirm that data quality is good but they don't no how to proceed. Microsoft has provided new task within SSIS 2008 called "Data Profiler Task". It solve th…
    Slowly Changing Dimension Transformation component in data task flow is very useful for us to manage and control how data changes in SSIS.
    This videos aims to give the viewer a basic demonstration of how a user can query current session information by using the SYS_CONTEXT function
    Via a live example, show how to backup a database, simulate a failure backup the tail of the database transaction log and perform the restore.

    746 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    18 Experts available now in Live!

    Get 1:1 Help Now