Preparing for 100 million users

I am preparing to work on a web project that will have 30 million users within 6 months and 100 million globally within 2 years.

Can anyone recommend best practices for this type of web site.

1) what kind of hardware will it take to handle this load? What should I look for?
2) what coding changes are necessary within the code to handle this? or does IIS handle it automatically via the server application?
3) I'll be using ASP.NET 4.0, SQL Server 2008, and Windows 2008 Web Server with IIS7.

Thanks for your help.
LVL 2
Starr DuskkASP.NET VB.NET DeveloperAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Jens FiedererTest Developer/ValidatorCommented:
It doesn't really matter much if you have 100 million users and each one only uses your project once, you store no data about each user, and they keep themselves spaced a minute apart, but if you have 100 million users every minute you are going to have to buy thousands of servers to handle the load.

You have to know how many requests/unit of time to expect, how big the requests are going to be, whether you are maintaining sessions with each user. and what kind of data you are going to store.

If you don't already have more expertise than almost all of the experts on this site, odds are very high that you should be looking for a new job as early as possible.
0
Imran Javed ZiaConsultant Software Engineer - .NET ArchitectCommented:
Hi,

Indeed it is very open query. You may consider following:

You have to think about a Distributed Systems may be a Clould.
     Distributed DataBase
     Multiple Web Servers
Page Caching.
Proper Layered Approach.

Thanks
0
Starr DuskkASP.NET VB.NET DeveloperAuthor Commented:
The site might be equated with a site as big and busy as something like google. But would store detailed data, and search detailed data like sites such as Match.com, or Monster.com - a site that collects information and does matching to find data. So yes, I will be maintaining sessions for each user, which should be retained when swapping servers. I would attempt to keep the sessions small with just the table identification keys for the records they are viewing, editing. I wouldn't be storing big blocks of content, etc.

I will not be in charge of the hardware. I will be in charge of the development. I have programmed numerous web applications for businesses, but have done nothing special in the code to handle millions of users. Is it necessary in .NET to alter the code for millions of users? I programmed a site that was once on one server and then they put it on two servers with load balancing and I made changes in the config file to allow identification of which server the person was actuallly on for an admin page, but nothing else changed.

Are there coding changes required when programming for a large site? If so, please direct me to an article or book that discusses these kinds of needs. Maybe someone else can help me rather than telling me I'm too stupid to keep this job and can't learn.
0
Cloud Class® Course: C++ 11 Fundamentals

This course will introduce you to C++ 11 and teach you about syntax fundamentals.

Starr DuskkASP.NET VB.NET DeveloperAuthor Commented:
Thank you IJZ.

Yes, I will be using layered architecture and will, as with all my applications, keep presentation, business, and data layers separate. I will be working with Visual Studio 2010 projects with DLL libraries and an ORM.

I will look into the distributed systems - I know it allows separate machines to handle different processing, such as solving a large complex problem. But wouldn't multiple servers, load balanced, accomplish the same purpose without requiring a code change? For instance, if I code for a one server application that intends to grow, wouldn't the hard ware load balancing take care of the growth without requiring coding changes or separation of code blocks onto a distributed system?

Does an expert on here actually have experience working on a large website such as google or youtube or linkedin such as I'm describing? What considerations were necessary in the coding process?

Thanks.
0
Jens FiedererTest Developer/ValidatorCommented:
Nobody is saying you are too stupid and can't learn - but sites like google are put together by large teams of people.  

I have only worked on sites with thousands of users at best, and it is already hard, and I am not stupid - but if I were responsible for doing such a thing in a few months by myself, I would be looking for a new job, because I would know I could not do it by myself ... I am saying the task is VERY hard.

You need to know a lot of details about what you are going to do, because things that are done to make things efficient also impose limitations.  The best performance is usually in big web farms where any one request can be handled by ANY available server, and all necessary context information is actually in the request (cookies, form fields, such form of storage).  When session information must be kept on the servers, usually either a client has to be steered toward one specific server (making load balancing harder) or the information has to be kept in the database (slowing each request down with database requests).

Even many experts that have worked on google, etc., might not know much about this unless they were specificallly involved with that part of the architecture.

Things that you need to be concerned about in your own coding include not only session storage but caching, and if there are concerns about multiple users making changes to the same resources you are going to have to do transactional processing.

If you are very lucky, and your application is the right sort, things will scale automatically with no worries on your end at all.
0
Jens FiedererTest Developer/ValidatorCommented:
If you could manage to get to talk with one of the architects of Myspace, that is (or at least was) one of the largest ASP.NET sites:

http://highscalability.com/blog/2009/2/12/myspace-architecture.html
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Jens FiedererTest Developer/ValidatorCommented:
By the way, the site that had that article might be useful:    http://highscalability.com/start-here/
0
RovastarCommented:
I will receive no points for this answer but it is the correct one:

"I am preparing to work on a web project that will have 30 million users within 6 months and 100 million globally within 2 years.

Can anyone recommend best practices for this type of web site."

Best practice -  get a team of professional consultants in that have experience with high volume websites.

Realistically your numbers are far too high you will not get those figures - no small scale business will get these numbers if it is a large scale project on the size of google.com you will need thousands of servers and then you will have tens of millions in setup costs and have professional consultants. If they have got tens of millions in budget for this project then you have blagged your way into a good position there, good look to you. If not they will not get these numbers. I have worked on large projects and you need a large infrastructure and teams of programmers for this.

And yes you have to have special configuration for large setups.

You need to know how many concurrent connections are they expecting per say second if they say 30 millions then and marry that up with your base application run that on the test server and then you will see how many servers you need.
0
Anthony PerkinsCommented:
Or too put it succinctly, assuming just for a minute those numbers are real, do you honestly believe that the complete answer to your questions can be provided in as short a space as provided here on Experts Exchange?  

Don't take this personally, but as everyone has pointed out here already, you are clearly misguided and out of your league.
0
Starr DuskkASP.NET VB.NET DeveloperAuthor Commented:
Thanks to those who offered direction.

The bottom line is, every single person who is now working on a large project didn't start out with all that knowledge. They were all noobs, just like, evidentally, you, and me.

Of course, I'll need to hire a team. But I like to do my research first to know what kind of a team to put together. Since you evidentally believe the experts here are so void of ability to work on such a team, I  won't be looking among the ranks here for contractors.

Experts-exchange.com is for answers. Not for people to slam someone because they are attempting to do something you wouldn't attempt to do because of your insecurities and inabilities.
0
RovastarCommented:
Seriously get serious figures  together about how much your employers/backers what to get concurrent users to the site. For every 1,000 concurrent users you will need at least another frond end web server.
0
Anthony PerkinsCommented:
>>Experts-exchange.com is for answers.<<
Experts Exchange.com is for serious questions. Consider this:
"I'll be using ASP.NET 4.0, SQL Server 2008, and Windows 2008 Web Server with IIS7.
...
I will not be in charge of the hardware."
The reason we cannot take this question seriously is because you evidentally have not done your homework.  For example: How do you know you will be able to use MS tools.  What if they (like Google, EBay, Amazon and Experts-Exchange for that matter) do not choose to use a Windows O/S? What then you are going to learn Java/Oracle in six months?

0
Starr DuskkASP.NET VB.NET DeveloperAuthor Commented:
acperkins,

Are you here to argue with the paid question askers? Seriously? Do you want to go down this route?

>>The reason we cannot take this question seriously is because you evidentally have not done your homework.

I guess I thought Experts-exchange.com was the place to go to do my homework. In fact, I know it is. I always come to Experts-exchange.com to do my homework. That's why I pay for a subscription and have been a paid member since 2005 and have asked 1120 questions. But Experts-exchange.com doesn't expect their so-called geniuses to be rude to their question askers.

>>How do you know you will be able to use MS tools.... What then you are going to learn Java/Oracle in six months?

Because I'm in charge. And what  makes you think I don't already know Java and Oracle. I've been working with Oracle since 1998 and with Java since 1999. My current 23 month project has been working with Oracle 10g-11g.

I do appreciate the links that jensfiederer provided and the non-judgmental comments made by IJZ. If the rest of you are in over your heads on this response then keep your mouths shut and move along.

I'd rather get no answers than these kinds of responses. What happened to "there are no dumb questions."
0
Anthony PerkinsCommented:
>>Are you here to argue with the paid question askers? <<
I am sorry you took it that way.  I thought I was stating the obvious.  As to the fact that you are a paid subscriber for many years, congratulations I am sure EE appreciates your loyalty.  I had no idea and to be quite frank it does not make any difference to me as to how address members here.  Does it to you?

>>I guess I thought Experts-exchange.com was the place to go to do my homework.<<
Actually no, we do not do homework questions, that is a violation of the EE Guidelines.

>>But Experts-exchange.com doesn't expect their so-called geniuses to be rude to their question askers.<<
I fail to see where I have been rude to you, but have you considered the fact that you may be a tad over-sensitive.  But feel free to take it up in Community Support that is a more appropriate venue for this type of discussion.

Good luck.
P.S. Make sure to respond here when you have reached your first million users and don't forget to include the URL.
0
Starr DuskkASP.NET VB.NET DeveloperAuthor Commented:
Rovastar, thanks for your responses.
0
Anthony PerkinsCommented:
BobCSD,

I am preparing to work on a web project that will have 30 million users within 6 months and 100 million globally within 2 years.
It is now two years and I am really curious how your project is going.  What tools did you end up using and did you reach the "100 million globally" ?

Anthony
0
Starr DuskkASP.NET VB.NET DeveloperAuthor Commented:
I found out the "owners" never actually had any cash. They were waiting for an investor which never did happen. So I never got paid and took another contract elsewhere.

Did you put this in your calendar to follow-up so you could gloat if it failed?

Gloat away. LOL
0
Jens FiedererTest Developer/ValidatorCommented:
I really don't think anybody was trying to slam you or argue with you.  Just trying to warn you....and to elicit more background (" how many requests/unit of time to expect, how big the requests are going to be, whether you are maintaining sessions with each user. and what kind of data you are going to store") that would have been needed to actually provide a useful answer.
0
Anthony PerkinsCommented:
I found out the "owners" never actually had any cash. They were waiting for an investor which never did happen.
That is exactly what I feared and tried to warn you about.

Did you put this in your calendar to follow-up so you could gloat if it failed?
Not at all, to be quite honest I had forgotten about it entirely.  I just happened to find an old email exchange I had with Netminder where he was asking for clarification on this thread and that reminded me to check back with you.

I am glad you got out of it OK and can now put it down to experience.  Trust me we have all gone through at least one of those fantastical projects (some of us more than once :) ).  I have yet to see any application that can run with that many users on MS O/S, so i was curious to see if I was wrong.

Best of luck.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
ASP.NET

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.