Hardware and Staffing Requirements for Growth

As the owner of a subscription based service provider marketing website, what types of things do I need to be mindful of or what type of person would I need on staff to manage the projected growth?

My rough projections:
6 months = 800 - 4,000 paying service providers & 5,000 - 20,000 clients,
1 year = 1,500 -  10,000 paying service providers & 25,000 - 100,000 of clients,
2 years = 10's of 1,000 of paying service providers & 100's of 1,000's of clients,
5 years = 80,000 paying service providers & millions of clients
After one year, or having met projections of even half the first six months low end projections, I will plan to upgrade.
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Thomas RushCommented:

As owner, do you have any current customers?   This should give you some idea of what you'll need as your customer base ramps up.  Not knowing anything about the load that one customer puts on your systems makes it pretty difficult to predict the load that 100 or 1,000 will put on your system.

What kind of person will you need?  Kind, dedicated, willing to learn, responsive, customer-focused, hard-working, intelligent, detail-oriented.

What skills will they need?  I guess that depends a lot on things you've not told us.  What OS are you running?  On what kind of hardware?  Which applications are you hosting?
DavidSenior Oracle Database AdministratorCommented:
Perhaps begin with what you expect this all-hands person to accomplish each day.  This is too broad to rattle off definitive answers -- and frankly, you will need professional advice rather than our volunteer comments.
Scott Fell, EE MVEDeveloper & EE ModeratorCommented:
@SelfGovern & @dvz there is a background to this question here http://www.experts-exchange.com/Web_Development/Miscellaneous/Q_28269303.html

The real issue is not so much how many service providers or clients you have, but how many users of the website in an average minute.  Data on 80,000 service providers and and 100,000 clients is probably going to be in the 100mb to 500mb area as far as the size of the database and that will vary depending on what other types of data and transactions you have.  

If you only have a 1 person searching in a given minute vs 100 vs 1000 is what is going to dictate how you set up your system.  

The choices today are growing.   You don't have to use traditional hosting.  Since the lines are getting muddy between native apps and websites, you are starting to see 3rd party services that manage cloud hosted data.  These services typically have a small window for free services but then a big jump to the paid which can run $1000 per month or more.  They are meant for scale.   There are also traditional hosted services that will have a private cloud where you can quickly adjust resources within an hours notice.  As example http://www.liquidweb.com/StormServers/dedicated.html.  Let's say Sunday through Wednesday you know traffic is slow. But then Wednesday night through Saturday night traffic increases.  You can flip a switch and provision more capacity.   The advantage with this type of service is you have more direct control over your data.  

There is also many services like http://www.windowsazure.com/en-us/ or amazon where your database is in the cloud.

If it were me, I would create a webservice for all my data functions and that way I could use html/js for my site and also create a native app that all uses the same data source.

As traffic gets heavy you can consider having your service provider search data on multiple servers across the country.  This will allow more people to access faster. This is easier to do when the searched data is not updated very often or does not have to be live.  In your case, I don't think this is going to be an issue.

I would start with a small dedicated hosting plan though.   These things do take a while to grow.  If you start off accessing your data via a webservice/end points, it should be easier to grow because the only thing you need to update is your endpoints as it will be your database what will be the weakest link.

Do you plan on hiring somebody to manage all of this and then hire somebody else or a development firm to create the site?

Keep in mind that we are either left or right brain heavy.   A developer/coder is typically not going to be the person you want to create a custom design although that person could probably at least manipulate a given template.  And a very creative right brainer that you might hire to create a logo, look and feel is not going to be the type of person you want to code or be a project manager.
Powerful Yet Easy-to-Use Network Monitoring

Identify excessive bandwidth utilization or unexpected application traffic with SolarWinds Bandwidth Analyzer Pack.

DavidSenior Oracle Database AdministratorCommented:
Padas, nice :)
craig1632Author Commented:
Padas! Awesome!

Again I am so impressed by the talent here. And to have dvz come along doubting and be impressed. I am blessed.

To answer your questions, I may have done this backwards. I have the website under development right now. I connected with a team in India who is now on day 140 of a 60 day project. (25% is my fault for adding features). My frustration is going to lead to another question I will post in a moment. But more relevant now, your question leaves me feeling like I have approached this project with the cart before the horse. And your comment about the database being the weakest link scares me. I am not even sure what to ask about that. I have limited db experience, but at little more than development experience.  You already have earned the points for this question. But if you can elaborate here, I will be indebted. Thanks.
craig1632Author Commented:
On the "original thread" padas had stated: it is not as much about the total users, but the amount of usage during the average minute and your biggest bottleneck is going to be your database.  (See U.S. Healthcare website)
   Do you also then mean the weight of the traffic like video that plays on the home page? each user will be on the website for only five to ten minutes a day on average. It is not content rich so I do not expect lengthy view times. Profiles however will have pictures which will impact upload usage.
   Thanks again.
Scott Fell, EE MVEDeveloper & EE ModeratorCommented:
Using a database takes up more resources than a static site.  Since there are multiple ways to get to the same end, some of those ways can be memory or cpu hogs. Even when done to the most optimized way, calls to the database are going to use the most resources as far as memory/cpu and at any given moment, the amount of resources you have to offer are fixed.    

Let's say every call to a static page takes up .1% of resources for 1 second and every call to a page that uses the database takes 1% of available resources for 3 seconds.  That means in the 3 second time span, there can only be 100 calls to the db before something happens like it freezes up.   Compare to 1000 calls to a static page.

The actual amount of resources your site/application actually takes will depend on image optimization, coding, db query optimization, cpu and memory available and even the type of hard drive.

I'm sure you have heard of a DDoS attack where the attacker will try and use up as many resources available until the thing breaks.   http://en.wikipedia.org/wiki/Denial-of-service_attack  There is not much differnce between a DDoS attack and just too many users at the same time.

Going back to an example where 1% of resources are used for 3 seconds.  That is 100 calls at the same 3 second period of which there are 20 of those in a minute meaning 2000 calls to the db in a minute and 120,000 constant db calls in an hour.  As you see things are getting maxed out, you find ways to add more resources and probably find more ways to optimize where possible.
Scott Fell, EE MVEDeveloper & EE ModeratorCommented:
>Do you also then mean the weight of the traffic like video that plays on the home page?

Video can take up resources and it is best to have your video hosted on a CDN http://en.wikipedia.org/wiki/Content_delivery_network or even better youtube since they have the video thing pretty much figured out.  

One of the features of a content delivery network is you upload the video once and the CDN sends the file out to multiple servers in their network that are probably global.  This is what Youtube or Vimeo already do.  If you are in Seattle WA, the CDN detects your location and finds a server closest to you vs the person that is in Miami FL is accessing the video closest to them.  Otherwise if your service has one server and lets say it is in AZ, the person in Miami has to access a video that is 2,500 miles away where the Seattle person is only 1,500 miles away and will probably have better experience with the video.  

According to wikipedia, the Akamai network has 250,000 servers in 80 countries assuring your video has a good chance of being served within a short distance from any user http://en.wikipedia.org/wiki/Akamai_Technologies.

>each user will be on the website for only five to ten minutes a day on average.  It is not content rich

You can sit and stare at a web page for 10 minutes, but it only loaded once.  And perhaps took 1 to 3 seconds of resources.  In that 10 minute time you may view 10 pages or maybe one page but searched 20 times. What matters is each time the page loads or the database is accessed.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
craig1632Author Commented:
awesome padas! Thanks again! I am officially overwhelmed
craig1632Author Commented:
very clear answers. Thank you!
Scott Fell, EE MVEDeveloper & EE ModeratorCommented:
Thank you!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Server Hardware

From novice to tech pro — start learning today.