The importance of a test lab


I'm not sure if this is the right forum, so please excuse me if it is not.

I work as a senior sysadmin for a company serving 500-600 users and computers. I'm in a situation where we are implementing a range of system center and SQL solutions. I'm looking for the best arguments to convince my supervisor about the need for a test environment - specifically for system center.

The testing will be for general config and management changes. Some changes that cannot be tested in a production environment (which is where we test stuff today) e.g. configuration of items that are global in scope.

Counter arguments vary from "What is the point of testing? We should know what to do." to "We have snapshots and backup".

Among other things I'm trying to avoid using days restoring servers and (parts of) databases because some ones testing went wrong.

So i'd like to create a document with convincing arguments from reliable sources to hand to my supervisor.
i486dx266Senior DevOps ConsultantAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Mohammed KhawajaManager - Infrastructure:  Information TechnologyCommented:
Few arguments I would come back with:

1. Keep skills updated:  Use this area for keeping skills updated
2. Minimize downtime:  Test and validate in an isolated environment, document and then apply the lessons learned in a learn world environment
3. Proof of concept:  Sandbox area to try new solutions
4. Snapshots ARE NOT BACKUPS:  Reverting to a snapshot is not a replacement for backups.  Example would be that settings were changed in Windows server running SQL (i.e. Windows patches, registry changes, COM+, etc.) and now the server is now working as expected.  Reverting to a snapshot will also revert all disks (and hence SQL data loss).  You need proper backups where you will restore system state only
5. Cost: Argue with him that cost is negligible.  If you are under Microsoft Select or Enterprise agreement or have a TechNet agreement then you can test all the software you want for free.  With virtualization, VLANs and thin-provisioning, no investment to very little capital investment is required

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Kash2nd Line EngineerCommented:
this is a problem in any typical environment where manager are not willing to fork out money for vital testing etc.

If you are covering over 500 workstations then it becomes more important.

A few very important points can be:

1. Software updates to start off with. If a software patch is released overnight and workstations just install them and there is some problem with update and compatibility among the computers then you are stuffed because it will take weeks and weeks before it can be rectified. with test environment, you can test updates for problems before you roll them out.

2. You need to convince your bosses that setting up a test env isn't an expense its an investment and can become very handy for may be up to 5 years or so. and with virtualisation every where you just want a couple of servers and off you go.

3. ask your boss, what happens if a new version of his favorite phone app comes out and he installs it and tries to access sql server on his phone and it don't work with existing sql server version. well, you install the latest version on your test servers and it works so another point.

hope this helps. others may add
i486dx266Senior DevOps ConsultantAuthor Commented:

Thanks for the replies so far.

What is common is supervisors and colleagues saying things like we did it this way for 10 years so why should we do anything different.

Other arguments against a test environment are "It takes to long time and generates twice as much work". Of cource i would argue that the extra time spendt working with a test environment is time you would save when a disaster strikes.

This never happenes to us is a commonly used statement. ;-)

Agree, snapshots are not backups. They are also useless on a range of servers. We do have regular backup, of cource, but that's not an argument to do testing in a production environment.
top-of-head and without lots of thought:

Loss of revenue.
Production systems support the business - if lost (even temporarily) revenue is at risk, and losses do in fact occur.

Reputational Risk
Consider the headlines when a bank loses it's ATM's - loss of reputation can be mercilessly hard to recover.

(ps: This is an argument that may help win - but is rarely executed.)
The hardware involved can be used to enhance/prop-up production in case of extreme emergency. (aka insurance)

Good Luck - I cannot imagine working in an environment like this today. Mgr possibly spends more on his/her car maintenance than the hardware needed.
Dave BaldwinFixer of ProblemsCommented:
we did it this way for 10 years so why should we do anything different.

Maybe because everything else has changed in that time?  A refusal to learn and change should be a 'fire-able' offense.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
IT Administration

From novice to tech pro — start learning today.