• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 463
  • Last Modified:

The importance of a test lab


I'm not sure if this is the right forum, so please excuse me if it is not.

I work as a senior sysadmin for a company serving 500-600 users and computers. I'm in a situation where we are implementing a range of system center and SQL solutions. I'm looking for the best arguments to convince my supervisor about the need for a test environment - specifically for system center.

The testing will be for general config and management changes. Some changes that cannot be tested in a production environment (which is where we test stuff today) e.g. configuration of items that are global in scope.

Counter arguments vary from "What is the point of testing? We should know what to do." to "We have snapshots and backup".

Among other things I'm trying to avoid using days restoring servers and (parts of) databases because some ones testing went wrong.

So i'd like to create a document with convincing arguments from reliable sources to hand to my supervisor.
3 Solutions
Mohammed KhawajaManager - Infrastructure:  Information TechnologyCommented:
Few arguments I would come back with:

1. Keep skills updated:  Use this area for keeping skills updated
2. Minimize downtime:  Test and validate in an isolated environment, document and then apply the lessons learned in a learn world environment
3. Proof of concept:  Sandbox area to try new solutions
4. Snapshots ARE NOT BACKUPS:  Reverting to a snapshot is not a replacement for backups.  Example would be that settings were changed in Windows server running SQL (i.e. Windows patches, registry changes, COM+, etc.) and now the server is now working as expected.  Reverting to a snapshot will also revert all disks (and hence SQL data loss).  You need proper backups where you will restore system state only
5. Cost: Argue with him that cost is negligible.  If you are under Microsoft Select or Enterprise agreement or have a TechNet agreement then you can test all the software you want for free.  With virtualization, VLANs and thin-provisioning, no investment to very little capital investment is required
Kash2nd Line EngineerCommented:
this is a problem in any typical environment where manager are not willing to fork out money for vital testing etc.

If you are covering over 500 workstations then it becomes more important.

A few very important points can be:

1. Software updates to start off with. If a software patch is released overnight and workstations just install them and there is some problem with update and compatibility among the computers then you are stuffed because it will take weeks and weeks before it can be rectified. with test environment, you can test updates for problems before you roll them out.

2. You need to convince your bosses that setting up a test env isn't an expense its an investment and can become very handy for may be up to 5 years or so. and with virtualisation every where you just want a couple of servers and off you go.

3. ask your boss, what happens if a new version of his favorite phone app comes out and he installs it and tries to access sql server on his phone and it don't work with existing sql server version. well, you install the latest version on your test servers and it works so another point.

hope this helps. others may add
i486dx266Senior IT Driftskonsulent/Senior System AdministratorAuthor Commented:

Thanks for the replies so far.

What is common is supervisors and colleagues saying things like we did it this way for 10 years so why should we do anything different.

Other arguments against a test environment are "It takes to long time and generates twice as much work". Of cource i would argue that the extra time spendt working with a test environment is time you would save when a disaster strikes.

This never happenes to us is a commonly used statement. ;-)

Agree, snapshots are not backups. They are also useless on a range of servers. We do have regular backup, of cource, but that's not an argument to do testing in a production environment.
top-of-head and without lots of thought:

Loss of revenue.
Production systems support the business - if lost (even temporarily) revenue is at risk, and losses do in fact occur.

Reputational Risk
Consider the headlines when a bank loses it's ATM's - loss of reputation can be mercilessly hard to recover.

(ps: This is an argument that may help win - but is rarely executed.)
The hardware involved can be used to enhance/prop-up production in case of extreme emergency. (aka insurance)

Good Luck - I cannot imagine working in an environment like this today. Mgr possibly spends more on his/her car maintenance than the hardware needed.
Dave BaldwinFixer of ProblemsCommented:
we did it this way for 10 years so why should we do anything different.

Maybe because everything else has changed in that time?  A refusal to learn and change should be a 'fire-able' offense.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

WEBINAR: GDPR Implemented - Tips & Lessons Learned

Join the WatchGuard team on Thursday, March 29th as we recount some valuable lessons learned in weighing the needs of a business against the new regulatory environment, look ahead at the two months left before implementation, and help you understand the steps you can take today!

Tackle projects and never again get stuck behind a technical roadblock.
Join Now