• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 257
  • Last Modified:

Enterprise class Uninterruptable Power Supply Systems

I have been searching online for some inforamtion about best practices for enterprise UPS power backup. We have over 200 servers, I know that is not alot in comparison, but most of our equipment is quite old and has some very high power requirements.

We currently run a very large (and old) Mitsubishi battery backup system that is supposed to keep the whole datacenter up for up to an hour. However its out of warrenty and we have recently been having some issues with it.

What are some good solutions as far as efficiency, redundancy, and cost? Im not asking for product advice Im just asking for solutions in general. For example would it more more efficient or better to have one smaller UPS per rack, or one larger rack per row, or one huge system that backs up everything?
0
Halonix666
Asked:
Halonix666
  • 4
  • 4
  • 3
  • +4
1 Solution
 
jhanceCommented:
I believe that Liebert is a leader in this type of equipment.

http://www.liebert.com/
0
 
Halonix666Author Commented:
Im not asking for brands... Im asking for practices. Like would it be better to use one big UPS as opposed to several smaller ones, would it be better to  have a UPS in each rack, and so on.I dont want to start calling vendors they are obviously going to have their own biased solutions based on their own products.
0
 
crnzCommented:
Ok, I would employ (and have employed) an approach where you have a battery based UPS for maybe 15-20 minutes only.  This gives you what you need to survive the  small power brown outs and other little disturbances.  But then I'd fire up a diesel generator that can keep things running for hours on end (make sure you have spare diesel on site :).  Also make sure you have a well ventilated exhaust that doesn't get sucked back into the aircon intakes.  Diesel doesn't like sitting still, so you need to run it at least once a month and shake rattle and roll the spare diesel storage; replace totally (if not used) in one year.  And also learn how to safely put diesel into a running generator... ;)  You can get some relatively small units on a trolley that are well suited to sitting in storage until you pull it out.
0
Transaction-level recovery for Oracle database

Veeam Explore for Oracle delivers low RTOs and RPOs with agentless transaction log backup and transaction-level recovery of Oracle databases. You can restore the database to a precise point in time, even to a specific transaction.

 
Halonix666Author Commented:
We have a large diesel generator onsite (outside) that is supposed to kick on and run the entire building. We also have a 3000 gallon tank of diesel fuel on hand. They run the generator once a month for a day just to cycle the fuel through. So as far as that is concerned we are good to go. But it doesnt kick on immediately and would require a battery backup to give it the time to get going. So what is the most power efficient and redundant method for providing batter backup? It seems to me from your statement that you think one large UPS system is ideal. Am I correct in that assumption?

 Is that really the most effective way to do it from a cost benefit standpoint? I have not yet done any sort of analysis on any alternative solutions so I am not really sure yet. Just want to get some opinions. Would it be more of a hassle, for example, to have say one rack mounted 4U UPS on each rack to prevent the huge draw from the primary battery backup? Are there any kind of distributed backup sytesms that intercommunicate and have interfaces with the servers to start shutdown processes and stuff if need be? What do large companies like google do for these kinds of things?
0
 
crnzCommented:
I think it is the best way to go from a system management standpoint.  Each UPS also needs to be plugged into mains and have systems plugged into it.  It's a bigger mess in my opinion.  I favour keeping things simple.  One supply for the lot, one mains into it, one source of power out of it.  You asked for opinions, opinions aren't right or wrong in and of themselves, just something that has formed over time through experience and influence I guess.  

It depends on your operating environment as to what sort of interruptions you get and what you are willing to pay for.  I know of one data centre locally that has 3 redundant battery UPS systems for the whole place (1 fails, 2nd kicks in, 2nd fails, 3rd kicks in) and then has a house sized diesel generator for indefinite power supply.  Power went.  UPS 1 failed, 2 kicked in.  Failed.  3 kicked in. Failed.  Whole data centre dead until someone could run outside and fire up the generator.

The lesson there is that even with tripple redundancy and a permament backup generator, it still failed.  And it wasn't through neglect.  This is a serious data centre that clears banking information, so they do their monthly drills and maintenance religiously.

You decide what sort of downtime you can (or can't) afford, and then pay for the solution.  Distributed UPSs would give you apparently less impact when one UPS failed, but just to keep a rack running isn't enough to keep the service up, it is dependent on other racks and other UPSs also.

I still prefer a single UPS for wiring, maintenance and operational similicity reasons.   That's personal preference.  I can equally accept there is nothing inherently wrong with distributed UPS system except perhaps for the complexity it brings with it.
0
 
Don S.Commented:
From an up front cost perspective, it will generally be less expensive to go the multiple UPS route.  However, I agree, the management of multiple UPSs is taxing in the long run.  One big up side to multiple UPSs is that when one fails, it does not take out everything.  In fact, if you size them so they can run multiple servers and then cross connect your server's redundant power supplys, you shouldn't have any downtime due to UPS or Power supply failure unless multiple one fail at the same time.
0
 
Halonix666Author Commented:
Well we stage our rows based on company with the exception of what we call the defcon row which houses out switching equipment. So one of our clients  has a whole row or a at least a section of a row. We dont share servers between clients for the most part so we can basically group all related servers together physically. I was basically think something along the lines of what dons suggested even though it would be inherently more complex. I didnt really think about cross connecting the power supplies which sounds like a brilliant idea.

I would still like to hear some more suggestions or advice from the rest of you but I think that is something we might try to look into. As although it seems like more of a hassle then one huge system I think the redundancy it provides would be worth it.
0
 
giltjrCommented:
I'm not sure about Google, but most large companies use UPS's that power the data center.  Of course most of these companies also run mainframes and large Unix boxes which at one time needed a lot of batteries.

I have heard of issues of sites that have a UPS powering the data center and then install indvidual/rack UPS.  Some thing about harmonics and feed back damaging some of the control boards on the UPS system.

Its been years since I looked, but I know that APC had a serial interface that you could connect to serial port on a computer and software that you install.  The UPS can tell the software to shutdown the server if it (UPS) starts getting low.  I would assume in this day in age you might find one that has a LAN interface and you could do the same thing but with many servers.

I personally beleive in one large UPS, but you would need to look at the purchase/install costs, on going maintenance costs, est. lifetime of the batteries, and replacement costs.  

We have a Data Center UPS and in the 20 years I have been there I beleive we have had to replace one, maybe two, of the 20+ batteries we have.   The last I knew the batteries in a server/rack UPS had a life span of about 5 years.
0
 
Don S.Commented:
Another thing to consider is growth.  Since an investment in a Datacenter sized UPS is considerable and you would likely intend on having it last at least 10 years, you would need to plan for capacity growth in the UPS and likely buy most of that capacity initially.  Multiple UPSs are inherently flexible growthwise.  

I also have an installation that has a large Datacenter UPS backed up by an automatic start generator.  Its very nice to be able to just plug something in to the wall which is powered by the UPS and not worry about it.  However, if you do this you MUST do regular UPS and generator testing to ensure it will work when needed.

I've been burned badly relying on the single large UPS solution.  I'm also very tired of the ongoing maintenance of the multiple UPS solution.  Maybe there is some middle ground?  Maybe using something like a handfull of medium sized UPS - something mudular like the APC Symetrix?  And cross connecting things for redundancy?
0
 
giltjrCommented:
Our Data Center UPS is upgradable,  all we do is add more batteries.  With smaller UPS you need to plan, if you out grow one, you have to either replace it or get another one.

What we are finding is that some devices we have (mainframes, large scale RISC boxes, and enterprise class SAN's) are including UPS's inside the boxes that once they see input power gone, they to a clean shutdown of all internal functions.  The mainframe and RICS OS's are not shutdown, they crash, but specific internal functions are cleanly shutdown.    This is because the vendors have seen to many issue where even with data center UPS's and generators the data center will loose power.

Due to a mis-connected generator (after we replace the UPS control panel) we have actually had a power outage when the generator kicked on because it blew out the control/tranfer circuts and few circut breakers in our PDU's.  

We also have come NOT to rely on testing of generators by just starting them.  You have to put a load on them.  In fact there was on major hosting data center that had problems with their generators after a major storm and they were without comercial power for about 1 week .  They had done a "full" test the week before the storm.  We found out their full test was to make sure they started.  Luckly we only hosted one small web site there, more of a sales page of our services.  

We run all of our own applications in our own data center, but , we have had our share of issues when actually putting a load on the generator.
0
 
crnzCommented:
I second the testing procedure being REAL.  ie: pull the plug on the mains.  It's a nerve wracking thing to do, but unless you do it, you never know for real if it will work for you.  You may also want to graph the "remaining time left" against actual time over months to see if the batteries are losing quality.  Otherwise you get to the "20 minutes left" and suddenly it has nothing left :)  You will see, over time, it degrades in its ability, and when it reduces to the level you find unacceptable, you get new batteries.  DO NOT RELY ON COMPANY LITERATURE for UPS testing.  The only test is a REAL test, no matter how annoying it is to create your own power cut.  Better than way than the tears when after a few years of "fake" testing you rely on it and it goes down when you're all at a party.  (fate has this way of choosing cool times to annoy you :)
0
 
scrathcyboyCommented:
A diesel generator linked to a power fail-over switch (we are talking a 200 Amp switch here, no toy stuff) is the way almost ALL major companies do it, hospitals, data centers, webhosts, you name it, diesel generators and precision switches running massive relays, are the way to go.  minimum cost is about $100,000.  Now if you want to go down from that, you could go with a battery farm, but batteries are crummy reliability compared to a big generator.
0
 
giltjrCommented:
Scratchyboy: So you are suggesting no batteries?  Just how do you power the equipment for the time it takes to kick on the generator?  I must be missing something, I doubt if we could run 1/3 of our data center on a 200 Amp circut and we do not have that large of a data center.
0
 
Halonix666Author Commented:
Using a Flywheel I would assume...
0
 
crnzCommented:
We used 2000 military trained hamsters.
0
 
pgm554Commented:
Check out APC.
Great products and excellent support.
They scale very well.
So depending upon your needs,I would check them out first.

http://www.apcc.com/products/category.cfm?id=13&subid=50
0
 
pgm554Commented:
M$ uses the APC solutions at their remote campuses server rooms.
I trained at their Silcon Vally campus in Mountain View on APC products.
They can afford anything and they use APC.
0

Featured Post

A Cyber Security RX to Protect Your Organization

Join us on December 13th for a webinar to learn how medical providers can defend against malware with a cyber security "Rx" that supports a healthy technology adoption plan for every healthcare organization.

  • 4
  • 4
  • 3
  • +4
Tackle projects and never again get stuck behind a technical roadblock.
Join Now