AS400 battery issue

I think the engineer is right, certainly he would know more than an eBay seller. I found this for another model and IBM's policy has always to be able to service live wherever possible on their bigger boxes.

Models 830/SB2 with FC 9074 - Battery (T01 through T04) For use by authorized service providers. Use this procedure to remove or replace the battery (T01 through T04) in the Model 830/SB2 with FC 9074. Attention: Removing the battery power unit while the system is running on battery power will cause the system to fail and may damage the battery power unit and the card enclosure. To remove or replace the battery (T01 through T04): 1. If the system is powered on, you do not have to power off the system to perform this procedure. 2. Remove the rear cover. See Models 830/SB2 with FC 9074 - Covers. 3. Refer to Figure 4. Models 830 and SB2 with FC 9074 - Final assembly, rear (continued) (See page 406) and perform the following steps: a. Remove the EMC access plate that covers the battery packs. b. Remove the mounting screw from the battery pack that you are replacing. c. Remove the battery pack. CAUTION:The weight of this part or unit is between 18 and 32 kilograms (39.7 and 70.5 pounds). It takes two persons to safely lift this part or unit. (RSFTC204) CAUTION:The battery is a lead-acid battery. To avoid possible explosion, do not burn. Exchange only with the IBM-approved part. Recycle or discard the battery as instructed by local regulations.

From PDF, I can't post link because how my browser is configured but Google for "Hardware (Remove and Replace; Part Locations and Listings)"

EDIT: google for "Battery (T01 through T04)"

Or google for  "batteries (T01, T02, T03, and T04)" - why can't they keep the same format in each manual!

Well IBM makes robust products that can work for ever but if you have a window to shutdown the system and perform the change without being on power its always better.

IBM service procedure requires an IPL to disable CPM:

Look at procedure BATRY, which requires SPCN-PIP5 to disable CPM.

This box is way, way past safe production usable life.  As I'm sure you know, you really need to upgrade.

http://ps-2.kev009.com/as400/manuals/Model__6xx_7xx/600_620_720_S10_S20_Problem_Analysis_Repair_Parts.pdf

A few notes:  

Swollen UPS batteries indicate that the charging circuit is running too hot.  You should replace the UPS base unit as well (UPS base unit — part 44H7784), and at the same time to protect the new batteries and avoid another IPL.

Make sure you neutralize and clean up any spilled battery acid.  Use chemical-resistant gloves and protective eyewear - this stuff is nasty.

If it was mine I would have just taken the in-chassis UPS out completely, no doubt there are some commands you have to run to tell it not to monitor the integrated UPS any more. The batteries are just as good for other UPS so you can use them in APC etc.

Not sure about "Swollen UPS batteries indicate that the charging circuit is running too hot...." but shorted cells in a battery may indeed cause the UPS to get hotter than normal.

Hot in this context means operating at higher voltage than spec.  Makes the batteries overheat and swell.

I sympathize with the need to keep the system running when it's old and cranky, being myself in love with PDP-11s -- but in this case you may be dealing with leaked acid in addition to the battery issue itself.

As stated above, wipe down the battery enclosure with a solution of baking soda in water (2 tbsp/quart, the concentration isn't critical) and look for fizzing.  If there is fizzing, there was an acid leak and you must (a) clean that up and (b) deal with the possibility that some of the leak got out and is on its way to corrode something critical.

There is now also a secondary problem:  gasified acid has been floating around through that machine, blown around by the fans and is now slowly eating away at everything it landed on.  The issue of replace the AS400 or go to a virtual machine has become more pressing.

SLA ("sealed" lead-acid) cells are still wet cells and in my own experience should be periodically checked for electrolyte level even though this is supposedly not necessary.  In this particular situation where there are failing fans then the air movement is impaired and as stated above, lead-acid cells run over nominal temperature fail quickly due to electrolyte evaporation.

>>  I'm reluctant to down the unit because it failed to restart one time because the old fans locked up.  <<  time to repair or replace them then

Gasified acid doesn't get blown through the rest of the machine; it goes out of the back of the integrated UPS, through the room, through the HVAC through a bit more of the room and only then into the front of of the rest of the machine. It's no different than an external UPS in that respect, would you really take all the servers in a rack apart and mop them all down with a solution of baking soda because a UPS in the bottom of the rack smelt of vinegar or hydrogen sulphide?

As far as virtualizing it you can but you have to buy another AS400 or a newer Power series server to do that, there's no emulator to run the code on x86 hardware.

I support a number of customers with these old IBM AS/400 / iSeries boxes.  My company also does upgrades, migrations, replacements, cloud migrations, and application modernization on IBM AS/400, iSeries, and IBM i..  

This is an IBM AS/400e 720 running OS/400 V5R2 - system, operating system, and application software (importantly) is close to 30 years old.  Yes, these systems can run just about forever, with proper maintenance and occasional disk, fan, tape drive, and battery replacement.  But this box went EOSL (End Of Service Life) with IBM on 09/30/2019, which means aftermarket support is the only option.

Shops running systems this old are often "stuck" on an old release due to inability to up upgrade software past a certain system OS version.  V5R2 cannot run virtualized on modern Power hardware (though it can run virtualized on old Power hardware).

Most application software that runs on V5R2 will run on V5R3 and V5R4, but V6R1 and later versions (current version is V7R4) require that applications be compiled "observable" to support automatic migration without source code, or recompilation is necessary from source code.  It is also possible (likely) that the software producer is no longer around on such an old application, so it may not be possible to obtain license keys to run under a new OS version or on a different hardware serial number or virtualization frame, since many AS/400 application use licenses that are keyed to the system serial number.

On the battery acid side:  I'm not a fan of drizzling liquid around inside the frame, and especially if you are trying to do a hot swap.  If you have acid to neutralize, I suggest using a commercial absorbent for battery acid, and then carefully spot treating with a commercial or homemade paste (3 parts baking soda, 1 part water). Unlikely you have a widespread contamination problem in the frame - just under and around the batteries, and possibly damaged ventilation fans in the path from the batteries out of the frame.  While you're in there, replace any damaged fans.

Disconnecting and attempting to remove the internal UPS yourself may cause problem: there may be components or configurable features that depend on presence of the internal UPS.  Even if these dependencies can be dealt with, if removal isn't done properly, the system may fail to start.  Best to leave this to an experienced tech.  This machine is out of support, and if you break something, there is no "calling IBM" to get service or a part.  

Best approach is to backup thoroughly and regularly; maintain it properly, keep it clean and swap out UPS batteries, cache batteries, disk units, fans, and tape drive at recommended intervals; and get off this old hardware as fast as you can - even if you can just upgrade a model or two right now - buy yourself time while you work on a long-term solution: new software, new platform, port of existing software (or reverse engineering), new development, acquisition of source code for porting to a new OS version, etc.

If you're going to run on a system this old, we generally suggest picking up a similar system on the aftermarket (they are usually very inexpensive), and keeping it on hand as a restore target in the event of a catastrophic failure of the primary system.  It is also a good idea to test your backup/restore process to verify that your applications function properly on the backup system. - we can discuss this option more if you like.

- Gary

Nobody answered my question.

I changed the batteries per the advice from the retired IBM hardware engineer.  pulled one tray of batteries out at a time, replace the batteries and put it back in.  the box stayed up with no problems.  

So the answer was YES, you can pull the internal batteries and replace one set at a time without the AS400 failing.

I found the following statement from IBM to be rather amusing, "Attention: Removing the battery power unit while the system is running on battery power will cause the system to fail"  (sort of like telling someone to not try to change the flat tire while they are still driving.)

BATRY - not applicable, no CPMs to clear
SLA - sealed batteries electrolyte level, not applicable to these sealed batteries.

I did use chemical resistant gloves, the batteries were leaking, they were hard to get out of the enclosure due to the swelling.
But other than that everything went fine.

I want to thank all of you for your efforts to help.

Wiley

The BATRY procedure described in the 720 service manual is completely self-guiding, and answers the question very specifically (it isn't a straight "yes" or "no" in all cases).  So I respectfully disagree that no answer was provided.  I not only provided the answer, I provided a direct reference to the manufacturer's service procedure, for my protection, for yours, and for anyone in the future who might read this and follow the advice here.  

My main point is that these service procedures are generally well-documented in the service guide, and you should follow it when performing service procedures, as long as an appropriate procedure exists, as opposed to following verbal or textual advice from anyone, including me.

Imagine this conversation:

Boss:  "You broke the system!"

System admin:  "I followed the manufacturer's service procedure.  Here is the reference."
   -vs-
System admin:  "Some guy on the internet told me it was safe to do that."

According to IBM, if you plan to replace the UPS batteries, have if you have CPM enabled, you must disable CPM, and IPL before replacing batteries to prevent possible system damage.  If you don't have CPM enabled, then neither a shutdown not an IPL is required.   You just proceed with the BATRY procedure and replace the batteries like any oher FRU.  The service manual will tell you when a power-down or IPL is required.  That's why I like to refer users to the service manual, instead of providing an overly-simplified 'yes' or "no" that just might not be accurate in a particular case.

When in doubt, just follow the service manual guidance.

Anyway, glad you got it done successfully.  Fingers crossed this old box holds together for you.

I provided the same reference several hours before you.

FYI - I have and previously read the same service manual that you gave me a link to...  

"do I have to power off the AS400 to change the internal batteries?" & "Anybody change these before know the answer?"

So, no.  These questions weren't answered.  I was given information that I already considered and had discarded the same as the other information that was not applicable.  So yes, I found my own solution.

"a retired AS400 hardware engineer told me that the internal batteries can be changed one unit at a time without downing the unit because it has an external UPS."  

This IBM hardware engineer is the man that installed our original AS400 back in '92, and I've worked with him for 24 years before he retired about six months ago.  I'm the only 'old' customer that he still takes calls from.  So I decided to go with his answer.

So your hypothetical conversation "Some guy on the internet told me it was safe to do that."  That guy won't be him.

Wiley

You misinterpreted my comment - "that guy" would be me, or the others posting here that you don't know (or your ebay seller, for that matter), not your retired hardware guy.

I misinterpreted one of your comments too, should have guessed that you meant "over-voltage" when you put "hot".

Never much liked AS/400s, having to sleep in the same bedroom as a one was bad enough (a friend was an RPG programmer) but then when I was a field engineer I kept having to do 3 hour trips to fix some AS/400e which were only running Windows terminal server.

Lol - at least the room was warm in the winter!.  They don't even make that old Intel Server add-in card for AS/400 / iSerie / IBM i any more.  They were a pain.