Hot Swappable?

I walked in our server room on Thursday afternoon to find one of our APS UPS’ had a battery fault.  They have been in two years, so it was about time to be honest.  So I pop on the APS website to find the RBC (replacement battery cartridge) number, and ordered two (so I had a spare).  They arrived late yesterday (Monday) and so this morning I am ready to replace them.

I would normally power down the attached equipment, shut off the UPS and swap the cartridge, but then why should I?  My UPS claims to have hot swappable batteries. Leave everything attached and running, undo four screws, disconnect the battery cable and pull out the cartridge.  Reverse for fitting the new one.  Fantastic.

However, with mission critical devices I prefer to do my research.  I came across this blog from a consultation and support business Trueblade Systems Inc, who have been through the same issue.  They write that on completion of a battery hot swap, the UPS does not clear the battery failure light.  To turn this light off, you must power-cycle the device!

In my environment, I have built in as much resilience as we could afford.  This UPS only powers one of my three pooled XenServers (although it is the master in this case). So I can migrate my VMs to the other two hosts (because I have also designed it so we can run on just two hosts), power down the physical server, replace the battery on or offline and boot the server back up, without anybody knowing.

However, I wanted to test the theory of Trueblade Systems Inc. So I completed the battery change with a hot swap.  They seemed to be correct, the battery error light does not clear when you change the battery, even though whilst no battery is connected the state of the light changes to flashing, and there is an error bleep.

It was at this point that I was just about to power down my server, and switch off the UPS.  But something stopped me…

I hardly ever read bits of paper any more.  If it’s not documented on the internet, and brought up with a google search within the first couple of results pages, it’s not worth knowing.  However, for some reason, I had a look at the three bits of paper which came with the RBC.  One about physically swapping the batteries, one about disposing of the old one correctly, and the final one, an  addendum.

Point III. Clearing the Replace Battery LED

A. In order for the replace bettery LED to be cleared, it is necessary to run a valid self test. For a self test to be considered valid, the following conditons must be satisfied:

• battery capacity must be greater than 75% (minimum of four battery bar graph LEDs illuminated)
• Smart-UPS must have been charging for at least 8 hours from the last low battery condition.

B. Ways to initiate a self test

• Start up of the UPS
• Smart-UPS front panel test button
• Powerchute PLUS software
• SmartSlot Accessory
• Automatic timer- every two weeks from turn on

I’ve made the important bits bold.

So I gave it a try and it works.  Press the power on / self test button for about three seconds and the self test clears the light after about 5 seconds.

So don’t believe everything you read on the web, and read the bits and pieces that come with your equipment.  It might just save you some time!

http://youtu.be/_ENh7n80NcU

What do you think about hot swapping?  Is it a last resort for you?