[Webinar] Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 2558
  • Last Modified:

Stange Odor from Server Rack?

Hello,

OK, I could really use some help with a very stinky problem.  I have a 400 square foot office with a 25 square foot closet that houses a server rack.  A few days ago I started to notice a strange odor in the office.  Over the course of the day the odor became so powerful that I needed to leave the office a few times.  Due to chronic allergies, I could not locate the source of the smell.  It was like a cross between burning wires and a dead rat.  

The smell seems to be worse between 7-9 am and 5-9 pm.  A colleague said it smelled like a sewer.  We checked the bathroom and everything is working fine there and the smell is not coming from there.  I thought it was an exhaust fan in my server closet that was burning out, but we eliminated that.

I had Orkin Pest Control inspect the office for dead pests.  They  (thankfully) did not find any dead animals and the technician said to his trained nose the smell was more like burning wires then any dead animal.  

I am 90% sure the smell is, in fact, coming from my rack.  I just can't determine exactly where.  I have several sets of rack fans that I disassembled and checked - no odor.  They were very dusty so I cleaned them out.  I then pulled my PowerEdge R300 server (the only active server) and opened it up.  The hard drive air intakes and all of the in case fans were coated in dust.  I pulled the CPU heat sink and looked at it closely - the thin metal fins(?) we so caked with dust that no matter what angle you used, light was blocked.  

I very carefully opened the server, pulled out the fans and used compressed air to blow out the dust, cleaned out the inside of the server and the hard drive cages.  The server is back online now.  I do not think this was the source of the smell.

What could cause this type of odor?  Is there any tool that would help me locate the source of the smell?

Please let me know if you need any further information.  Thanks!
0
MrChip2
Asked:
MrChip2
  • 5
  • 5
  • 3
  • +4
5 Solutions
 
Ernie BeekCommented:
But the smell is gone now?

Heated dust can smell like that but it could also be that due to all the dust in the machines parts/wires/etc get over heated and start to melt (and stink).
Did you also clean the power supply units in the server?
0
 
RGRodgersCommented:
If it is a sulfur smell, you need to check your UPS.  Your batteries are bad, overheated and likely swelling.  I've had this happen when UPS fans failed.  ...RG
0
 
rindiCommented:
Adequate cooling is essential for servers and other electronic equipment. If it isn't cooled adequately things overheat and will get burnt. As has been mentioned above, excessive dust blocks the airflow and thus prevents your system from getting cooled properly. I suggest you inspect all your electronic devices and PC's and carefully clean out all the dust you can find. If the server was so dusty it'll be very likely that not only it will be full of dust.

Once all the dust is cleaned out you should inspect the system for signs of charred parts, and if you see any have either the part or the electronic device exchanged, as otherwise you are in risk of getting a fire.

Another thing to is to check if the fans are all running smoothly, as dust can clot or even corrode their bearings, and if the don't run smoothly, replace them. Sometimes a drop of oil onto their bearings can also help, but I'd only regard that as a temporary fix before you replace them, as a combination of oil and dust can eventually make things worse.

Also, thoroughly clean off all the thermal transfer paste or pads from your heatsinks and the CPUs/Chips onto which the heatsinks attach to, and after that add a very small drop of fresh thermal transfer paste to the CPU and firmly and properly reattach the heatsinks.

Make sure your Air-conditioning in the server room is fully functional, and in future plan such hardware service days regularly, where you at least check for fresh dust, and clean it out. Based on the experience you get from those tests you can then make out a time frame for those regular cleaning days.
0
 The Evil-ution of Network Security Threats

What are the hacks that forever changed the security industry? To answer that question, we created an exciting new eBook that takes you on a trip through hacking history. It explores the top hacks from the 80s to 2010s, why they mattered, and how the security industry responded.

 
Nagendra Pratap SinghCommented:
Are your servers having any Dell OMSA type tool? That can tell you the component temperatures.
0
 
CallandorCommented:
I think you had a cooling problem and some components heated up beyond what they normally operate at.  Electrical burning smells are usually a sign that a fan isn't cooling something properly, such as a cpu; you should check for blockage or non-working fans.
0
 
MrChip2Author Commented:
Thanks everyone for such great advice.  In terms of the server, about the only part I did not clean was the PSU.  I do not think it is hot swappable.  I will check the user manual to see what they say about checking it.  How do you clean a PSU?  Should I take it out of the case and just blow compressed air through it?  I assume I should not try to open the PSU itself.

I used Dell Open Manage to check on temps and fan speed in the R300.  This is after the server was running for 10 hours.  The two temps both got green check marks.  System Board Ambient is 27C (max threshold is 38C) and System Board Planar is 34C (max threshold is 48 C).

In terms of fans, there are 8 check marks.  The server has four sets of redundant fans.  The first fans in each set are spinning at 6400 rpm (+/- 200 rpm) while the second backup fans in each set are spinning at 4200 rpm (+/- 100 rpm).

I am hoping/praying it is not the UPS - but have a bad feeling this is the case as the smell did seem to be strongest behind the UPS.  I am next going to run some diagnostics on it.  I can tell you it is an APC Smart-UPS 1400XL with three additional battery packs.  All of the batteries were replaced in 2008 - so they are getting up there in age.  I pray its not the UPS because those batteries have historically been super expensive.  Any suggestions on tests I should run on the UPS itself?

One question - could pulling too much current through a PDU cause a smell like this?  Or would the PDUs circuit breaker just trip in a case of overload?

I should have more news in a few hours.
0
 
MrChip2Author Commented:
rindi,

you wrote: Also, thoroughly clean off all the thermal transfer paste or pads from your heatsinks and the CPUs/Chips onto which the heatsinks attach to, and after that add a very small drop of fresh thermal transfer paste to the CPU and firmly and properly reattach the heatsinks.:

When cleaning the heat sink I noticed that the paste residue did not seem to cover the entire surface area.  I have never worked with cooling paste before.  Is this easy and safe for a novice to use?  If yes, can you recommend a particular paste from NewEgg that I should buy?  I was careful when reattaching the heat sink and checked that it was firmly in place.
0
 
Nagendra Pratap SinghCommented:
There are third party battery packs that can be used and cost a fraction of the original.

http://www.atbatt.com/ups-backup-batteries/b/apc/m/smartups-xl-1400va-rm-3u.asp
0
 
pgm554Commented:
Never underestimate the power of a rodent.

Found many a dead varmint using the CPU box as a bathroom/bedroom back in my FE days.
0
 
rindiCommented:
Arctic Silver is a good product:

http://www.newegg.com/Product/Product.aspx?Item=N82E16835100007

It is easy to clean and apply. First just wipe off the old paste with toilet paper or a kitchen cloth, and then use another such cloth you have dipped in cleaning alcohol and wipe off anything that is left. Then apply a small drop of thermal paste to the center, metal part of the CPU and after that firmly re-attach the heatsink. The pressure will disperse the paste so any uneven gaps are filled.
0
 
CallandorCommented:
When you use thermal paste, you don't want to use too much.  Take an old credit card and press down hard and draw across to distribute the paste.  The purpose of the paste is to fill in microscopic air gaps; you want metal-to-metal contact to do most of the heat transfer.
0
 
RGRodgersCommented:
I am betting strong that it is the UPS.  All other efforts are probably wasted until you check that.  Batteries give off a strong and nasty smell.  They only last about 3 years max, too, so yours are overdue.  However, if they have gone this bad, I'd look for other problems too.  It's unlikely they will swell and outgas unless they heat up pretty badly.

Your servers should be connected to the UPS either through a USB cable or, more likely, a network interface.  The servers should have APC software that communicates with the UPS that allows you to check them out, including the battery state.  That software probably told you that the batteries needed replaced in about 2010...

...RG
0
 
MrChip2Author Commented:
Update on UPS -

I am running PowerChute on the server and was able to log in.  According to PCBE, I think the batteries are OK even though they are over 4 years old.  Here are two screen shots from the software.  The UPS runs self tests every two weeks and has passed all of them for a while.  

Should I do a runtime calibration test?
Is there another way to test the batteries as the source of the odor?
UPS-1.jpg
UPS-2.jpg
0
 
RGRodgersCommented:
Interesting for sure.  I guess I could be wrong.  But, even so, I'd pull the batteries and physically inspect them.  If you can.  That can be hard to do if they are swelling.  I still think it is the source. ...RG
0
 
CallandorCommented:
If the batteries were bad, that voltage would be below 24v, which is the rated value for charging.  Internally, they are pairs of 12v batteries in series.  Being above 24v is a good indication that they are ok.
0
 
RGRodgersCommented:
I don't disagree.  However, especially since they are outdated, physically inspecting them is the prudent course.  ...RG
0
 
MrChip2Author Commented:
RGRodgers,

I will inspect the batteries tomorrow.  It shouldn't be too hard to inspect the ones in the add-on packs.  I plan to disconnect the pack from the UPS then pull out the batteries.

To check the ones in the UPS, I assume I will need to shut down the server for safety reasons.  Is this correct?  

If I do need to shut down the server, should I just inspect all batteries with it shutdown?

So far the smell has not returned.  Keeping fingers crossed.
0
 
Ernie BeekCommented:
How do you clean a PSU?  Should I take it out of the case and just blow compressed air through it?  I assume I should not try to open the PSU itself.

If you're not completely sure/confident about it, don't open it. Try using a vacuum cleaner to get the dust out, blow some air through it to loosing remaining dust and vacuum again. You could void warranty (if any) when opening the PSU.
0
 
Ernie BeekCommented:
When checking the batteries I'd suggest, for safety reasons as well, to shut everything down.
0
 
RGRodgersCommented:
OMG, yes!  Even with that, handle the unit and batteries carefully according to the manual.

http://www.apcmedia.com/salestools/ASTE-6Z8LBV_R0_EN.pdf

...RG
0
 
MrChip2Author Commented:
Thank you all for helping with this.  The smell has not returned after all these days so I think the problem was in fact excessive dust in the R300 server caused some components to over heat.  The server has been running fine so I think I cleaned it out in time before there was any damage.  it was very hard to assign points because I learned so much in this thread (about thermal paste and heat sink cleaning, alternate 3rd party batteries, etc.).  Wish I had 1,000 points to distribute. :)
0

Featured Post

Microsoft Certification Exam 74-409

VeeamĀ® is happy to provide the Microsoft community with a study guide prepared by MVP and MCT, Orin Thomas. This guide will take you through each of the exam objectives, helping you to prepare for and pass the examination.

  • 5
  • 5
  • 3
  • +4
Tackle projects and never again get stuck behind a technical roadblock.
Join Now