Stange Odor from Server Rack?
Hello,
OK, I could really use some help with a very stinky problem. I have a 400 square foot office with a 25 square foot closet that houses a server rack. A few days ago I started to notice a strange odor in the office. Over the course of the day the odor became so powerful that I needed to leave the office a few times. Due to chronic allergies, I could not locate the source of the smell. It was like a cross between burning wires and a dead rat.
The smell seems to be worse between 7-9 am and 5-9 pm. A colleague said it smelled like a sewer. We checked the bathroom and everything is working fine there and the smell is not coming from there. I thought it was an exhaust fan in my server closet that was burning out, but we eliminated that.
I had Orkin Pest Control inspect the office for dead pests. They (thankfully) did not find any dead animals and the technician said to his trained nose the smell was more like burning wires then any dead animal.
I am 90% sure the smell is, in fact, coming from my rack. I just can't determine exactly where. I have several sets of rack fans that I disassembled and checked - no odor. They were very dusty so I cleaned them out. I then pulled my PowerEdge R300 server (the only active server) and opened it up. The hard drive air intakes and all of the in case fans were coated in dust. I pulled the CPU heat sink and looked at it closely - the thin metal fins(?) we so caked with dust that no matter what angle you used, light was blocked.
I very carefully opened the server, pulled out the fans and used compressed air to blow out the dust, cleaned out the inside of the server and the hard drive cages. The server is back online now. I do not think this was the source of the smell.
What could cause this type of odor? Is there any tool that would help me locate the source of the smell?
Please let me know if you need any further information. Thanks!
OK, I could really use some help with a very stinky problem. I have a 400 square foot office with a 25 square foot closet that houses a server rack. A few days ago I started to notice a strange odor in the office. Over the course of the day the odor became so powerful that I needed to leave the office a few times. Due to chronic allergies, I could not locate the source of the smell. It was like a cross between burning wires and a dead rat.
The smell seems to be worse between 7-9 am and 5-9 pm. A colleague said it smelled like a sewer. We checked the bathroom and everything is working fine there and the smell is not coming from there. I thought it was an exhaust fan in my server closet that was burning out, but we eliminated that.
I had Orkin Pest Control inspect the office for dead pests. They (thankfully) did not find any dead animals and the technician said to his trained nose the smell was more like burning wires then any dead animal.
I am 90% sure the smell is, in fact, coming from my rack. I just can't determine exactly where. I have several sets of rack fans that I disassembled and checked - no odor. They were very dusty so I cleaned them out. I then pulled my PowerEdge R300 server (the only active server) and opened it up. The hard drive air intakes and all of the in case fans were coated in dust. I pulled the CPU heat sink and looked at it closely - the thin metal fins(?) we so caked with dust that no matter what angle you used, light was blocked.
I very carefully opened the server, pulled out the fans and used compressed air to blow out the dust, cleaned out the inside of the server and the hard drive cages. The server is back online now. I do not think this was the source of the smell.
What could cause this type of odor? Is there any tool that would help me locate the source of the smell?
Please let me know if you need any further information. Thanks!
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Nagendra Pratap Singh
Are your servers having any Dell OMSA type tool? That can tell you the component temperatures.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Thanks everyone for such great advice. In terms of the server, about the only part I did not clean was the PSU. I do not think it is hot swappable. I will check the user manual to see what they say about checking it. How do you clean a PSU? Should I take it out of the case and just blow compressed air through it? I assume I should not try to open the PSU itself.
I used Dell Open Manage to check on temps and fan speed in the R300. This is after the server was running for 10 hours. The two temps both got green check marks. System Board Ambient is 27C (max threshold is 38C) and System Board Planar is 34C (max threshold is 48 C).
In terms of fans, there are 8 check marks. The server has four sets of redundant fans. The first fans in each set are spinning at 6400 rpm (+/- 200 rpm) while the second backup fans in each set are spinning at 4200 rpm (+/- 100 rpm).
I am hoping/praying it is not the UPS - but have a bad feeling this is the case as the smell did seem to be strongest behind the UPS. I am next going to run some diagnostics on it. I can tell you it is an APC Smart-UPS 1400XL with three additional battery packs. All of the batteries were replaced in 2008 - so they are getting up there in age. I pray its not the UPS because those batteries have historically been super expensive. Any suggestions on tests I should run on the UPS itself?
One question - could pulling too much current through a PDU cause a smell like this? Or would the PDUs circuit breaker just trip in a case of overload?
I should have more news in a few hours.
I used Dell Open Manage to check on temps and fan speed in the R300. This is after the server was running for 10 hours. The two temps both got green check marks. System Board Ambient is 27C (max threshold is 38C) and System Board Planar is 34C (max threshold is 48 C).
In terms of fans, there are 8 check marks. The server has four sets of redundant fans. The first fans in each set are spinning at 6400 rpm (+/- 200 rpm) while the second backup fans in each set are spinning at 4200 rpm (+/- 100 rpm).
I am hoping/praying it is not the UPS - but have a bad feeling this is the case as the smell did seem to be strongest behind the UPS. I am next going to run some diagnostics on it. I can tell you it is an APC Smart-UPS 1400XL with three additional battery packs. All of the batteries were replaced in 2008 - so they are getting up there in age. I pray its not the UPS because those batteries have historically been super expensive. Any suggestions on tests I should run on the UPS itself?
One question - could pulling too much current through a PDU cause a smell like this? Or would the PDUs circuit breaker just trip in a case of overload?
I should have more news in a few hours.
ASKER
rindi,
you wrote: Also, thoroughly clean off all the thermal transfer paste or pads from your heatsinks and the CPUs/Chips onto which the heatsinks attach to, and after that add a very small drop of fresh thermal transfer paste to the CPU and firmly and properly reattach the heatsinks.:
When cleaning the heat sink I noticed that the paste residue did not seem to cover the entire surface area. I have never worked with cooling paste before. Is this easy and safe for a novice to use? If yes, can you recommend a particular paste from NewEgg that I should buy? I was careful when reattaching the heat sink and checked that it was firmly in place.
you wrote: Also, thoroughly clean off all the thermal transfer paste or pads from your heatsinks and the CPUs/Chips onto which the heatsinks attach to, and after that add a very small drop of fresh thermal transfer paste to the CPU and firmly and properly reattach the heatsinks.:
When cleaning the heat sink I noticed that the paste residue did not seem to cover the entire surface area. I have never worked with cooling paste before. Is this easy and safe for a novice to use? If yes, can you recommend a particular paste from NewEgg that I should buy? I was careful when reattaching the heat sink and checked that it was firmly in place.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Never underestimate the power of a rodent.
Found many a dead varmint using the CPU box as a bathroom/bedroom back in my FE days.
Found many a dead varmint using the CPU box as a bathroom/bedroom back in my FE days.
Arctic Silver is a good product:
http://www.newegg.com/Product/Product.aspx?Item=N82E16835100007
It is easy to clean and apply. First just wipe off the old paste with toilet paper or a kitchen cloth, and then use another such cloth you have dipped in cleaning alcohol and wipe off anything that is left. Then apply a small drop of thermal paste to the center, metal part of the CPU and after that firmly re-attach the heatsink. The pressure will disperse the paste so any uneven gaps are filled.
http://www.newegg.com/Product/Product.aspx?Item=N82E16835100007
It is easy to clean and apply. First just wipe off the old paste with toilet paper or a kitchen cloth, and then use another such cloth you have dipped in cleaning alcohol and wipe off anything that is left. Then apply a small drop of thermal paste to the center, metal part of the CPU and after that firmly re-attach the heatsink. The pressure will disperse the paste so any uneven gaps are filled.
When you use thermal paste, you don't want to use too much. Take an old credit card and press down hard and draw across to distribute the paste. The purpose of the paste is to fill in microscopic air gaps; you want metal-to-metal contact to do most of the heat transfer.
I am betting strong that it is the UPS. All other efforts are probably wasted until you check that. Batteries give off a strong and nasty smell. They only last about 3 years max, too, so yours are overdue. However, if they have gone this bad, I'd look for other problems too. It's unlikely they will swell and outgas unless they heat up pretty badly.
Your servers should be connected to the UPS either through a USB cable or, more likely, a network interface. The servers should have APC software that communicates with the UPS that allows you to check them out, including the battery state. That software probably told you that the batteries needed replaced in about 2010...
...RG
Your servers should be connected to the UPS either through a USB cable or, more likely, a network interface. The servers should have APC software that communicates with the UPS that allows you to check them out, including the battery state. That software probably told you that the batteries needed replaced in about 2010...
...RG
ASKER
Update on UPS -
I am running PowerChute on the server and was able to log in. According to PCBE, I think the batteries are OK even though they are over 4 years old. Here are two screen shots from the software. The UPS runs self tests every two weeks and has passed all of them for a while.
Should I do a runtime calibration test?
Is there another way to test the batteries as the source of the odor?
UPS-1.jpg
UPS-2.jpg
I am running PowerChute on the server and was able to log in. According to PCBE, I think the batteries are OK even though they are over 4 years old. Here are two screen shots from the software. The UPS runs self tests every two weeks and has passed all of them for a while.
Should I do a runtime calibration test?
Is there another way to test the batteries as the source of the odor?
UPS-1.jpg
UPS-2.jpg
Interesting for sure. I guess I could be wrong. But, even so, I'd pull the batteries and physically inspect them. If you can. That can be hard to do if they are swelling. I still think it is the source. ...RG
If the batteries were bad, that voltage would be below 24v, which is the rated value for charging. Internally, they are pairs of 12v batteries in series. Being above 24v is a good indication that they are ok.
I don't disagree. However, especially since they are outdated, physically inspecting them is the prudent course. ...RG
ASKER
RGRodgers,
I will inspect the batteries tomorrow. It shouldn't be too hard to inspect the ones in the add-on packs. I plan to disconnect the pack from the UPS then pull out the batteries.
To check the ones in the UPS, I assume I will need to shut down the server for safety reasons. Is this correct?
If I do need to shut down the server, should I just inspect all batteries with it shutdown?
So far the smell has not returned. Keeping fingers crossed.
I will inspect the batteries tomorrow. It shouldn't be too hard to inspect the ones in the add-on packs. I plan to disconnect the pack from the UPS then pull out the batteries.
To check the ones in the UPS, I assume I will need to shut down the server for safety reasons. Is this correct?
If I do need to shut down the server, should I just inspect all batteries with it shutdown?
So far the smell has not returned. Keeping fingers crossed.
How do you clean a PSU? Should I take it out of the case and just blow compressed air through it? I assume I should not try to open the PSU itself.
If you're not completely sure/confident about it, don't open it. Try using a vacuum cleaner to get the dust out, blow some air through it to loosing remaining dust and vacuum again. You could void warranty (if any) when opening the PSU.
If you're not completely sure/confident about it, don't open it. Try using a vacuum cleaner to get the dust out, blow some air through it to loosing remaining dust and vacuum again. You could void warranty (if any) when opening the PSU.
When checking the batteries I'd suggest, for safety reasons as well, to shut everything down.
OMG, yes! Even with that, handle the unit and batteries carefully according to the manual.
http://www.apcmedia.com/salestools/ASTE-6Z8LBV_R0_EN.pdf
...RG
http://www.apcmedia.com/salestools/ASTE-6Z8LBV_R0_EN.pdf
...RG
ASKER
Thank you all for helping with this. The smell has not returned after all these days so I think the problem was in fact excessive dust in the R300 server caused some components to over heat. The server has been running fine so I think I cleaned it out in time before there was any damage. it was very hard to assign points because I learned so much in this thread (about thermal paste and heat sink cleaning, alternate 3rd party batteries, etc.). Wish I had 1,000 points to distribute. :)