Urgent - What to Do About Synology NAS Disk Failure
Hello,
I have a roughly one year old Synology DS412+ NAS that started sounding an alarm. The NAS is configured with 4 hard drives, all 2TB Western Digital Red NAS drives. The drives are configured in RAID 6 with data protection.
I store music on one of the volumes and noticed that when I went to play song lists starting a couple of days ago Windows Media Player would sometimes crash. A few minutes ago the NAS started sounding an alarm. Apparently Disk #4 has failed. When I log in I see the message in my first image attached.
I also saw that my notifications log exploded in size with many I/O error messages. See this screenshot:
Here are my questions.
1. What should I do now? I need to see if I have a spare drive. I kind of remember buying an extra - but need to check.
Assume I do not have one on hand. Should I shut the NAS down? If so, what is the best way to shut it down?
Assume I do have a spare drive, how do I replace / rebuild disk 4?
Hearing this constant alarm is a bit unnerving. I also have all my home videos and photos on the NAS and pray I will not lose them! Thanks!
Update: As far as I can tell, I do NOT have a spare drive. My bad. So I think I need to shut the unit down, but will wait to hear advice.
I have a roughly one year old Synology DS412+ NAS that started sounding an alarm. The NAS is configured with 4 hard drives, all 2TB Western Digital Red NAS drives. The drives are configured in RAID 6 with data protection.
I store music on one of the volumes and noticed that when I went to play song lists starting a couple of days ago Windows Media Player would sometimes crash. A few minutes ago the NAS started sounding an alarm. Apparently Disk #4 has failed. When I log in I see the message in my first image attached.
I also saw that my notifications log exploded in size with many I/O error messages. See this screenshot:
Here are my questions.
1. What should I do now? I need to see if I have a spare drive. I kind of remember buying an extra - but need to check.
Assume I do not have one on hand. Should I shut the NAS down? If so, what is the best way to shut it down?
Assume I do have a spare drive, how do I replace / rebuild disk 4?
Hearing this constant alarm is a bit unnerving. I also have all my home videos and photos on the NAS and pray I will not lose them! Thanks!
Update: As far as I can tell, I do NOT have a spare drive. My bad. So I think I need to shut the unit down, but will wait to hear advice.
As long as you keep on making your regular backups, you should be fine.
You should just be able to swap the disk while the NAS is running. The new disk should then start syncing automatically. If it doesn't, check the NAS' utility and manual on how to add the new disk to the array so it starts syncing. With many RAID systems, if it doesn't sync automatically, you have to assign the new disk as a hot-spare.
Be aware that the sync can take very long, and during that time performance will be lower. Also, don't shut the system down while it is syncing.
You should just be able to swap the disk while the NAS is running. The new disk should then start syncing automatically. If it doesn't, check the NAS' utility and manual on how to add the new disk to the array so it starts syncing. With many RAID systems, if it doesn't sync automatically, you have to assign the new disk as a hot-spare.
Be aware that the sync can take very long, and during that time performance will be lower. Also, don't shut the system down while it is syncing.
No need to shut down the NAS. Simply replace it the failed one with similar disk and set it online. After that the drive will be added to the RAID.
ASKER
Hi rindi,
I have a 3TB HDD connected to the NAS via eSATA. The NAS includes Backup software, but I have been negligent about really setting it up. It almost looks like the auto backups have filled the external drive, but I can't be sure.
Tomorrow when I am awake I will disconnect the external drive and plug it into a desktop and explore its contents. I think I have a spare 2TB or 3TB HDD lying around. If I find one, I will format it and plug it into the NAS and run a full backup.
I have a 3TB HDD connected to the NAS via eSATA. The NAS includes Backup software, but I have been negligent about really setting it up. It almost looks like the auto backups have filled the external drive, but I can't be sure.
Tomorrow when I am awake I will disconnect the external drive and plug it into a desktop and explore its contents. I think I have a spare 2TB or 3TB HDD lying around. If I find one, I will format it and plug it into the NAS and run a full backup.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Hi rindi,
You are absolutely correct - I have been lax with backing up this NAS. Do you have any advice on the best way to back it up?
One idea is to get buy two external HDDs like this one from Seagate. It is 4TB, the size of my RAID volumes and USB 3.0. I would then store one offsite and leave one connected to the NAS and rotate the two say once a month.
http://www.newegg.com/Product/Product.aspx?Item=N82E16822178354
In a review someone said this worked perfectly with their DS412+. Some reviews talk about the drives failing at a high rate. They say this is plug and play for Windows, but as you know the NAS is not running Windows. I looked but could not find if this is a 5400rpm or 7200 rpm drive. Not sure if that matters for backup, but I will be backing up 3TB of data and don't want a super slow solution. My guess is this must be a 5400 rpm drive given the price.
Or, would I be better off buying a 4 TB WD Green drive (it's variable speed is supposed to give a fairly high throughput) and putting the drive in am enclosure. The bare drive costs $149 and I already have a couple of USB 3.0 enclosures (and an eSATA enclosure currently connected to the NAS). I am leaning towards the bare drive solution and let the NAS format the drive however it likes. Then I can always check the contents by booting my PC into Ubuntu and connecting the drive to it.
You are absolutely correct - I have been lax with backing up this NAS. Do you have any advice on the best way to back it up?
One idea is to get buy two external HDDs like this one from Seagate. It is 4TB, the size of my RAID volumes and USB 3.0. I would then store one offsite and leave one connected to the NAS and rotate the two say once a month.
http://www.newegg.com/Product/Product.aspx?Item=N82E16822178354
In a review someone said this worked perfectly with their DS412+. Some reviews talk about the drives failing at a high rate. They say this is plug and play for Windows, but as you know the NAS is not running Windows. I looked but could not find if this is a 5400rpm or 7200 rpm drive. Not sure if that matters for backup, but I will be backing up 3TB of data and don't want a super slow solution. My guess is this must be a 5400 rpm drive given the price.
Or, would I be better off buying a 4 TB WD Green drive (it's variable speed is supposed to give a fairly high throughput) and putting the drive in am enclosure. The bare drive costs $149 and I already have a couple of USB 3.0 enclosures (and an eSATA enclosure currently connected to the NAS). I am leaning towards the bare drive solution and let the NAS format the drive however it likes. Then I can always check the contents by booting my PC into Ubuntu and connecting the drive to it.
For backing up the speed shouldn't matter. Besides, as far as I know the default backup tool of Synology is based on Rsync, and so after the first full backup which will take a long time, the next ones should be short. But don't take my word on that, as I don't have a Synology and could be mistaken. It is also possible that you can install more backup modules.
ASKER
Update: Today I received four new hard drives. Western Digital delivered a 2 TB Red drive to replace the RMA. I also ordered a new identical drive to be used as a spare if this ever happens again. I also ordered two 4TB Western Digital Green drives that I plan on using in a rotation scheme to backup the NAS.
Slight hiccup this afternoon. My PC has two X-docks in it. I put one of the Green Drives in it to start the backup. Well it gave me the choice of formatting it with a MBR or a GPT. I figured I would go with GPT so I could have a single 4TB volume. After the drive was formatted I needed to leave so I shutdown the computer. Next time I tried to power up the PC would not boot - it asked for me to insert a bootable disk. I rebooted into BIOS and for some reason it placed the new drive ahead of my C drive in the boot order. I switched the order and still no success. Only when I disabled the second choice and forced the PC to boot from C did it work.
My plan was to use RoboCopy to copy the files from the NAS to one of these 4TB drives BEFORE I swap out the failed drive and have the NAS rebuild the volume. The problem? I did a small test of copying 8 large video files totaling roughly 55GB to the drive. As a point of reference, my NAS has about 2.5TB of data on it. Well the job started 3 hours ago and it is still not done. It is averaging 9 GB an hour. I don't know if the network is a bottleneck - everything is running 100BaseT if not 1000BaseT. At this rate it would take 277 hours to backup the NAS.
There must be a better way. Now I am thinking about putting the other green drive in a USB3.0 enclosure, connecting it directly to the NAS, and running the local backup. I don't like that the files will not be readable on a Windows PC, but maybe it will go a little faster so I can swap out the bad drive. I am told the NAS may need a day to rebuild the volume. Any suggestions?
Slight hiccup this afternoon. My PC has two X-docks in it. I put one of the Green Drives in it to start the backup. Well it gave me the choice of formatting it with a MBR or a GPT. I figured I would go with GPT so I could have a single 4TB volume. After the drive was formatted I needed to leave so I shutdown the computer. Next time I tried to power up the PC would not boot - it asked for me to insert a bootable disk. I rebooted into BIOS and for some reason it placed the new drive ahead of my C drive in the boot order. I switched the order and still no success. Only when I disabled the second choice and forced the PC to boot from C did it work.
My plan was to use RoboCopy to copy the files from the NAS to one of these 4TB drives BEFORE I swap out the failed drive and have the NAS rebuild the volume. The problem? I did a small test of copying 8 large video files totaling roughly 55GB to the drive. As a point of reference, my NAS has about 2.5TB of data on it. Well the job started 3 hours ago and it is still not done. It is averaging 9 GB an hour. I don't know if the network is a bottleneck - everything is running 100BaseT if not 1000BaseT. At this rate it would take 277 hours to backup the NAS.
There must be a better way. Now I am thinking about putting the other green drive in a USB3.0 enclosure, connecting it directly to the NAS, and running the local backup. I don't like that the files will not be readable on a Windows PC, but maybe it will go a little faster so I can swap out the bad drive. I am told the NAS may need a day to rebuild the volume. Any suggestions?
ASKER
OK. I read that the theoretical USB 3.0 transfer rate maxes out at 5 Gigabit/sec or 600 MB per second. Is this correct? 0.6 GB per second is 36 GB per minute or 2,160 GB per hour or 2.2TB per hour. I am sure nothing goes that fast that involves spinning disks. But that would be way faster. I believe the WD Greens max out at 150MB/sec. So maybe my idea would work in something like 5-8 hours as opposed to 277 hours?
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
When booting the PC, just make sure you don't have the external disk connected. Some chipsets and BIOS's have problems when there are external disks attached.
ASKER
Update
Thank you all for your help and patience. I think I see the light at the end of the tunnel :)
So after much research I ran over to the local Frys to buy a USB 3.0 enclosure. I was going to use it to backup my NAS before attempting the repair. Long story short, the design of the wiring/connectors on this enclosure was horrible. I have purchased 3-4 different ones over the years but never had such a difficult time installing a HDD. Once installed, the drive refused to install and be recognized. I ended up taking the HDD out of my eSATA enclosure, replacing it with one of my new 4TB WD Green drives and backing up the NAS. The whole backup process took about 10 hours. It would have been faster but I split it into four parts. Once the NAS was backed up, I replaced the failed drive with the new one purchased from NewEgg. I then used DSM to repair the volume (the Wizard was extremely easy to use). The LED for drive 4 is now solid green and the NAS is now repairing itself. No idea how many hours this will take. I have turned off all my scheduled/automated jobs that use the NAS so traffic/use should be minimal. I will keep the question open until I the repair is completed.
NAS-repairing.jpg
Thank you all for your help and patience. I think I see the light at the end of the tunnel :)
So after much research I ran over to the local Frys to buy a USB 3.0 enclosure. I was going to use it to backup my NAS before attempting the repair. Long story short, the design of the wiring/connectors on this enclosure was horrible. I have purchased 3-4 different ones over the years but never had such a difficult time installing a HDD. Once installed, the drive refused to install and be recognized. I ended up taking the HDD out of my eSATA enclosure, replacing it with one of my new 4TB WD Green drives and backing up the NAS. The whole backup process took about 10 hours. It would have been faster but I split it into four parts. Once the NAS was backed up, I replaced the failed drive with the new one purchased from NewEgg. I then used DSM to repair the volume (the Wizard was extremely easy to use). The LED for drive 4 is now solid green and the NAS is now repairing itself. No idea how many hours this will take. I have turned off all my scheduled/automated jobs that use the NAS so traffic/use should be minimal. I will keep the question open until I the repair is completed.
NAS-repairing.jpg
ASKER
Great news. It ended up taking about 20 hours, but the new drive is fully integrated into the RAID array and the parity check passed! I am now scheduling daily quick SMART tests on all four drives and will run extended tests on each one to be sure all is OK.
I also have a much better BACKUP plan for the NAS that I will start to implement.
I also have a much better BACKUP plan for the NAS that I will start to implement.
ASKER
1) I figured out how to turn off the audible alarm in the software. That is a relief! Listening to that sound non-stop for nearly an hour will get on your nerves.
2) I registered my drives at WD and requested an RMA for the failed drive. I opted for the pre-pay RMA so they will ship the drive before I need to return the bad one. Hope it ships Monday.
3) I also ordered another identical drive from NewEgg to be me onsite spare. That drive will arrive on Tuesday.
Bottom line, I won't have a replacement drive for about 48 hours. According to Synology my RAID configuration means 4 TB is for storage and 4 TB is for data protection. For now I am leaving the NAS on. Is there any risk in this? My NAS includes hot swappable drives.
For my notes, I checked and Drive Bay #4 is the furthest on the right, closest to the indicator lights.