Solved

the disk is offline because it is out of capacity ISCSI LUN

Posted on 2015-02-02
26
2,416 Views
Last Modified: 2016-03-28
I have a Synology DS1513+ with 5 x 2TB HDDs. the whole volume (RAID5) is used as a single ISCSI LUN (with thin provisioning is enabled) and used as a backup drive for the server (Server 2012R2) due to some unexpected data transfers it looks like we have maxed out the capacity and need to delete some files. However Disk management has marked the disk as offline as its out of capacity. The problem I have is when I try to enable it again disk management immediately disables it and marks it as offline so I cannot get to the files and free up some space. Any suggestions? Thanks in advance
0
Comment
Question by:Dead_Eyes
  • 8
  • 7
  • 4
  • +4
26 Comments
 
LVL 76

Expert Comment

by:arnold
ID: 40585508
Do not have Synology.

Try going through the iscsi initiator and disconnect/reconnect there or at least make sure the connection is still active.

How was the LUN partitioned as a GPT volume?

Check whether the device, synology is reporting any issues/errors on its end.
Check the vendor site to see whether there is a firmware update that addresses an issue such as yours.

8TB of backups filled up?
0
 
LVL 22

Expert Comment

by:rickhobbs
ID: 40585538
What version of DSM are you using?
0
 
LVL 76

Expert Comment

by:arnold
ID: 40585553
Newer versions include the 2012 UNMAP support dealing with file deletion/reclamation described in the newest release notes:


https://www.synology.com/en-us/releaseNote/DS1513+

Re: rickhobs, ......

https://www.synology.com/en-us/support/download/DS1513%2B

Depending on the current version, you may have to go through several updates.
0
 
LVL 42

Expert Comment

by:Davis McCarn
ID: 40586089
Because it is already offline, it should be safe to use the web interface to delete some files and free up space.  Have you tried that?
0
 

Author Comment

by:Dead_Eyes
ID: 40586207
Hi all sorry for late reply, Its running the latest DSM version and I can't seem to find any way to view the contents of a LUN. File manager seems to only deal in shares ect :(. I am looking at the Synology forums as and when I get 5 but I have not managed to find any useful info so far
0
 
LVL 76

Expert Comment

by:arnold
ID: 40586645
Could you try the disconnect/connect method with the iscsi initiator?  It may clear out stale records and free up files and will bring it online.
0
 

Author Comment

by:Dead_Eyes
ID: 40586734
Hi Arnold, that's for the suggestion but I have already tried that and rebooting the server and rebooting the NAS
0
 
LVL 76

Expert Comment

by:arnold
ID: 40586757
Can you disconnect, and attach the LUN to a different system?
0
 

Author Comment

by:Dead_Eyes
ID: 40586772
Still getting the same issue, seems like I need some form of force online command :(
0
 
LVL 76

Expert Comment

by:arnold
ID: 40586848
diskpart
list disk
select disk <number of 8TB>
online
see if that helps.
0
 
LVL 42

Expert Comment

by:Davis McCarn
ID: 40586851
Geeze! The users manual says you can use FTP or FileStation to access it and you should be able to delete files with either: http://www.use-ip.co.uk/datasheets/synologyds1513+userguide.pdf
0
 

Author Comment

by:Dead_Eyes
ID: 40586947
Hi Davis, I cannot find a reference in the PDF you linked. What page where you looking at?
0
 
LVL 42

Expert Comment

by:Davis McCarn
ID: 40586969
Page 50 and 53
0
How to Backup Ubuntu to Amazon S3

CloudBerry Backup offers automatic cloud backup and restoration for Linux. It has both GUI and command line interface (CLI) ensuring its flexibility in use. Find out more

 

Author Comment

by:Dead_Eyes
ID: 40586991
Pages 50 and 53 cover accessing files on a share from FTP but nothing about accessing an iSCSI LUN or connecting to one
0
 
LVL 2

Accepted Solution

by:
Jim_Nim earned 500 total points
ID: 40586994
If you're using thin-provisioning on the storage array and ran out of disk space to accommodate growth of new space allocation on the virtual disk, the array likely set the volume into a read-only state (as there is no space to accommodate writes). I don't think the volume is in an "offline" state on the storage side if you still have a connection to the disk device and it's showing up in disk management - the disk is likely just in a read-only state.

Here's the problem: You can't online the disk in disk management (which auto-mounts the file system) if it's in a read-only state. Windows requires the ability to write to the volume in order to mount it normally (there is a small amount of metadata that is immediately written when mounting).

You may be able to mount the volume in a read-only state within windows (e.g. disable auto-mount with mountvol.exe /N then set the read-only flag with diskpart), but that still doesn't allow you the write access you need to delete files (or "rethin" with SCSI UNMAP, which latest software/firmware versions appear to support).

Since the storage array isn't going to have any "visibility" into the file system layer, your only option here is likely going to be increasing the block-level storage that the enclosure has available to allocate for this volume, even if that means moving data off of another volume so that it can be deleted. If you have absolutely no remaining space on the array outside this volume (i.e. 100% usable capacity allocated to this single volume), I'd suggest looking into one of these options:
Add disks to any available drive slots and expand the RAID (if the system supports this action)
Look into what this enclosure supports in terms of live-upgrading hard drives to larger capacity models.
Consider adding an expansion enclosure - this system supports expanding out to 60TB max with expansion enclosures
Look into whether or not this enclosure supports RAID type conversion. If it does, you may be able to convert from whatever RAID level you've chosen to one that provides more storage capacity (e.g. RAID6 -> RAID5, RAID5 -> RAID0). This will affect the system's redundancy though, so this may only work as a temporary solution until you can free up space at a file system level

If/when you're able to get the volume back online by some means, I would strongly recommend that you start off by "rethinning" the volume using the Windows "Defragment & Optimize Drives" tool. This runs once per week by default already though, so it may not make a big difference. Next, start searching for files to delete to free up space. If they're small enough, Windows won't send SCSI UNMAP commands on the fly to reflect all the changes on the block level & storage array side, so you may need to re-run another optimization.

Note: I don't have any experience with this particular enclosure/system - my comments draw from a few years' experience with Enterprise-class storage systems that also deal with thin-provisioning support.

Good luck!
0
 
LVL 42

Expert Comment

by:kevinhsieh
ID: 40587745
This is only tangentially related, but RAID 5 is not safe with hard drives larger than 1 TB. If a drive were to fail you have a non-trivial risk of having a second drive fail or encountering an unrecoverable read error (URE).

I don't consider Synology anywhere close to enterprise class.
0
 

Author Comment

by:Dead_Eyes
ID: 40587805
Hi Kevin, thanks for comment but I don't see the relevancy.
1.) "not safe" is hardly a technically precise term. May I suggest not fit for the purpose of X would have been a more relevant way to start that comment. (by the way the purpose of the NAS is as one of two backups)
2.) The only data I can find to support that claim is woefully outdated and based on error rates staying the same as they where 10, 15 years ago while volume size increases
3.) I have 100 plus systems in our company / client base running RAID 5 setups and have never had data corruption or URE's reported by engineers or clients in 6+ years so I cannot relate that to a real life POV
4.) Considering the importance of the data, backup routine and track record of data loss  (0% in 6+ years). I would consider RAID 5 perfectly adequate for our needs and higher RAID levels not justified for most of our clients needs. (those that do have need of greater redundancy use RAID 6 and 6 +hotspare)
5.) While RAID 5, 6, 10, 50, 60 ect have their intended uses different products by different company also have their place. While you might not consider Synologys products to be relevant to your company's needs many people do and I don't think your use of the term "enterprise class" is in any way a POV many people would share.
0
 

Author Comment

by:Dead_Eyes
ID: 40587810
@ Jim_Nim thanks for a great explanation Jim I am just exploring a few options today but I suspect your 100% right and will award you the points in the next 12 hours or so I just need to try a couple of last ditch things but am taking it a lesson learned: Never setup a LUN with 100% of your available capacity
0
 
LVL 76

Expert Comment

by:arnold
ID: 40587815
Any possibility to access the LUN from a non windows system to see whether it will remain active

Look at testsisk to see whether it could access the partition while the disk reflected os offline.

Not sure whether diskpart can be used to check the status of the LUN.
0
 
LVL 42

Expert Comment

by:Davis McCarn
ID: 40588500
Sine the iSCSI LUN is offline, there is no point in attempting to connect to it using iSCSI.  FTP directly into the Synology and see if it will let you delete files!
0
 
LVL 76

Expert Comment

by:arnold
ID: 40588591
I think the following , over allocation, might explain your situation.
http://forum.synology.com/wiki/index.php/Best_practices_of_using_the_Synology_DiskStation_to_host_iSCSI_Storage

Linux can mount partitions read-only and this might let you get the data off.

Have you reached out to Synology to see what their options are?
I believe it was suggested to attach an additional external drive/resource if possible to allow it to be used for the excess data so the LUN can "expand" and return to a mode where it once again will be seen as online on the windows system.

...
0
 
LVL 2

Expert Comment

by:Jim_Nim
ID: 40588888
@Dead_Eyes - In regards to the use of RAID5, "unsafe" really isn't too far off from the truth... though more technically accurate terminology would be that its use for storing business-critical data is not recommended & discouraged according to industry standard best practices. The fact of the matter is, RAID5 has never been "safe" - it's RAID. No RAID should be solely entrusted with the only existing copy of any critical data, it should always be backed up elsewhere.

It sounds like you've already weighed the risks and benefits though, and you have a backup strategy in place, so you've not by any means made a wrong decision. Typically the people who get hit hard by RAID5 problems are those who are both unaware of the potential risks and unprepared for them.
0
 

Author Closing Comment

by:Dead_Eyes
ID: 40589541
Good explanation and although I decided to trash the LUN as the data was non-critical and replaced weekly I hope someone can make use of this to fix or plan and avoid this situation.
1
 
LVL 42

Expert Comment

by:kevinhsieh
ID: 40589575
My comment about Synology not being enterprise class is that I wouldn't assume that techniques available on tier 1 enterprise arrays are available to Synology.

Current industry practice is to not use RAID 5 for primary storage when drives are 1 TB or larger. URE rates haven't really changed over the years, but drives have gotten larger, making the probability of a URE taking down the entire array non-trivial. If you can afford to lose the entire array, such as in this case, then it is less of a concern.
0
 

Expert Comment

by:linkeast
ID: 41525651
i have found the exact issue i have elsewhere on this site... think they blew it away though..will look now..

http://www.experts-exchange.com/questions/28935767/Iscsi-target-on-a-synology-Nas.html#a41524528

i have attached a doc with the requested screenshos...  i will need to find a away to get the disk online even tough it does not have enough free space... once i get it online i will be able to free space immediately..
synology.docx
0
 

Expert Comment

by:linkeast
ID: 41525671
sorry..!  posed in wrong question
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

This article is an update and follow-up of my previous article:   Storage 101: common concepts in the IT enterprise storage This time, I expand on more frequently used storage concepts.
The recent Microsoft changes on update philosophy for Windows pre-10 and their impact on existing WSUS implementations.
This video teaches viewers how to encrypt an external drive that requires a password to read and edit the drive. All tasks are done in Disk Utility. Plug in the external drive you wish to encrypt: Make sure all previous data on the drive has been …
This tutorial will walk an individual through the process of installing the necessary services and then configuring a Windows Server 2012 system as an iSCSI target. To install the necessary roles, go to Server Manager, and select Add Roles and Featu…

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now