Corrupt size cluster shared volume (CSV) on Hyper-V 2012 R2

Posted on 2014-08-18
Last Modified: 2014-11-12
Unable to use extra disk space on my cluster shared volume (CSV) after extending.

EMC storage: VNXe3150
6 physical servers configured in a Hyper-V 2012 R2 cluster
connected to the storage through iSCSI
CSV is formatted with REFS

Events leading up to the issue:
CSV initially was 1 TB
I was copying data to one of the VMs (using robocopy)
more data then I expected
I had assigned more space to the VM then was actually available
During the robocopy process, suddenly all VMs on the cluster grinded to a halt

My response:
* Increased LUN size on EMC from 1024GB to 1950GB
* Sorted out which hyper-v host was in control of the CSV
* RDPed to that particular hyper-v host
* Opened an elevated CMD
* started 'diskpart':
screen shot diskpartAs you can see, I tried to 'extend' the volume. Executing the extend command returned an error but also reported a successfully extended volume.

The best way to explain the current problem is by a screen shot of windows disk manager:
screen shot disk managerJudging from the information in the top panel, the capacity of the volume is still the old size of 1023.87 GB
But the bottom panel shows a disk and volume size of 1996.67 GB
Most importantly the newly claimed disk space is assigned from the storage but unavailable to servers on the cluster!
Fortunately the data on CSV (Hyper-V disk images) is still accessible. So no data has been lost (yet).

Tried (but failed) so far:
Chkdsk /scan (finds nothing wrong)
Chkdsk /fix
Repeating the diskpart process (reports not enough space available to extend)
shrink volume (unrecognized FS error, threatened to remove all data when proceeding)
Re-attaching in iSCSI initiator
Re-scan in disk manager

If I'm unable to 'fix' the problem within the next few hours I'll be forced to take these (time consuming) steps:
* backup data from CSV
* Destroy current volume
* Re-create larger volume
* Restore data
* resume migration

Circumstances / considerations:
* Not enough space on local media at the customer to backup the CSV data
* Customer is a high school of with 2000 students (end-users)
* Their holiday ends and school starts within next 4 days (no pressure)

Does anybody have an alternative suggestion to my plan B?

Thank you in advance!
Question by:Joost Kuin
    LVL 34

    Expert Comment

    I think you should turn off all VMs placed on CSV

    Then remove volume from csv

    Then check from disk manager and command prompt if its showing proper size

    Reboot all Hyper-V servers one by one

    Readd the volume as CSV in Hyper-V cluster and check.

    This should not imapct your data stored on CSV, you might want to take CSV volume backup 1st
    LVL 1

    Author Comment

    by:Joost Kuin
    Hi Mahesh,

    Thanx for replying! Your suggestion sounds very do-able and at worse pretty harmless to the data.
    I'm currently backing up the data to a NAS that my colleague dropped on-site. It is estimated to finish in about 3 hours from now. I'll be sure to give your suggestion a go before I waste more time restoring the data to a new volume.

    PS: can you elaborate on why you think that this might work? Have you had a similar experience?

    Kind regards,
    LVL 1

    Accepted Solution

    I tried everything you instructed but the difference between the two size values would not be corrected.

    With the pressure of getting this project finished on time, Idecided to go with plan B (starting over), which is currently in progress.

    I wish that I would have had a bit more time to figure out the issue. ...
    LVL 1

    Author Closing Comment

    by:Joost Kuin
    Check my last comment.
    I appreciated Mahesh's input but unfortunately his suggestions have not contributed to the solution.

    I feel the need for an additional grading option. Just to indicate, that in the interest of time, I have decided to not await further advice and canceled the rescue process.

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    Looking for New Ways to Advertise?

    Engage with tech pros in our community with native advertising, as a Vendor Expert, and more.

    Lets start to have a small explanation what is VAAI(vStorage API for Array Integration ) and what are the benefits using it. VAAI is an API framework in VMware that enable some Storage tasks. It first presented in ESXi 4.1, but only after 5.x sup…
    The article will include the best Data Recovery Tools along with their Features, Capabilities, and their Download Links. Hope you’ll enjoy it and will choose the one as required by you.
    This tutorial will walk an individual through the process of installing of Data Protection Manager on a server running Windows Server 2012 R2, including the prerequisites. Microsoft .Net 3.5 is required. To install this feature, go to Server Manager…
    This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…

    779 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    14 Experts available now in Live!

    Get 1:1 Help Now