FC Shared Storage Issue (Windows Server 2012 R2)

I am building a new hyper-v cluster with Windows Server 2012 R2 to replace my 2008 R2 cluster.

I have the following:

- 2 x Dell Servers
- QLogic SANbox 5800 FC switch
- Winsys FC storage array (will be shared cluster volume)

I have connected each Windows node and storage array to the switch and configured those ports/WWN's with their own zone on the FC switch. On each node, I can see the shared storage in Disk Management and was able to bring the storage online for both and initialize. However, the problem I am having is that each node isn't seeing the changes that the other is making. If I create a folder on one node, I don't see the folder show up on the other node and vice versa. I had this same problem years ago with the 2008 R2 cluster at first but sadly didn't document how I fixed it (shame on me). I am fairly certain it is a switch problem but I can't recall for the life of me what it was, plus I don't see anything on the switch that I could be missing.

Anyone else ever run into this problem with FC? I would appreciate any advice anyone might have.
BSG IT TeamIT ManagerAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Cliff GaliherCommented:
I can't help but notice you said your storage "will be" a cluster shared volume (CSV.) If it *isn't currently* a CSV then changes made by one node will not show up on the other. NTFS is not a shared file system so windows us not expecting or looking fr changes it didn't make. That means the nodes *not* making changes won't try to remember fresh and changes don't show up (and, can in fact he overwritten and data can get corrupted.) A CSV is *required* for shared storage in a cluster to work properly.
BSG IT TeamIT ManagerAuthor Commented:
So, in my 2008 R2 Cluster, there's the Cluster folder that gets created (ClstorageStorage for 2012) when you create a cluster.

I am able to create folders, copy data, delete, etc.. within the 2008 Volume1/Cluster folder on any of the nodes. When I copy something into that folder, it immediately shows up on all the other nodes in that same location. If I try make anything at the top level (same level at the Volume1 folder). I get an "Access Denied error. Not sure how that Volume1 folder got there to begin with.

On the 2012 clusterstorage folder, there is no Volume1 folder and I cannot create anything at the top level without getting an access denied error.

What am I missing here? I don't remember this much drama during the making of my 2008 cluster.
Cliff GaliherCommented:
You created a CSV in 2008. And yes, a folder is created under clusterstorage for each CSV you create. 5 CSVs = 5 subfolders. This has not changed between 2008 and 2012 and as I said above, you need to create CSVs to share storage. This is consistent with your experiences as described.
Problems using Powershell and Active Directory?

Managing Active Directory does not always have to be complicated.  If you are spending more time trying instead of doing, then it's time to look at something else. For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why

BSG IT TeamIT ManagerAuthor Commented:
So.. I recreated the cluster but told the wizard to NOT automatically add disks.

When it does this, it not only adds it but auto configures the quorum settings as well.  The cluster was setting my storage as a disk witness (will have to do some reading as to what that means.. I am new with Server 2012 R2). This was somehow preventing me from setting up a proper CSV.

When I added the storage manually, I was able to configure it EXACTLY as I did on my 2008 R2 cluster.

I followed this article on setting up the CSV in 2012 R2 and now everything works great:


In short, letting the cluster make all the decisions was throwing me off :o)

Thanks for the responses.
Cliff GaliherCommented:
You definitely want a quorum disk. And ss long as you have one (which you'd make sure you have in the planning phase before starting a deployment), letting the cluster make decisions for you is fine. And, in fact, it made a decision you *really truly * want, which is to have a witness. What you have may work more as you expected, but is actually less stable and less fault tolerant because you manually overrode the default and don't have a witness.
BSG IT TeamIT ManagerAuthor Commented:
When the defaults were selected, I wasn't able to do ANYTHING with the share storage. I couldn't create vms, copy existing/exported vm folders, etc... without getting an access denied error. Again, I will need to do some reading on the specifics of the new 2012 Quorum configurations and understand them before making any further changes,
Cliff GaliherCommented:
There isn't anything particularly new in quorum configurations. The most significant change is that now it is recommended to always configure a witness and the clustering service will decide whether to use it based on the total number of nodes. Basically an ease of use so you don't need to configure a witness or deconfigure it when adding/evicting nodes. Much more intelligent.

And yes, I understand the problem you ran into. But the point is, had two disks been made available when you initially configured clustering, the wizard would have asked you which disk to use as a witness. It defaults to always configuring a witness (again, a VERY good thing and great default!), so if only one disk is available, it'll use that.  Which, in your case, was the disk you wanted to use for the CSV.  

All back to my the first sentence of my last post...had the witness disk been made available as it should have been during the planning phase, you could have stuck to the wizard and its recommendations without hitting the problem you did. You hit the problem because you didn't plan for a witness disk *and* stuck to the defaults (a bad combination.)  So you really had two choices to resolve the problem. Plan for a witness disk *and* stick to the defaults; problem solved because your intended CSV would not have been selected for witness and you'd have had access to it.  Or forcibly ignore the defaults and not create a witness. Which I've also indicated reduces fault-tolerance significantly.

In short, what you did worked. But it isn't recommended and doesn't work *well.*  There is a fundamental difference there.  I'm just trying to give advice that gets you to a *good* configuration, not just one that sorta kinda works...until it doesn't.
BSG IT TeamIT ManagerAuthor Commented:
Why can I not add the disk as a disk witness after I've configured it as a CSV?

Why can I also not configure a disk witness to be a CSV?
Cliff GaliherCommented:
That is absolutely by design and is fundamental to what a disk witness does (and hasn't changed much at all in 2012 from 2008.)  The CSV holds a full copy of the cluster configuration *and* acts as a tie-breaker in case your cluster experiences a failure that causes an even split.

Imagine a four-node cluster.  Two nodes are plugged into switch A.  Two nodes are plugged into switch B.  Switch A and Switch B are connected to a core switch, called switch C.

Now if switch C were to fail, you'd have two nodes that can still talk to each other on switch A. And two more that can still talk to each other on Switch B. But the two halves cannot talk to each other in totality because of the failure on Switch C.

You have two ways to handle such a situation in a default cluster. Have the cluster go down, or have one half of the cluster stay up. But which half should stay up?  The usual logic is the half with more than half the nodes (great if you have an odd number of nodes or of the failure leads to an uneven split.  But in this case the failure leaves both halves with two, which means both halves will *want* to stay up.  And thus the "witness" disk. It witnesses the failure and, through some interesting logic not worth going into at the moment, only one half will "win" and stay up. The other half will voluntarily shut down.

Which brings me back to the backup of the cluster config.  The half that shut down *is* recorded in the database, and that copy is on the witness disk. Which means even upon reboot, each node knows the status, and when the switch is replaced. the nodes can check the witness disk to make sure they have the most up-to-date copy.  That way any changes made to the cluster while the one half was down are also still known (nodes added, changed, removed, etc.)

It is a *very* robust design. But if it were a CSV, the whole thing would break down because the disk, being available to all nodes at all times, could inadvertently have changes overwritten or could not act as a tie-breaker (one node most "own" the disk for that to work.)  

So the disk *needs* to be separate from your CSVs.  And it has been this way since CSVs were introduced in 2008 R2.  Technically actually earlier, but since CSVs didn't exist in 2003 or 2008, this particular restriction didn't matter.
BSG IT TeamIT ManagerAuthor Commented:
So, you are proposing I sacrifice over 11 TB of storage to simply act as a disk witness? Sounds like an awful waste to me.

Then where does one put the vm storage that supposed to go in the CSV? In a completely separate storage array? Trouble is, I don't have another fiber channel disk to use, plus I don't have the $$$ to buy another one.

Sounds to me (for my particular situation) like the better answer would be to create some kind of virtual disk to use as the witness disk (maybe 1-5 GB in size) and keep the 11 TB storage as CSV space.

You say things haven't changed much and keep harping that, yet in 2008 I did not run into this problem when configuring my quorum there. The wizard in 2008 didn't require me to sacrifice my entire storage device as a disk witness. It is a 3 node cluster and has worked perfectly for 4 years.

I get what you're saying with "planning" which is what I always do in fact, but also understand I don't have infinite budget. I have to work with what I've got, which is very little and anything I obtain is usually used scraps from a different IT department. Hell, the last time my department was allowed to buy a new server was 2011.
Cliff GaliherCommented:
Every FC array I've seen in the last 20 years allows you to create multiple logical disks and assign a LUN to each. Heck, even low end RAID controllers allow this.  You can easily carve out a small logical disk just for witness. It has its own LUN so the OS will see it as separate storage. You don't need to sacrifice 11TB of storage.  So the budget argument is a non-starter.

And yes, nothing regarding how witness disks work or their need to be exclusive has changed (from a configuration standpoint.)  So only a few possibilities exist for your difference of experience:

1) You created a separate LUN and used it as a witness disk and then forgot.
2) You were running without a witness disk.
3) You indeed grabbed one of those "used scraps" and used it as a witness disk.

None of those possibilities requires that Microsoft changed how witness disks behave between 2008 R2 and 2012.  Now I'm sorry you hit trouble, and I appreciate your frustration. But I was explaining *why* things are (and were) the way they were/are. If you insist on (mis)placing blame on Microsoft and insist things must've been different in the product, the argument becomes academic and is not something I feel further compelled to participate in.

The original answer stands. For changes to be visible on both nodes, the storage *must* be a CSV. And for fault tolerance, you *should* have a witness disk, but nobody will force you to have one if you want to go against best practices. I believe the original question was answered, both completely and with great detail, and a fair amount of background information was provided.
BSG IT TeamIT ManagerAuthor Commented:
I wasn't attempting to place blame anywhere. I fully understand that the need for further understanding of how quorums work and I am sure I will eventually get to that point after I complete my MCSA. Please try not to take offense to my sometimes frustrated nature, it has nothing to do with you. Thank you for thoroughly answering my questions.

One last question; is there a minimum amount of space needed for a disk witness? The cluster I am building now will "eventually" have 7 nodes (adopting more "used scraps" from elsewhere in the near future). Most of the nodes will have extra local storage disk that I won't be using since I will be storing everything on the CSV. I was thinking I could add a witness disk later that would/could be one of the unused local disks but I am not aware of any disk size requirements for witnesses.
Cliff GaliherCommented:
Well, let's get the first thing out of the way. The witness disk must still reside on shared storage. So take any thoughts of using a local disk and toss 'em out.  

Which also means you'll want to do this sooner than later. Since there are no single 11TB disks on the market, I can only assume you've already created a logical disk on your FC array.  Creating another logical disk for the witness is done the exact same way. But if you've already used all of your physical space, you'll want to redo that *now* leaving enough space to create your witness disk. Better to do that *before* you've started loading data on that CSV, eh?  That's why I recommend doing this sooner than later.  It is easier to find *and fix* mistakes than it is to go back later. Don't want to have a live cluster with workloads and be unable to add a witness disk because you've allocated all your space to the CSV.  That's a painful (and costly) mistake to try and undo!  

As far as the size.  Small.  A 1GB logical disk would be sufficient.  Leaving 10.99 TB for your CSV. Hence my earlier statement that cost (in this instance) isn't a good argument. There just isn't any added cost and almost no lost space to getting your array configured for proper quorum.

Hope that helps.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
BSG IT TeamIT ManagerAuthor Commented:
Definitely helped. Created a 10 GB LUN (10 GB is the minimum my storage array will allow you to make) and am now using it as a disk witness. Now I can sleep tonight :o)

Thanks for all your help.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.