Solved

Storage Bay Bridge from Supermicro drives allocation

Posted on 2013-01-02
41
1,135 Views
Last Modified: 2013-01-20
We are trying to setup a Supermicro Storage Bay Bridge with Windows Storage Enterprise 2008 R2 in the two nodes.

When we try to update/install the operating system in one node, the other node goes down (it gets corrupted in some way, as it does not boot).

The SBB server has 12 disks: 2 disk in RAID 1 for node 1 operating system, 2 disk for the other node OS, and 8 disk in RAID 10 for shared storage.

Any help in how to properly configure this hardware is greatly appreciated.
0
Comment
Question by:JavBorr
  • 16
  • 15
  • 9
  • +1
41 Comments
 
LVL 18

Expert Comment

by:Andrej Pirman
ID: 38736429
I don't know for Supermicro, but in general, OS will get corrupted if you do not use Multipath driver AND you share LUNs among more than 1 single OS.
A LUN can only be used in ACTIVE state by 1 single OS at a time. So if, for example, SHARED storeage you intend to use in Active state in both OS installations at once, it is a no-go. Each one will try to mount it in read-write mode and lock it for itself, so freezing is inavoidable.

Now, one thing I do not understand.
You say SBB STORAGE has 3 LUNs:
- RAID1 array for OS1
- RAID1 array for OS2
- and RAID10 for shared storage

Do you meant that OS1 server and OS2 server are connected to this SBB STORAGE, and they both boot from SBB RAID disk arrays? Does not OS1 and OS2 server have their own disks, not even for booting?


If I understand this particular storage device corectly, it is:
- either with PREINSTALLED management utility, without some known OS in background, which enables you to create RAID arrays and present LUNs to connected servers
- or with an option for you to install, for example, Windows Storage Server 2008, where only from Windows Storage Server you will create iSCSI targets, which will then be used by connected servers to use as a storage

So I do not understand how did you try to install TWO operating systems on storage itself...if I've got your description correctly.
0
 

Author Comment

by:JavBorr
ID: 38736485
Hi Labsy, that's right, both servers are connected to a backplane with all storage. They do not have their own OS disks, as in other hardware like the HP X5520.

You have to configure hard disks for the OS, and hard disks for the shared storage. But the LSI Megaraid card has no indications for this procedure, and we have no aid from the supplier.

This equipment is an integrated 3U high availability storage server, with 2 nodes connected to a single backplane with all the storage. The two nodes should be in configured in a cluster, providing HA storage.
0
 
LVL 55

Expert Comment

by:andyalder
ID: 38737039
If there's no option to mask off dedicated disks for the OS/boot I don't see how you can use it with Windows. You could use VMware hypervisor on a USB stick (or SD card if there's a socket for it since that's dedicated to the node it booted from)
0
 

Author Comment

by:JavBorr
ID: 38737055
There should be a way to mask disk for the OS. But we have not found it, nor the people of the technical service which is assisting us.

The system came with the two nodes running, but once we tried to update one node, or reinstall the OS, everything failed.
0
 
LVL 55

Expert Comment

by:andyalder
ID: 38737074
It had RELDATA OS pre-installed or Windows?
0
 

Author Comment

by:JavBorr
ID: 38737090
Windows Storage Server 2008 R2 Enterprise
0
 
LVL 55

Expert Comment

by:andyalder
ID: 38737118
So you must be just expected to pick the right drive to install onto, so for the left server you would chose to install onto disk 0 and for the right server install onto disk 2 assuming disk 1 contains the data (disk as seen by the OS, not as seen by the RAID software). Sounds risky since you would always have to avoid accidentally writing a signature/giving a drive letter to the disk "owned" by the other OS and you would also have to disable any write cache on the active/passive shared storage.
0
 

Author Comment

by:JavBorr
ID: 38737138
There should be a way to block drives for one node or the other. Problem is we do not know how to do it, and the documentation also does not include details about it.
0
 
LVL 55

Expert Comment

by:andyalder
ID: 38737206
That would be nice but I don't think so since it just has a dual expander backplane (one expander connects one server to one port of the dual-ported* SAS disks and the other expander connects the other port of the SAS disks to the other server. Although expanders can be told to mask off PHYs that would be forgotten at power off. It wasn't supplied to you with disks masked off by the sound of it.

*SATAs aren't dual-ported, that's why it needs interposers for SATA.
0
 

Author Comment

by:JavBorr
ID: 38737236
Hi andyalder, you might be right, but there has to be some way to do it. This is a server built to be a high availability storage solution, so it should work in one way or the other.
0
 
LVL 55

Expert Comment

by:andyalder
ID: 38737495
If you look in BIOS can you set one controller to boot off a different logical disk than the other? That may be how they do it since it can't be stored on the disk metadata or both controllers would read the same info.

Been looking into Storage Bridge Bay and it's a standard rather than just a SuperMicro product, in theory you could have two different makes of controller/motherboard in the chassis although I don't think anyone else makes the kit. The backplane spec is passive as far as the data paths go so no LUN masking available there, it would have to be done on the LSI controllers or the system BIOS. Hopefully dlethe pops in since he knows lots more than I do about the LSI SAS 2008 chip the SCSI controllers use.
0
 

Author Comment

by:JavBorr
ID: 38739362
Hi andyalder, thanks for your time. That's right, Storage Bay Bridge is a standard (pushed by Intel I think).

The problem with this equipment from Supermicro is how to mark the drives in the backplane for one node or the other. The shared drives should not be a problem, provided that the two nodes are configured in cluster (the reseller did not even send us the nodes with Windows cluster enabled and configured).

I have gone through the BIOS and the LSI card, and did not find anything that apparently do this procedure.

The HP X5520 is very similiar, but each node has its own OS pair of disks in RAID 1 (and costs 3 times as much as the Supermicro). The Quanta CB2200 I think is pretty similar to the Supermicro, but I have not seen this hardware.
0
 
LVL 55

Expert Comment

by:andyalder
ID: 38753052
Have you got to the bottom of this yet?

SuperMicro SBB server as a hardware solution is valid and supports redundancy if you use the right OS even down to both controllers booting off the same read-only LUN. Burn your own BIOS/firmware onto the SuperMicro SBB twin-controller box and you can create a fully redundant one-box wonder pretty similar to many on the market.

 It's just a bit of tin for OEMs to configure and badge though and for Windows (and windows storage server) you might have to cut paths on the midplane to customise it.


>The HP X5520 is very similiar
No it isn't. The HP thing is two fully working blades plus shared storage in one box.
0
 

Author Comment

by:JavBorr
ID: 38754178
Andyalder, I do not fully understand the differences between the Supermicro SBB and the HP X5520. The Supermicro SBB has two full nodes (blades, or however it is called), with their memory, microprocessors, USB controller, etc, and a LSI card to connect to the backplane (where the shared storage is located).

You can even install a SATADOM in each node for the operating system. Both nodes are interconnected through a internal 10Gb connection for the heartbeat. How is that different from the HP X5520?

Thanks a lot.
0
 
LVL 55

Expert Comment

by:andyalder
ID: 38754302
The main difference is the HP blades have two RAID controllers in them, one for the internal disks with the OS on and one for the shared storage. Thus neither host can access the other host's OS disks.
0
 

Author Comment

by:JavBorr
ID: 38754381
And in the Supermicro SBB both nodes share the same OS disk? If this is the case, how this affect performance, updates, etc?
0
 
LVL 55

Expert Comment

by:andyalder
ID: 38754536
They don't necessarily share the same OS disks (you would need a read-only OS for that) but they do have access to each other's disks unless you can find a way to mask disks off.

As RELvos from reldata which the box comes with as standard is available as an embedded OS it's quite likely to be read-only so could share the same boot device.
0
 

Author Comment

by:JavBorr
ID: 38754548
Hi andyalder, that is exactly our question: how this equipment handles disks on the backplane, and how can we configure the nodes and the hard disks in the back so that they work flawlesly.
0
 
LVL 55

Expert Comment

by:andyalder
ID: 38754577
You probably need to install RELvos on it, you might be able to use VMware on SD card or USB sticks and then install storage server ontop of that.

Did SuperMicro say it supported Windows as the host OS?
0
 

Author Comment

by:JavBorr
ID: 38754757
Supermicro sent us the equipment with Windows installed. It was working untill a windows update took place. And now none is able to reinstall operating system, or make it work.
0
Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 55

Expert Comment

by:andyalder
ID: 38762716
I checked on the Quanta CB220 and like the HP box it has internal disks dedicated to each OS. It also has RAID controllers that mirror each other's write cache.

I suppose you could install a boot manager to control which node booted from which set of disks but even then I don't see an easy way for it to tell whether to boot the left or right controller's disks.
0
 
LVL 47

Expert Comment

by:dlethe
ID: 38769521
For what it is worth, this is a solid design, and i know of customers that literally have thousands of them. If you haven't already done so, make sure you have the latest backplane firmware, and MAKE SURE you update the LSI firmware.  Lots of issues with the older firmware/BIOS that you do not want to deal with.

In addition, actually read the release notes of the controller/drivers.  You can make some registry tweaks.
0
 
LVL 55

Expert Comment

by:andyalder
ID: 38772492
Indeed, but is there a way to mask off the OS LUNs from one controller or the other because JavBorr can't find the setting, documentation is sparse and I haven't got one to take apart.

It's fairly simple if you use USB sticks to boot the individual nodes from since a small boot environment can control what LUN to boot from and then you just don't give a drive letter to the other node's boot LUN and disable disk manager at least for normal users who might accidentally acess the wrong LUN.

You mention the backplane firmware but surely that hasn't got any LUN masking ability since then it would be a SPOF, I thought it just identified part number and capability to the controllers.
0
 

Author Comment

by:JavBorr
ID: 38773721
Thanks dlethe for jumping in with this information. As andyalder says, the documentation is almost nonexistent.

We have now the two nodes working, each with its operating system RAID in the backplane. We made a RAID 1 with disks 0 and 1, and activated it in one node (node 1). We then made a RAID 1 with disks 2 and 3, and activated it with the other node (node 2).

Node 1 does not access RAID disks 2 and 3 simply because it is not activated in this node. Is the only thing to be done? Isn't there a way to mask some disks, so the setup is more secure?

For the rest of disks, should we setup a cluster in Windows to share it, or is there any provision in the machine to use another software or hardware solution?

Supermicro says in the website: "With the dual 10GbE connection between the Serverboards via the midplane, if one serverboard fails, the other serverboard is able to take over control and access the HDD's (both controllers can also work as Active-active mode), keeping the system up and running. Storage software is the key to enable this feature, which is available from several Supermicro's partners"

What are the partners? What is the software to do this? How is control transfered from one node to the other? Is Windows Clustering service enough?

Thanks a lot.
0
 
LVL 47

Expert Comment

by:dlethe
ID: 38774531
You're going to have to buy something extra to do this.

I know of a storage OEM that was having a lot of problems getting a SBB working in a windows cluster, but they were using Win2012 and it was for a different SBB vendor, and they won't be selling it stand-alone, so that wouldn't help you anyway. (Plus it is NDA so I can't tell you who they are).

Google. I assure you there are some 3rd-party SBB clustering products out there.
0
 
LVL 55

Expert Comment

by:andyalder
ID: 38776242
The Supermicro documentation really sucks although I suppose if you tell them you want 1000 of them they'd be more help. Take http://supermicro.biz/support/faqs/faq.cfm?faq=11137 for example: "it doesn't run VMware* but it does run Nexenta". Why the heck can't they produce a list of supported OSs instead of just one without even giving version details for that?

I think I'd put a couple of PCIe SSD cards in it for the OSs just to be sure the other node has no access, but not activating the disks for a particular node seems to be good enough for LUN masking so long as you can replace a failed controller with a new one without the risk of activating the wrong ones, otherwise instead of increasing reliability by having redundancy you end up doubling the chance of failure since misconfiguring either node can kill it.

*I asked on EE if anyone was running VMware on one of them but got spammed so closed the question, I cannot believe Supermicro's use of the word "cannot" in that FAQ, more like not supported as far as I can see.
0
 

Author Comment

by:JavBorr
ID: 38787852
Hi andyalder, dlethe,

I have bee talking to a dealer, and it seems as if this SBB equipment is designed to be working with NexentarStor, the storage operating system from Nexenta.

I do not know whether this is 100% true, and if this hardware supports any other OS. As andyalder says the Supermicro documentation is deficient, and it nowhere says which are the supported OS and how they should be configured. There are a lot of voids in the docs regarding this equipment.

Have you had experience with Nexenta? Does it work as intended?

Thanks a lot.
0
 
LVL 47

Accepted Solution

by:
dlethe earned 500 total points
ID: 38794324
I have lots of experience with Nexenta professionally, and many of my customers who make NAS appliances as well as some larger ISPs have THOUSANDS of Nexeta systems.

Since you bought this as a DIY type of installation instead of going through a VAR who can help you then I suggest you talk to Nexenta and half them get you in touch with one of the Supermicro/Nexenta OEMs they have and pay them to help configure it.

It isn't so much that the documentation is insufficient, it is that Supermicro SBBs, or anybody's SBB systems are not what I call whitebox solutions.  It is compliant with the SBB spec, and that is all anybody should need to configure it outside of asking them for some additional programming information like interrupt numbers, SES Page 0A data structures, and other things developers need.

Win2012 is the appropriate O/S to use with this equipment, along with their new software RAID, since you are a windows person.  If you had unix skills then I say you will be MUCH better off with Nexenta as Win2012 is still a bit premature.   My suggestion is to go to the Supermicro site, look for the info on the "Nexenta Partners", and call up one of them and see what it they would charge you to set this up, plus you would have to buy the clustering software.

If you can get your money back for a restocking fee, do so.  You are in over your head and need to pay somebody to set this up for Nexenta.  (But that would be a wonderful solution and if time/budget allows, will be much better than a Win2008 or Win2012 in terms of performance, flexibility and features .. assuming you want this to be a file server and not some windows-centric "services" server.

(P.S. I am not implying I am offering any professional services on the side to help you through this.  I am just saying you need to pay somebody to make this work or just send it back an buy something that is "windows-friendly") ... P.S. I do have some LINUX-centric OEMs who use the supermicro SBBs that we wrote management software for.  LINUX works, but again, learning curve and configuration for you is cost prohibitive-- to get them work well under LINUX requires additional code and kernel modifications)
0
 

Author Comment

by:JavBorr
ID: 38794370
Hi dlethe, thanks a lot for your answer. I will try to do so.

I did not buy this SBB server as a whitebox server. I bought this as a solution; I indicated the VAR that I wanted this equipment, but they never mentioned this options and compatibilities. Had I know it, I would have been happy to pay for the complete setup with Nexenta had they introduced it and made a presentation.

Thanks again,
Javier.
0
 
LVL 55

Expert Comment

by:andyalder
ID: 38794760
It would still be nice to know the answer to the initial question, i.e. whether LUN masking via backplane or controller is possible or not on that box rather than the accepted answer just being "get someone else to do it for you".
0
 
LVL 47

Expert Comment

by:dlethe
ID: 38795243
Yes it is possible.
0
 

Author Comment

by:JavBorr
ID: 38796603
Hi dlethe, and can you do it in the BIOS, or with the MegaRaid software, or do you need to develop special software for each operating system?

For what I have worked with the documentation and the provided software it seem impossible. So it is good to know that it is. And it could be great if you could share, unless it is part of the services you provide for a fee, of course.
0
 
LVL 47

Expert Comment

by:dlethe
ID: 38796634
Not with the megaRAID.  You need to go host-based RAID, like Nexenta ZFS and/or LINUX md file systems, or even Windows2012 new host-based RAID.   (But Nexenta is the tried-and-true solution).

But I would break a NDA if I gave specifics when it comes to Win2K12 and LINUX, since I am involved in development.  Sorry.   But Clustered solutions for Nexenta is common knowledge and there are a fair number of vendors out there.  

Google "Nexenta Cluster HA" and you will find lots of vendors and many are familiar with SuperMicro.  Of course all of them sell whitebox bundled solutions with the hardware and you will have to convince them to sell you software & services, but you really need that anyway.  The Nexenta is a much better solution.  Things like hot snapshot backup, online expansion, you can add SSDs into the same pools for performance;  data compression that can be turned on and off, on live data at the filesystem level.   De-duplication, etc ..

Much more flexibility and data integrity than you can get with NTFS.

But are you sure you have a MegaRAID?  LSI has 2 families, and it is easy to confuse. You don't use the LSI MegaRAID with nexenta, you use the LSI MPT family.  I would be surprised if you have the MegaRAID because that is wrong controller.
0
 
LVL 55

Expert Comment

by:andyalder
ID: 38796673
LSI SAS2008 is the controller that comes with the SuperMicro SBB, integrated onto the motherboards. It's just the LUN masking I'm interested in.
0
 
LVL 47

Expert Comment

by:dlethe
ID: 38796708
The LSI2008 is a SAS-2 controller.  Per the ANSI spec,  If any SAS-2 target device can "see" another SAS-2 target or initiator, then they can be programmed to not see it.  SAS-2 has zoning just like Fibre channel does, but with some differences in implementation.  Still same concept.  Also one can do persistent addressing within the LSI chipset and that works with both SAS-1 and SAS-2.

In fact, one can write code to program any SAS-2 make/model of disk to be zoned to any controller or controllers and mix and match at will.  Where it gets tricky is dealing with expanders.  The Supermicro SBB expanders are based on the LSI chipset and they behave (well, some do).  If you are curious about whether I or anybody can be creative with zoning on HP expanders, I'll tell you I have never been contracted to look into that, so honestly don't know.

Dell's products do NOT play nice, BTW.
0
 

Author Comment

by:JavBorr
ID: 38797248
As anyalder says, the card is a LSI 2008 SAS-2. When I mentioned MegaRAID, I mean the software under Windows. I thought maybe some could be done here, but as you mentioned it looks as if everything has to be controlled in the OS layer.
0
 
LVL 47

Expert Comment

by:dlethe
ID: 38797347
No, the LSI2008 is a controller chip that stands on its own and is used by both LSI non-RAID and RAID controllers (and many other vendors use the chip as well).

You need to figure out what you have by checking out packing slips and/or BIOS if you want correct info.  But in any event, everything is controlled by the chip firmware, not O/S layer.  If a HDD is zoned then it is zoned and the O/S layer has nothing to do with it.   The O/S level software is used for the convenience of an end-user, as well as housekeeping to define how services and paths need to get reworked and such.

But bottom line, figure out exactly what you have and contact a reseller at the supermicro site and see what they will charge to turnkey this into configured Nexenta solution.    (Or send it back and pay a restocking fee.  If you have what you indicate you have, then it isn't a good fit for Windows or Linux because what you need isn't productized for just any hardware combinations)
0
 
LVL 55

Expert Comment

by:andyalder
ID: 38798300
Oh, the persistence mappings are configured with LSIutil, pity nobody mentioned that earlier. Documentation is on Oracle's website under Sun Blade 6000 Modular System. With that setup properly the SuperMicro SBB ought to be able to run any x86 OS although it could be a bit hairy replacing a failed controller card.

Glad I found that out so next time I can answer similar questions without reverting to name dropping and claims of I won't tell you due to NDA.
0
 
LVL 47

Expert Comment

by:dlethe
ID: 38798474
Andy, not my fault I am privy to things under non-disclosure. But when you are a developer who writes the code that people like you just use or read about, but somebody else puts their name on it ... then non disclosure agreements go with the territory.

Deal with it. You're lucky I took the time to respond at all.

P.S. persistance and "lun maskng" as you call it (which is not the right term here anyway, I was being kind by letting that term go) have nothing to do with zoning, which is what needs to be done for this to work.  The target operating systems and software-based RAID can deal with non-persistant device mapping just fine.   LSIutil adds nothing to this particular solution, which is why I didn't mention it.
0
 
LVL 55

Expert Comment

by:andyalder
ID: 38798709
Unfortunately needling you is sometimes the only way to get information out of you since you don't bother to check whether NDA info is now in the public domain. When I need to use something that I learned under NDA I feel justified in re-releasing that info if the manufacturer has already released it, otherwise you could be told that 2+2=4 under NDA and you would then be unable to ever tell anyone how to add.
0
 
LVL 47

Expert Comment

by:dlethe
ID: 38798735
It is not my responsibility to check to see whether something is in public domain.  Even if it is in the public domain, it must be released by the owners of the intellectual property in the form of a public announcement and I must know (in writing) if it is OK to comment on.

Even then, what is announced is seldom the whole story, and my inability to elaborate on the whole story, or really do anything beyond explain what is in an announcement is why I often don't say anything at all on a subject, or just let incorrect information slide.

In fact, In this particular situation, I know certain things are possible, because the SAS-2 programming spec defines zoning.  Whether or not I or anybody else productized it is proprietary information beyond Nexenta which is public info and shipping product. Some vendors in Windows/LINUX may have announced something, so use a search engine and find out.
0

Featured Post

Network it in WD Red

There's an industry-leading WD Red drive for every compatible NAS system to help fulfill your data storage needs. With drives up to 8TB, WD Red offers a wide array of solutions for customers looking to build the biggest, best-performing NAS storage solution.  

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
Netapp Snapshots 3 45
Recommendation for External Drives 6 36
Hyper V host drive space issue 16 135
Time Machine backup from iCloud 7 51
Hyper-convergence systems have taken the IT world by storm and have quickly started to change our point of view of how the data center should and could be architected. In this article, I’ll explain the benefits of employing a hyper-converged system …
AWS Glacier is Amazons cheapest storage option and is their answer to a ‘Cold’ storage service.  Customers primarily use this service for archival purposes and storage of infrastructure backups.  Its unlimited storage potential and low storage cost …
This video teaches viewers how to encrypt an external drive that requires a password to read and edit the drive. All tasks are done in Disk Utility. Plug in the external drive you wish to encrypt: Make sure all previous data on the drive has been …
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now