Link to home
Start Free TrialLog in
Avatar of JavBorr
JavBorrFlag for Spain

asked on

Storage Bay Bridge from Supermicro drives allocation

We are trying to setup a Supermicro Storage Bay Bridge with Windows Storage Enterprise 2008 R2 in the two nodes.

When we try to update/install the operating system in one node, the other node goes down (it gets corrupted in some way, as it does not boot).

The SBB server has 12 disks: 2 disk in RAID 1 for node 1 operating system, 2 disk for the other node OS, and 8 disk in RAID 10 for shared storage.

Any help in how to properly configure this hardware is greatly appreciated.
Avatar of Andrej Pirman
Andrej Pirman
Flag of Slovenia image

I don't know for Supermicro, but in general, OS will get corrupted if you do not use Multipath driver AND you share LUNs among more than 1 single OS.
A LUN can only be used in ACTIVE state by 1 single OS at a time. So if, for example, SHARED storeage you intend to use in Active state in both OS installations at once, it is a no-go. Each one will try to mount it in read-write mode and lock it for itself, so freezing is inavoidable.

Now, one thing I do not understand.
You say SBB STORAGE has 3 LUNs:
- RAID1 array for OS1
- RAID1 array for OS2
- and RAID10 for shared storage

Do you meant that OS1 server and OS2 server are connected to this SBB STORAGE, and they both boot from SBB RAID disk arrays? Does not OS1 and OS2 server have their own disks, not even for booting?

If I understand this particular storage device corectly, it is:
- either with PREINSTALLED management utility, without some known OS in background, which enables you to create RAID arrays and present LUNs to connected servers
- or with an option for you to install, for example, Windows Storage Server 2008, where only from Windows Storage Server you will create iSCSI targets, which will then be used by connected servers to use as a storage

So I do not understand how did you try to install TWO operating systems on storage itself...if I've got your description correctly.
Avatar of JavBorr


Hi Labsy, that's right, both servers are connected to a backplane with all storage. They do not have their own OS disks, as in other hardware like the HP X5520.

You have to configure hard disks for the OS, and hard disks for the shared storage. But the LSI Megaraid card has no indications for this procedure, and we have no aid from the supplier.

This equipment is an integrated 3U high availability storage server, with 2 nodes connected to a single backplane with all the storage. The two nodes should be in configured in a cluster, providing HA storage.
Avatar of Member_2_231077

If there's no option to mask off dedicated disks for the OS/boot I don't see how you can use it with Windows. You could use VMware hypervisor on a USB stick (or SD card if there's a socket for it since that's dedicated to the node it booted from)
Avatar of JavBorr


There should be a way to mask disk for the OS. But we have not found it, nor the people of the technical service which is assisting us.

The system came with the two nodes running, but once we tried to update one node, or reinstall the OS, everything failed.
It had RELDATA OS pre-installed or Windows?
Avatar of JavBorr


Windows Storage Server 2008 R2 Enterprise
So you must be just expected to pick the right drive to install onto, so for the left server you would chose to install onto disk 0 and for the right server install onto disk 2 assuming disk 1 contains the data (disk as seen by the OS, not as seen by the RAID software). Sounds risky since you would always have to avoid accidentally writing a signature/giving a drive letter to the disk "owned" by the other OS and you would also have to disable any write cache on the active/passive shared storage.
Avatar of JavBorr


There should be a way to block drives for one node or the other. Problem is we do not know how to do it, and the documentation also does not include details about it.
That would be nice but I don't think so since it just has a dual expander backplane (one expander connects one server to one port of the dual-ported* SAS disks and the other expander connects the other port of the SAS disks to the other server. Although expanders can be told to mask off PHYs that would be forgotten at power off. It wasn't supplied to you with disks masked off by the sound of it.

*SATAs aren't dual-ported, that's why it needs interposers for SATA.
Avatar of JavBorr


Hi andyalder, you might be right, but there has to be some way to do it. This is a server built to be a high availability storage solution, so it should work in one way or the other.
If you look in BIOS can you set one controller to boot off a different logical disk than the other? That may be how they do it since it can't be stored on the disk metadata or both controllers would read the same info.

Been looking into Storage Bridge Bay and it's a standard rather than just a SuperMicro product, in theory you could have two different makes of controller/motherboard in the chassis although I don't think anyone else makes the kit. The backplane spec is passive as far as the data paths go so no LUN masking available there, it would have to be done on the LSI controllers or the system BIOS. Hopefully dlethe pops in since he knows lots more than I do about the LSI SAS 2008 chip the SCSI controllers use.
Avatar of JavBorr


Hi andyalder, thanks for your time. That's right, Storage Bay Bridge is a standard (pushed by Intel I think).

The problem with this equipment from Supermicro is how to mark the drives in the backplane for one node or the other. The shared drives should not be a problem, provided that the two nodes are configured in cluster (the reseller did not even send us the nodes with Windows cluster enabled and configured).

I have gone through the BIOS and the LSI card, and did not find anything that apparently do this procedure.

The HP X5520 is very similiar, but each node has its own OS pair of disks in RAID 1 (and costs 3 times as much as the Supermicro). The Quanta CB2200 I think is pretty similar to the Supermicro, but I have not seen this hardware.
Have you got to the bottom of this yet?

SuperMicro SBB server as a hardware solution is valid and supports redundancy if you use the right OS even down to both controllers booting off the same read-only LUN. Burn your own BIOS/firmware onto the SuperMicro SBB twin-controller box and you can create a fully redundant one-box wonder pretty similar to many on the market.

 It's just a bit of tin for OEMs to configure and badge though and for Windows (and windows storage server) you might have to cut paths on the midplane to customise it.

>The HP X5520 is very similiar
No it isn't. The HP thing is two fully working blades plus shared storage in one box.
Avatar of JavBorr


Andyalder, I do not fully understand the differences between the Supermicro SBB and the HP X5520. The Supermicro SBB has two full nodes (blades, or however it is called), with their memory, microprocessors, USB controller, etc, and a LSI card to connect to the backplane (where the shared storage is located).

You can even install a SATADOM in each node for the operating system. Both nodes are interconnected through a internal 10Gb connection for the heartbeat. How is that different from the HP X5520?

Thanks a lot.
The main difference is the HP blades have two RAID controllers in them, one for the internal disks with the OS on and one for the shared storage. Thus neither host can access the other host's OS disks.
Avatar of JavBorr


And in the Supermicro SBB both nodes share the same OS disk? If this is the case, how this affect performance, updates, etc?
They don't necessarily share the same OS disks (you would need a read-only OS for that) but they do have access to each other's disks unless you can find a way to mask disks off.

As RELvos from reldata which the box comes with as standard is available as an embedded OS it's quite likely to be read-only so could share the same boot device.
Avatar of JavBorr


Hi andyalder, that is exactly our question: how this equipment handles disks on the backplane, and how can we configure the nodes and the hard disks in the back so that they work flawlesly.
You probably need to install RELvos on it, you might be able to use VMware on SD card or USB sticks and then install storage server ontop of that.

Did SuperMicro say it supported Windows as the host OS?
Avatar of JavBorr


Supermicro sent us the equipment with Windows installed. It was working untill a windows update took place. And now none is able to reinstall operating system, or make it work.
I checked on the Quanta CB220 and like the HP box it has internal disks dedicated to each OS. It also has RAID controllers that mirror each other's write cache.

I suppose you could install a boot manager to control which node booted from which set of disks but even then I don't see an easy way for it to tell whether to boot the left or right controller's disks.
Avatar of David
For what it is worth, this is a solid design, and i know of customers that literally have thousands of them. If you haven't already done so, make sure you have the latest backplane firmware, and MAKE SURE you update the LSI firmware.  Lots of issues with the older firmware/BIOS that you do not want to deal with.

In addition, actually read the release notes of the controller/drivers.  You can make some registry tweaks.
Indeed, but is there a way to mask off the OS LUNs from one controller or the other because JavBorr can't find the setting, documentation is sparse and I haven't got one to take apart.

It's fairly simple if you use USB sticks to boot the individual nodes from since a small boot environment can control what LUN to boot from and then you just don't give a drive letter to the other node's boot LUN and disable disk manager at least for normal users who might accidentally acess the wrong LUN.

You mention the backplane firmware but surely that hasn't got any LUN masking ability since then it would be a SPOF, I thought it just identified part number and capability to the controllers.
Avatar of JavBorr


Thanks dlethe for jumping in with this information. As andyalder says, the documentation is almost nonexistent.

We have now the two nodes working, each with its operating system RAID in the backplane. We made a RAID 1 with disks 0 and 1, and activated it in one node (node 1). We then made a RAID 1 with disks 2 and 3, and activated it with the other node (node 2).

Node 1 does not access RAID disks 2 and 3 simply because it is not activated in this node. Is the only thing to be done? Isn't there a way to mask some disks, so the setup is more secure?

For the rest of disks, should we setup a cluster in Windows to share it, or is there any provision in the machine to use another software or hardware solution?

Supermicro says in the website: "With the dual 10GbE connection between the Serverboards via the midplane, if one serverboard fails, the other serverboard is able to take over control and access the HDD's (both controllers can also work as Active-active mode), keeping the system up and running. Storage software is the key to enable this feature, which is available from several Supermicro's partners"

What are the partners? What is the software to do this? How is control transfered from one node to the other? Is Windows Clustering service enough?

Thanks a lot.
You're going to have to buy something extra to do this.

I know of a storage OEM that was having a lot of problems getting a SBB working in a windows cluster, but they were using Win2012 and it was for a different SBB vendor, and they won't be selling it stand-alone, so that wouldn't help you anyway. (Plus it is NDA so I can't tell you who they are).

Google. I assure you there are some 3rd-party SBB clustering products out there.
The Supermicro documentation really sucks although I suppose if you tell them you want 1000 of them they'd be more help. Take for example: "it doesn't run VMware* but it does run Nexenta". Why the heck can't they produce a list of supported OSs instead of just one without even giving version details for that?

I think I'd put a couple of PCIe SSD cards in it for the OSs just to be sure the other node has no access, but not activating the disks for a particular node seems to be good enough for LUN masking so long as you can replace a failed controller with a new one without the risk of activating the wrong ones, otherwise instead of increasing reliability by having redundancy you end up doubling the chance of failure since misconfiguring either node can kill it.

*I asked on EE if anyone was running VMware on one of them but got spammed so closed the question, I cannot believe Supermicro's use of the word "cannot" in that FAQ, more like not supported as far as I can see.
Avatar of JavBorr


Hi andyalder, dlethe,

I have bee talking to a dealer, and it seems as if this SBB equipment is designed to be working with NexentarStor, the storage operating system from Nexenta.

I do not know whether this is 100% true, and if this hardware supports any other OS. As andyalder says the Supermicro documentation is deficient, and it nowhere says which are the supported OS and how they should be configured. There are a lot of voids in the docs regarding this equipment.

Have you had experience with Nexenta? Does it work as intended?

Thanks a lot.
Avatar of David
Flag of United States of America image

Link to home
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of JavBorr


Hi dlethe, thanks a lot for your answer. I will try to do so.

I did not buy this SBB server as a whitebox server. I bought this as a solution; I indicated the VAR that I wanted this equipment, but they never mentioned this options and compatibilities. Had I know it, I would have been happy to pay for the complete setup with Nexenta had they introduced it and made a presentation.

Thanks again,
It would still be nice to know the answer to the initial question, i.e. whether LUN masking via backplane or controller is possible or not on that box rather than the accepted answer just being "get someone else to do it for you".
Yes it is possible.
Avatar of JavBorr


Hi dlethe, and can you do it in the BIOS, or with the MegaRaid software, or do you need to develop special software for each operating system?

For what I have worked with the documentation and the provided software it seem impossible. So it is good to know that it is. And it could be great if you could share, unless it is part of the services you provide for a fee, of course.
Not with the megaRAID.  You need to go host-based RAID, like Nexenta ZFS and/or LINUX md file systems, or even Windows2012 new host-based RAID.   (But Nexenta is the tried-and-true solution).

But I would break a NDA if I gave specifics when it comes to Win2K12 and LINUX, since I am involved in development.  Sorry.   But Clustered solutions for Nexenta is common knowledge and there are a fair number of vendors out there.  

Google "Nexenta Cluster HA" and you will find lots of vendors and many are familiar with SuperMicro.  Of course all of them sell whitebox bundled solutions with the hardware and you will have to convince them to sell you software & services, but you really need that anyway.  The Nexenta is a much better solution.  Things like hot snapshot backup, online expansion, you can add SSDs into the same pools for performance;  data compression that can be turned on and off, on live data at the filesystem level.   De-duplication, etc ..

Much more flexibility and data integrity than you can get with NTFS.

But are you sure you have a MegaRAID?  LSI has 2 families, and it is easy to confuse. You don't use the LSI MegaRAID with nexenta, you use the LSI MPT family.  I would be surprised if you have the MegaRAID because that is wrong controller.
LSI SAS2008 is the controller that comes with the SuperMicro SBB, integrated onto the motherboards. It's just the LUN masking I'm interested in.
The LSI2008 is a SAS-2 controller.  Per the ANSI spec,  If any SAS-2 target device can "see" another SAS-2 target or initiator, then they can be programmed to not see it.  SAS-2 has zoning just like Fibre channel does, but with some differences in implementation.  Still same concept.  Also one can do persistent addressing within the LSI chipset and that works with both SAS-1 and SAS-2.

In fact, one can write code to program any SAS-2 make/model of disk to be zoned to any controller or controllers and mix and match at will.  Where it gets tricky is dealing with expanders.  The Supermicro SBB expanders are based on the LSI chipset and they behave (well, some do).  If you are curious about whether I or anybody can be creative with zoning on HP expanders, I'll tell you I have never been contracted to look into that, so honestly don't know.

Dell's products do NOT play nice, BTW.
Avatar of JavBorr


As anyalder says, the card is a LSI 2008 SAS-2. When I mentioned MegaRAID, I mean the software under Windows. I thought maybe some could be done here, but as you mentioned it looks as if everything has to be controlled in the OS layer.
No, the LSI2008 is a controller chip that stands on its own and is used by both LSI non-RAID and RAID controllers (and many other vendors use the chip as well).

You need to figure out what you have by checking out packing slips and/or BIOS if you want correct info.  But in any event, everything is controlled by the chip firmware, not O/S layer.  If a HDD is zoned then it is zoned and the O/S layer has nothing to do with it.   The O/S level software is used for the convenience of an end-user, as well as housekeeping to define how services and paths need to get reworked and such.

But bottom line, figure out exactly what you have and contact a reseller at the supermicro site and see what they will charge to turnkey this into configured Nexenta solution.    (Or send it back and pay a restocking fee.  If you have what you indicate you have, then it isn't a good fit for Windows or Linux because what you need isn't productized for just any hardware combinations)
Oh, the persistence mappings are configured with LSIutil, pity nobody mentioned that earlier. Documentation is on Oracle's website under Sun Blade 6000 Modular System. With that setup properly the SuperMicro SBB ought to be able to run any x86 OS although it could be a bit hairy replacing a failed controller card.

Glad I found that out so next time I can answer similar questions without reverting to name dropping and claims of I won't tell you due to NDA.
Andy, not my fault I am privy to things under non-disclosure. But when you are a developer who writes the code that people like you just use or read about, but somebody else puts their name on it ... then non disclosure agreements go with the territory.

Deal with it. You're lucky I took the time to respond at all.

P.S. persistance and "lun maskng" as you call it (which is not the right term here anyway, I was being kind by letting that term go) have nothing to do with zoning, which is what needs to be done for this to work.  The target operating systems and software-based RAID can deal with non-persistant device mapping just fine.   LSIutil adds nothing to this particular solution, which is why I didn't mention it.
Unfortunately needling you is sometimes the only way to get information out of you since you don't bother to check whether NDA info is now in the public domain. When I need to use something that I learned under NDA I feel justified in re-releasing that info if the manufacturer has already released it, otherwise you could be told that 2+2=4 under NDA and you would then be unable to ever tell anyone how to add.
It is not my responsibility to check to see whether something is in public domain.  Even if it is in the public domain, it must be released by the owners of the intellectual property in the form of a public announcement and I must know (in writing) if it is OK to comment on.

Even then, what is announced is seldom the whole story, and my inability to elaborate on the whole story, or really do anything beyond explain what is in an announcement is why I often don't say anything at all on a subject, or just let incorrect information slide.

In fact, In this particular situation, I know certain things are possible, because the SAS-2 programming spec defines zoning.  Whether or not I or anybody else productized it is proprietary information beyond Nexenta which is public info and shipping product. Some vendors in Windows/LINUX may have announced something, so use a search engine and find out.