Redundancy with a Point to Point (pt to pt) SAN
Hello Experts,
I have an old SAN that I bought to familiarize myself with the technology and configuration options outside of a production environment. (I am a network Engineer and we are in the process of moving to Blades and soon a SAN). This SAN has proved to not only be an invaluable learning tool but also much more usefulness that I had originally anticipated. I am now trying to determine the best redundancy configuration that I can put it in.
The SAN is a Infortrend IFT-6330-12. 2GB Fibre channel model, 128mb Cache Raid Controller, 12 IDE ATA Drives, Point to Point (pt to pt) topology. The SAN is populated with (12) x 200GBs drives, out of which I configured (2) x RAID 5 arrays; 6 Disks in each, giving me (2) x 1TB Volumes. I currently have the SAN connected to a Emulex LP902-DC FC 2GB HBA, connected to a Dell PowerEdge Tower Server.
It took some learning and hard trial & error but I have the unit running quite well. I used 1 array for my nightly backups and was using the other array as random storage. With the upcoming RTM release of Server 2008 R2, I have been working a lot with the R2 Beta and Hyper-V (moving VMs from Virtual Server 2005 to Hyper-V, etc). On a whim I threw a VHD on the catch all SAN array and was shocked at the performance. (ATA with a decent cached controller runs better than SATA through onboard mobo port. Who'd have thought it?)
Since then, I have been researching SAN Point to Point topology and trying to determine my options in regard to redundancy/availability. It is my understanding that from an availability standpoint, Point to Point SAN is pretty close to Direct Attached storage. Access to storage area network is dependant on the Host (Host's OS). From a configuration standpoint, I defined LDs and/or LVs, setup the Raid configuration, Mask/Map the LUNS, then configure the FC access. HBA connects to the SAN to the host. The SAN's arrays are formatted by the Host's OS. From that point on the access (reads, writes, etc) are done by or through a connect to the Host.
My issue is that I would like to use the storage on the SAN to house VHDs/VMs that will be running on machines other than the host. (I know, technically NAS, Network Storage, etc is a no-no with VMs but it can be done. Have even though of using an opensource iSCSI solution to manage the SAN arrays but that is getting off topic).
My problem boils down to if the SAN is housing data being used by multiple machines, and its host needs to reboot, gets turned off, HBA has an fault etc; it would present an issue. Ultimately I'd like to configure some redundancy to mitigate this.
Although it was my understanding that this is one of the draw backs with Pt to Pt SAN topology, I have done my homework and read about a number of scenarios that accomplish some of this. (1 Pt to Pt SAN, using multiple PIDs for each LUN, multiple HBAs, and multiple fibre switches for mirrored fibre fabric, etc). I have also read about various SAN/HBA multipath/multi-pathing solutions. Additionally I recently took the certification exams for SharePoint and read a decent amount about NLB and Clustering and wondered it that could also lend to this.
Although I believe in theory there may be a combination of these technologies that accomplish my desired out come, I haven't come across anything that covered a Point to Point configuration, with Multiple HBA's, and hosts being configured in exactly this manner.
I know my goal is reaching and before I purchase any additional components I wanted to get some feedback.
Thanks,
jbhaire2004
I have an old SAN that I bought to familiarize myself with the technology and configuration options outside of a production environment. (I am a network Engineer and we are in the process of moving to Blades and soon a SAN). This SAN has proved to not only be an invaluable learning tool but also much more usefulness that I had originally anticipated. I am now trying to determine the best redundancy configuration that I can put it in.
The SAN is a Infortrend IFT-6330-12. 2GB Fibre channel model, 128mb Cache Raid Controller, 12 IDE ATA Drives, Point to Point (pt to pt) topology. The SAN is populated with (12) x 200GBs drives, out of which I configured (2) x RAID 5 arrays; 6 Disks in each, giving me (2) x 1TB Volumes. I currently have the SAN connected to a Emulex LP902-DC FC 2GB HBA, connected to a Dell PowerEdge Tower Server.
It took some learning and hard trial & error but I have the unit running quite well. I used 1 array for my nightly backups and was using the other array as random storage. With the upcoming RTM release of Server 2008 R2, I have been working a lot with the R2 Beta and Hyper-V (moving VMs from Virtual Server 2005 to Hyper-V, etc). On a whim I threw a VHD on the catch all SAN array and was shocked at the performance. (ATA with a decent cached controller runs better than SATA through onboard mobo port. Who'd have thought it?)
Since then, I have been researching SAN Point to Point topology and trying to determine my options in regard to redundancy/availability. It is my understanding that from an availability standpoint, Point to Point SAN is pretty close to Direct Attached storage. Access to storage area network is dependant on the Host (Host's OS). From a configuration standpoint, I defined LDs and/or LVs, setup the Raid configuration, Mask/Map the LUNS, then configure the FC access. HBA connects to the SAN to the host. The SAN's arrays are formatted by the Host's OS. From that point on the access (reads, writes, etc) are done by or through a connect to the Host.
My issue is that I would like to use the storage on the SAN to house VHDs/VMs that will be running on machines other than the host. (I know, technically NAS, Network Storage, etc is a no-no with VMs but it can be done. Have even though of using an opensource iSCSI solution to manage the SAN arrays but that is getting off topic).
My problem boils down to if the SAN is housing data being used by multiple machines, and its host needs to reboot, gets turned off, HBA has an fault etc; it would present an issue. Ultimately I'd like to configure some redundancy to mitigate this.
Although it was my understanding that this is one of the draw backs with Pt to Pt SAN topology, I have done my homework and read about a number of scenarios that accomplish some of this. (1 Pt to Pt SAN, using multiple PIDs for each LUN, multiple HBAs, and multiple fibre switches for mirrored fibre fabric, etc). I have also read about various SAN/HBA multipath/multi-pathing solutions. Additionally I recently took the certification exams for SharePoint and read a decent amount about NLB and Clustering and wondered it that could also lend to this.
Although I believe in theory there may be a combination of these technologies that accomplish my desired out come, I haven't come across anything that covered a Point to Point configuration, with Multiple HBA's, and hosts being configured in exactly this manner.
I know my goal is reaching and before I purchase any additional components I wanted to get some feedback.
Thanks,
jbhaire2004
ASKER
I agree that there is a single point of failure with the SAN controller but my focus is not redundancy to avoid an internal failure of the SAN itself; but rather of a failure of the SAN's Host machine.
The most likely issues I see is that the Server acting as the SAN's Host will reboot, get turn off, or be disrupted in some way. This interruption in the SANs Host availability would sever access to the storage on the SAN.
AND MAYBE THERE LIES MY ISSUE...
I am working under the assumption, per the IFT-6330-12's manual, that the point to point san can have only 1 physical Host. It is also documented in the manual that the manner to allow access to the SANs Storage is by creating a FC connect between the SANs FC controller and the SAN Hosts FC HBA, after which the HBA and SAN controller are configured to communicated, the SAN Host then sees the connected volume. The IFT-6630-12 Manual then states the method to allow access to the SAN storage from other servers is to create a network share for the drive with the SAN Host managing the volume.
The root of my thinking is based on the assumption that there can be only 1 SAN Host. If that is not a false assumption, I would like to determine if there is a way to cluster the SANs Host so that there are 2 separate boxes managing the SAN.
Barring something I may have missed or miss understood, I want to wind up with 2 Hosts for a given LUN, (Cluster, Active/Passive, however), so that if SAN Host 1 needs a maintenance reboot, SAN Host 2 can keep the access up uninterrupted.
If I am missing something here or should be looking at this differently please let me know.
The most likely issues I see is that the Server acting as the SAN's Host will reboot, get turn off, or be disrupted in some way. This interruption in the SANs Host availability would sever access to the storage on the SAN.
AND MAYBE THERE LIES MY ISSUE...
I am working under the assumption, per the IFT-6330-12's manual, that the point to point san can have only 1 physical Host. It is also documented in the manual that the manner to allow access to the SANs Storage is by creating a FC connect between the SANs FC controller and the SAN Hosts FC HBA, after which the HBA and SAN controller are configured to communicated, the SAN Host then sees the connected volume. The IFT-6630-12 Manual then states the method to allow access to the SAN storage from other servers is to create a network share for the drive with the SAN Host managing the volume.
The root of my thinking is based on the assumption that there can be only 1 SAN Host. If that is not a false assumption, I would like to determine if there is a way to cluster the SANs Host so that there are 2 separate boxes managing the SAN.
Barring something I may have missed or miss understood, I want to wind up with 2 Hosts for a given LUN, (Cluster, Active/Passive, however), so that if SAN Host 1 needs a maintenance reboot, SAN Host 2 can keep the access up uninterrupted.
If I am missing something here or should be looking at this differently please let me know.
It seems rather unlikely that it would only be able to support one host. Do you have an URL to the manual and reference to where it says that. It may only support one host directly connected but if you use a fabric switch it ought to support more and they ought to be able to share LUNs.
Well for production environments I'll go always for an Infiniband fabric network. Usually they provide good reliability and fast speed. For redundancy you have to go for a redundant file system (like GFS).
Obviously is a very hard work to build up a project like that, I'll suggest you to check these documents to improve your skills:
http://www.redhat.com/gfs/
https://www.redhat.com/docs/manuals/csgfs/pdf/rh-gfs-en-6_0.pdf
and inifiniband fabric switched networks:
http://en.wikipedia.org/wiki/Infiniband
Obviously is a very hard work to build up a project like that, I'll suggest you to check these documents to improve your skills:
http://www.redhat.com/gfs/
https://www.redhat.com/docs/manuals/csgfs/pdf/rh-gfs-en-6_0.pdf
and inifiniband fabric switched networks:
http://en.wikipedia.org/wiki/Infiniband
ASKER
The IFT-6330-12 Manual can be found at: http://www.a1computer.de/download/ift_6330_12.pdf
As I re-read this again I think that I was not understanding some elements properly.
"2.1.7 RAID Sharing (Network)
RAID sharing with IFT-6330-12 subsystems involves allowing more
than just the host computer to store and retrieve data from the unit.
Once a RAID array has been created and assigned a drive
designation, 2-it can be shared to a LAN or WAN using the same
OS-dependent method as other host storage devices (e.g., 2-"local"
hard drives) are shared. Consult your OS documentation for
information as to how to share storage devices.
In order to connect the array to different host computers, 2-host
channel IDs or LUN numbers should be mapped with different
logical configurations (logical drives). If there is only one logical
configuration, 2-different ID/LUNs can be mapped with the same
array capacity. However, 2-file-locking mechanism must be
implemented with management software running to handle the
sharing."
"1.3.5 Notice for Fibre Channel Connection
The system Fibre port is not equipped with by-pass circuits. Bypass
services should be provided outside the system. Connection
to the host should be made using a 3rd party hub or switch, unless
the system is connected point-to-point to a single host computer."
"4.1.1 Host Requirements
In order for the IFT-6330-12F2D (Fibre models) to be connected to
the host computer, the host must have a fibre host bus adapter
installed and a SFP module. Most commonly, a host bus adapter
(HBA) is an add-on card that includes external fibre ports used to
connect with fibre devices, including the IFT-6330-12 array. It is
necessary to use an SFP module to connect the HBA to the IFT-
6330-12. If your HBA does not come with an SFP module, you will
not be able to connect it to the IFT-6330-12. Please contact our
technical support to obtain IFTs Quality Vendor List (QVL) to
determine which SFP modules and HBAs are compatible with the
IFT-6330-12.
Use Fiber optic cables to connect to host. If connection to host is
made to a GBICs converter, an LC-SC cable may be used. You may
use cables recommended by your HBA manufacturer. Do not bend
or allow the cable to be damaged following the instructions by the
manufacturer of the cable you select.
Please note that there is no by-pass circuits along the host port. Bypass
circuitry must be provided with a Fibre hub or switch.
Once the array has been connected, you must reboot the host and
then assign the RAID array a drive letter designation. Each OS has a
different procedure for assigning drive letters to storage devices.
Consult your particular OSs documentation and the documentation
for your HBA for more information."
As I re-read this again I think that I was not understanding some elements properly.
"2.1.7 RAID Sharing (Network)
RAID sharing with IFT-6330-12 subsystems involves allowing more
than just the host computer to store and retrieve data from the unit.
Once a RAID array has been created and assigned a drive
designation, 2-it can be shared to a LAN or WAN using the same
OS-dependent method as other host storage devices (e.g., 2-"local"
hard drives) are shared. Consult your OS documentation for
information as to how to share storage devices.
In order to connect the array to different host computers, 2-host
channel IDs or LUN numbers should be mapped with different
logical configurations (logical drives). If there is only one logical
configuration, 2-different ID/LUNs can be mapped with the same
array capacity. However, 2-file-locking mechanism must be
implemented with management software running to handle the
sharing."
"1.3.5 Notice for Fibre Channel Connection
The system Fibre port is not equipped with by-pass circuits. Bypass
services should be provided outside the system. Connection
to the host should be made using a 3rd party hub or switch, unless
the system is connected point-to-point to a single host computer."
"4.1.1 Host Requirements
In order for the IFT-6330-12F2D (Fibre models) to be connected to
the host computer, the host must have a fibre host bus adapter
installed and a SFP module. Most commonly, a host bus adapter
(HBA) is an add-on card that includes external fibre ports used to
connect with fibre devices, including the IFT-6330-12 array. It is
necessary to use an SFP module to connect the HBA to the IFT-
6330-12. If your HBA does not come with an SFP module, you will
not be able to connect it to the IFT-6330-12. Please contact our
technical support to obtain IFTs Quality Vendor List (QVL) to
determine which SFP modules and HBAs are compatible with the
IFT-6330-12.
Use Fiber optic cables to connect to host. If connection to host is
made to a GBICs converter, an LC-SC cable may be used. You may
use cables recommended by your HBA manufacturer. Do not bend
or allow the cable to be damaged following the instructions by the
manufacturer of the cable you select.
Please note that there is no by-pass circuits along the host port. Bypass
circuitry must be provided with a Fibre hub or switch.
Once the array has been connected, you must reboot the host and
then assign the RAID array a drive letter designation. Each OS has a
different procedure for assigning drive letters to storage devices.
Consult your particular OSs documentation and the documentation
for your HBA for more information."
ASKER
Additionally I found another Infortrend manual with more specific information that may help point me in the right directions.
http://ftp4.de.freesbie.org/pub/misc/infortrend/ftp.infortrend.com.tw/EonRAID_SentinelRAID/Manual/GMv162.pdf
http://ftp4.de.freesbie.org/pub/misc/infortrend/ftp.infortrend.com.tw/EonRAID_SentinelRAID/Manual/GMv162.pdf
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
That is odd about the server being unavailable. I have attached the pdf to this post.
I will look at picking up a Silkworm 3200. In the interim, if I were to pick up a less expensive switch like a Vixel Emulex Inspeed 330 would I be able to accomplish the majority of the same functions?
Also when I connect 2 hosts to a switch and then (as you suggested), "list the WWPNs or WWNNs of both hosts against the LUN masking configuration so the particular (array in their terms) is presented to both", will there be any additional steps required so that the hosts can both work with the array when required? I am green in this area so you have to excuse me. I do remember specifying WWPNs and WWNNs in the SAN controller software and the HBA management software when I set up the point to point configuration. It was to my understanding that if 2 hosts connected to a given array, Host 1 would be unaware if Host 2 was making changes to the array/volume, and such if Host 1 and Host 2 both tried to write to the volume/array at the same time there is the possibility that they could write over data that was written by the other causing corruption. Is that correct? Additionally, the Infortrend manual made a reference to, "If there is only one logical configuration, 2-different ID/LUNs can be mapped with the same array capacity. However, 2-file-locking mechanism must be implemented with management software running to handle the sharing." Would that be handled by the HBAs, Switch or Hosts' OS's; or would I need to take additional steps? I do not intend to have the 2 Hosts working with the array at the same time, rather I would like for the array to remain available by Host 2 if Host 1 was shut down.
GMv162.pdf
I will look at picking up a Silkworm 3200. In the interim, if I were to pick up a less expensive switch like a Vixel Emulex Inspeed 330 would I be able to accomplish the majority of the same functions?
Also when I connect 2 hosts to a switch and then (as you suggested), "list the WWPNs or WWNNs of both hosts against the LUN masking configuration so the particular (array in their terms) is presented to both", will there be any additional steps required so that the hosts can both work with the array when required? I am green in this area so you have to excuse me. I do remember specifying WWPNs and WWNNs in the SAN controller software and the HBA management software when I set up the point to point configuration. It was to my understanding that if 2 hosts connected to a given array, Host 1 would be unaware if Host 2 was making changes to the array/volume, and such if Host 1 and Host 2 both tried to write to the volume/array at the same time there is the possibility that they could write over data that was written by the other causing corruption. Is that correct? Additionally, the Infortrend manual made a reference to, "If there is only one logical configuration, 2-different ID/LUNs can be mapped with the same array capacity. However, 2-file-locking mechanism must be implemented with management software running to handle the sharing." Would that be handled by the HBAs, Switch or Hosts' OS's; or would I need to take additional steps? I do not intend to have the 2 Hosts working with the array at the same time, rather I would like for the array to remain available by Host 2 if Host 1 was shut down.
GMv162.pdf
ASKER
I bought a Silkworm 3200. Should be here in a few days.
>It was to my understanding that if 2 hosts connected to a given array, Host 1 would be unaware if Host 2 was making changes to the array/volume, and such if Host 1 and Host 2 both tried to write to the volume/array at the same time there is the possibility that they could write over data that was written by the other causing corruption. Is that correct?
It's inevitable, hosts store some of the file information in RAM, so data will become corrupted if two hosts have access to the same LUN unless you use clustering or a shared file system. However you can create several LUNs and present them to different servers so they are not shared.
Various options exist to share data between more than one server, active/passive clustering like MSCS, VMware HA etc. and there are shared filesystems that use file locking or special clients like Quantum's StorNext.
It's inevitable, hosts store some of the file information in RAM, so data will become corrupted if two hosts have access to the same LUN unless you use clustering or a shared file system. However you can create several LUNs and present them to different servers so they are not shared.
Various options exist to share data between more than one server, active/passive clustering like MSCS, VMware HA etc. and there are shared filesystems that use file locking or special clients like Quantum's StorNext.
point to point in the SAN world implies switched fabric, not direct attached. Since your infortrend box ofers LUN masking and supports up to 1024 hosts it must support fabric.