Solved

Disaster recovery location \ San Replication questions

Posted on 2012-03-30
14
693 Views
Last Modified: 2012-04-18
We are opening a disaster recovery location soon, in a remote datacenter.  We are using san replication, vmware vmotion, etc.  These sites will be connected initially using ipsec vpn and later some mpls avpn solution from AT&T.  That could take a while though.

We have hired consultants to help us with this, but they have not been very helpful.  

I'm new to DR and san replication, so I have a couple questions.  Obviously if our servers are being replicated, they have the same private ip addresses.  When I set up the vpn, is this going to cause problems?  The fact that we will have the same ip scheme on both sides?  If so, what is the usual way to remedy this?  

Also, we will have two redundant switches at each location connecting the sans.  San controllers A &B on both sides will have their own vlans, and they all need to be able to route to each other.  I've set it up on layer 3 switches so that A&B can route to each other, but how will A & B on the DR side route to A & B on the office side?  Is it just more static routes that are needed?

thanks!
0
Comment
Question by:readymade
  • 7
  • 4
  • 3
14 Comments
 
LVL 42

Expert Comment

by:kevinhsieh
ID: 37790074
It sounds like you need to hire different consultants...

Your network needs to have different IP subnets at the different locations. You can not use the SAME IP addresses on both sides of a VPN. The usual thing to do is to change the IP addresses of your VMs when you bring them up on the other side. If they are using DHCP, then you shouldn't have issues since they will just get the new IP information when they boot.

If you are using unique IP subnets everywhere and have routing setup on your L3 switches so that they know how to route over the VPN to the remote subnets, the SAN controllers should be able to communicate across the VPN as long as their default gateway is properly set.
0
 
LVL 8

Expert Comment

by:gsmartin
ID: 37790796
The are a number of ways to architect your DC/DR infrastructure.  

For me, I choose to go with a point-to-point fiber connection between my primary DC and DR sites.  I architected the sites as if they were a single site by spanning Layer 2 across both sites along with a Layer 3 VRRP-Extended (similar to HSRP in the Cisco world) between the sites.  I spanned my addressing scheme (10.SITE.VLAN.HOST) and VLANs across both sites.  This allowed for a much more flexible environment in my case providing the ability to L2 traffic transverse between sites as well as Layer 3 traffic.

I architected a somewhat mirrored environment between the sites with Blade Enclosures, servers, a similarly sized SAN, as well as all other appropriate network devices.   We are a 98% Virtualized (VMware 4.x) with SRM.  For Replication, we are replicating data on several levels SAN-to-SAN, Database (Sybase and DB2), Exchange 2010 DAWG, AD, DNS, and DFSR.  With our SAN replication LUNs are snapshotted (Compellent Continous Replay) then replicated using 'Compellent Thin Replication' to the remote site in a dismounted state; all other replication methods are with live data to dedicated physical and virtual machines  (with unique IP addresses).  

As far as replicated dismounted Virtual Machine LUNs, the LUNs are only mounted and Virtual Machines are configured (via VMware's SRM) to come online only in the event of a true disaster.  At which point, it would be assumed the primary site would be completely unavailable and would require activating all virtual machines that were replicated from the primary site.  At which point, you would encounter or expect IP conflicts.

If your objective is to replicate live servers from your primary site and to have them running on the secondary site this would be an issue in many ways.  You won't be able to this with a live machine unless your only replicating the data from the servers.  And best, way to do this is by creating a mirrored version of the server with a unique IP address on the same VLAN and then replicate the data to the DR servers.  In my case, I have both solutions Active and Dismounted identical servers where applicable.  Also, I leverage load balancers VIPs as the primary IP addresses for most resources that then can redirect to the appropriate server/service on either site.  

All while avoiding IP conflicts.
0
 
LVL 8

Accepted Solution

by:
gsmartin earned 500 total points
ID: 37790810
My recommendation is when replicating your LUNs that they are dismounted or at least the Virtual Machines are not running on the secondary site and then leverage VMware SRM for recovery from a Disaster.  

Otherwise, If you require live servers with live replicated data then you will need to leverage a different replication strategy, as I have with my additional replication methods.  In this case, you would only need to modify DNS settings to direct hosts to the set of duplicate VMs (unique IP addresses) or implement a set of Load Balancers to alleviate this task, in such an event since it becomes your primary IP for server services (i.e. VIP - Virtual IP).
0
 
LVL 8

Expert Comment

by:gsmartin
ID: 37790819
If both sites have same IP scheme then you need to bridge them at layer 2.  Otherwise, it will be a difficult solution.  Spanning Layer 2 for DC/DR DataCenter scenarios is typically how most enterprises configure their DATA centers.  You can either do this with a point-to-point thernet or fiber telco connection or via VPLS over MPLS, depending on the provider; VPLS will allow you to do Layer 2.  With Cisco routers, you can also leverage GRE (VPN) to enable layer 2 connectivity.
0
 

Author Comment

by:readymade
ID: 37796493
We are using VMware SRM but I don't totally understand the capabilities, when it comes to the individual vm ip addresses.  

Ok so we will use different IPs at the DR site.  Here's what I don't understand:  if this DR network will have different IPs and the firewall is set up for those IPs.....when I bring up the DR site in a disaster, and even if I change the IPs of the vm's....   how do I set up the firewall for this scenario?  

Many of our clients will be connecting via client-to-site vpn and they will be looking to use the IPs they are used to.  I'm not understanding what SRM does to remedy this and how I would set the firewall so the changing all the IPs would solve the problem.

thanks!
0
 
LVL 42

Expert Comment

by:kevinhsieh
ID: 37798456
Hopefully your clients can use name resolution instead of hard coded IP addresses. If they are looking at static IPs, your job is going to be a bit harder. Also, what happens when the VPN endpoint that they are connecting to is down because of a site-wide failure?
0
 

Author Comment

by:readymade
ID: 37798606
We know we will have to call in and get our public DNS records changed.  That should take care of the VPN issue.  That is set up on the DR firewall as well.  

I'm just not getting the SRM part:

DNS servers are being "REPLICATED" to DR site
However they will have a different IP, somehow
Also, the DNS records stored in the replicated DNS server will be for the original ip range.  

I don't get how that is all going to work in a failover.
0
Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

 
LVL 42

Expert Comment

by:kevinhsieh
ID: 37798703
I am not a SRM expert, but I would think that you would have some VMs running at the DR site, including domain controllers. You would not be replicating domain controllers to the DR site, because they should already be running, and would already have their IP addresses. The VMs should use dynamic DNS to update their IP addresses in DNS when they boot with the new IP addresses.

My advice is free. If you are paying consultants to help you with this and they can't, I really suggest finding some different consultants.
0
 

Author Comment

by:readymade
ID: 37798711
Getting other consultants isn't really an option at this stage in the project.  I was hired recently, so not really my call anyways.  

All servers that are replicated are running.  This is real time replication.  DNS servers are replicated.  

Is there anybody that is familiar with SRM that can explain how this will work?  I know it does, I just need to know how so I can do the networking.  Thanks!
0
 
LVL 8

Expert Comment

by:gsmartin
ID: 37798725
With VMware SRM, your SAN LUNs are in a dismounted state.  SRM will first mount the LUNs and then boot the respective VMs in the proper order you define with the original IP and hostnames.  SRM failover assumes the primary sites is completely unavailable.
0
 
LVL 8

Expert Comment

by:gsmartin
ID: 37798744
"We know we will have to call in and get our public DNS records changed.  That should take care of the VPN issue."

Keep in mind, the update can takes days for the changes to fully propagate across the internet to all of the various name servers.  ISP typically use a default TTL setting for DNS records of 7200 seconds, which is approximately 5 days.

In my case, I manage my own SOA name servers via a set of internet load balancers/WAN aggregators and use a 30 second TTL for all DNS records.  This enables me to have better control on DNS changes with almost immediate propagation.
0
 

Author Comment

by:readymade
ID: 37798889
So if my DR network is a different IP range than my office, and these vm's come up with the original ip's (different than dr network).....how do I get that to work?

That would essentially be bringing up servers with IPs that don't match the network they are on.  This is my issue.
0
 
LVL 8

Expert Comment

by:gsmartin
ID: 37799015
In my case, I spanned IP Addressing scheme (at Layer 2), and VLANs across both sites that eliminates the need to change  IP and DNS settings.    

For your configuration, VMware has a white paper that shows you how to automate and customize Virtual Machine network properties.  So when systems come online IP, hostnames, and DNS settings are updated on each system as you define in the VMware SRM - VMware Infrastructure Client Windows Guest Customization Wizard.  The systems will then update Dynamic DNS as systems become available.  Please see the referenced links below for details.

http://communities.vmware.com/docs/DOC-11516
http://communities.vmware.com/servlet/JiveServlet/download/11516-1-32121/VMW_09Q2_WP_VCENTER_SRM_EN_P37_R2.zip

FYI... Caveat to automating via VMware's SRM is if you have applications, scripts, and other resources pointing to the original IP or Host name you will most likely encounter issues.  Primarily, this could be a problem for Active Directory, Mail, databases, and other services that have host name and IP dependencies.  To help with some of the DNS issues you can create DNS entries that are independent of a systems host name where only the IP address needs to be changed.  

However, the VMware SRM customization is not my preferred approach for a good DR strategy.  Spanning your IP addressing at Layer 2 across both sites allows for a more seamless configuration for all servers and services involved without all of the headache and additional work.  

Otherwise, you will most likely need to add a section in your DR plan that factors in the time needed to address any configuration changes.  You will need to do a VMware SRM test that brings everything online in at the DR site in a VMware test bubble with the new settings.  You will need test all services in the bubble to ensure you don't encounter issues.  However, I wouldn't expect this configuration to be seamless.
0
 
LVL 8

Expert Comment

by:gsmartin
ID: 37799028
In my case, I am not using the customization.  However, you will need to customize your setting in SRM as indicated in my previous post to avoid this issue.
0

Featured Post

Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

Join & Write a Comment

Moving your enterprise fax infrastructure from in-house fax machines and servers to the cloud makes sense — from both an efficiency and productivity standpoint. But does migrating to a cloud fax solution mean you will no longer be able to send or re…
In the modern office, employees tend to move around the workplace a lot more freely. Conferences, collaborative groups, flexible seating and working from home require a new level of mobility. Technology has not only changed the behavior and the expe…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now