Replication software for 15 terabytes of data

Posted on 2011-05-12
Medium Priority
Last Modified: 2012-05-11
We are looking for either a hard-ware or software solution that can replicate large amounts of data 10-15 terabytes from one physical server to another. The servers themselves have raid 6 and are on a gigabyte switched network. We are hoping to find something that can replicate the changes of one windows volume to another volume on a separate physical server. It doesn't have to be an imaging solution and could be a flat file copy solution.

Speed/convergence  is our biggest concern as our current solutions takes several weeks to sync all of the data and repeats this process constantly which means we have weeks at a time in which the data on our front end server isn't sync'd with the back end server.

If anyone has any recommendations whether it be hardware, software, virtual or physical I am open to all solutions.. Thank-you.
Question by:Axis52401

Author Comment

ID: 35748804

The front end server and backup server are in different racks and connected over a switched gigabyte network.

Expert Comment

ID: 35748926
I would say a de-duplication SAN with a replicate on the other server. NetApp makes a good de-duplication SAN and their replication is great.
LVL 47

Expert Comment

ID: 35749090
Just put in a pair if infiniband controller cards and run a cable.  Depending on the O/S; bus structure; and model you get, you can easily transfer 10+ GB/sec, (much more than the capacity of the hardware to read or write from the drives).

Nothing ever in the clear!

This technical paper will help you implement VMware’s VM encryption as well as implement Veeam encryption which together will achieve the nothing ever in the clear goal. If a bad guy steals VMs, backups or traffic they get nothing.

LVL 12

Expert Comment

ID: 35749554
i would second dlethe's suggestion.

infiniband FDR 12X or EDR 12X is the way to go for you.

there are many manufacturers. Mellanox and Texas Memory comes to mind right now.

Expert Comment

ID: 35750356
What is your budget to fix the issue?  Any ballpack number will do, it just helps to give an idea of what kind of solution to propose.

What is the purpose of the data copy?  Is the secondary copy read/write (i.e. a dev/test server with current data)?  Or is it jread-only?  Do you want the secondary server to be in synch as close to the primary server possible, or do you want data copied over from points-in-time of the primary?

Author Comment

ID: 35750737
I really appreciate everyone's responses so far as I am trying to gather as many options as possible.
Ballpark budget is hard to say. The owner makes those decisions but if the solution was able to keep us within a few days then we would be willing  to spend about $2000 per server or $18,000 total.

The purpose of the data copy is to provide redundancy on for our front end servers. So in the event something catastrophic happens we can flip-a-switch and the backup server takes over for the primary server. So the backup server should be as close to real-time as possible.
LVL 56

Expert Comment

ID: 35753814
doubletake is probably the most popular Windows to Windows replicator, doesn't take much bandwidth after the initial replication since only changes are sent across the wire.
LVL 21

Expert Comment

ID: 35754115
I agree with Doubletake (also sold by HP as Storage Mirroring) -- it gives you an efficient copy of changed blocks to keep a copy of your data on another server, and you can set the second server up as a fail-over in case the first server fails (many-to-one if desired).

Do make sure that the owner knows that you still need backup.  These copy/mirroring solutions won't do you much good if there's accidental deletion, or corruption (whether by HW or SW) of the original data.  And... should something happen to the data center (fire, flood, etc.)... you'll appreciate having a second copy of the data offsite.
LVL 25

Expert Comment

ID: 35755377
Implement your core system (DB/Apps) on a SAN-SAN replica and via LUN copying have online replication from Prod site to DR site via dark fiber or Giga media or high speed (depends on your traffic).  Check all vendors for replication method:

- EMC Symmetrix Remote Data Facility (SRDF) for both synchronous and asynchronous replication, and EMC MirrorView for synchronous and asynchronous replication of Clariion systems.

- Hitachi Data Systems TrueCopy for synchronous replication and Hitachi Data Systems Universal Replicator software for asynchronous replication.

-HP StorageWorks XP Continuous Access and Continuous Access EVA for both synchronous and asynchronous replication for HP XP and EVA arrays.
- IBM  Metro Mirror for synchronous replication and IBM Global Mirror for asynchronous replication.

- NetApp SnapMirror for synchronous and asynchronous block-based replication, and NetApp SnapVault for file-based replication.

I do Metro Mirror / Global Mirror and MGM from IBM.

Accepted Solution

driskollt earned 2000 total points
ID: 35777203
EMC also has RepliStor.  We use it for some of our smaller sites that only have one or two servers that need a small amount of data replicated.

I don't know if you need all of the bandwidth people are talking about... It looks like you're doing a weekly full copy to keep it in synch.

Any Replication software you use will only need one full copy, then everything else after that will be changes.  
You could probably just use the 1 gig, or buy a small 10GB Ethernet switch.   Or just buy a 10GB Ethernet card for each server and direct-connect them to their DR server.

If this stuff is attached to a storage array (SAN/NAS), then you could use some of the native tools within the array to replicate.

Featured Post

What is SQL Server and how does it work?

The purpose of this paper is to provide you background on SQL Server. It’s your self-study guide for learning fundamentals. It includes both the history of SQL and its technical basics. Concepts and definitions will form the solid foundation of your future DBA expertise.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The main intent of this article is to make you aware of ‘Exchange fail to mount’ error, its effects, causes, and solution.
Steps to fix error: “Couldn’t mount the database that you specified. Specified database: HU-DB; Error code: An Active Manager operation fail”
This tutorial will walk an individual through the process of configuring basic necessities in order to use the 2010 version of Data Protection Manager. These include storage, agents, and protection jobs. Launch Data Protection Manager from the deskt…
Despite its rising prevalence in the business world, "the cloud" is still misunderstood. Some companies still believe common misconceptions about lack of security in cloud solutions and many misuses of cloud storage options still occur every day. …
Suggested Courses

850 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question