Replication software for 15 terabytes of data

Posted on 2011-05-12
Last Modified: 2012-05-11
We are looking for either a hard-ware or software solution that can replicate large amounts of data 10-15 terabytes from one physical server to another. The servers themselves have raid 6 and are on a gigabyte switched network. We are hoping to find something that can replicate the changes of one windows volume to another volume on a separate physical server. It doesn't have to be an imaging solution and could be a flat file copy solution.

Speed/convergence  is our biggest concern as our current solutions takes several weeks to sync all of the data and repeats this process constantly which means we have weeks at a time in which the data on our front end server isn't sync'd with the back end server.

If anyone has any recommendations whether it be hardware, software, virtual or physical I am open to all solutions.. Thank-you.
Question by:Axis52401
    LVL 2

    Author Comment


    The front end server and backup server are in different racks and connected over a switched gigabyte network.
    LVL 8

    Expert Comment

    I would say a de-duplication SAN with a replicate on the other server. NetApp makes a good de-duplication SAN and their replication is great.
    LVL 46

    Expert Comment

    Just put in a pair if infiniband controller cards and run a cable.  Depending on the O/S; bus structure; and model you get, you can easily transfer 10+ GB/sec, (much more than the capacity of the hardware to read or write from the drives).

    LVL 12

    Expert Comment

    i would second dlethe's suggestion.

    infiniband FDR 12X or EDR 12X is the way to go for you.

    there are many manufacturers. Mellanox and Texas Memory comes to mind right now.
    LVL 3

    Expert Comment

    What is your budget to fix the issue?  Any ballpack number will do, it just helps to give an idea of what kind of solution to propose.

    What is the purpose of the data copy?  Is the secondary copy read/write (i.e. a dev/test server with current data)?  Or is it jread-only?  Do you want the secondary server to be in synch as close to the primary server possible, or do you want data copied over from points-in-time of the primary?
    LVL 2

    Author Comment

    I really appreciate everyone's responses so far as I am trying to gather as many options as possible.
    Ballpark budget is hard to say. The owner makes those decisions but if the solution was able to keep us within a few days then we would be willing  to spend about $2000 per server or $18,000 total.

    The purpose of the data copy is to provide redundancy on for our front end servers. So in the event something catastrophic happens we can flip-a-switch and the backup server takes over for the primary server. So the backup server should be as close to real-time as possible.
    LVL 55

    Expert Comment

    doubletake is probably the most popular Windows to Windows replicator, doesn't take much bandwidth after the initial replication since only changes are sent across the wire.
    LVL 20

    Expert Comment

    I agree with Doubletake (also sold by HP as Storage Mirroring) -- it gives you an efficient copy of changed blocks to keep a copy of your data on another server, and you can set the second server up as a fail-over in case the first server fails (many-to-one if desired).

    Do make sure that the owner knows that you still need backup.  These copy/mirroring solutions won't do you much good if there's accidental deletion, or corruption (whether by HW or SW) of the original data.  And... should something happen to the data center (fire, flood, etc.)... you'll appreciate having a second copy of the data offsite.
    LVL 25

    Expert Comment

    Implement your core system (DB/Apps) on a SAN-SAN replica and via LUN copying have online replication from Prod site to DR site via dark fiber or Giga media or high speed (depends on your traffic).  Check all vendors for replication method:

    - EMC Symmetrix Remote Data Facility (SRDF) for both synchronous and asynchronous replication, and EMC MirrorView for synchronous and asynchronous replication of Clariion systems.

    - Hitachi Data Systems TrueCopy for synchronous replication and Hitachi Data Systems Universal Replicator software for asynchronous replication.

    -HP StorageWorks XP Continuous Access and Continuous Access EVA for both synchronous and asynchronous replication for HP XP and EVA arrays.
    - IBM  Metro Mirror for synchronous replication and IBM Global Mirror for asynchronous replication.

    - NetApp SnapMirror for synchronous and asynchronous block-based replication, and NetApp SnapVault for file-based replication.

    I do Metro Mirror / Global Mirror and MGM from IBM.
    LVL 3

    Accepted Solution

    EMC also has RepliStor.  We use it for some of our smaller sites that only have one or two servers that need a small amount of data replicated.

    I don't know if you need all of the bandwidth people are talking about... It looks like you're doing a weekly full copy to keep it in synch.

    Any Replication software you use will only need one full copy, then everything else after that will be changes.  
    You could probably just use the 1 gig, or buy a small 10GB Ethernet switch.   Or just buy a 10GB Ethernet card for each server and direct-connect them to their DR server.

    If this stuff is attached to a storage array (SAN/NAS), then you could use some of the native tools within the array to replicate.

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    What Security Threats Are You Missing?

    Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

    Are you looking to recover an email message or a contact you just deleted mistakenly? Or you are searching for a contact that you erased from your MS Outlook ‘Contacts’ folder and now realized that it was important.
    The article will include the best Data Recovery Tools along with their Features, Capabilities, and their Download Links. Hope you’ll enjoy it and will choose the one as required by you.
    To efficiently enable the rotation of USB drives for backups, storage pools need to be created. This way no matter which USB drive is installed, the backups will successfully write without any administrative intervention. Multiple USB devices need t…
    This tutorial will show how to inventory, catalog, and restore media from legacy versions of Backup Exec into both 2012 and 2014 versions of the software. Select Storage from the tabs along the ribbon bar as the top: Ensure the proper storage devi…

    759 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    11 Experts available now in Live!

    Get 1:1 Help Now