Solved

Cloning a remote server (Ubuntu) into a local VM

Posted on 2013-05-16
22
670 Views
Last Modified: 2013-05-17
We have a server held in a datacenter a few hundred miles away.

We need to clone the hard disks machine to a local server (We have a matching machine locally) in order to run some testing on it without affecting the live server. The question is, how can we do this?

The server is running Ubuntu 10.04 and has 3 hard disks (sda, sdb, sdc) - it boots from sdc.

I have no issue with sda and sdb as I can use dd to clone them, but I'm not sure how to go about cloning sdc.

My questions are..

1) Best way to clone over ssh using dd - preferably compressing the stream to make the transfer quicker (About 120GB to transfer)?
2) Best way to clone the main hard disk (sdc) over ssh (About 80GB used). I cannot shut the server down and boot from a live CD or usb stick?

Thanks
0
Comment
Question by:Rabbit80
  • 8
  • 7
  • 4
  • +1
22 Comments
 
LVL 8

Assisted Solution

by:Pepe2323
Pepe2323 earned 200 total points
ID: 39171535
Try this way , dd is cloning and sending the info by ssh


dd if=/dev/sda | gzip | ssh root@target 'gzip -d | dd of=/dev/sda'
0
 
LVL 1

Author Comment

by:Rabbit80
ID: 39171542
Thanks for that!

Will this work for the OS drive?
0
 
LVL 8

Expert Comment

by:Pepe2323
ID: 39171568
Yes but you will need to boot the target server with a live CD linux version

For example if your plan is to have like an image of the disk on a third server you can do this:

This will copy the entire drive into a file on the remote machine:

$ dd if=/dev/sda | ssh user@server 'dd of=sda.img'

To restore a local drive from the image on the server, reverse the command:

$ ssh user@server 'dd if=sda.img' | dd of=/dev/sda

As you can see is very similar as i described previously
0
 
LVL 1

Author Comment

by:Rabbit80
ID: 39171577
That's the point I was trying to make in my original post.. I cannot boot the server from a live CD as a) we cannot take it offline for that long and b) we don't have easy or convenient physical access to it.
0
 
LVL 8

Expert Comment

by:Pepe2323
ID: 39171617
I thought you ware talking about the source server and not target server.

So in that case how you plan to put the information on the target server if there is not a Os runnig ??
0
 
LVL 1

Author Comment

by:Rabbit80
ID: 39171647
Ah - I see.. Reading comprehension failure on my part!

I thought you were talking about booting the source server from the Live CD!

Will dd create a suitable copy from the source if it is live?
0
 
LVL 8

Expert Comment

by:Pepe2323
ID: 39171667
correct  dd is raw copy of the entry disk so if you are doing a clone copy the target disk also need to be identical  as source disk
0
 
LVL 3

Assisted Solution

by:GhostInTheMacheen
GhostInTheMacheen earned 100 total points
ID: 39172820
I have to disagree. dd is a raw copy of the information on disk as it is being read.

A live disk is changing its contents as you're reading it. Best case, you miss a few changed blocks, worst case you miss the filesystem's journal updates (assuming a journaled filesystem like ext3/4).

If you're running your source machine's sdc partitions on XFS or using logical volumes (LVM) to provide snapshot capability then just pull a snapshot.

Assuming that isn't the case, though, I usually default to multiple back-to-back rsync calls in cases like this. Duplicate the disk as closely as possible (same speed, partition size(s), etc.), then rsync the content over (use -z to enable compression). As soon as the rsync completes, run it again.

The second (and sometimes best to do a third) rsync run will catch any files that had read locks on them the first time around.

Is it a perfect clone? Certainly not. But then again every drive itself will perform a little differently. If you're really concerned about getting an exact copy, then you either need the source machine to be running a snapshot-capable partition scheme or you need to be running a RAID mirror and have a drive available to pull offline to treat like a snapshot.
0
 
LVL 3

Expert Comment

by:GhostInTheMacheen
ID: 39172839
As an additional note, if this is a production web server odds are you aren't updating much on it except logs and possibly the DB (though that should really be housed in another machine, or at least be snapshot capable).

Rsync will only pull the changed files on subsequent runs. You could probably just use rsync's --exclude option to skip /var/log (and your DB storage directory if necessary), as the existing logs will have no practical impact on the test server's performance. If a subsequent run returns no changes then no need to go any further.

If your database is on the same machine and is not running on a snapshot-capable partition (1) change that when you can and (2) you can always perform a binary export, import it, and recreate the database locally.
0
 
LVL 1

Author Comment

by:Rabbit80
ID: 39173064
Thanks Ghost - you confirmed what I was thinking which is why I asked the question in the first place. Any guidance on using rsync? Its not a tool I have used before.
0
Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

 
LVL 27

Accepted Solution

by:
serialband earned 200 total points
ID: 39173536
rsync -az / remote_host_name_or_IP:/


If you don't have rsync services turned on, just tunnel over ssh
rsync -az -e 'ssh -l root' / remote_host_name_or_IP:/

You may want to exclude /etc/ if you've changed hostname and IP settings or other configurations.  I've included /var/tmp to show how to have multiple excludes.
rsync -az -e 'ssh -l root' --exclude '/etc'/ --exclude '/var/tmp' remote_host_name_or_IP:/


From man rsync
-a = archive
-z = zip (compress)
-v = verbose
-e = execute rsh command
--exclude = exclude directive  You can have multiple excludes.
0
 
LVL 1

Author Comment

by:Rabbit80
ID: 39173550
Have split the points as there were 2 parts to the question. Thanks guys - your help is appreciated!
0
 
LVL 3

Expert Comment

by:GhostInTheMacheen
ID: 39173629
I suppose I don't really care much about the points, but considering I answered the question I would have appreciated at least a bit of credit as an assist...

I guess that's what happens when I take the wife out for date night, heh.

So long as your problem's solved that's what's really important ;)
0
 
LVL 1

Author Comment

by:Rabbit80
ID: 39173643
Sorry Ghost - I just assumed it was you that posted the final solution.. didn't look at the name! (confession - its nearly 4am and i'm a little blurry eyed!)

Thanks again for your help.
0
 
LVL 3

Expert Comment

by:GhostInTheMacheen
ID: 39173666
Ouch, late night. Get some sleep. But in the morning, perhaps you'll remember that RA button up top can summon a mod to change your selections... if you care about karma and the like ;)

The experts love to help, but the "street cred" is a nice reward. Be kind, rewind =)
0
 
LVL 1

Author Closing Comment

by:Rabbit80
ID: 39174285
Points reassigned and Ghost given credit!

Thanks to everyone!
0
 
LVL 3

Expert Comment

by:GhostInTheMacheen
ID: 39174603
Thank you, and I mean that with all honesty.

I've become a little too accustomed to askers bailing out on questions (in other forums) as soon as a workable solution is posted. Being able to scan through the solutions and quickly pick out the useful answers is an invaluable time saver for most users.

The attention to completion from both sides is what I appreciate about EE =)
0
 
LVL 27

Expert Comment

by:serialband
ID: 39175328
@Ghost

It wasn't my intention to come in to take the points.  I'm not even really here for points except to keep a minimum to be able to read the solutions to the other questions.  I sometimes just respond after a solution is accepted just to provide information.

You'll also find that up to half the questions become neglected.  They've only recently deleted some very old neglected questions (over 10 years) when they updated the interface a short while ago to allow the mods to more easily get to them.  I got a few emails about them a while back.
0
 
LVL 3

Expert Comment

by:GhostInTheMacheen
ID: 39175971
@serialband

I know that wasn't your intent. I catch up on people's histories pretty quickly when I see their names pop up a few times in the same questions I'm interested in =)

I don't have any interest in the points or even attaining/maintaining Expert status for this account. My interests are solely placed in helping people get their problems solved, getting to know the community, and maintaining incentive for others to do the same.

I've been around... I know a good number of the newer Experts really do care about the points, and more importantly, those searching for answers want to find them quickly and get on with their day.

Ensuring the right solutions are flagged as answers / assists facilitates both, and I think it's beneficial to occasionally remind the community about keeping a tidy house so we can all enjoy it -- and make the the CVs' jobs a little easier at the same time ;)
0
 
LVL 1

Author Comment

by:Rabbit80
ID: 39175997
I just simply appreciate that all you guys took the time to help. . It would have taken me a lot longer otherwise. It was simply a mistake on my part not realising we had a third expert in the conversation else I would have credited each of you - I had some other unrelated issues which required urgent attention hence the late night. . And its a pretty easy mistake to make.

The copy job is going to happen over the weekend although I have tested the methods laid out here and now have the confidence to perform the task.

Thanks again to all of you.
0
 
LVL 3

Expert Comment

by:GhostInTheMacheen
ID: 39176052
Well if you do run into any problems with the data migration, I'm positive there will be someone here that can help.

Best of luck to you =)
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Suggested Solutions

The purpose of this article is to demonstrate how we can use conditional statements using Python.
The Delta outage: 650 cancelled flights, more than 1200 delayed flights, thousands of frustrated customers, tens of millions of dollars in damages – plus untold reputational damage to one of the world’s most trusted airlines. All due to a catastroph…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now