[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 999
  • Last Modified:

Cloning a remote server (Ubuntu) into a local VM

We have a server held in a datacenter a few hundred miles away.

We need to clone the hard disks machine to a local server (We have a matching machine locally) in order to run some testing on it without affecting the live server. The question is, how can we do this?

The server is running Ubuntu 10.04 and has 3 hard disks (sda, sdb, sdc) - it boots from sdc.

I have no issue with sda and sdb as I can use dd to clone them, but I'm not sure how to go about cloning sdc.

My questions are..

1) Best way to clone over ssh using dd - preferably compressing the stream to make the transfer quicker (About 120GB to transfer)?
2) Best way to clone the main hard disk (sdc) over ssh (About 80GB used). I cannot shut the server down and boot from a live CD or usb stick?

Thanks
0
Rabbit80
Asked:
Rabbit80
  • 8
  • 7
  • 4
  • +1
3 Solutions
 
Pepe2323Commented:
Try this way , dd is cloning and sending the info by ssh


dd if=/dev/sda | gzip | ssh root@target 'gzip -d | dd of=/dev/sda'
0
 
Rabbit80Author Commented:
Thanks for that!

Will this work for the OS drive?
0
 
Pepe2323Commented:
Yes but you will need to boot the target server with a live CD linux version

For example if your plan is to have like an image of the disk on a third server you can do this:

This will copy the entire drive into a file on the remote machine:

$ dd if=/dev/sda | ssh user@server 'dd of=sda.img'

To restore a local drive from the image on the server, reverse the command:

$ ssh user@server 'dd if=sda.img' | dd of=/dev/sda

As you can see is very similar as i described previously
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 
Rabbit80Author Commented:
That's the point I was trying to make in my original post.. I cannot boot the server from a live CD as a) we cannot take it offline for that long and b) we don't have easy or convenient physical access to it.
0
 
Pepe2323Commented:
I thought you ware talking about the source server and not target server.

So in that case how you plan to put the information on the target server if there is not a Os runnig ??
0
 
Rabbit80Author Commented:
Ah - I see.. Reading comprehension failure on my part!

I thought you were talking about booting the source server from the Live CD!

Will dd create a suitable copy from the source if it is live?
0
 
Pepe2323Commented:
correct  dd is raw copy of the entry disk so if you are doing a clone copy the target disk also need to be identical  as source disk
0
 
GhostInTheMacheenCommented:
I have to disagree. dd is a raw copy of the information on disk as it is being read.

A live disk is changing its contents as you're reading it. Best case, you miss a few changed blocks, worst case you miss the filesystem's journal updates (assuming a journaled filesystem like ext3/4).

If you're running your source machine's sdc partitions on XFS or using logical volumes (LVM) to provide snapshot capability then just pull a snapshot.

Assuming that isn't the case, though, I usually default to multiple back-to-back rsync calls in cases like this. Duplicate the disk as closely as possible (same speed, partition size(s), etc.), then rsync the content over (use -z to enable compression). As soon as the rsync completes, run it again.

The second (and sometimes best to do a third) rsync run will catch any files that had read locks on them the first time around.

Is it a perfect clone? Certainly not. But then again every drive itself will perform a little differently. If you're really concerned about getting an exact copy, then you either need the source machine to be running a snapshot-capable partition scheme or you need to be running a RAID mirror and have a drive available to pull offline to treat like a snapshot.
0
 
GhostInTheMacheenCommented:
As an additional note, if this is a production web server odds are you aren't updating much on it except logs and possibly the DB (though that should really be housed in another machine, or at least be snapshot capable).

Rsync will only pull the changed files on subsequent runs. You could probably just use rsync's --exclude option to skip /var/log (and your DB storage directory if necessary), as the existing logs will have no practical impact on the test server's performance. If a subsequent run returns no changes then no need to go any further.

If your database is on the same machine and is not running on a snapshot-capable partition (1) change that when you can and (2) you can always perform a binary export, import it, and recreate the database locally.
0
 
Rabbit80Author Commented:
Thanks Ghost - you confirmed what I was thinking which is why I asked the question in the first place. Any guidance on using rsync? Its not a tool I have used before.
0
 
serialbandCommented:
rsync -az / remote_host_name_or_IP:/


If you don't have rsync services turned on, just tunnel over ssh
rsync -az -e 'ssh -l root' / remote_host_name_or_IP:/

You may want to exclude /etc/ if you've changed hostname and IP settings or other configurations.  I've included /var/tmp to show how to have multiple excludes.
rsync -az -e 'ssh -l root' --exclude '/etc'/ --exclude '/var/tmp' remote_host_name_or_IP:/


From man rsync
-a = archive
-z = zip (compress)
-v = verbose
-e = execute rsh command
--exclude = exclude directive  You can have multiple excludes.
0
 
Rabbit80Author Commented:
Have split the points as there were 2 parts to the question. Thanks guys - your help is appreciated!
0
 
GhostInTheMacheenCommented:
I suppose I don't really care much about the points, but considering I answered the question I would have appreciated at least a bit of credit as an assist...

I guess that's what happens when I take the wife out for date night, heh.

So long as your problem's solved that's what's really important ;)
0
 
Rabbit80Author Commented:
Sorry Ghost - I just assumed it was you that posted the final solution.. didn't look at the name! (confession - its nearly 4am and i'm a little blurry eyed!)

Thanks again for your help.
0
 
GhostInTheMacheenCommented:
Ouch, late night. Get some sleep. But in the morning, perhaps you'll remember that RA button up top can summon a mod to change your selections... if you care about karma and the like ;)

The experts love to help, but the "street cred" is a nice reward. Be kind, rewind =)
0
 
Rabbit80Author Commented:
Points reassigned and Ghost given credit!

Thanks to everyone!
0
 
GhostInTheMacheenCommented:
Thank you, and I mean that with all honesty.

I've become a little too accustomed to askers bailing out on questions (in other forums) as soon as a workable solution is posted. Being able to scan through the solutions and quickly pick out the useful answers is an invaluable time saver for most users.

The attention to completion from both sides is what I appreciate about EE =)
0
 
serialbandCommented:
@Ghost

It wasn't my intention to come in to take the points.  I'm not even really here for points except to keep a minimum to be able to read the solutions to the other questions.  I sometimes just respond after a solution is accepted just to provide information.

You'll also find that up to half the questions become neglected.  They've only recently deleted some very old neglected questions (over 10 years) when they updated the interface a short while ago to allow the mods to more easily get to them.  I got a few emails about them a while back.
0
 
GhostInTheMacheenCommented:
@serialband

I know that wasn't your intent. I catch up on people's histories pretty quickly when I see their names pop up a few times in the same questions I'm interested in =)

I don't have any interest in the points or even attaining/maintaining Expert status for this account. My interests are solely placed in helping people get their problems solved, getting to know the community, and maintaining incentive for others to do the same.

I've been around... I know a good number of the newer Experts really do care about the points, and more importantly, those searching for answers want to find them quickly and get on with their day.

Ensuring the right solutions are flagged as answers / assists facilitates both, and I think it's beneficial to occasionally remind the community about keeping a tidy house so we can all enjoy it -- and make the the CVs' jobs a little easier at the same time ;)
0
 
Rabbit80Author Commented:
I just simply appreciate that all you guys took the time to help. . It would have taken me a lot longer otherwise. It was simply a mistake on my part not realising we had a third expert in the conversation else I would have credited each of you - I had some other unrelated issues which required urgent attention hence the late night. . And its a pretty easy mistake to make.

The copy job is going to happen over the weekend although I have tested the methods laid out here and now have the confidence to perform the task.

Thanks again to all of you.
0
 
GhostInTheMacheenCommented:
Well if you do run into any problems with the data migration, I'm positive there will be someone here that can help.

Best of luck to you =)
0

Featured Post

Important Lessons on Recovering from Petya

In their most recent webinar, Skyport Systems explores ways to isolate and protect critical databases to keep the core of your company safe from harm.

  • 8
  • 7
  • 4
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now