Need to find a reliable way to download and migrate a large 40+ GB WordPress site

I have a very large WordPress site that's set up on a Linux based shared hosting environment (40+ GB)  .. and I need to find a proper way to migrate it -- in it's entirety -- to a different web hosting provider.  

The problem that I'm having is that I can't seem to download the entire site in one sitting.  Every time I try to download to site to my local machine via FTP (using Filezilla) .. it's not indexing all of the files, and what I end up with is a directory that's only about 15GB in size.

Note that I've also tried creating a full backup archive of the site via the site's hosting cpanel .. but that isn't working at all.  I've tried initiating the backup archive in the early evening, and in the morning, the ZIP file that I'm expecting to find in the site root is no where to be found.  

I'm reluctant to experiment or waste time with any of those WordPress "Backup" or "Duplicator" plugins, as they work pretty much the same way, where they attempt to create giant ZIP file archives, which I'm nearly certain will fail .. and will also hog all of the server's CPU's resources in the process.

I've already contacted technical support about this problem, and they're annoyingly clueless as to why this is happening. It's almost like they're holding the site hostage or something, and this is quickly turning into an emergency situation.  We need to get this site off of the shared hosting environment and set up on a VPS of some kind on another host ASAP.

Why is this happening?  And what alternatives do I have besides FTP to get these files onto my local machine --- or transferred directly over to the other host?  

Thanks,
- Yvan
egoselfaxisAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Daniel CenziCommented:
Hi Yvan,
Have you tried some Wordpress Backup plugin? Take a look at Vaultpress (https://vaultpress.com)
It seems to be nice, and costs only $3.50/month. Here you can find how to migrate your site to a new host:
https://help.vaultpress.com/migrate-to-a-new-host/

Best Regads,
egoselfaxisAuthor Commented:
>> I'm reluctant to experiment or waste time with any of those WordPress "Backup" or "Duplicator" plugins, as they work pretty much the same way, where they attempt to create giant ZIP file archives, which I'm nearly certain will fail .. and will also hog all of the server's CPU's resources in the process.
Daniel CenziCommented:
Another option could be zipping your website content and database dump in multiple 2GB files and then copy them to the new VPS using scp or rsync. But it will require some work to put everything working again, like restoring the database and credentials, setting filesystem permissions, etc.
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

egoselfaxisAuthor Commented:
That wouldn't be possible on a shared hosting environment, as they don't allow root or shell access.
Jesse BruffettM&F-ing SorcererCommented:
SFTP the files off in chunks. That's your best bet. audit the files that are there, and break it up into moves chunks of 1-2gb. That's or tar/zip the files on the server and move the tar/zip file off the server as one large file. don't forget to get a DB dump too. When you zip the files do NOT use cpanel, ssh in and do it via terminal. you may still have to do it in chunks though, 40gb is WAY outside the norm of ANY wordpress install i've ever seen.
Lucas BishopClick TrackerCommented:
Instead of FTP'ing to your local machine, transfer the site from server-to-server. The connection between the servers will be much faster since it won't need to touch any last-mile networks.

You could do this by SSH'ing into one of the servers and using Rsync, to sync the data from one to the other.
https://en.wikipedia.org/wiki/Rsync
...you'll be able to maintain all the file permissions, directory structures, etc. It's really the "right" way to do this. There are even rsync gui clients on the market if you're not comfortable with shell prompts: https://acrosync.com/home.html

Alternatively you could use an FTP client that supports FXP (assuming both hosts also support it):
https://en.wikipedia.org/wiki/File_eXchange_Protocol
...however FXP has been known to be a security issue, so there's a chance your host doesn't allow it.
Andrew LeniartIT Consultant & Freelance JournalistCommented:
I purchased Duplicator Pro by Snap Creek and find it excellent. I've found the author to be very responsive to support requests too.

To avoid the oversized failing zip size, you could simply copy your website in chunks, then restore them all at the other host.

In so far as this problem - "and will also hog all of the server's CPU's resources in the process"

It can be avoided by setting Duplicator Pro's limit to throttle the amount of server resources you want it to use.

Duplicator_Pro_Settings.png
Hope that's helpful.

Regards, Andrew
BRMarketingCommented:
Dear egoselfaxis
It seems like you have already tried everything

I used to have a very big web site, then i checked why it was so big...

You don't probably need all the 40 gb files.

If i were you, i would list all my files according to their size and delete unnecessary files. You will be amazed after you see that you do not have to keep most of the old files.

Also try to resize the picture files with php.

I would recommend you to dowload all big video files and put it on your disk or another server so that you can delete them before the back up to lighten your backup process
Shaun VermaakTechnical SpecialistCommented:
I have used Updraft Premium for large backups/migrations. It created a bunch of smaller zip files during the transmission.
https://updraftplus.com/
egoselfaxisAuthor Commented:
Shaun .. would I need to get the "Premium" version in order to be able to create the multiple, smaller zip files during the transmission? Or is that achievable using the Free version?

-- Yvan
Shaun VermaakTechnical SpecialistCommented:
If the URL stays the same you should be able to use the free version
egoselfaxisAuthor Commented:
Thanks Shaun.  Although it appears that the free version of the plugin might do the trick .. as it turns out, my hosting company is causing me headaches by claiming that I'm violating their "Unlimited Disk Space" policy by trying to store and server up files which are "not essential to the proper operation of the website".  in other words .. if I can't get the 28+ gb worth of ZIP files off the server on the same day .. they end up physically deleting the files.  I'm incredibly annoyed, .. but that's the problem I'm dealing with.

I think what I'm going to need to do is set up a VPS on another host .. and somehow make use of it's shell access to download the data from the current shared hosting environment onto it .. preferably without having the first create one or more ZIP files.  

I believe that I can use "wget" via the command line to do this .. although I'm not sure as to what the specific syntax is that I should use to retrieve an entire directory tree and all of it's subfolders and contents.  For example  .. what's the correct way to do something like this?

wget https://mydomain.com/wp-content/uploads/*

(Where  "*" represents all folders, subfolders and files)


Thanks,
_ Yvan
egoselfaxisAuthor Commented:
Anyone?
Shaun VermaakTechnical SpecialistCommented:
Updraft can stream the backup to other server so the hosting issue should not occur
Lucas BishopClick TrackerCommented:
I believe that I can use "wget" via the command line to do this .. although I'm not sure as to what the specific syntax is that I should use to retrieve an entire directory tree and all of it's subfolders and contents.

The recursive flag:
https://www.gnu.org/software/wget/manual/html_node/Recursive-Retrieval-Options.html

For example:
wget -r https://example.com/wp-content/uploads/

Open in new window


There are other things to consider. For example you probably want the --no-parent flag invoked as well:
https://www.gnu.org/software/wget/manual/wget.html#Directory_002dBased-Limits

wget -r --no-parent https://example.com/wp-content/uploads/

Open in new window


Wget is a fairly taxing command though. Don't be surprised if your current host bans your new VPS' IP address when it notices this massive amount of http bandwidth being pulled.

If you do get banned, it probably means you should have limited the download rate (ie. --limit-rate=x)
https://www.gnu.org/software/wget/manual/wget.html#Download-Options

And if your wget http transfer fails (for any reason), I'm fairly certain you can't resume. You'll have to start all over again.

This brings me back to recommending rsync.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
egoselfaxisAuthor Commented:
>> Just to reiterate my original comment, you should consider using rsync for this. Let your current host know that you need to migrate these files and you'd like to have shell access temporarily to accommodate the request.

Lucas --- I really wish it was that simple.  I've already inquired with them about this, and they've made it very clear to me that they refuse to cooperate in this regard.  So as much as I agree with you, .. it just simply isn't an option.

Shaun -- I saw no indication that I'd be able to stream the data directly from one server to the other.  It appears that I'd first need to create multiple 400gb zip files .. and only once those file have been created would I be able to transfer them anywhere.  What's more -- I was recently forced to permanently disable CRON in my wp-config.php file -- using "define('DISABLE_WP_CRON', true)" --- as it was resulting in exceedingly high CPU usage and was crashing the site.  Unfortunately, the UpdraftPlus plugin is apparently reliant upon CRON, which means it's no longer a viable option for me.  

As best as I can tell, I'm going to have to request the technical assistance of the new web hosting provider to retrieve the contents of our uploads folder and transfer them over to the new VPS .. using whatever means necessary.  

Thanks again for your help.

- Yvan
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux

From novice to tech pro — start learning today.