egoselfaxis
asked on
Need to find a reliable way to download and migrate a large 40+ GB WordPress site
I have a very large WordPress site that's set up on a Linux based shared hosting environment (40+ GB) .. and I need to find a proper way to migrate it -- in it's entirety -- to a different web hosting provider.
The problem that I'm having is that I can't seem to download the entire site in one sitting. Every time I try to download to site to my local machine via FTP (using Filezilla) .. it's not indexing all of the files, and what I end up with is a directory that's only about 15GB in size.
Note that I've also tried creating a full backup archive of the site via the site's hosting cpanel .. but that isn't working at all. I've tried initiating the backup archive in the early evening, and in the morning, the ZIP file that I'm expecting to find in the site root is no where to be found.
I'm reluctant to experiment or waste time with any of those WordPress "Backup" or "Duplicator" plugins, as they work pretty much the same way, where they attempt to create giant ZIP file archives, which I'm nearly certain will fail .. and will also hog all of the server's CPU's resources in the process.
I've already contacted technical support about this problem, and they're annoyingly clueless as to why this is happening. It's almost like they're holding the site hostage or something, and this is quickly turning into an emergency situation. We need to get this site off of the shared hosting environment and set up on a VPS of some kind on another host ASAP.
Why is this happening? And what alternatives do I have besides FTP to get these files onto my local machine --- or transferred directly over to the other host?
Thanks,
- Yvan
The problem that I'm having is that I can't seem to download the entire site in one sitting. Every time I try to download to site to my local machine via FTP (using Filezilla) .. it's not indexing all of the files, and what I end up with is a directory that's only about 15GB in size.
Note that I've also tried creating a full backup archive of the site via the site's hosting cpanel .. but that isn't working at all. I've tried initiating the backup archive in the early evening, and in the morning, the ZIP file that I'm expecting to find in the site root is no where to be found.
I'm reluctant to experiment or waste time with any of those WordPress "Backup" or "Duplicator" plugins, as they work pretty much the same way, where they attempt to create giant ZIP file archives, which I'm nearly certain will fail .. and will also hog all of the server's CPU's resources in the process.
I've already contacted technical support about this problem, and they're annoyingly clueless as to why this is happening. It's almost like they're holding the site hostage or something, and this is quickly turning into an emergency situation. We need to get this site off of the shared hosting environment and set up on a VPS of some kind on another host ASAP.
Why is this happening? And what alternatives do I have besides FTP to get these files onto my local machine --- or transferred directly over to the other host?
Thanks,
- Yvan
ASKER
>> I'm reluctant to experiment or waste time with any of those WordPress "Backup" or "Duplicator" plugins, as they work pretty much the same way, where they attempt to create giant ZIP file archives, which I'm nearly certain will fail .. and will also hog all of the server's CPU's resources in the process.
Another option could be zipping your website content and database dump in multiple 2GB files and then copy them to the new VPS using scp or rsync. But it will require some work to put everything working again, like restoring the database and credentials, setting filesystem permissions, etc.
ASKER
That wouldn't be possible on a shared hosting environment, as they don't allow root or shell access.
SFTP the files off in chunks. That's your best bet. audit the files that are there, and break it up into moves chunks of 1-2gb. That's or tar/zip the files on the server and move the tar/zip file off the server as one large file. don't forget to get a DB dump too. When you zip the files do NOT use cpanel, ssh in and do it via terminal. you may still have to do it in chunks though, 40gb is WAY outside the norm of ANY wordpress install i've ever seen.
Instead of FTP'ing to your local machine, transfer the site from server-to-server. The connection between the servers will be much faster since it won't need to touch any last-mile networks.
You could do this by SSH'ing into one of the servers and using Rsync, to sync the data from one to the other.
https://en.wikipedia.org/wiki/Rsync
...you'll be able to maintain all the file permissions, directory structures, etc. It's really the "right" way to do this. There are even rsync gui clients on the market if you're not comfortable with shell prompts: https://acrosync.com/home.html
Alternatively you could use an FTP client that supports FXP (assuming both hosts also support it):
https://en.wikipedia.org/wiki/File_eXchange_Protocol
...however FXP has been known to be a security issue, so there's a chance your host doesn't allow it.
You could do this by SSH'ing into one of the servers and using Rsync, to sync the data from one to the other.
https://en.wikipedia.org/wiki/Rsync
...you'll be able to maintain all the file permissions, directory structures, etc. It's really the "right" way to do this. There are even rsync gui clients on the market if you're not comfortable with shell prompts: https://acrosync.com/home.html
Alternatively you could use an FTP client that supports FXP (assuming both hosts also support it):
https://en.wikipedia.org/wiki/File_eXchange_Protocol
...however FXP has been known to be a security issue, so there's a chance your host doesn't allow it.
I purchased Duplicator Pro by Snap Creek and find it excellent. I've found the author to be very responsive to support requests too.
To avoid the oversized failing zip size, you could simply copy your website in chunks, then restore them all at the other host.
In so far as this problem - "and will also hog all of the server's CPU's resources in the process"
It can be avoided by setting Duplicator Pro's limit to throttle the amount of server resources you want it to use.
Hope that's helpful.
Regards, Andrew
To avoid the oversized failing zip size, you could simply copy your website in chunks, then restore them all at the other host.
In so far as this problem - "and will also hog all of the server's CPU's resources in the process"
It can be avoided by setting Duplicator Pro's limit to throttle the amount of server resources you want it to use.
Hope that's helpful.
Regards, Andrew
Dear egoselfaxis
It seems like you have already tried everything
I used to have a very big web site, then i checked why it was so big...
You don't probably need all the 40 gb files.
If i were you, i would list all my files according to their size and delete unnecessary files. You will be amazed after you see that you do not have to keep most of the old files.
Also try to resize the picture files with php.
I would recommend you to dowload all big video files and put it on your disk or another server so that you can delete them before the back up to lighten your backup process
It seems like you have already tried everything
I used to have a very big web site, then i checked why it was so big...
You don't probably need all the 40 gb files.
If i were you, i would list all my files according to their size and delete unnecessary files. You will be amazed after you see that you do not have to keep most of the old files.
Also try to resize the picture files with php.
I would recommend you to dowload all big video files and put it on your disk or another server so that you can delete them before the back up to lighten your backup process
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Shaun .. would I need to get the "Premium" version in order to be able to create the multiple, smaller zip files during the transmission? Or is that achievable using the Free version?
-- Yvan
-- Yvan
If the URL stays the same you should be able to use the free version
ASKER
Thanks Shaun. Although it appears that the free version of the plugin might do the trick .. as it turns out, my hosting company is causing me headaches by claiming that I'm violating their "Unlimited Disk Space" policy by trying to store and server up files which are "not essential to the proper operation of the website". in other words .. if I can't get the 28+ gb worth of ZIP files off the server on the same day .. they end up physically deleting the files. I'm incredibly annoyed, .. but that's the problem I'm dealing with.
I think what I'm going to need to do is set up a VPS on another host .. and somehow make use of it's shell access to download the data from the current shared hosting environment onto it .. preferably without having the first create one or more ZIP files.
I believe that I can use "wget" via the command line to do this .. although I'm not sure as to what the specific syntax is that I should use to retrieve an entire directory tree and all of it's subfolders and contents. For example .. what's the correct way to do something like this?
wget https://mydomain.com/wp-content/uploads/*
(Where "*" represents all folders, subfolders and files)
Thanks,
_ Yvan
I think what I'm going to need to do is set up a VPS on another host .. and somehow make use of it's shell access to download the data from the current shared hosting environment onto it .. preferably without having the first create one or more ZIP files.
I believe that I can use "wget" via the command line to do this .. although I'm not sure as to what the specific syntax is that I should use to retrieve an entire directory tree and all of it's subfolders and contents. For example .. what's the correct way to do something like this?
wget https://mydomain.com/wp-content/uploads/*
(Where "*" represents all folders, subfolders and files)
Thanks,
_ Yvan
ASKER
Anyone?
Updraft can stream the backup to other server so the hosting issue should not occur
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
>> Just to reiterate my original comment, you should consider using rsync for this. Let your current host know that you need to migrate these files and you'd like to have shell access temporarily to accommodate the request.
Lucas --- I really wish it was that simple. I've already inquired with them about this, and they've made it very clear to me that they refuse to cooperate in this regard. So as much as I agree with you, .. it just simply isn't an option.
Shaun -- I saw no indication that I'd be able to stream the data directly from one server to the other. It appears that I'd first need to create multiple 400gb zip files .. and only once those file have been created would I be able to transfer them anywhere. What's more -- I was recently forced to permanently disable CRON in my wp-config.php file -- using "define('DISABLE_WP_CRON',
As best as I can tell, I'm going to have to request the technical assistance of the new web hosting provider to retrieve the contents of our uploads folder and transfer them over to the new VPS .. using whatever means necessary.
Thanks again for your help.
- Yvan
Lucas --- I really wish it was that simple. I've already inquired with them about this, and they've made it very clear to me that they refuse to cooperate in this regard. So as much as I agree with you, .. it just simply isn't an option.
Shaun -- I saw no indication that I'd be able to stream the data directly from one server to the other. It appears that I'd first need to create multiple 400gb zip files .. and only once those file have been created would I be able to transfer them anywhere. What's more -- I was recently forced to permanently disable CRON in my wp-config.php file -- using "define('DISABLE_WP_CRON',
As best as I can tell, I'm going to have to request the technical assistance of the new web hosting provider to retrieve the contents of our uploads folder and transfer them over to the new VPS .. using whatever means necessary.
Thanks again for your help.
- Yvan
Have you tried some Wordpress Backup plugin? Take a look at Vaultpress (https://vaultpress.com)
It seems to be nice, and costs only $3.50/month. Here you can find how to migrate your site to a new host:
https://help.vaultpress.com/migrate-to-a-new-host/
Best Regads,