CoffeeBlack
asked on
2 Million files, 48 hours to copy
Recently I had to move a large number of files from one 64bit win system to another. I was moving around 2 million files, 300kb to 16mb in size, most were around 4mb. About 1.4tb total. And I was trying to do it with a simple copy and paste over a weekend. That worked about as well as you might guess...
The reasons for moving the data, were simple, we were out of space on that server. I moved the data from a number of 600gb 10k drives, to a number of 4tb 7.2k drives. The rate of growth was such that it just didn't make sense to leave the files on original setup, in the server that they were housed in. Eventually i'll have to do this again, as we move them to a larger scale SAN.
This process took 3 weekends, because the process failed over and again, then didn't leave me enough time to get in another run at it. It would get part way through the files (16-30 hours in) and just crash out. I looked at couple other programs, and eventually used SyncToy to avoid making it 4 weekends, by simply copying what hadn't been copied already. However even with only copying a lesser amount of the total files, it crashed out as well, and I eventually gave up on trying to do it in one set, and broke up the group of files in to 100k file increments (ish). And that worked, I could verify that the files came across, that I had the same amount of files on both sides upon completion, and that the size was the same as well (also ran chkdsk against the files).
So getting back to "Eventually i'll have to do this again...", I don't want to break the files up, I want to start the job and it finish on its own. I don't want to go in every 3 hours and start a new batch of 100k files. Do you have any recommendations for moving a large amount of files from one 64 bit windows system to another?
The reasons for moving the data, were simple, we were out of space on that server. I moved the data from a number of 600gb 10k drives, to a number of 4tb 7.2k drives. The rate of growth was such that it just didn't make sense to leave the files on original setup, in the server that they were housed in. Eventually i'll have to do this again, as we move them to a larger scale SAN.
This process took 3 weekends, because the process failed over and again, then didn't leave me enough time to get in another run at it. It would get part way through the files (16-30 hours in) and just crash out. I looked at couple other programs, and eventually used SyncToy to avoid making it 4 weekends, by simply copying what hadn't been copied already. However even with only copying a lesser amount of the total files, it crashed out as well, and I eventually gave up on trying to do it in one set, and broke up the group of files in to 100k file increments (ish). And that worked, I could verify that the files came across, that I had the same amount of files on both sides upon completion, and that the size was the same as well (also ran chkdsk against the files).
So getting back to "Eventually i'll have to do this again...", I don't want to break the files up, I want to start the job and it finish on its own. I don't want to go in every 3 hours and start a new batch of 100k files. Do you have any recommendations for moving a large amount of files from one 64 bit windows system to another?
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER