robocopy failing on large copy

Hi,

I am after some help with Robocopy.

Basically I have a share with 10 million flat files and 369k folders. this is actually a document management system file store.  I need to copy the entire store entirely not missing a single file and I need to leave the source files in place as well.

I need to move this file store to a new source directory. I have done this in the past and been successful using the command below.


@ehho off
@ECHO     !!!!!!!!!!!!!    RoboCopy - Large file copy - Do you want to continue    !!!!!!!!!!!!!
PAUSE

@echo Copy \\server1\copy-fixing-issue
C:\Windows\SysWOW64\Robocopy.exe \\server1\WSDOCS\DB1 \\server2\WSDOCS2\db1 /MIR /MT *.* *.*  /LOG:"\\server2\WSDOCS2\db1.log"

pause


however now I am hitting a brick wall in that the copy is bombing out complaining robocopy is running out of memory, the sever has adequate memory 32GB, with nothing else on it apps wise etc.  I tried upping it to 128 GB Because the server is VM and it makes no difference. The memory never seems to be busy either unless its re-realising back to the OS once its failed.

I was originally doing this on a server 2012 server so I tried it on  2008 R2 server but it made no difference I still get the same error in the robocopy log

the servers are all on the same network in the same office etc so its not like there is link issues to overcome, any ideas?

once I have done the first initial copy, I then have to offline the document management system out of hours and do the same copy again but a differential so pick up any differences. this has to be right, i cant miss any files...





-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows                              
-------------------------------------------------------------------------------

  Started : Sun Jul 12 17:18:23 2015

   Source : \\server1\WSDOCS\db1\
     Dest : \\server2\WSDOCS2\db1\

    Files : *.*
          
  Options : *.* /S /E /COPY:DAT /PURGE /MIR /MT:8 /R:1000000 /W:30

------------------------------------------------------------------------------

ERROR : Robocopy ran out of memory, exiting.
ERROR : Invalid Parameter #%d : "%s"

ERROR : Invalid Job File, Line #%d :"%s"


  Started : %hs

   Source %c

     Dest %c
speechlysAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

David Johnson, CD, MVPOwnerCommented:
C:\Windows\SysWOW64\Robocopy.exe ?????
Why are you using the 32bit version and not simply robocopy.exe or c:\windows\system32\robocopy.exe ??

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Gerald NegrotaIT ProfessionalCommented:
If server1 and server2 are virtual servers, you can:

1) add another disk to server1 in mirror with the FS store
2) sync
3) brake the mirror
4) move disk to server2

For Robocopy, run batch as Administrator.  I guess that is User Rights related.
Qlemo"Batchelor", Developer and EE Topic AdvisorCommented:
You are calling RoboCopy from the 32bit folder, so it is the 32bit RoboCopy. That one can only use 4 GB of memory.
Though you should think that 4GB is enough to track file info, it might still be too less. Run the native RoboCopy (C:\Windows\System32) in a 64bit cmd.exe to see if it can handle that much files.
I would also not use /MT with more than 4 threads. Usually 2 - 4 are useful, anything more is slowing down the copy and using up more resources without any benefit.

Also always make sure not to use the defaults of /R and /W.
In the first (full) copy, don't use /MIR (replace with /E), and keep the retry parameters at e.g.  /R:5 /W:1 (5 retries, one second pause for retries), so you are not stuck on files you cannot copy at the moment for any reason.

With the final or intermediate (differential) copy it is different in that you should use /MIR, and allow for more retries. But be prepared there might still be files you cannot copy - check the log!
speechlysAuthor Commented:
you don't know how stupid I feel right now, the old server was 32 bit and I have just assumed my old script should work from before as it worked fine. I didn't even think about changing the variable to 64bit completely overlooked ! thanks for pointing it out.....

I have done it before with robocopy with a similar number of flat files it handled it ok...thanks again will let you know if the 64 bit doesn't work but I think it will.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Server 2008

From novice to tech pro — start learning today.