Solved

Why does RoboCopy run out of memory when copying millions of files from machine "a" to machine "b" over LAN

Posted on 2008-10-18
5
3,602 Views
Last Modified: 2012-05-05
I have 2 machines.
Both machines are windows XP Pro.
I copy from one machine to another a folder.
Using the switch on robocopy that tells it to only copy over NEWER files.
I have 256 folders - split up into 16 batch files of 16 lines of robocopy each.
I run all 16 at once.
Over a LAN.
No drive mapping is used.
I use: " robocopy c:\55 \\10.0.0.99\c$\55 /E /XO "
it will do fine unless there are a lot of files.
By "a lot" i mean about 37 million.
It will say "unable to connect to server...waiting 30 secs..."
or sometimes it will say "insuffecient resources waiting 30 secs"
it will wait 30 secs - then it looks like it will send about 5 - 10 more files over
or compare 5 - 10 more files
then it will lock up again. and say "waiting 30 secs"
like the cache or page file or memory pool is not clearing temp memory.

Total files come to about 40 gigs, not very big - but millions of small files.
that need to be compared - and only the newer ones copied over.

Suggestions?
other apps to do this?
or if anyone knows how to do this in VB.net I would appreciate it.

=======================
I am a beginner in Vb.net

Folder A
Folder B

compare folder A to Folder B -
copy all the f iles form Folder B to Folder A that are newer.

and do it for all 256 folders
=======================
0
Comment
Question by:CanvasYou
  • 3
5 Comments
 
LVL 44

Assisted Solution

by:Darr247
Darr247 earned 200 total points
ID: 22749330
Does it give you the same problem if you run the batches one-at-a-time?

Because you could make yet-another batch that 'call's all the other batches in sequence, if they run ok singly.

e.g. say, allcopybatch.bat
CALL BATCH1
CALL BATCH2
CALL BATCH3
et cetera ...

as each one finishes, it returns control to allcopybatch.bat which then invokes the next batch.
You could use FOR and DO, but with 16 batches it's probably just as fast to build it line-by-line.
0
 

Author Comment

by:CanvasYou
ID: 22749705
Thanks for the response.
I have tried that using process.start
and Do while process is not ended.
but i am not that good w/ VB
I have 16 batch files
each one has 16 lines of robocopy
so I am running 16 one after another of this: " robocopy c:\55 \\10.0.0.99\c$\55 /E /XO "

So Batch file #1 has:

robocopy c:\01 \\10.0.0.99\c$\01 /E /XO
robocopy c:\02 \\10.0.0.99\c$\02 /E /XO
robocopy c:\03 \\10.0.0.99\c$\03 /E /XO
robocopy c:\04 \\10.0.0.99\c$\04 /E /XO
robocopy c:\05 \\10.0.0.99\c$\05 /E /XO
ect. all the way to folder 16

then Batch #2 has 16 more lines:
robocopy c:\17 \\10.0.0.99\c$\17 /E /XO

------------------
Total of 16 batch files (256 folders need to be copied)
I want to run them all at once.
------------------
when I do run less of them at once it does not seem to have the error...
------------------
What about in VB though - not using robocopy?
do you know how to search a folder on \\10.0.0.99\c$\FOLDER
and complare it to local folder c:\FOLDER
then compare all the files in both folders
and copy the newer files or non-existant files over?
and still not have it run out of memory but be pretty fast?

Thanks!

0
 
LVL 18

Accepted Solution

by:
sk_raja_raja earned 300 total points
ID: 22749803
1.Ref the below KB for the registry fix for the same issue,
http://support.microsoft.com/kb/304101
0
 

Author Comment

by:CanvasYou
ID: 22750150
sk raja raja:
Will this work on XP Pro You think?
I have seen this same thing on other posts but the KB solution shows SERVER 2k3 and win2k
Thanks for the help
0
 

Author Closing Comment

by:CanvasYou
ID: 31507446
If you have VB tips for file comparison -
ie copy one folder to another folder - and ONLY copy files that are newer than the other machine over -
I would appreciat it.
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

You may have already been in the need to update a whole folder stucture using a script. Robocopy does it well and even provides a list of non-updated files in a log (if asked to). Generally those files that were locked by a user or a process by the …
For both online and offline retail, the cross-channel business is the most recent pattern in the B2C trade space.
This video demonstrates how to create an example email signature rule for a department in a company using CodeTwo Exchange Rules. The signature will be inserted beneath users' latest emails in conversations and will be displayed in users' Sent Items…
You have products, that come in variants and want to set different prices for them? Watch this micro tutorial that describes how to configure prices for Magento super attributes. Assigning simple products to configurable: We assigned simple products…

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now