Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

Why does RoboCopy run out of memory when copying millions of files from machine "a" to machine "b" over LAN

Posted on 2008-10-18
5
Medium Priority
?
4,310 Views
Last Modified: 2012-05-05
I have 2 machines.
Both machines are windows XP Pro.
I copy from one machine to another a folder.
Using the switch on robocopy that tells it to only copy over NEWER files.
I have 256 folders - split up into 16 batch files of 16 lines of robocopy each.
I run all 16 at once.
Over a LAN.
No drive mapping is used.
I use: " robocopy c:\55 \\10.0.0.99\c$\55 /E /XO "
it will do fine unless there are a lot of files.
By "a lot" i mean about 37 million.
It will say "unable to connect to server...waiting 30 secs..."
or sometimes it will say "insuffecient resources waiting 30 secs"
it will wait 30 secs - then it looks like it will send about 5 - 10 more files over
or compare 5 - 10 more files
then it will lock up again. and say "waiting 30 secs"
like the cache or page file or memory pool is not clearing temp memory.

Total files come to about 40 gigs, not very big - but millions of small files.
that need to be compared - and only the newer ones copied over.

Suggestions?
other apps to do this?
or if anyone knows how to do this in VB.net I would appreciate it.

=======================
I am a beginner in Vb.net

Folder A
Folder B

compare folder A to Folder B -
copy all the f iles form Folder B to Folder A that are newer.

and do it for all 256 folders
=======================
0
Comment
Question by:CanvasYou
  • 3
5 Comments
 
LVL 44

Assisted Solution

by:Darr247
Darr247 earned 600 total points
ID: 22749330
Does it give you the same problem if you run the batches one-at-a-time?

Because you could make yet-another batch that 'call's all the other batches in sequence, if they run ok singly.

e.g. say, allcopybatch.bat
CALL BATCH1
CALL BATCH2
CALL BATCH3
et cetera ...

as each one finishes, it returns control to allcopybatch.bat which then invokes the next batch.
You could use FOR and DO, but with 16 batches it's probably just as fast to build it line-by-line.
0
 

Author Comment

by:CanvasYou
ID: 22749705
Thanks for the response.
I have tried that using process.start
and Do while process is not ended.
but i am not that good w/ VB
I have 16 batch files
each one has 16 lines of robocopy
so I am running 16 one after another of this: " robocopy c:\55 \\10.0.0.99\c$\55 /E /XO "

So Batch file #1 has:

robocopy c:\01 \\10.0.0.99\c$\01 /E /XO
robocopy c:\02 \\10.0.0.99\c$\02 /E /XO
robocopy c:\03 \\10.0.0.99\c$\03 /E /XO
robocopy c:\04 \\10.0.0.99\c$\04 /E /XO
robocopy c:\05 \\10.0.0.99\c$\05 /E /XO
ect. all the way to folder 16

then Batch #2 has 16 more lines:
robocopy c:\17 \\10.0.0.99\c$\17 /E /XO

------------------
Total of 16 batch files (256 folders need to be copied)
I want to run them all at once.
------------------
when I do run less of them at once it does not seem to have the error...
------------------
What about in VB though - not using robocopy?
do you know how to search a folder on \\10.0.0.99\c$\FOLDER
and complare it to local folder c:\FOLDER
then compare all the files in both folders
and copy the newer files or non-existant files over?
and still not have it run out of memory but be pretty fast?

Thanks!

0
 
LVL 18

Accepted Solution

by:
sk_raja_raja earned 900 total points
ID: 22749803
1.Ref the below KB for the registry fix for the same issue,
http://support.microsoft.com/kb/304101
0
 

Author Comment

by:CanvasYou
ID: 22750150
sk raja raja:
Will this work on XP Pro You think?
I have seen this same thing on other posts but the KB solution shows SERVER 2k3 and win2k
Thanks for the help
0
 

Author Closing Comment

by:CanvasYou
ID: 31507446
If you have VB tips for file comparison -
ie copy one folder to another folder - and ONLY copy files that are newer than the other machine over -
I would appreciat it.
0

Featured Post

Important Lessons on Recovering from Petya

In their most recent webinar, Skyport Systems explores ways to isolate and protect critical databases to keep the core of your company safe from harm.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article was inspired by a question here at Experts Exchange (http://www.experts-exchange.com/Software/Photos_Graphics/Images_and_Photos/Q_28629170.html). The requirements stated in that question are (1) reduce the file size of a large number of…
For both online and offline retail, the cross-channel business is the most recent pattern in the B2C trade space.
Two types of users will appreciate AOMEI Backupper Pro: 1 - Those with PCIe drives (and haven't found cloning software that works on them). 2 - Those who want a fast clone of their boot drive (no re-boots needed) and it can clone your drive wh…
Please read the paragraph below before following the instructions in the video — there are important caveats in the paragraph that I did not mention in the video. If your PaperPort 12 or PaperPort 14 is failing to start, or crashing, or hanging, …

885 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question