Solved

Why does RoboCopy run out of memory when copying millions of files from machine "a" to machine "b" over LAN

Posted on 2008-10-18
5
3,672 Views
Last Modified: 2012-05-05
I have 2 machines.
Both machines are windows XP Pro.
I copy from one machine to another a folder.
Using the switch on robocopy that tells it to only copy over NEWER files.
I have 256 folders - split up into 16 batch files of 16 lines of robocopy each.
I run all 16 at once.
Over a LAN.
No drive mapping is used.
I use: " robocopy c:\55 \\10.0.0.99\c$\55 /E /XO "
it will do fine unless there are a lot of files.
By "a lot" i mean about 37 million.
It will say "unable to connect to server...waiting 30 secs..."
or sometimes it will say "insuffecient resources waiting 30 secs"
it will wait 30 secs - then it looks like it will send about 5 - 10 more files over
or compare 5 - 10 more files
then it will lock up again. and say "waiting 30 secs"
like the cache or page file or memory pool is not clearing temp memory.

Total files come to about 40 gigs, not very big - but millions of small files.
that need to be compared - and only the newer ones copied over.

Suggestions?
other apps to do this?
or if anyone knows how to do this in VB.net I would appreciate it.

=======================
I am a beginner in Vb.net

Folder A
Folder B

compare folder A to Folder B -
copy all the f iles form Folder B to Folder A that are newer.

and do it for all 256 folders
=======================
0
Comment
Question by:CanvasYou
  • 3
5 Comments
 
LVL 44

Assisted Solution

by:Darr247
Darr247 earned 200 total points
ID: 22749330
Does it give you the same problem if you run the batches one-at-a-time?

Because you could make yet-another batch that 'call's all the other batches in sequence, if they run ok singly.

e.g. say, allcopybatch.bat
CALL BATCH1
CALL BATCH2
CALL BATCH3
et cetera ...

as each one finishes, it returns control to allcopybatch.bat which then invokes the next batch.
You could use FOR and DO, but with 16 batches it's probably just as fast to build it line-by-line.
0
 

Author Comment

by:CanvasYou
ID: 22749705
Thanks for the response.
I have tried that using process.start
and Do while process is not ended.
but i am not that good w/ VB
I have 16 batch files
each one has 16 lines of robocopy
so I am running 16 one after another of this: " robocopy c:\55 \\10.0.0.99\c$\55 /E /XO "

So Batch file #1 has:

robocopy c:\01 \\10.0.0.99\c$\01 /E /XO
robocopy c:\02 \\10.0.0.99\c$\02 /E /XO
robocopy c:\03 \\10.0.0.99\c$\03 /E /XO
robocopy c:\04 \\10.0.0.99\c$\04 /E /XO
robocopy c:\05 \\10.0.0.99\c$\05 /E /XO
ect. all the way to folder 16

then Batch #2 has 16 more lines:
robocopy c:\17 \\10.0.0.99\c$\17 /E /XO

------------------
Total of 16 batch files (256 folders need to be copied)
I want to run them all at once.
------------------
when I do run less of them at once it does not seem to have the error...
------------------
What about in VB though - not using robocopy?
do you know how to search a folder on \\10.0.0.99\c$\FOLDER
and complare it to local folder c:\FOLDER
then compare all the files in both folders
and copy the newer files or non-existant files over?
and still not have it run out of memory but be pretty fast?

Thanks!

0
 
LVL 18

Accepted Solution

by:
sk_raja_raja earned 300 total points
ID: 22749803
1.Ref the below KB for the registry fix for the same issue,
http://support.microsoft.com/kb/304101
0
 

Author Comment

by:CanvasYou
ID: 22750150
sk raja raja:
Will this work on XP Pro You think?
I have seen this same thing on other posts but the KB solution shows SERVER 2k3 and win2k
Thanks for the help
0
 

Author Closing Comment

by:CanvasYou
ID: 31507446
If you have VB tips for file comparison -
ie copy one folder to another folder - and ONLY copy files that are newer than the other machine over -
I would appreciat it.
0

Featured Post

Ransomware-A Revenue Bonanza for Service Providers

Ransomware – malware that gets on your customers’ computers, encrypts their data, and extorts a hefty ransom for the decryption keys – is a surging new threat.  The purpose of this eBook is to educate the reader about ransomware attacks.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

cPanel is a Unix based web hosting control panel that provides a graphical interface and automation tools designed to simplify the process of hosting a web site. cPanel utilizes a 3 tier structure that provides functionality for administrators, rese…
You may have already been in the need to update a whole folder stucture using a script. Robocopy does it well and even provides a list of non-updated files in a log (if asked to). Generally those files that were locked by a user or a process by the …
This Micro Tutorial will teach you how to censor certain areas of your screen. The example in this video will show a little boy's face being blurred. This will be demonstrated using Adobe Premiere Pro CS6.
Many functions in Excel can make decisions. The most simple of these is the IF function: it returns a value depending on whether a condition you describe is true or false. Once you get the hang of using the IF function, you will find it easier to us…

910 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now