Solved

Why does RoboCopy run out of memory when copying millions of files from machine "a" to machine "b" over LAN

Posted on 2008-10-18
5
3,794 Views
Last Modified: 2012-05-05
I have 2 machines.
Both machines are windows XP Pro.
I copy from one machine to another a folder.
Using the switch on robocopy that tells it to only copy over NEWER files.
I have 256 folders - split up into 16 batch files of 16 lines of robocopy each.
I run all 16 at once.
Over a LAN.
No drive mapping is used.
I use: " robocopy c:\55 \\10.0.0.99\c$\55 /E /XO "
it will do fine unless there are a lot of files.
By "a lot" i mean about 37 million.
It will say "unable to connect to server...waiting 30 secs..."
or sometimes it will say "insuffecient resources waiting 30 secs"
it will wait 30 secs - then it looks like it will send about 5 - 10 more files over
or compare 5 - 10 more files
then it will lock up again. and say "waiting 30 secs"
like the cache or page file or memory pool is not clearing temp memory.

Total files come to about 40 gigs, not very big - but millions of small files.
that need to be compared - and only the newer ones copied over.

Suggestions?
other apps to do this?
or if anyone knows how to do this in VB.net I would appreciate it.

=======================
I am a beginner in Vb.net

Folder A
Folder B

compare folder A to Folder B -
copy all the f iles form Folder B to Folder A that are newer.

and do it for all 256 folders
=======================
0
Comment
Question by:CanvasYou
  • 3
5 Comments
 
LVL 44

Assisted Solution

by:Darr247
Darr247 earned 200 total points
ID: 22749330
Does it give you the same problem if you run the batches one-at-a-time?

Because you could make yet-another batch that 'call's all the other batches in sequence, if they run ok singly.

e.g. say, allcopybatch.bat
CALL BATCH1
CALL BATCH2
CALL BATCH3
et cetera ...

as each one finishes, it returns control to allcopybatch.bat which then invokes the next batch.
You could use FOR and DO, but with 16 batches it's probably just as fast to build it line-by-line.
0
 

Author Comment

by:CanvasYou
ID: 22749705
Thanks for the response.
I have tried that using process.start
and Do while process is not ended.
but i am not that good w/ VB
I have 16 batch files
each one has 16 lines of robocopy
so I am running 16 one after another of this: " robocopy c:\55 \\10.0.0.99\c$\55 /E /XO "

So Batch file #1 has:

robocopy c:\01 \\10.0.0.99\c$\01 /E /XO
robocopy c:\02 \\10.0.0.99\c$\02 /E /XO
robocopy c:\03 \\10.0.0.99\c$\03 /E /XO
robocopy c:\04 \\10.0.0.99\c$\04 /E /XO
robocopy c:\05 \\10.0.0.99\c$\05 /E /XO
ect. all the way to folder 16

then Batch #2 has 16 more lines:
robocopy c:\17 \\10.0.0.99\c$\17 /E /XO

------------------
Total of 16 batch files (256 folders need to be copied)
I want to run them all at once.
------------------
when I do run less of them at once it does not seem to have the error...
------------------
What about in VB though - not using robocopy?
do you know how to search a folder on \\10.0.0.99\c$\FOLDER
and complare it to local folder c:\FOLDER
then compare all the files in both folders
and copy the newer files or non-existant files over?
and still not have it run out of memory but be pretty fast?

Thanks!

0
 
LVL 18

Accepted Solution

by:
sk_raja_raja earned 300 total points
ID: 22749803
1.Ref the below KB for the registry fix for the same issue,
http://support.microsoft.com/kb/304101
0
 

Author Comment

by:CanvasYou
ID: 22750150
sk raja raja:
Will this work on XP Pro You think?
I have seen this same thing on other posts but the KB solution shows SERVER 2k3 and win2k
Thanks for the help
0
 

Author Closing Comment

by:CanvasYou
ID: 31507446
If you have VB tips for file comparison -
ie copy one folder to another folder - and ONLY copy files that are newer than the other machine over -
I would appreciat it.
0

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Scheduled task not running 5 80
Batch to Restart Remote Computers 4 96
Need some help in Batch file assigning variable value ? 4 76
Windows 7 Share with XP 22 198
We have adopted the strategy to use Computers in Student Labs as the bulletin boards. The same target can be achieved by using a Login Notice feature in Group policy but it’s not as attractive as graphical wallpapers with message which grabs the att…
When you start your Windows 10 PC and got an "Operating system not found" error or just saw  "Auto repair for startup" or a blinking cursor with black screen. A loop for Auto repair will start but fix nothing.  You will be panic as there are no back…
Two types of users will appreciate AOMEI Backupper Pro: 1 - Those with PCIe drives (and haven't found cloning software that works on them). 2 - Those who want a fast clone of their boot drive (no re-boots needed) and it can clone your drive wh…
Although Jacob Bernoulli (1654-1705) has been credited as the creator of "Binomial Distribution Table", Gottfried Leibniz (1646-1716) did his dissertation on the subject in 1666; Leibniz you may recall is the co-inventor of "Calculus" and beat Isaac…

856 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question