Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Why does RoboCopy run out of memory when copying millions of files from machine "a" to machine "b" over LAN

Posted on 2008-10-18
5
Medium Priority
?
4,168 Views
Last Modified: 2012-05-05
I have 2 machines.
Both machines are windows XP Pro.
I copy from one machine to another a folder.
Using the switch on robocopy that tells it to only copy over NEWER files.
I have 256 folders - split up into 16 batch files of 16 lines of robocopy each.
I run all 16 at once.
Over a LAN.
No drive mapping is used.
I use: " robocopy c:\55 \\10.0.0.99\c$\55 /E /XO "
it will do fine unless there are a lot of files.
By "a lot" i mean about 37 million.
It will say "unable to connect to server...waiting 30 secs..."
or sometimes it will say "insuffecient resources waiting 30 secs"
it will wait 30 secs - then it looks like it will send about 5 - 10 more files over
or compare 5 - 10 more files
then it will lock up again. and say "waiting 30 secs"
like the cache or page file or memory pool is not clearing temp memory.

Total files come to about 40 gigs, not very big - but millions of small files.
that need to be compared - and only the newer ones copied over.

Suggestions?
other apps to do this?
or if anyone knows how to do this in VB.net I would appreciate it.

=======================
I am a beginner in Vb.net

Folder A
Folder B

compare folder A to Folder B -
copy all the f iles form Folder B to Folder A that are newer.

and do it for all 256 folders
=======================
0
Comment
Question by:CanvasYou
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
5 Comments
 
LVL 44

Assisted Solution

by:Darr247
Darr247 earned 600 total points
ID: 22749330
Does it give you the same problem if you run the batches one-at-a-time?

Because you could make yet-another batch that 'call's all the other batches in sequence, if they run ok singly.

e.g. say, allcopybatch.bat
CALL BATCH1
CALL BATCH2
CALL BATCH3
et cetera ...

as each one finishes, it returns control to allcopybatch.bat which then invokes the next batch.
You could use FOR and DO, but with 16 batches it's probably just as fast to build it line-by-line.
0
 

Author Comment

by:CanvasYou
ID: 22749705
Thanks for the response.
I have tried that using process.start
and Do while process is not ended.
but i am not that good w/ VB
I have 16 batch files
each one has 16 lines of robocopy
so I am running 16 one after another of this: " robocopy c:\55 \\10.0.0.99\c$\55 /E /XO "

So Batch file #1 has:

robocopy c:\01 \\10.0.0.99\c$\01 /E /XO
robocopy c:\02 \\10.0.0.99\c$\02 /E /XO
robocopy c:\03 \\10.0.0.99\c$\03 /E /XO
robocopy c:\04 \\10.0.0.99\c$\04 /E /XO
robocopy c:\05 \\10.0.0.99\c$\05 /E /XO
ect. all the way to folder 16

then Batch #2 has 16 more lines:
robocopy c:\17 \\10.0.0.99\c$\17 /E /XO

------------------
Total of 16 batch files (256 folders need to be copied)
I want to run them all at once.
------------------
when I do run less of them at once it does not seem to have the error...
------------------
What about in VB though - not using robocopy?
do you know how to search a folder on \\10.0.0.99\c$\FOLDER
and complare it to local folder c:\FOLDER
then compare all the files in both folders
and copy the newer files or non-existant files over?
and still not have it run out of memory but be pretty fast?

Thanks!

0
 
LVL 18

Accepted Solution

by:
sk_raja_raja earned 900 total points
ID: 22749803
1.Ref the below KB for the registry fix for the same issue,
http://support.microsoft.com/kb/304101
0
 

Author Comment

by:CanvasYou
ID: 22750150
sk raja raja:
Will this work on XP Pro You think?
I have seen this same thing on other posts but the KB solution shows SERVER 2k3 and win2k
Thanks for the help
0
 

Author Closing Comment

by:CanvasYou
ID: 31507446
If you have VB tips for file comparison -
ie copy one folder to another folder - and ONLY copy files that are newer than the other machine over -
I would appreciat it.
0

Featured Post

Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

If like me you are one who spends a lot of time working and scripting with cmd.exe, sometimes it is handy to be able to quickly view a calendar for a given month and year. This script will quickly do just that!  Save the code posted below to a .bat …
This article was inspired by a question here at Experts Exchange (http://www.experts-exchange.com/Software/Photos_Graphics/Images_and_Photos/Q_28629170.html). The requirements stated in that question are (1) reduce the file size of a large number of…
Two types of users will appreciate AOMEI Backupper Pro: 1 - Those with PCIe drives (and haven't found cloning software that works on them). 2 - Those who want a fast clone of their boot drive (no re-boots needed) and it can clone your drive wh…
We’ve all felt that sense of false security before—locking down external access to a database or component and feeling like we’ve done all we need to do to secure company data. But that feeling is fleeting. Attacks these days can happen in many w…

688 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question