- Windows Server 2003
- Storage Software
- Windows Batch
Robocopy to copy only Changed / New Files ?
Hello Gurus,
here is my situation.
I have around 300Gb of Data on my Server that I need to backup and keep daily copies for a 7 days (Company's Policy).
Normally , I would need a 300GB *10 space but since we all know that no way that all data changes every day... So I was thinking of having the following scheme...
Robocopy ALL Files on a Folder called ALL
and create folders Mon/Tue/Wed...etc and do an incremental backup of the CHANGED / NEW Files only to those folders...
I created the following batch file :
**************************
C:
CD \
FOR /F "TOKENS=1* DELIMS= " %%A IN ('DATE/T') DO SET day=%%A
MD C:\Daily\%day%
ROBOCOPY \\DATA-SRV\D$ R:\Daily\%day% /TEE /E /R:5 /W:15 /FFT /NP
**************************
My problem with this script is that Mon will contain all files.
Tue also ...Wed also....
What I want is the following.
Once a month I create the ALL folder that is :
ROBOCOPY \\DATA-SRV\D$ R:\ALL /TEE /E /R:500 /W:500 /FFT /NP
and then every day the robocopy command should check what are the files changed/added between the
<ALL> folder AND the <DATA-SRV\D$> to be copied to the Folder Mon/Tue/... according to the date...
in that case I would have the full backup in the ALL folder...and any changes in the Mon/Tue/Wed...folder...
Can this be done ?
Thanks Gurus
here is my situation.
I have around 300Gb of Data on my Server that I need to backup and keep daily copies for a 7 days (Company's Policy).
Normally , I would need a 300GB *10 space but since we all know that no way that all data changes every day... So I was thinking of having the following scheme...
Robocopy ALL Files on a Folder called ALL
and create folders Mon/Tue/Wed...etc and do an incremental backup of the CHANGED / NEW Files only to those folders...
I created the following batch file :
**************************
C:
CD \
FOR /F "TOKENS=1* DELIMS= " %%A IN ('DATE/T') DO SET day=%%A
MD C:\Daily\%day%
ROBOCOPY \\DATA-SRV\D$ R:\Daily\%day% /TEE /E /R:5 /W:15 /FFT /NP
**************************
My problem with this script is that Mon will contain all files.
Tue also ...Wed also....
What I want is the following.
Once a month I create the ALL folder that is :
ROBOCOPY \\DATA-SRV\D$ R:\ALL /TEE /E /R:500 /W:500 /FFT /NP
and then every day the robocopy command should check what are the files changed/added between the
<ALL> folder AND the <DATA-SRV\D$> to be copied to the Folder Mon/Tue/... according to the date...
in that case I would have the full backup in the ALL folder...and any changes in the Mon/Tue/Wed...folder...
Can this be done ?
Thanks Gurus
I'm afraid you can't do that. ROBOCOPY is an utility for mantaining a "mirrored" copy of a directory structure. The command structure is <source> ---> <destination>. Everything is happening between these two parameters (check between them, copy/move/or delete between them).
u can integrate a validation in your script to check if file has changed and then perform the robocopy as usual.
XXCOPY can do that easily for you
www.xxcopy.com/
for sync operation of updated files
http://www.xxcopy.com/xxgroup/m03/msg03254.htm
hope this helps
www.xxcopy.com/
for sync operation of updated files
http://www.xxcopy.com/xxgroup/m03/msg03254.htm
hope this helps
>>I have around 300Gb of Data on my Server that I need to backup and keep daily copies for a 7 days (Company's Policy).
do u need to delete them after 7 days?
do u need to delete them after 7 days?
can you tell me if this is what you are looking for.
you have 300GB data in one location and you want to back it up in another location to cycle the backup over the 7 days of week? is that means you will have 300GB backup copy in 7 archives based on Weekday Name? (eg. SAT, SUN, MON)
you have 300GB data in one location and you want to back it up in another location to cycle the backup over the 7 days of week? is that means you will have 300GB backup copy in 7 archives based on Weekday Name? (eg. SAT, SUN, MON)
nbass668, the thing is as follows :
suppose I have 10 files to backup file1....file10
Now day0 I will backup up all files file1...file10 in a folder <ALL>
Now day1 (monday) I will backup the files that changed on monday to a folder <MON>
so if only 1 file was changed , the MON folder will have 1 file only
Now day2 (tuesday) I will backup the files that changed on tuesday to a folder <TUE>
so if only 3 file was changed , the TUE folder will have 3 files only
so on wednesday my whole data would be the <ALL> folder + <WED> folder.
etc..
suppose I have 10 files to backup file1....file10
Now day0 I will backup up all files file1...file10 in a folder <ALL>
Now day1 (monday) I will backup the files that changed on monday to a folder <MON>
so if only 1 file was changed , the MON folder will have 1 file only
Now day2 (tuesday) I will backup the files that changed on tuesday to a folder <TUE>
so if only 3 file was changed , the TUE folder will have 3 files only
so on wednesday my whole data would be the <ALL> folder + <WED> folder.
etc..
find below a batch file which simply does the following.
Check which day it is Today! example MON or TUE or SUN etc.
Check for new modified files for the source directory and copy only the new modified to a folder corresponding to the day of week
if today is Monday... the copy folder will be C:\backup\MON\
I used the windows command XCOPY, you can use instead any similar copy commnads
I have tested the batch file it worked....
let me know your thoughts
Check which day it is Today! example MON or TUE or SUN etc.
Check for new modified files for the source directory and copy only the new modified to a folder corresponding to the day of week
if today is Monday... the copy folder will be C:\backup\MON\
I used the windows command XCOPY, you can use instead any similar copy commnads
I have tested the batch file it worked....
let me know your thoughts
@ECHO OFF
:: Find which day it is today (eg. Mon, Tue, Wed etc.)
for /F %%A in ('date /t') do set DAY=%%A
:: Use XCOPY command to find files modified today and copy them to the specified date folder
for /F "tokens=2-4* delims=/ " %%A in ('DATE/T') do set MMDDYY=%%A-%%B-%%C
XCOPY c:\test\*.* c:\backup\%DAY% /D:%MMDDYY% /S /C
PAUSE
nbass668 thanks for the script but it doesn't do exactly what I need mainly because I can't rely on the modified date !!!
What if the user gets an old file from somewhere else and copies it to his folder ?
that file would never be backed up !!
What if the user gets an old file from somewhere else and copies it to his folder ?
that file would never be backed up !!
@ammounpierre;
yeah indeed you are correct. and how I had the big picture of what you facing.
however, it seems your request is kind of complicated.
but, I have two solutions. let me know which one is more close to your request.
Solution 1.
- In order to achieve your request, you will need 8 folders/locations for backup.
- an <ALL> folder that will maintain a clone/sync copy of the destination.. coz this is the only way to compare and find new changed/updated/added files from two directories.
- once compare is complete... the batch file will move the new files from the source to the <ALL> folder to keep it updated and on the same time... move those same specific files to a <DAY> folder. such as MON, TUE, SUN etc.
- this way... the batch file will always sync the destination folder with <ALL> folder and maintain the file changes over 7 folders representing days of week. and cycle them weekly.
Solution 2.
if you dont wish to maintain <ALL> folder daily.
instead of comparing the destination folder with <ALL> folder.. you can compare the destination folder with a database file that gets updated daily with all the folder structure, file sizes, modified date etc. to find out new changes and updates.. and those changes will be copied to the 7 folders representing days of week. and cycle them weekly. and perhaps maintain a full copy only on Mondays if you prefer.
however "Solution 2" does not seem achievable with a simple batch file.. and perhaps you will need a much more advanced scripting... this is my humble opinion.
as for "solution 1".. I think we can easily do it in a batch file
please advice...
yeah indeed you are correct. and how I had the big picture of what you facing.
however, it seems your request is kind of complicated.
but, I have two solutions. let me know which one is more close to your request.
Solution 1.
- In order to achieve your request, you will need 8 folders/locations for backup.
- an <ALL> folder that will maintain a clone/sync copy of the destination.. coz this is the only way to compare and find new changed/updated/added files from two directories.
- once compare is complete... the batch file will move the new files from the source to the <ALL> folder to keep it updated and on the same time... move those same specific files to a <DAY> folder. such as MON, TUE, SUN etc.
- this way... the batch file will always sync the destination folder with <ALL> folder and maintain the file changes over 7 folders representing days of week. and cycle them weekly.
Solution 2.
if you dont wish to maintain <ALL> folder daily.
instead of comparing the destination folder with <ALL> folder.. you can compare the destination folder with a database file that gets updated daily with all the folder structure, file sizes, modified date etc. to find out new changes and updates.. and those changes will be copied to the 7 folders representing days of week. and cycle them weekly. and perhaps maintain a full copy only on Mondays if you prefer.
however "Solution 2" does not seem achievable with a simple batch file.. and perhaps you will need a much more advanced scripting... this is my humble opinion.
as for "solution 1".. I think we can easily do it in a batch file
please advice...
nbass668 thank you solution 1 would be great.
So that we don't get confused... here what would happen.
we have the folder <SOURCE> that we need to sync with <DESTINATION>
the batch would do the following :
1- SYNC between <SOURCE> and <DESTINATION> (always 1 way from Source to Destination)
2-MOVE or COPY the new files to <MON> on monday ....<TUE> on Tuesday...etc...
Thanks
So that we don't get confused... here what would happen.
we have the folder <SOURCE> that we need to sync with <DESTINATION>
the batch would do the following :
1- SYNC between <SOURCE> and <DESTINATION> (always 1 way from Source to Destination)
2-MOVE or COPY the new files to <MON> on monday ....<TUE> on Tuesday...etc...
Thanks
the trick is, to log all new/changed files in to a log file then step into the log file and individually copy each new file to the DAY folder.
however, I was not able to achieve this with ROBOCOPY. as its LOG file or screen output is filled with so many unwanted spaces and characters.
in details;
so I have used XCOPY which output for me only the new and changed files between the source and destination in to a LOG file... XCOPY will not do any copy but to report new changes... I used your own ROBOCOPY command to take care of updating the destination as it is more robust and effective.
once ROBOCOPY finish updating the destination and source folder... XCOPY will step again into the log file and individually copy the new changed files to MON folder or SUN folder (which ever the day falls into)
I have tested it and it worked perfectly.
this method is not bullet proof, try to test it and let me know your concerns.
if you get the idea... you can further customize it or someone can shed more light and make the batch file much more effective
however, I was not able to achieve this with ROBOCOPY. as its LOG file or screen output is filled with so many unwanted spaces and characters.
in details;
so I have used XCOPY which output for me only the new and changed files between the source and destination in to a LOG file... XCOPY will not do any copy but to report new changes... I used your own ROBOCOPY command to take care of updating the destination as it is more robust and effective.
once ROBOCOPY finish updating the destination and source folder... XCOPY will step again into the log file and individually copy the new changed files to MON folder or SUN folder (which ever the day falls into)
I have tested it and it worked perfectly.
this method is not bullet proof, try to test it and let me know your concerns.
if you get the idea... you can further customize it or someone can shed more light and make the batch file much more effective
@ECHO OFF
REM Set to current working directory
CD /d %~dp0
REM Find which day it is today (eg. Mon, Tue, Wed etc.)
for /F %%A in ('date /t') do set DAY=%%A
REM Compare the two destination dirctory and [ALL] folder (this command will ONLY output changed/new files to a log file)
XCOPY "C:\DestinationFolder" "C:\backup\[ALL]" /D /E /L /Y /C > logme.txt
REM Copy new files
ROBOCOPY "C:\DestinationFolder" "C:\backup\[ALL]" *.* /TEE /E /R:5 /W:15 /FFT /NP
REM Read successive lines from the log file and copy the new files into specific DAY folder
for /f "delims=] tokens=1*" %%a in ('find /v /n "" ^<logme.txt') do (
IF EXIST "%%b" XCOPY "%%b" "C:\backup\%DAY%" /C /Y
)
PAUSE
EXIT
nbass668 sorry for the late comments... but it is not working...
here is what I did to test it
I created 2 folders <SOURCE> and <DESTINATION>
Now in the <DESTINATION> folder I Created the following subfolders
<ALL> <Mon> <Tue> <Wed> <Thu> <Fri> <Sat> <Sun>
Now in the source I put 5 files...
Now I run the script but modified it a bit
**this line is giving me an error...So I removed it and put the batch file in the <SOURCE> folder
CD /d %~dp0
HERE IS THE SCRIPT I RUN
**************************
@ECHO OFF
REM Set to current working directory
REM Find which day it is today (eg. Mon, Tue, Wed etc.)
for /F %%A in ('date /t') do set DAY=%%A
REM Compare the two destination dirctory and [ALL] folder (this command will ONLY output changed/new files to a log file)
XCOPY "C:\SOURCE" "C:\DESTINATION\ALL" /D /E /L /Y /C > logme.txt
REM Copy new files
ROBOCOPY "C:\SOURCE" "C:\DESTINATION\ALL" *.* /TEE /E /R:5 /W:15 /FFT /NP
REM Read successive lines from the log file and copy the new files into specific DAY folder
for /f "delims=] tokens=1*" %%a in ('find /v /n "" ^<logme.txt') do (
IF EXIST "%%b" XCOPY "%%b" "C:\DESTINATION\%DAY%" /C /Y
)
PAUSE
EXIT
**************************
MY PROBLEMS ARE AS FOLLOWS:
1-it is xcopying... instead of robocopy ...so it is much much slower
2-what if I run it more than once in the same day ? It will re-copy again all files for today ?
thanks
here is what I did to test it
I created 2 folders <SOURCE> and <DESTINATION>
Now in the <DESTINATION> folder I Created the following subfolders
<ALL> <Mon> <Tue> <Wed> <Thu> <Fri> <Sat> <Sun>
Now in the source I put 5 files...
Now I run the script but modified it a bit
**this line is giving me an error...So I removed it and put the batch file in the <SOURCE> folder
CD /d %~dp0
HERE IS THE SCRIPT I RUN
**************************
@ECHO OFF
REM Set to current working directory
REM Find which day it is today (eg. Mon, Tue, Wed etc.)
for /F %%A in ('date /t') do set DAY=%%A
REM Compare the two destination dirctory and [ALL] folder (this command will ONLY output changed/new files to a log file)
XCOPY "C:\SOURCE" "C:\DESTINATION\ALL" /D /E /L /Y /C > logme.txt
REM Copy new files
ROBOCOPY "C:\SOURCE" "C:\DESTINATION\ALL" *.* /TEE /E /R:5 /W:15 /FFT /NP
REM Read successive lines from the log file and copy the new files into specific DAY folder
for /f "delims=] tokens=1*" %%a in ('find /v /n "" ^<logme.txt') do (
IF EXIST "%%b" XCOPY "%%b" "C:\DESTINATION\%DAY%" /C /Y
)
PAUSE
EXIT
**************************
MY PROBLEMS ARE AS FOLLOWS:
1-it is xcopying... instead of robocopy ...so it is much much slower
2-what if I run it more than once in the same day ? It will re-copy again all files for today ?
thanks
The reason I used xcopy because I had issues logging the new changes to a logfile. however, with a bit of asking I found the solution and there is no need to use Xcopy
I have used multiple FOR loops to read the log file then to clean the log file from unwanted spaces and finally to parse the filenames to ROBOCOPY again to copy the new changed files to the respective DAY folder
you can run the batch file multiple times and if ROBOCOPY did not find any changes then it will simply skip.
please note the extra switches for ROBOCOPY and the log file (logtxt.txt) should always be stored in the same folder the batch file running or else this process
I have tested this batch and works..
please test from your end
I have used multiple FOR loops to read the log file then to clean the log file from unwanted spaces and finally to parse the filenames to ROBOCOPY again to copy the new changed files to the respective DAY folder
you can run the batch file multiple times and if ROBOCOPY did not find any changes then it will simply skip.
please note the extra switches for ROBOCOPY and the log file (logtxt.txt) should always be stored in the same folder the batch file running or else this process
I have tested this batch and works..
please test from your end
@ECHO OFF
REM Find which day it is today (eg. Mon, Tue, Wed etc.)
for /F %%A in ('date /t') do set DAY=%%A
REM Copy new files
ROBOCOPY "C:\SOURCE" "C:\DESTINATION\ALL" *.* /TEE /E /R:5 /W:15 /FFT /NP /LOG:logtxt.txt /NS /NC /NJH /NJS /NDL
REM Clean up the log file and Read successive lines from the log file and copy the new files into specific DAY folder
for /f "delims=] tokens=1*" %%a in ('find /v /n "" ^<logtxt.txt') do (
for /f "tokens=*" %%s in ("%%b") do (
for %%a in ("%%s") do ROBOCOPY "C:\SOURCE" "C:\DESTINATION\%DAY%" "%%~nxa" /TEE /R:5 /W:15 /NP /NS /NC /NJH /NJS /NDL
)
)
PAUSE
EXIT
please ignore the above batch as I found a small bug that if you have multiple sub folders the files inside them may not get copied to the DAY folder
@ECHO OFF
REM Find which day it is today (eg. Mon, Tue, Wed etc.)
for /F %%A in ('date /t') do set DAY=%%A
REM Copy new files
ROBOCOPY "C:\SOURCE" "C:\DESTINATION\ALL" *.* /TEE /E /R:5 /W:15 /FFT /NP /LOG:logtxt.txt /NS /NC /NJH /NJS /NDL
REM Clean up the log file and Read successive lines from the log file and copy the new files into specific DAY folder
for /f "delims=] tokens=1*" %%a in ('find /v /n "" ^<logtxt.txt') do (
for /f "tokens=*" %%s in ("%%b") do (
for %%a in ("%%s") do ROBOCOPY %%~da%%~pa "C:\DESTINATION\%DAY%" "%%~nxa" /TEE /R:5 /W:15 /NP /NS /NC /NJH /NJS /NDL
)
)
PAUSE
EXIT
this is another final corrected version.
I am actually correcting line 12.
and after few live backups I performed with this batch.. i found another bug that subfolders are not created inside the DAY folder...
below is the final version please discard previous
I am actually correcting line 12.
and after few live backups I performed with this batch.. i found another bug that subfolders are not created inside the DAY folder...
below is the final version please discard previous
@ECHO OFF
REM Find which day it is today (eg. Mon, Tue, Wed etc.)
for /F %%A in ('date /t') do set DAY=%%A
REM Copy new files
ROBOCOPY "C:\SOURCE" "C:\DESTINATION\ALL" *.* /TEE /E /R:5 /W:15 /FFT /NP /LOG:logtxt.txt /NS /NC /NJH /NJS /NDL
REM Clean up the log file and Read successive lines from the log file and copy the new files into specific DAY folder
for /f "delims=] tokens=1*" %%a in ('find /v /n "" ^<logtxt.txt') do (
for /f "tokens=*" %%s in ("%%b") do (
for %%a in ("%%s") do ROBOCOPY "%%~da%%~pa\" "C:\DESTINATION\%DAY%%%~pa\" "%%~nxa" /TEE /R:5 /W:15 /NP /NS /NC /NJH /NJS /NDL
)
)
PAUSE
EXIT
nbass668 I did try it and we are getting close...
there is another problem now.. it does copy also the destination folder ?
I don't know why but I have a sub-folder called "DESTINATION" in the DESTINATION folder
I think it needs some more fine tuning...
thanks
there is another problem now.. it does copy also the destination folder ?
I don't know why but I have a sub-folder called "DESTINATION" in the DESTINATION folder
I think it needs some more fine tuning...
thanks
can you please replace line 12 with the below mentioned line..
it seems the folder variables getting mixed up. Iam trying to visualize the scenario you getting but not sure if the new changes in the said line will help fix it.
plz try and let me know
it seems the folder variables getting mixed up. Iam trying to visualize the scenario you getting but not sure if the new changes in the said line will help fix it.
plz try and let me know
for %%a in ("%%s") do ROBOCOPY "%%~da%%~pa " "C:\DESTINATION\%DAY%%%~pa " "%%~nxa" /XJD /XJF /TEE /R:5 /W:15 /NP /NS /NC /NJH /NJS /NDL
-
![]()
received a Solution
Need a way of using Robocopy to copy files into a destination folder that has the current date in its folder name -
![]()
wrote an Article
![]()
A Guide to Volume Shadow Copy
-
![]()
created a Video
![]()
Xpdf - PDFtoHTML - Command Line Utility to Convert a PDF File to HTML
-
Explore Cloud Class® ![]()
Certification: MCP Microsoft Certified Partner - Installing and Configuring Windows 7
Premium Content
You need an Expert Office subscription to comment.Start Free Trial