Robocopy - copying empty folders even when it is told not to
Summary of question:
Is Robocopy working as expected, or should I be using different command line parameters to prevent empty folder creation?
Detail Issue:
Below is the RoboCopy Job File, which is saved using the /SAVE parameter.
Note the /S parameter is used and not the /E parameter.
When the command is run, via a VBScript which assembles the command line, it works without issue EXCEPT it is creating empty folders. From what I can tell from the documentation and every site I have looked at on the Internet for this issue, RoboCopy should NOT be creating these empty folders..
From what I understand, if a folder does not contain one of the desired file types, it should be considered empty.
If I have 1,200 folders nested 30 levels deep and those folders contain files of type .FUBAR, it should NOT create empty destination folders of the same name, yet it is.
If Robocopy is, in fact, performing as expected, I presume I am going to need to run something like the following to recursively remove empty folders. This is a step I would rather not take, though would write as a secondary housekeeping script due to time constraints on the primary script.
Is Robocopy working as expected, or should I be using different command line parameters to prevent empty folder creation?
Detail Issue:
Below is the RoboCopy Job File, which is saved using the /SAVE parameter.
Note the /S parameter is used and not the /E parameter.
When the command is run, via a VBScript which assembles the command line, it works without issue EXCEPT it is creating empty folders. From what I can tell from the documentation and every site I have looked at on the Internet for this issue, RoboCopy should NOT be creating these empty folders..
From what I understand, if a folder does not contain one of the desired file types, it should be considered empty.
If I have 1,200 folders nested 30 levels deep and those folders contain files of type .FUBAR, it should NOT create empty destination folders of the same name, yet it is.
::
::
::
:: Source Directory :
::
/SD:\\ComputerName\c$\ :: Source Directory.
::
:: Destination Directory :
::
/DD:E:\Migrated Data\ccnllp\Username\ComputerName\Root\ByExtension\ :: Destination Directory.
::
:: Include These Files :
::
/IF :: Include Files matching these names
*.log
*.txt
*.acl
*.ade
*.asd
*.cnv
*.doc
*.dot
*.grv
*.h1q
*.iaf
*.maf
*.mam
*.maq
*.mar
*.mat
*.maw
*.mda
*.mdb
*.mde
*.mdt
*.mdw
*.mpd
*.mpp
*.mpt
*.mso
*.oab
*.obi
*.oft
*.olm
*.one
*.ops
*.ost
*.pip
*.pot
*.ppa
*.pps
*.ppt
*.prf
*.pst
*.pub
*.puz
*.slk
*.snp
*.svd
*.vdx
*.vsd
*.vss
*.vst
*.vsx
*.vtx
*.wbk
*.wll
*.xar
*.xla
*.xlb
*.xlc
*.xll
*.xlm
*.xls
*.xlt
*.xlw
*.xsf
*.xsn
*.dic
*.lex
*.nk2
*.pdf
*.rtf
*.thmx
*.ccda
*.ccdb
*.ccdb
*.ccdc
*.ccde
*.ccdp
*.ccdr
*.ccdt
*.ccdu
*.crtx
*.docm
*.docx
*.dotm
*.dotx
*.epkg
*.pa
*.pmsg
*.potm
*.potx
*.ppam
*.ppsm
*.ppsx
*.pptm
*.pptx
*.sldm
*.sldx
*.vsdx
*.xl
*.xlam
*.xlsb
*.xlsm
*.xlsx
*.xltm
*.xltx
*.xslb
*.db
*.dbf
*.ldb
*.mdb
*.sdf
*.sqlite
*.bmp
*.emf
*.gif
*.ico
*.jpg
*.png
*.tif
*.wmf
*.dat
*.kml
*.kmz
*.xml
*.wav
*.wmdb
*.wpl
*.cfg
*.config
*.icc
*.ini
*.properties
*.zip
*.msg
/XJD :: eXclude Junction points for Directories.
::
:: Exclude These Directories :
::
/XD :: eXclude Directories matching these names
temp
akamai
cache
unified_cache_leveldb_leveldb2
\\ComputerName\c$\Users\Username\WiredRed
\\ComputerName\c$\Users\Username\Videos
\\ComputerName\c$\Users\Username\Music
\\ComputerName\c$\Users\Username\Contacts
\\ComputerName\c$\Users\Username\Pictures
\\ComputerName\c$\Users\Username\Downloads
\\ComputerName\c$\Users\Username\Documents
\\ComputerName\c$\Users\Username\Desktop
\\ComputerName\c$\Users\Username\Appdata\Roaming\Zeon
\\ComputerName\c$\Users\Username\Appdata\Roaming\Litera
\\ComputerName\c$\Users\Username\Appdata\Local\Litera
\\ComputerName\c$\Users\Username\Appdata\Local\Interwoven\QuickSearch\Config
\\ComputerName\C$\dbmakerW64
\\ComputerName\C$\epop
\\ComputerName\C$\epopAdmin
\\ComputerName\C$\epopremove
\\ComputerName\C$\epopremove-verified
\\ComputerName\C$\ePopServer
\\ComputerName\C$\inetpub
\\ComputerName\C$\KB3AIK_EN
\\ComputerName\C$\Robocopy
\\ComputerName\C$\robocopy.bak
\\ComputerName\C$\Scripts
\\ComputerName\C$\StaffData
\\ComputerName\C$\treesize
\\ComputerName\C$\Visio
\\ComputerName\C$\WiredRed
content.ie*
::
:: Exclude These Files :
::
/XF :: eXclude Files matching these names
*.tmp
*cache*
*akamai*
*.lock
::
:: Copy options :
::
/S :: copy Subdirectories, but not empty ones.
/PF :: check run hours on a Per File (not per pass) basis.
/COPY:DAT :: what to COPY for files (default is /COPY:DAT).
/ZB :: use restartable mode; if access denied use Backup mode.
/EFSRAW :: copy all encrypted files in EFS RAW mode.
::
:: Retry Options :
::
/R:1 :: number of Retries on failed copies: default 1 million.
/W:10 :: Wait time between retries: default is 30 seconds.
::
:: Logging Options :
::
/V :: produce Verbose output, showing skipped files.
/X :: report all eXtra files, not just those selected.
/TS :: include source file Time Stamps in the output.
/FP :: include Full Pathname of files in the output.
/NP :: No Progress - don't display percentage copied.
/LOG+:C:\Robocopy\Username_ComputerName_YYYYMMDD_HHMMSS.log :: output status to LOG file (append to existing log).If Robocopy is, in fact, performing as expected, I presume I am going to need to run something like the following to recursively remove empty folders. This is a step I would rather not take, though would write as a secondary housekeeping script due to time constraints on the primary script.
'Sub to recursively delete all empty folders.
Sub RecursiveDeleteEmptyFolders(ByVal strDirectory)
On Error Resume Next
Dim objFolder, objSubFolder
Set objFolder = objFSO.GetFolder(strDirectory)
WScript.Echo "Checking Folder: " & strDirectory & ". Contains " & objFolder.Files.Count & " files and " & objFolder.SubFolders.Count & " folders."
'If the RemoveDesktopIni or RemoveThumbsDB Flag is set to True then remove any files called Desktop.ini or thumbs.db
If objFSO.FileExists(strDirectory & "\desktop.ini") Then
objFSO.DeleteFile(strDirectory & "\desktop.ini")
If Err Then
WScript.Echo "Error deleting:" & strDirectory & "\desktop.ini" & " - " & Err.Description
intErrorCount = intErrorCount + 1
Err.Clear
End If
End If
If objFSO.FileExists(strDirectory & "\thumbs.db") Then
objFSO.DeleteFile(strDirectory & "\thumbs.db")
If Err Then
WScript.Echo "Error deleting:" & strDirectory & "\thumbs.db" & " - " & Err.Description
intErrorCount = intErrorCount + 1
Err.Clear
End If
End If
'Check if there are any subfolders, and if so go through each of those recursively deleting them
If objFolder.SubFolders.Count > 0 Then
For Each objSubFolder in objFolder.SubFolders
RecursiveDeleteEmptyFolders objSubFolder.Path
Next
End If
'Now check if the folder contains any files.
If objFolder.Files.Count = 0 And objFolder.SubFolders.Count = 0 Then
'Check that the folder name does not begin with a tildeand if it does not then delete it
If Left(objFolder.Name,1) <> "~" Then
WScript.Echo "Deleting: " & objFolder.Path
objFolder.Delete
If Err Then
WScript.Echo "Error deleting:" & objFolder.Name & " – " & Err.Description
intErrorCount = intErrorCount + 1
Err.Clear
End If
End If
End If
End Sub
yo_bee
Even though you have /IF set maybe there are hidden files in the empty directories.
ASKER
Even though you have /IF set maybe there are hidden files in the empty directories.
I wish it were that easy. I have done extensive analysis on the data and the vast majority of the folders have no hidden files AND have no files which meet the criteria being set to copy.
Now, another solution, if anyone is aware of it, would be a tool which does exactly what the Windows Easy Transfer Wizard or USMT does in a non-proprietary format, providing individual file/folder copying and registry settings export and copying (they'd just be REG files at that point).
I never us the /ZB , /EFSRAW or /XJ.
I would remove all those and see if things change
I would remove all those and see if things change
ASKER
The /XJ excludes copying/following file junctions - have tried removing this with no change in behavior.
The /EFSRAW copies encrypted files in RAW format - this is required because, if the user has encrypted files on their local hard drive, the server performing the backup would not be able to back these files up - have tried removing this with no change in behavior (except that encrypted files are not copied and Robocopy generates errors when the script runs against machines with encrypted files.
The /ZB uses restartable mode, and if that fails, uses Backup mode. This increases the probability that files opened by the user at the time Robocopy runs actually get backed up/copied - have tried removing this with no change in behavior.
The /EFSRAW copies encrypted files in RAW format - this is required because, if the user has encrypted files on their local hard drive, the server performing the backup would not be able to back these files up - have tried removing this with no change in behavior (except that encrypted files are not copied and Robocopy generates errors when the script runs against machines with encrypted files.
The /ZB uses restartable mode, and if that fails, uses Backup mode. This increases the probability that files opened by the user at the time Robocopy runs actually get backed up/copied - have tried removing this with no change in behavior.
For testing can you remove the /xd. I just saw someone else have a similar issue and removing the /xd returned the desired results minus the excluded directories.
ASKER
The issue with removing the /XD, which specifies which folders to exclude from the copy, is one of the folders not listed in the above RCJ file is C:\Users. If this folder is included, the Robocopy job, on average, goes from 2 to 9 GB to 14 to 58 GB. Granted the network data storage is not an issue, but the time required per workstation increases dramatically even for an incremental backup,
Is the /XD switch responsible for the behavior of Robocopy creating these empty folders?
Is the /XD switch responsible for the behavior of Robocopy creating these empty folders?
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
So the empty folders being there are more beneficial.Yes, as I have a script which deletes these empty folders.
I would prefer it if these were not created and my log wasn't filled with erroneous messages indicating new folder creation without file copy, but copying the excluded folders is NOT and option.
ASKER
Alexandre, the advertisements on that website are offensive beyond reason. I believe the site operators do this intentionally to garner undeserved click-through revenue.
WalkaboutTigger, sorry about that!
I did not see many adds when I visited the site as I use an ad blocker and a script blocker
Have a look at http://alternativeto.net/software/robocopy/. However, it will be difficult to get a non proprietary software that does everything that you want to do.
After putting so much effort in Robocopy, you might try something else just to find that it too is only 99% perfect
Have you looked into RSYNC (https://rsync.samba.org/ and https://download.samba.org/pub/rsync/rsync.html). It is very powerful ... but it might have the same shortfall as robocopy: copying a source folder that is not empty but whose content is completly excluded and thus whose destination is empty...
Here is a solution of sort: use your script above to copy from source to temporary destination. Use Robocopy /S to copy from temporary destination to final distination. Empty folders will be skipped! Your script to delete empty folders is probably simpler...
I did not see many adds when I visited the site as I use an ad blocker and a script blocker
Have a look at http://alternativeto.net/software/robocopy/. However, it will be difficult to get a non proprietary software that does everything that you want to do.
After putting so much effort in Robocopy, you might try something else just to find that it too is only 99% perfect
Have you looked into RSYNC (https://rsync.samba.org/ and https://download.samba.org/pub/rsync/rsync.html). It is very powerful ... but it might have the same shortfall as robocopy: copying a source folder that is not empty but whose content is completly excluded and thus whose destination is empty...
Here is a solution of sort: use your script above to copy from source to temporary destination. Use Robocopy /S to copy from temporary destination to final distination. Empty folders will be skipped! Your script to delete empty folders is probably simpler...
ASKER
As an experiment, I removed the /XD parameter. In addition to copying an additional 56 GB of data, Robocopy still created over 70,000 empty folders.
The performance impact this is causing to the script is becoming unacceptable. What is worse is that, when an incremental job is run to update any changes, Robocopy recreates all of the now-deleted empty folders.
This is mind-mindbogglingly stupid behavior on the part of Robocopy. It should NOT be creating folders and then looking in the source folder to see if there is anything to copy. It should be examining the source folder first and only creating a destination folder if it is supposed to be doing something.
The biggest challenge with using USMT or WETW is that users have historically saved data all over their local drives - we had one person saving files in \Windows\System32. In the new environment, users are NOT local admins and have redirected folders and document management integration, but we have to scan their entire drive for possible data.
The performance impact this is causing to the script is becoming unacceptable. What is worse is that, when an incremental job is run to update any changes, Robocopy recreates all of the now-deleted empty folders.
This is mind-mindbogglingly stupid behavior on the part of Robocopy. It should NOT be creating folders and then looking in the source folder to see if there is anything to copy. It should be examining the source folder first and only creating a destination folder if it is supposed to be doing something.
The biggest challenge with using USMT or WETW is that users have historically saved data all over their local drives - we had one person saving files in \Windows\System32. In the new environment, users are NOT local admins and have redirected folders and document management integration, but we have to scan their entire drive for possible data.
This is crazy i agree. What version of RC you running?
Also have you tried running your RC with direct switches rather than a job. I know you have a very complex structure, but it is something worth trying
Also have you tried running your RC with direct switches rather than a job. I know you have a very complex structure, but it is something worth trying
ASKER
Robocopy version 5.1.10.1027
It doesn't matter if I use a CMD prompt, run it from a VBSCRIPT or launch it from Powershell, the results are the same.
It doesn't matter if I use a CMD prompt, run it from a VBSCRIPT or launch it from Powershell, the results are the same.
Are you 100% sure that there are no hidden system level files with in the sub-directories that you visibility know are empty?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
No other presented solution provided a complete answer.
I do appreciate yo_bee's and Alexabdre's efforts.
I do appreciate yo_bee's and Alexabdre's efforts.