[Last Call] Learn about multicloud storage options and how to improve your company's cloud strategy. Register Now

x
?
Solved

How to split files in one directory into 2 or more directories?

Posted on 2015-01-09
5
Medium Priority
?
279 Views
Last Modified: 2015-01-12
Hi all!

I have some directories with thousands of files and I need to split those files into directories of 4,000 files or less. I'm fairly new to scripting so please bear with me. I've searched for a solution and I see that you can copy ALL files from a directory to a new directory but I want to copy only some of the files. All the files are the same type.

#split files into different directories if directory has more than 4,000 files

if $totalNumOfFiles > 4000

#Determine how many new directories are needed
numOfNewDir = int($totalNumOfFiles/4000)


#The new directories will be like this: original_folder_name_1, original_folder_name_2, etc.
#depending on how many new directories have to be created

sourceDirName = $DIR

while [ $numOfNewDir > 0 ]
do

destinationDirName = $sourceDirName_$numOfNewDir
mkdir $destinationDirName

#this is where I would split the files and copy them into a new directory but I don't know how to do it
#start copying 4,000 files

#cp $sourceDirName/files to $destinationDirName/files

#end copying, the next iteration would copy the next set of files starting at file number 4001

--numOfNewDir
done


done
0
Comment
Question by:afrocelt
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2
5 Comments
 
LVL 84

Expert Comment

by:ozo
ID: 40541540
sourceDirName=$DIR
files=( $sourceDirName/* )
if (( ${#files[*]} > 4000 )) ; then
  for (( i = 0 ; i < ${#files[@]} ; i++ )) do
    destinationDirName=${sourceDirName}_$((1+$i/4000))
    if [[ ! -d $destinationDirName ]] ; then  mkdir $destinationDirName ; fi
    mv "${files[$i]}" "$destinationDirName/"
  done
fi
0
 

Author Comment

by:afrocelt
ID: 40545348
Hi Ozo,

Thank you so much for the quick reply. I almost have it working except when the path contains spaces. I know I need to put quotes somewhere along here:

destinationDirName=${sourceDirName}_$((1+$i/4000))
f [[ ! -d $destinationDirName ]] ; then  mkdir $destinationDirName ;

because it tries multiple times to create a directory where there is a space:

If the directory is:  /Volumes/DirA/DirB/Personal Folders/me/THE FILES/image1.JPG

I get this:
mkdir: /Volumes/DirA/DirB/Personal: File exists
mkdir: Folders/me: No such file or directory
mkdir: FILES_5: File exists

Strangely, it still creates the correct path:

/Volumes/DirA/DirB/Personal Folders/me/THE FILES_5/

Do I need to put quotes around ${sourceDirName}? I tried a few iterations but I think I'm doing something wrong. Thanks in advance.
0
 
LVL 84

Accepted Solution

by:
ozo earned 2000 total points
ID: 40545355
files=( "$sourceDirName"/* )
if (( ${#files[*]} > 4000 )) ; then
  for (( i = 0 ; i < ${#files[@]} ; i++ )) do
    destinationDirName=${sourceDirName}_$((1+$i/4000))
    if [[ ! -d "$destinationDirName" ]] ; then  mkdir "$destinationDirName" ; fi
    mv "${files[$i]}" "$destinationDirName/"
  done
fi
0
 

Author Comment

by:afrocelt
ID: 40545393
Yes! That worked. I really appreciate your help.
0
 

Author Closing Comment

by:afrocelt
ID: 40545396
The response was quick as well as the answer to my follow-up question. Really helpful!
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction Regular patching is part of a system administrator's tasks. However, many patches require that the system be in single-user mode before they can be installed. A cluster patch in particular can take quite a while to apply if the machine…
How to remove superseded packages in windows w60 or w61 installation media (.wim) or online system to prevent unnecessary space. w60 means Windows Vista or Windows Server 2008. w61 means Windows 7 or Windows Server 2008 R2. There are various …
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
Suggested Courses

650 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question