Sort a list of files and move duplicate files to another directory and unique files to a different directory

Have a list of files in a directory and the only difference in the file name is the date/time stamp.  The first 4 values of the file name will be the same for all files except the date/time and 6th value.  Would need to check the 6 value to see if it is the same as another file because the only difference would be the date/stamp.  If the 6th value is the same then move to a dup directory else move to another directory.

Would like to use a file to store the 6th value in a file and check each file against this file.

This is what I have.  Not sure if this is a good approach.

File name  --  parm1_parm2_parm3_parm4_date/time_parm6.csv

      FILELIST=`ls -1 2> /dev/null`
  for file in $FILELIST
do
      messageid=`echo $file | cut -d "_" -f6`
      echo ${messageid} >> /app/tmp/msg_list.txt
if      [ ` grep -c -f /app/tmp/msg_list.txt $messageid ` -ne 0 ]
      then
            mv $file  /somedirectory
      else
            echo mv $file /dup_directory
fi            
done
bjeAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

simon3270Commented:
The "if" will always find the messageid because you've just added it to the msg_list.txt file!

To fix, move the "echo $messageid" line to just after the "mv" line (i.e. between the "mv" and the "else").
0
bjeAuthor Commented:
Thanks,
I am getting an error "No such file or directory".  Do I need to rename the files to check and then rename them back to move them or is there a better way to do this?

cut -d _ -f6
messageid=Value6.csv
grep -c /tmp/msg_list.txt Value6.csv
grep: Value6.csv: No such file or directory
'[' '!=' 0 ']'
./test_sort.sh: line 12: [: !=: unary operator expected
0
simon3270Commented:
Thr text you arr searching for comes first, followed by the file(s) to search in. So,

   grep -c "$messageId" /tmp/msg/list.txt
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
bjeAuthor Commented:
This worked well.  Thanks for the help
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Shell Scripting

From novice to tech pro — start learning today.