Solved

How can I compare several text files and keep only 1 copy of a file without duplication?

Posted on 2014-09-24
9
127 Views
Last Modified: 2014-10-01
I would like to compare several text files, ie new1.txt and new2.txt, new3.txt .. regardless of the name of the files.
If the 'content's of the files compared with others is the same then delete one of the files which is a duplicate.
0
Comment
Question by:100questions
  • 4
  • 3
  • 2
9 Comments
 
LVL 17

Expert Comment

by:Emmanuel Adebayo
ID: 40342088
Hi,

You can use Multi-File Compare

To download files and see other information about the project, go to http://sourceforge.net/projects/multi-fcompare.

Rgds
0
 

Author Comment

by:100questions
ID: 40342173
Will this work in an existing batch script?
0
 
LVL 17

Expert Comment

by:Emmanuel Adebayo
ID: 40342192
No this is an executable.
0
3 Use Cases for Connected Systems

Our Dev teams are like yours. They’re continually cranking out code for new features/bugs fixes, testing, deploying, testing some more, responding to production monitoring events and more. It’s complex. So, we thought you’d like to see what’s working for us.

 

Author Comment

by:100questions
ID: 40342243
Then I would need something that I can insert in an existing Windows Batch file or a new Powershell or VBScript which can perform the function.
0
 
LVL 9

Accepted Solution

by:
dlb6597 earned 500 total points
ID: 40342313
barebones, inefficient...definately test this with a subset of your data.
basically for every text file this launches another for loop that compares each .txt file to every other .txt file and deletes if there is a match. The script starts over after a deletion because the (*.txt) set changes...

:start
for %%i in (*.txt) do (
	for %%j in (*.txt) do (
	if not "%%i" == "%%j" fc %%i %%j && del %%i && goto start
)
)

Open in new window

0
 

Author Comment

by:100questions
ID: 40342336
Does this script look into the contents of the txt file?
0
 
LVL 9

Expert Comment

by:dlb6597
ID: 40342351
yes, it compares file contents using the fc command.
0
 

Author Comment

by:100questions
ID: 40344239
This seems to work, however the problem is that one of the files it compares contains a small right arrow at the end of the data (an ASCII EOF marker) and if it sees that then it does not deduplicate properly.  

Is there a way your script can be modified so as to ignore the an ASCII EOF marker?
0
 
LVL 9

Expert Comment

by:dlb6597
ID: 40344258
then the files aren't identical then are they?  There is a /L parameter for FC, but I doubt it will make any difference since the files are truly different.
0

Featured Post

3 Use Cases for Connected Systems

Our Dev teams are like yours. They’re continually cranking out code for new features/bugs fixes, testing, deploying, testing some more, responding to production monitoring events and more. It’s complex. So, we thought you’d like to see what’s working for us.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The following is a collection of cases for strange behaviour when using advanced techniques in DOS batch files. You should have some basic experience in batch "programming", as I'm assuming some knowledge and not further explain the basics. For some…
If like me you are one who spends a lot of time working and scripting with cmd.exe, sometimes it is handy to be able to quickly view a calendar for a given month and year. This script will quickly do just that!  Save the code posted below to a .bat …
Two types of users will appreciate AOMEI Backupper Pro: 1 - Those with PCIe drives (and haven't found cloning software that works on them). 2 - Those who want a fast clone of their boot drive (no re-boots needed) and it can clone your drive wh…
Finds all prime numbers in a range requested and places them in a public primes() array. I've demostrated a template size of 30 (2 * 3 * 5) but larger templates can be built such 210  (2 * 3 * 5 * 7) or 2310  (2 * 3 * 5 * 7 * 11). The larger templa…

831 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question