Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

How can I compare several text files and keep only 1 copy of a file without duplication?

Posted on 2014-09-24
9
Medium Priority
?
140 Views
Last Modified: 2014-10-01
I would like to compare several text files, ie new1.txt and new2.txt, new3.txt .. regardless of the name of the files.
If the 'content's of the files compared with others is the same then delete one of the files which is a duplicate.
0
Comment
Question by:100questions
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 3
  • 2
9 Comments
 
LVL 18

Expert Comment

by:Emmanuel Adebayo
ID: 40342088
Hi,

You can use Multi-File Compare

To download files and see other information about the project, go to http://sourceforge.net/projects/multi-fcompare.

Rgds
0
 

Author Comment

by:100questions
ID: 40342173
Will this work in an existing batch script?
0
 
LVL 18

Expert Comment

by:Emmanuel Adebayo
ID: 40342192
No this is an executable.
0
Hire Technology Freelancers with Gigs

Work with freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely, and get projects done right.

 

Author Comment

by:100questions
ID: 40342243
Then I would need something that I can insert in an existing Windows Batch file or a new Powershell or VBScript which can perform the function.
0
 
LVL 9

Accepted Solution

by:
dlb6597 earned 2000 total points
ID: 40342313
barebones, inefficient...definately test this with a subset of your data.
basically for every text file this launches another for loop that compares each .txt file to every other .txt file and deletes if there is a match. The script starts over after a deletion because the (*.txt) set changes...

:start
for %%i in (*.txt) do (
	for %%j in (*.txt) do (
	if not "%%i" == "%%j" fc %%i %%j && del %%i && goto start
)
)

Open in new window

0
 

Author Comment

by:100questions
ID: 40342336
Does this script look into the contents of the txt file?
0
 
LVL 9

Expert Comment

by:dlb6597
ID: 40342351
yes, it compares file contents using the fc command.
0
 

Author Comment

by:100questions
ID: 40344239
This seems to work, however the problem is that one of the files it compares contains a small right arrow at the end of the data (an ASCII EOF marker) and if it sees that then it does not deduplicate properly.  

Is there a way your script can be modified so as to ignore the an ASCII EOF marker?
0
 
LVL 9

Expert Comment

by:dlb6597
ID: 40344258
then the files aren't identical then are they?  There is a /L parameter for FC, but I doubt it will make any difference since the files are truly different.
0

Featured Post

How to Use the Help Bell

Need to boost the visibility of your question for solutions? Use the Experts Exchange Help Bell to confirm priority levels and contact subject-matter experts for question attention.  Check out this how-to article for more information.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When you receive another warning that your shared drive is almost full and you have asked your users to clean out old files again and again, here is a single command that may help. This command will place all the files that have not been used rec…
This is a fine trick which I've found useful many times, when you just don't want to accidentally run a batch script or the commands needs administrator rights.
Video by: ITPro.TV
In this episode Don builds upon the troubleshooting techniques by demonstrating how to properly monitor a vSphere deployment to detect problems before they occur. He begins the show using tools found within the vSphere suite as ends the show demonst…
Visualize your data even better in Access queries. Given a date and a value, this lesson shows how to compare that value with the previous value, calculate the difference, and display a circle if the value is the same, an up triangle if it increased…

721 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question