Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
?
Solved

How can I compare several text files and keep only 1 copy of a file without duplication?

Posted on 2014-09-24
9
Medium Priority
?
143 Views
Last Modified: 2014-10-01
I would like to compare several text files, ie new1.txt and new2.txt, new3.txt .. regardless of the name of the files.
If the 'content's of the files compared with others is the same then delete one of the files which is a duplicate.
0
Comment
Question by:100questions
  • 4
  • 3
  • 2
9 Comments
 
LVL 18

Expert Comment

by:Emmanuel Adebayo
ID: 40342088
Hi,

You can use Multi-File Compare

To download files and see other information about the project, go to http://sourceforge.net/projects/multi-fcompare.

Rgds
0
 

Author Comment

by:100questions
ID: 40342173
Will this work in an existing batch script?
0
 
LVL 18

Expert Comment

by:Emmanuel Adebayo
ID: 40342192
No this is an executable.
0
Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

 

Author Comment

by:100questions
ID: 40342243
Then I would need something that I can insert in an existing Windows Batch file or a new Powershell or VBScript which can perform the function.
0
 
LVL 9

Accepted Solution

by:
dlb6597 earned 2000 total points
ID: 40342313
barebones, inefficient...definately test this with a subset of your data.
basically for every text file this launches another for loop that compares each .txt file to every other .txt file and deletes if there is a match. The script starts over after a deletion because the (*.txt) set changes...

:start
for %%i in (*.txt) do (
	for %%j in (*.txt) do (
	if not "%%i" == "%%j" fc %%i %%j && del %%i && goto start
)
)

Open in new window

0
 

Author Comment

by:100questions
ID: 40342336
Does this script look into the contents of the txt file?
0
 
LVL 9

Expert Comment

by:dlb6597
ID: 40342351
yes, it compares file contents using the fc command.
0
 

Author Comment

by:100questions
ID: 40344239
This seems to work, however the problem is that one of the files it compares contains a small right arrow at the end of the data (an ASCII EOF marker) and if it sees that then it does not deduplicate properly.  

Is there a way your script can be modified so as to ignore the an ASCII EOF marker?
0
 
LVL 9

Expert Comment

by:dlb6597
ID: 40344258
then the files aren't identical then are they?  There is a /L parameter for FC, but I doubt it will make any difference since the files are truly different.
0

Featured Post

[Webinar] Database Backup and Recovery

Does your company store data on premises, off site, in the cloud, or a combination of these? If you answered “yes”, you need a data backup recovery plan that fits each and every platform. Watch now as as Percona teaches us how to build agile data backup recovery plan.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article was inspired by a question here at Experts Exchange (http://www.experts-exchange.com/Software/Photos_Graphics/Images_and_Photos/Q_28629170.html). The requirements stated in that question are (1) reduce the file size of a large number of…
Use this article to create a batch file to backup a Microsoft SQL Server database to a Windows folder.  The folder can be on the local hard drive or on a network share.  This batch file will query the SQL server to get the current date & time and wi…
Whether it be Exchange Server Crash Issues, Dirty Shutdown Errors or Failed to mount error, Stellar Phoenix Mailbox Exchange Recovery has always got your back. With the help of its easy to understand user interface and 3 simple steps recovery proced…
The Relationships Diagram is a good way to get an overall view of what a database is keeping track of. It is also where relationships are defined. A relationship specifies how two tables connect to each other. As you build tables in Microsoft Ac…

581 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question