Solved

Remove duplicate lines from text file using vbscript?

Posted on 2007-03-21
5
10,891 Views
Last Modified: 2008-03-06
Remove Duplicate Lines From Text File.

How can i remove duplicate lines from text file?
The format of the file is as follows

flipper.berkeley.edu
flipper.berkeley.edu
ftp.aao.gov.au
ftp.deet.gov.au
ftp.deet.gov.au
ftp.deetya.gov.au
ftp.deetya.gov.au
ftp.ece.jcu.edu.au
ftp.eng.jcu.edu.au
ftp.hq.nasa.gov

I am using VBS Script
0
Comment
Question by:dion_p1
  • 2
  • 2
5 Comments
 
LVL 67

Accepted Solution

by:
sirbounty earned 300 total points
ID: 18762947
Try this...

'SortIt.vbs
strFile="C:\YourFile.txt"
Dim objFSO: Set objFSO = CreateObject("Scripting.FileSystemObject")
Dim objFile: Set objFile = objFSO.OpenTextFile(strFile)
Dim dicSort: Set dicSort = CreateObject("Scripting.Dictionary")

Do While Not objFile.AtEndOfStream
    On Error Resume Next
    strData = objFile.ReadLine
    dicSort.Add strData, dicSort.Count
Loop

objFile.Close
Set objFile = objFSO.CreateTextFile(strFile)
For Each Item In dicSort
  objFile.WriteLine Item
Next
objFile.Close
Set objFile = Nothing
Set objFSO=Nothing
0
 
LVL 9

Assisted Solution

by:samopal
samopal earned 200 total points
ID: 18767582
  set cnCSV = CreateObject("ADODB.Connection")
   cnCSV.ConnectionString = "Driver={Microsoft Text Driver (*.txt; *.csv)};Initial Catalog=."
   cnCSV.Open
   cnCSV.Execute "SELECT distinct * INTO OUT.txt FROM IN.txt"
   cnCSV.Close
   Set cnCSV = Nothing
0
 
LVL 1

Expert Comment

by:Simon336697
ID: 25532303
Hi guys, hi Sir Bounty.
Sir Bounty, Im trying to following what you have done. Is it possible to clarify for me the following?

1) You create a dictionary object
2) You then read into the dictionary each line of the text file
What im not getting is, are you writing in every duplicate line into the dictionary object?

Any help clarifying much appreciated mate.
0
 
LVL 67

Expert Comment

by:sirbounty
ID: 25534576
Well Simon - this is quite old! :^)
But reading over the code above, it will error with duplicates - a dictionary won't allow duplicate indexes - which is why the 'on error' line (so any duplicates will simply be skipped).
0
 
LVL 1

Expert Comment

by:Simon336697
ID: 25541432
Hi Sirbounty,
As usual, you're everywhere and always in a good way.
Thanks for clearing that up.
Take care :>)
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Script to copy or move mouse-selected collection of files plus targets referenced by shortcuts (.lnk) The purpose of this article is to help illuminate the real challenges and options available (where they may exist) for utilizing simple scriptin…
Introduction During my participation as a VBScript contributor at Experts Exchange, one of the most common questions I come across is this: "I have a script that runs against only one computer. How can I make it run against a list of computers in …
When you create an app prototype with Adobe XD, you can insert system screens -- sharing or Control Center, for example -- with just a few clicks. This video shows you how. You can take the full course on Experts Exchange at http://bit.ly/XDcourse.
This video demonstrates how to create an example email signature rule for a department in a company using CodeTwo Exchange Rules. The signature will be inserted beneath users' latest emails in conversations and will be displayed in users' Sent Items…

948 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

22 Experts available now in Live!

Get 1:1 Help Now