Solved

Remove duplicate lines from text file using vbscript?

Posted on 2007-03-21
5
10,982 Views
Last Modified: 2008-03-06
Remove Duplicate Lines From Text File.

How can i remove duplicate lines from text file?
The format of the file is as follows

flipper.berkeley.edu
flipper.berkeley.edu
ftp.aao.gov.au
ftp.deet.gov.au
ftp.deet.gov.au
ftp.deetya.gov.au
ftp.deetya.gov.au
ftp.ece.jcu.edu.au
ftp.eng.jcu.edu.au
ftp.hq.nasa.gov

I am using VBS Script
0
Comment
Question by:dion_p1
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
5 Comments
 
LVL 67

Accepted Solution

by:
sirbounty earned 300 total points
ID: 18762947
Try this...

'SortIt.vbs
strFile="C:\YourFile.txt"
Dim objFSO: Set objFSO = CreateObject("Scripting.FileSystemObject")
Dim objFile: Set objFile = objFSO.OpenTextFile(strFile)
Dim dicSort: Set dicSort = CreateObject("Scripting.Dictionary")

Do While Not objFile.AtEndOfStream
    On Error Resume Next
    strData = objFile.ReadLine
    dicSort.Add strData, dicSort.Count
Loop

objFile.Close
Set objFile = objFSO.CreateTextFile(strFile)
For Each Item In dicSort
  objFile.WriteLine Item
Next
objFile.Close
Set objFile = Nothing
Set objFSO=Nothing
0
 
LVL 9

Assisted Solution

by:samopal
samopal earned 200 total points
ID: 18767582
  set cnCSV = CreateObject("ADODB.Connection")
   cnCSV.ConnectionString = "Driver={Microsoft Text Driver (*.txt; *.csv)};Initial Catalog=."
   cnCSV.Open
   cnCSV.Execute "SELECT distinct * INTO OUT.txt FROM IN.txt"
   cnCSV.Close
   Set cnCSV = Nothing
0
 
LVL 1

Expert Comment

by:Simon336697
ID: 25532303
Hi guys, hi Sir Bounty.
Sir Bounty, Im trying to following what you have done. Is it possible to clarify for me the following?

1) You create a dictionary object
2) You then read into the dictionary each line of the text file
What im not getting is, are you writing in every duplicate line into the dictionary object?

Any help clarifying much appreciated mate.
0
 
LVL 67

Expert Comment

by:sirbounty
ID: 25534576
Well Simon - this is quite old! :^)
But reading over the code above, it will error with duplicates - a dictionary won't allow duplicate indexes - which is why the 'on error' line (so any duplicates will simply be skipped).
0
 
LVL 1

Expert Comment

by:Simon336697
ID: 25541432
Hi Sirbounty,
As usual, you're everywhere and always in a good way.
Thanks for clearing that up.
Take care :>)
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Excel 2016 loop through 6 49
VB Script: restart service function 30 39
Need a simple script to SYNC/ Mirror a local folder to a network folder. 10 48
Modify file details 1 22
In this article we want to have a look at the directory attributes which are used by Microsoft to store the so called Security Identifiers (SID). These SIDs plays an important role in delegating and granting permissions and in authentication of trus…
Welcome, welcome!  If you are new to the series and haven't been following along, please take a brief moment to review the first three installments: Part 1 (http://www.experts-exchange.com/Programming/Languages/Visual_Basic/VB_Script/A_266-VBScri…
Nobody understands Phishing better than an anti-spam company. That’s why we are providing Phishing Awareness Training to our customers. According to a report by Verizon, only 3% of targeted users report malicious emails to management. With compan…

740 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question