Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people, just like you, are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
Solved

Remove duplicate lines from text file using vbscript?

Posted on 2007-03-21
5
10,954 Views
Last Modified: 2008-03-06
Remove Duplicate Lines From Text File.

How can i remove duplicate lines from text file?
The format of the file is as follows

flipper.berkeley.edu
flipper.berkeley.edu
ftp.aao.gov.au
ftp.deet.gov.au
ftp.deet.gov.au
ftp.deetya.gov.au
ftp.deetya.gov.au
ftp.ece.jcu.edu.au
ftp.eng.jcu.edu.au
ftp.hq.nasa.gov

I am using VBS Script
0
Comment
Question by:dion_p1
  • 2
  • 2
5 Comments
 
LVL 67

Accepted Solution

by:
sirbounty earned 300 total points
ID: 18762947
Try this...

'SortIt.vbs
strFile="C:\YourFile.txt"
Dim objFSO: Set objFSO = CreateObject("Scripting.FileSystemObject")
Dim objFile: Set objFile = objFSO.OpenTextFile(strFile)
Dim dicSort: Set dicSort = CreateObject("Scripting.Dictionary")

Do While Not objFile.AtEndOfStream
    On Error Resume Next
    strData = objFile.ReadLine
    dicSort.Add strData, dicSort.Count
Loop

objFile.Close
Set objFile = objFSO.CreateTextFile(strFile)
For Each Item In dicSort
  objFile.WriteLine Item
Next
objFile.Close
Set objFile = Nothing
Set objFSO=Nothing
0
 
LVL 9

Assisted Solution

by:samopal
samopal earned 200 total points
ID: 18767582
  set cnCSV = CreateObject("ADODB.Connection")
   cnCSV.ConnectionString = "Driver={Microsoft Text Driver (*.txt; *.csv)};Initial Catalog=."
   cnCSV.Open
   cnCSV.Execute "SELECT distinct * INTO OUT.txt FROM IN.txt"
   cnCSV.Close
   Set cnCSV = Nothing
0
 
LVL 1

Expert Comment

by:Simon336697
ID: 25532303
Hi guys, hi Sir Bounty.
Sir Bounty, Im trying to following what you have done. Is it possible to clarify for me the following?

1) You create a dictionary object
2) You then read into the dictionary each line of the text file
What im not getting is, are you writing in every duplicate line into the dictionary object?

Any help clarifying much appreciated mate.
0
 
LVL 67

Expert Comment

by:sirbounty
ID: 25534576
Well Simon - this is quite old! :^)
But reading over the code above, it will error with duplicates - a dictionary won't allow duplicate indexes - which is why the 'on error' line (so any duplicates will simply be skipped).
0
 
LVL 1

Expert Comment

by:Simon336697
ID: 25541432
Hi Sirbounty,
As usual, you're everywhere and always in a good way.
Thanks for clearing that up.
Take care :>)
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Not long ago I saw a question in the VB Script forum that I thought would not take much time. You can read that question (Question ID  (http://www.experts-exchange.com/Programming/Languages/Visual_Basic/VB_Script/Q_28455246.html)28455246) Here (http…
Deploying a Microsoft Access application in a Citrix environment is not difficult but takes a few steps. However, Citrix system people are often of little help, as they typically know next to nothing about Access. The script provided here will take …
Microsoft Active Directory, the widely used IT infrastructure, is known for its high risk of credential theft. The best way to test your Active Directory’s vulnerabilities to pass-the-ticket, pass-the-hash, privilege escalation, and malware attacks …
This video shows how to use Hyena, from SystemTools Software, to bulk import 100 user accounts from an external text file. View in 1080p for best video quality.

856 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question