Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Remove duplicate lines from text file using vbscript?

Posted on 2007-03-21
5
Medium Priority
?
11,213 Views
Last Modified: 2008-03-06
Remove Duplicate Lines From Text File.

How can i remove duplicate lines from text file?
The format of the file is as follows

flipper.berkeley.edu
flipper.berkeley.edu
ftp.aao.gov.au
ftp.deet.gov.au
ftp.deet.gov.au
ftp.deetya.gov.au
ftp.deetya.gov.au
ftp.ece.jcu.edu.au
ftp.eng.jcu.edu.au
ftp.hq.nasa.gov

I am using VBS Script
0
Comment
Question by:dion_p1
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
5 Comments
 
LVL 67

Accepted Solution

by:
sirbounty earned 1200 total points
ID: 18762947
Try this...

'SortIt.vbs
strFile="C:\YourFile.txt"
Dim objFSO: Set objFSO = CreateObject("Scripting.FileSystemObject")
Dim objFile: Set objFile = objFSO.OpenTextFile(strFile)
Dim dicSort: Set dicSort = CreateObject("Scripting.Dictionary")

Do While Not objFile.AtEndOfStream
    On Error Resume Next
    strData = objFile.ReadLine
    dicSort.Add strData, dicSort.Count
Loop

objFile.Close
Set objFile = objFSO.CreateTextFile(strFile)
For Each Item In dicSort
  objFile.WriteLine Item
Next
objFile.Close
Set objFile = Nothing
Set objFSO=Nothing
0
 
LVL 9

Assisted Solution

by:samopal
samopal earned 800 total points
ID: 18767582
  set cnCSV = CreateObject("ADODB.Connection")
   cnCSV.ConnectionString = "Driver={Microsoft Text Driver (*.txt; *.csv)};Initial Catalog=."
   cnCSV.Open
   cnCSV.Execute "SELECT distinct * INTO OUT.txt FROM IN.txt"
   cnCSV.Close
   Set cnCSV = Nothing
0
 
LVL 1

Expert Comment

by:Simon336697
ID: 25532303
Hi guys, hi Sir Bounty.
Sir Bounty, Im trying to following what you have done. Is it possible to clarify for me the following?

1) You create a dictionary object
2) You then read into the dictionary each line of the text file
What im not getting is, are you writing in every duplicate line into the dictionary object?

Any help clarifying much appreciated mate.
0
 
LVL 67

Expert Comment

by:sirbounty
ID: 25534576
Well Simon - this is quite old! :^)
But reading over the code above, it will error with duplicates - a dictionary won't allow duplicate indexes - which is why the 'on error' line (so any duplicates will simply be skipped).
0
 
LVL 1

Expert Comment

by:Simon336697
ID: 25541432
Hi Sirbounty,
As usual, you're everywhere and always in a good way.
Thanks for clearing that up.
Take care :>)
0

Featured Post

Important Lessons on Recovering from Petya

In their most recent webinar, Skyport Systems explores ways to isolate and protect critical databases to keep the core of your company safe from harm.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this article we want to have a look at the directory attributes which are used by Microsoft to store the so called Security Identifiers (SID). These SIDs plays an important role in delegating and granting permissions and in authentication of trus…
I met Paul Devereux (@pdevereux) today when I responded to his tweet asking “Anybody know how to automate adding files from disk to a folder in #outlook  ?”.  I replied back and told Paul that using automation, in this case scripting, to add files t…
In this video, Percona Director of Solution Engineering Jon Tobin discusses the function and features of Percona Server for MongoDB. How Percona can help Percona can help you determine if Percona Server for MongoDB is the right solution for …
In this video, Percona Solution Engineer Rick Golba discuss how (and why) you implement high availability in a database environment. To discuss how Percona Consulting can help with your design and architecture needs for your database and infrastr…

670 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question