troubleshooting Question

vb.net fix issue with dup remover

Avatar of XK8ER
XK8ERFlag for United States of America asked on
Visual Basic.NET.NET ProgrammingC#
2 Comments1 Solution449 ViewsLast Modified:
hello,
I am having a little issue this dup remover source I have..

For example:
> ids_database.txt has lines by lines like this

65098908|name1|last1
98046545|name2|last2
65098998|name3|last3
25989808|name4|last4

>new_ids.txt has lines by line like this

65098908|this_is|a_dup_id
98046584|name5|last5
65068945|name6|last6
32106890|name7|last7

when you run you should see this 3 lines in printed to "ids_to_fix.txt"

98046584|name5|last5
65068945|name6|last6
32106890|name7|last7

thats' because those 3 ids before the "|" pipe are not in the database.
everything after the pipe is ignored, this is just for checking IDs against from ids_database..if not found then create a text file..

its not happening whats going on ?


Imports System.Collections.Generic
Imports System.Linq
Imports System.Text
Imports System.IO

Module dupRemover
    Sub Main()
        Console.WriteLine("{0}: Start", DateTime.Now)

        Dim data As New HashSet(Of String)()

        Try
            ' read your main IDs
            Using reader As New StreamReader("ids_database.txt")
                While reader.Peek() > 0
                    data.Add(reader.ReadLine().Trim())
                End While
            End Using

            Dim ids_to_fix As New List(Of String)()

            'read new IDs and if they are not in data - add them
            Using reader As New StreamReader("new_ids.txt")
                Dim tempArr As New List(Of String)
                tempArr.AddRange(String.Join(",", data.ToArray()).ToLower.Split("|".ToCharArray(), StringSplitOptions.RemoveEmptyEntries))
                While reader.Peek() > -1
                    Dim sLine As String = reader.ReadLine().Trim()
                    Dim firstPart As String = sLine.Split(",")(0)
                    If Not tempArr.Contains(firstPart.ToLower) Then
                        data.Add(sLine)
                        ids_to_fix.Add(sLine)
                    End If
                End While
            End Using

            ' Write new IDs back to ids_to_fix
            Using writer As New StreamWriter("ids_to_fix.txt")
                For Each sLine As String In ids_to_fix
                    writer.WriteLine(sLine)
                Next
            End Using

            ' Eventually, write everything back to database
            Using writer As New StreamWriter("ids_database.txt")
                For Each sLine As String In data
                    writer.WriteLine(sLine)
                Next
            End Using
        Catch ex As Exception
            Console.WriteLine("An error occurred - {0}", ex.Message)
        End Try

        Console.WriteLine("{0}: End", DateTime.Now)
        Console.ReadLine() 'pause
    End Sub
End Module

Open in new window

ASKER CERTIFIED SOLUTION
ElrondCT

Our community of experts have been thoroughly vetted for their expertise and industry experience.

Join our community to see this answer!
Unlock 1 Answer and 2 Comments.
Start Free Trial
Learn from the best

Network and collaborate with thousands of CTOs, CISOs, and IT Pros rooting for you and your success.

Andrew Hancock - VMware vExpert
See if this solution works for you by signing up for a 7 day free trial.
Unlock 1 Answer and 2 Comments.
Try for 7 days

”The time we save is the biggest benefit of E-E to our team. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange.

-Mike Kapnisakis, Warner Bros