Find distinct value from values in a text file....

I have a text file of size >5,000,000 KB. It has a header and a trailer record, and all records between these two are body records. The records are pipe-delimited. Each body record has 236 columns. I want to find out the distinct values present in columns 168 and 169. The file is too large to import into Excel, otherwise I could have set a filter on each column and viewed the distinct values. It is also too large to open in MS-Word. How can I go about finding out the distinct values in the two columns I am interested in?

Thank you.
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

dsackerContract ERP Admin/ConsultantCommented:
VBScript or Powershell would work.

You'd read each record in, use the Split function to reference YourRecord.Split("|")(167) and YourRecord.Split("|")(168), concatenate them into another array, testing first whether you already placed it in that other array, sort it if you wish, then echo it out.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
dsackerContract ERP Admin/ConsultantCommented:
This VBScript template might work:
Option Explicit

Call Main

Sub Main
    Dim blnFound
    Dim cnt
    Dim ndx
    Dim objFSO
    Dim objFileIn
    Dim strArray()
    Dim strFileName
    Dim strKey
    Dim strRecord

    cnt = -1
    strFileName = "C:\Temp\YourInputFile.txt"
    Set objFSO     = CreateObject("Scripting.FileSystemObject")
    Set objFileIn  = objFSO.OpenTextFile(strFileName, 1, 2)

    Do While objFileIn.AtEndOfStream = False
        strRecord = objFileIn.ReadLine
        strKey = Split(strRecord, "|")(167) & "|" & Split(strRecord, "|")(168)
        If cnt = -1 Then
            cnt = cnt + 1
            Redim strArray(cnt)
            strArray(cnt) = strKey
            blnFound = False
            For ndx = 0 To UBound(strArray)
                If strArray(ndx) = strKey Then
                    blnFound = True
                    Exit For
                End If
            If blnFound = False Then
                cnt = cnt + 1
                Redim Preserve strArray(cnt)
                strArray(cnt) = strKey
            End If
        End If

    Set objFileIn  = Nothing
    Set objFSO     = Nothing

    For ndx = 0 To Ubound(strArray)
        WScript.Echo strArray(ndx)
End Sub

Open in new window

dsackerContract ERP Admin/ConsultantCommented:
Even sweeter is the small amount of Powershell needed:
$file = Get-Content C:\Temp\YourInputFile.txt
$array = @()
ForEach ($line in $file) {
    $fields = $line.Split("|");
    $key = $fields[167]+ "|" + $fields[168]
    If ($array.Contains($key) -eq $False) { $array += $key }

Open in new window

Acronis Data Cloud 7.8 Enhances Cyber Protection

A closer look at five essential enhancements that benefit end-users and help MSPs take their cloud data protection business further.

aej1973Author Commented:
Thanks dsacker, how will the output from this poweshell look? Thank you.

dsackerContract ERP Admin/ConsultantCommented:
The lone "$array" will spit out the two values you're interested in. If you need to change that and grab other values, just tack them on.

Play with it some. Powershell can be addicting. :)
aej1973Author Commented:
Will do, thank you.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Scripting Languages

From novice to tech pro — start learning today.