October 12, 2008 07:12pm pdt

My Account

Top 15 Experts

Overall Top Experts

1TheLearn... 4,264,878
2Idle_Mind 3,606,568
3Sancler 2,354,988
4Fernando... 2,104,478
5jpaulino 1,349,857
6RonaldBie... 1,159,695
7emoreau 1,098,034
8VBRocks 1,030,108
9iboutchkine 817,529
10angelIII 693,669
11Chaosian 688,334
12webtubbs 676,915
13Dhaest 644,774
14PaulHews 642,548
15planocz 634,195

	[x] Attachment Details

Best Way to Parse A Huge Text File

[x]

The Solution Rating System

With so many solutions, how can you tell which solutions are most likely to help you and which ones are not? To provide you with a tool to use, we rate our solutions based on various elements that most accurately determine if a solution is a quality solution. To explain what factors affect the solution rating, here are the elements we take into consideration when formulating our solution rating.

The Grade of the Solution
The Zone Rank of the Expert Providing the Solution
The Number of Author and Expert Comments
The Number of Experts Contributing
The Feedback of the Community

Your Input Matters
Because of the way the system is set up, the most important variable in this equation is you. As a member of Experts Exchange, you are able to cast your vote on the quality of the solutions in regard to how complete, accurate, helpful and easy to understand each solution is. When you provide your feedback, each rating is adjusted accordingly. So, if you see a solution that has a poor rating that you think is a good solution, let us know by rating it. As you do, the rating will be adjusted and will become more accurate for other members of our site.

If you have any suggestions that you would like to make for our rating system, please ask a question in the Suggestions Zone of Community Support.

Thank you!

9.4

Zone: Microsoft Visual Basic.Net

Tags: VB.NET, Text File, Parse Text File, .NET, Parsing Large Text File

Greetings Experts,

I have a huge csv file that has around 765,000 rows of part numbers with their respective inventory levels. Unfortunately, my distributor cannot supply another format or even a slimmed down version of this file. With that in mind, I am parsing through this huge file to get the current stock levels for 4,000 items. While it is working, IT IS SLOW!!!

What would be the most efficient approach to parsing this file keeping in mind that I only need the stock levels for 4,000 items?

I am currently do someting like this:

Thank you in advance,

Keith

I

         
           
             1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:
44:
45:
46:
47:
48:
49:
50:
51:
52:
53:
54:
55:
56:
57:
58:
59:
60:
61:
62:
63:
64:
65:
66:
67:
68:
69:
70:
71:
72:
73:
74:
75:
76:
77:
78:
79:

           
           
             Public Function ReadFile(ByVal fileName As String) As DataTable
 
        ' Initialize the return values
        
        Dim list As New List(Of String())
        Dim table As DataTable = Nothing
        Dim myCT As Integer = 0
 
        Using parser As New TextFieldParser(fileName)
 
            ' Setup the comma-delimited file parser.
            parser.TextFieldType = FieldType.Delimited
            parser.Delimiters = New String() {","}
            parser.HasFieldsEnclosedInQuotes = True
 
            While Not parser.EndOfData
 
                Try
 
                    ' Read the comma-delimited text as fields into a string array.
 
                    Dim input As String() = parser.ReadFields()
 
                    If table Is Nothing Then
                        table = CreateTable(Path.GetFileName(fileName), input)
                    End If
 
                    AddRow(table, input)
 
                Catch ex As MalformedLineException
 
                    ' Ignore invalid lines.
 
                End Try
 
            End While
 
        End Using
 
        Return table
 
    End Function
 
Private Function CreateTable(ByVal name As String, ByVal input As String()) As DataTable
 
        Dim table As New DataTable(name)
      
            ' Add the table columns
 
            With table
 
                .Columns.Add("ActionDef")
                .Columns.Add("VendorPartNo")
                .Columns.Add("MFGNAME")
                .Columns.Add("Desc1")
                .Columns.Add("Desc2")
                .Columns.Add("RetailPrice")
                .Columns.Add("MFGPartNo")
                .Columns.Add("Weight")
                .Columns.Add("UPC")
                .Columns.Add("LENGTH")
                .Columns.Add("WIDTH")
                .Columns.Add("HEIGHT")
                .Columns.Add("CHGFLAG")
                .Columns.Add("QLTCOST")
                .Columns.Add("SPPRICE")
                .Columns.Add("STOCKYN")
                .Columns.Add("STATUS")
                .Columns.Add("CPUCODE")
                .Columns.Add("PRIMCAT")
                .Columns.Add("SUBCAT")
                .Columns.Add("NA")
                .Columns.Add("NEWSTOCK")
                .Columns.Add("REBATE")
                .Columns.Add("SUBPART")
 
            End With              
 
        Return table
           
         

	#	Title
	1	PAR utility for newbie
	2	PAR Protocol
	3	par file
	4	Par files
	5	Good inventory/Stock software
	6	Par. Query