Solved

Start reading a File from a Specific Row or Position

Posted on 2004-09-17
6
419 Views
Last Modified: 2010-05-18
Basically I know how to loop through the file with StreamReader (i.e. sr.ReadLine) but what I want to do is start reading a row from a specific point in the file.

My routine basically reads a file into StreamReader, loops through the starting set of lines to get to my Starting Point, and then it reads the lines I want extracted to another file.

Here is some example code:
//iStart is declared as int32, line is a String
//oWrite is a class that creates the new file

Private Function openFile(ByVal fileName As String, ByVal getCount As Int32)
            Try
                Dim sr As StreamReader = New StreamReader(fileName)
                Dim sLines As New System.Text.StringBuilder
                Do While sCount <> iStart
                    line = sr.ReadLine
                    sCount += 1
                Loop
                sCount = 0
                Do While sCount <> getCount
                    line = sr.ReadLine
                    sLines.Append(line & vbCrLf)
                    sCount += 1
                Loop
                oWrite.WriteParse(sLines.ToString(), sfileName)
                sr.Close()
                sCount = 0
            Catch ex As Exception
                Throw ex
            End Try
        End Function

The worst situations occur when I am loading a file that has 2 million rows and I need something that started after the 500,000th row.  It takes a while to loop to the 500,000th position in the file, and when it finally gets there I am having memory problems slowing down my computer.  Any suggestions on how to jump to the position without reading each line?
0
Comment
Question by:deadferret
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 2
6 Comments
 
LVL 4

Expert Comment

by:vigrid
ID: 12089456
In order to know (for sure) the current line number you are reading, you must read the whole file from the beginning to the current point and count the CR/LF occurences.

Overloads Overrides Public Function Read(Char(), Integer, Integer) As Integer

You could read a file fragment to a predefined buffer, say in 16kb chunks and analyze them manually to find CR/LFs. That would solve your problem with memory.

HTH
0
 
LVL 4

Expert Comment

by:vigrid
ID: 12089462
ms-help://MS.MSDNQTR.2004JAN.1033/cpref/html/frlrfSystemIOStreamReaderClassReadTopic.htm

MSDN help link
0
 

Author Comment

by:deadferret
ID: 12101892
How would I go about doing this?  "You could read a file fragment to a predefined buffer, say in 16kb chunks and analyze them manually to find CR/LFs. That would solve your problem with memory."
0
Instantly Create Instructional Tutorials

Contextual Guidance at the moment of need helps your employees adopt to new software or processes instantly. Boost knowledge retention and employee engagement step-by-step with one easy solution.

 

Author Comment

by:deadferret
ID: 12102136
I also continue to get this message when trying to read for a Cr/Lf.

"Offset and length were out of bounds for the array or count is greater than the number of elements from index to the end of the source collection."

Do I have to give it the full length of the file?
0
 
LVL 4

Expert Comment

by:vigrid
ID: 12106447
Private Shared Sub Main(ByVal args As String())
      Dim fs As FileStream
      If (args.Length = 1) Then
            fs = New FileStream(args(0), FileMode.Open)
      Else
            Console.Write("Enter file name: ")
            Dim fileName As String = Console.ReadLine
            fs = New FileStream(fileName, FileMode.Open)
      End If
      Dim buffer As Byte() = New Byte(262144) {}
      Dim bufferSize As Integer = buffer.Length
      Dim offset As Integer = 0
      Dim readBytes As Integer = 0
      Dim lines As Integer = 0
      Do
            readBytes = fs.Read(buffer, 0, bufferSize)
            offset = (offset + readBytes)
            Console.Write("Read {0} bytes... ", offset)
            Dim counter As Integer = 0
            Do While (counter < readBytes)
                  lines = (lines + IIf((buffer(counter++) = 10), 1, 0))
            Loop
            Console.WriteLine("{0} lines found", lines)
      Loop ((readBytes = bufferSize) AndAlso (offset <> fs.Length))
      fs.Close
End Sub
0
 
LVL 4

Accepted Solution

by:
vigrid earned 250 total points
ID: 12106499
Above code is a translation from C# made by .NET Reflector. Sorry, I'm not an expert in VB.NET, but as the .NET Framework is the same to all of the languages, I'm trying to help :)

The magic number 262144 is 256kbytes of memory - size of the buffer.
The magic number 10 is ASCII code for the newline character.

Above reads the file in 0.25MB chunks and searches for the newline characters. When you find enough of them, you must note the offset where it did occur (should be something like: seekOffset = offset - readBytes + counter) and use the FileStream.Seek method to point your stream at the beginning of the line you need.

HTH
0

Featured Post

Enroll in June's Course of the Month

June’s Course of the Month is now available! Experts Exchange’s Premium Members, Team Accounts, and Qualified Experts have access to a complimentary course each month as part of their membership—an extra way to sharpen your skills and increase training.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The ECB site provides FX rates for major currencies since its inception in 1999 in the form of an XML feed. The files have the following format (reducted for brevity) (CODE) There are three files available HERE (http://www.ecb.europa.eu/stats/exch…
It was really hard time for me to get the understanding of Delegates in C#. I went through many websites and articles but I found them very clumsy. After going through those sites, I noted down the points in a easy way so here I am sharing that unde…
In this video we outline the Physical Segments view of NetCrunch network monitor. By following this brief how-to video, you will be able to learn how NetCrunch visualizes your network, how granular is the information collected, as well as where to f…
Monitoring a network: why having a policy is the best policy? Michael Kulchisky, MCSE, MCSA, MCP, VTSP, VSP, CCSP outlines the enormous benefits of having a policy-based approach when monitoring medium and large networks. Software utilized in this v…

688 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question