Solved

Start reading a File from a Specific Row or Position

Posted on 2004-09-17
6
408 Views
Last Modified: 2010-05-18
Basically I know how to loop through the file with StreamReader (i.e. sr.ReadLine) but what I want to do is start reading a row from a specific point in the file.

My routine basically reads a file into StreamReader, loops through the starting set of lines to get to my Starting Point, and then it reads the lines I want extracted to another file.

Here is some example code:
//iStart is declared as int32, line is a String
//oWrite is a class that creates the new file

Private Function openFile(ByVal fileName As String, ByVal getCount As Int32)
            Try
                Dim sr As StreamReader = New StreamReader(fileName)
                Dim sLines As New System.Text.StringBuilder
                Do While sCount <> iStart
                    line = sr.ReadLine
                    sCount += 1
                Loop
                sCount = 0
                Do While sCount <> getCount
                    line = sr.ReadLine
                    sLines.Append(line & vbCrLf)
                    sCount += 1
                Loop
                oWrite.WriteParse(sLines.ToString(), sfileName)
                sr.Close()
                sCount = 0
            Catch ex As Exception
                Throw ex
            End Try
        End Function

The worst situations occur when I am loading a file that has 2 million rows and I need something that started after the 500,000th row.  It takes a while to loop to the 500,000th position in the file, and when it finally gets there I am having memory problems slowing down my computer.  Any suggestions on how to jump to the position without reading each line?
0
Comment
Question by:deadferret
  • 4
  • 2
6 Comments
 
LVL 4

Expert Comment

by:vigrid
ID: 12089456
In order to know (for sure) the current line number you are reading, you must read the whole file from the beginning to the current point and count the CR/LF occurences.

Overloads Overrides Public Function Read(Char(), Integer, Integer) As Integer

You could read a file fragment to a predefined buffer, say in 16kb chunks and analyze them manually to find CR/LFs. That would solve your problem with memory.

HTH
0
 
LVL 4

Expert Comment

by:vigrid
ID: 12089462
ms-help://MS.MSDNQTR.2004JAN.1033/cpref/html/frlrfSystemIOStreamReaderClassReadTopic.htm

MSDN help link
0
 

Author Comment

by:deadferret
ID: 12101892
How would I go about doing this?  "You could read a file fragment to a predefined buffer, say in 16kb chunks and analyze them manually to find CR/LFs. That would solve your problem with memory."
0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 

Author Comment

by:deadferret
ID: 12102136
I also continue to get this message when trying to read for a Cr/Lf.

"Offset and length were out of bounds for the array or count is greater than the number of elements from index to the end of the source collection."

Do I have to give it the full length of the file?
0
 
LVL 4

Expert Comment

by:vigrid
ID: 12106447
Private Shared Sub Main(ByVal args As String())
      Dim fs As FileStream
      If (args.Length = 1) Then
            fs = New FileStream(args(0), FileMode.Open)
      Else
            Console.Write("Enter file name: ")
            Dim fileName As String = Console.ReadLine
            fs = New FileStream(fileName, FileMode.Open)
      End If
      Dim buffer As Byte() = New Byte(262144) {}
      Dim bufferSize As Integer = buffer.Length
      Dim offset As Integer = 0
      Dim readBytes As Integer = 0
      Dim lines As Integer = 0
      Do
            readBytes = fs.Read(buffer, 0, bufferSize)
            offset = (offset + readBytes)
            Console.Write("Read {0} bytes... ", offset)
            Dim counter As Integer = 0
            Do While (counter < readBytes)
                  lines = (lines + IIf((buffer(counter++) = 10), 1, 0))
            Loop
            Console.WriteLine("{0} lines found", lines)
      Loop ((readBytes = bufferSize) AndAlso (offset <> fs.Length))
      fs.Close
End Sub
0
 
LVL 4

Accepted Solution

by:
vigrid earned 250 total points
ID: 12106499
Above code is a translation from C# made by .NET Reflector. Sorry, I'm not an expert in VB.NET, but as the .NET Framework is the same to all of the languages, I'm trying to help :)

The magic number 262144 is 256kbytes of memory - size of the buffer.
The magic number 10 is ASCII code for the newline character.

Above reads the file in 0.25MB chunks and searches for the newline characters. When you find enough of them, you must note the offset where it did occur (should be something like: seekOffset = offset - readBytes + counter) and use the FileStream.Seek method to point your stream at the beginning of the line you need.

HTH
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Article by: Kraeven
Introduction Remote Share is a simple remote sharing tool, enabling you to see, add and remove remote or local shares. The application is written in VB.NET targeting the .NET framework 2.0. The source code and the compiled programs have been in…
Introduction When many people think of the WebBrowser (http://msdn.microsoft.com/en-us/library/2te2y1x6%28v=VS.85%29.aspx) control, they immediately think of a control which allows the viewing and navigation of web pages. While this is true, it's a…
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…
This video gives you a great overview about bandwidth monitoring with SNMP and WMI with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're looking for how to monitor bandwidth using netflow or packet s…

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

16 Experts available now in Live!

Get 1:1 Help Now