Solved

Start reading a File from a Specific Row or Position

Posted on 2004-09-17
6
417 Views
Last Modified: 2010-05-18
Basically I know how to loop through the file with StreamReader (i.e. sr.ReadLine) but what I want to do is start reading a row from a specific point in the file.

My routine basically reads a file into StreamReader, loops through the starting set of lines to get to my Starting Point, and then it reads the lines I want extracted to another file.

Here is some example code:
//iStart is declared as int32, line is a String
//oWrite is a class that creates the new file

Private Function openFile(ByVal fileName As String, ByVal getCount As Int32)
            Try
                Dim sr As StreamReader = New StreamReader(fileName)
                Dim sLines As New System.Text.StringBuilder
                Do While sCount <> iStart
                    line = sr.ReadLine
                    sCount += 1
                Loop
                sCount = 0
                Do While sCount <> getCount
                    line = sr.ReadLine
                    sLines.Append(line & vbCrLf)
                    sCount += 1
                Loop
                oWrite.WriteParse(sLines.ToString(), sfileName)
                sr.Close()
                sCount = 0
            Catch ex As Exception
                Throw ex
            End Try
        End Function

The worst situations occur when I am loading a file that has 2 million rows and I need something that started after the 500,000th row.  It takes a while to loop to the 500,000th position in the file, and when it finally gets there I am having memory problems slowing down my computer.  Any suggestions on how to jump to the position without reading each line?
0
Comment
Question by:deadferret
  • 4
  • 2
6 Comments
 
LVL 4

Expert Comment

by:vigrid
ID: 12089456
In order to know (for sure) the current line number you are reading, you must read the whole file from the beginning to the current point and count the CR/LF occurences.

Overloads Overrides Public Function Read(Char(), Integer, Integer) As Integer

You could read a file fragment to a predefined buffer, say in 16kb chunks and analyze them manually to find CR/LFs. That would solve your problem with memory.

HTH
0
 
LVL 4

Expert Comment

by:vigrid
ID: 12089462
ms-help://MS.MSDNQTR.2004JAN.1033/cpref/html/frlrfSystemIOStreamReaderClassReadTopic.htm

MSDN help link
0
 

Author Comment

by:deadferret
ID: 12101892
How would I go about doing this?  "You could read a file fragment to a predefined buffer, say in 16kb chunks and analyze them manually to find CR/LFs. That would solve your problem with memory."
0
Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

 

Author Comment

by:deadferret
ID: 12102136
I also continue to get this message when trying to read for a Cr/Lf.

"Offset and length were out of bounds for the array or count is greater than the number of elements from index to the end of the source collection."

Do I have to give it the full length of the file?
0
 
LVL 4

Expert Comment

by:vigrid
ID: 12106447
Private Shared Sub Main(ByVal args As String())
      Dim fs As FileStream
      If (args.Length = 1) Then
            fs = New FileStream(args(0), FileMode.Open)
      Else
            Console.Write("Enter file name: ")
            Dim fileName As String = Console.ReadLine
            fs = New FileStream(fileName, FileMode.Open)
      End If
      Dim buffer As Byte() = New Byte(262144) {}
      Dim bufferSize As Integer = buffer.Length
      Dim offset As Integer = 0
      Dim readBytes As Integer = 0
      Dim lines As Integer = 0
      Do
            readBytes = fs.Read(buffer, 0, bufferSize)
            offset = (offset + readBytes)
            Console.Write("Read {0} bytes... ", offset)
            Dim counter As Integer = 0
            Do While (counter < readBytes)
                  lines = (lines + IIf((buffer(counter++) = 10), 1, 0))
            Loop
            Console.WriteLine("{0} lines found", lines)
      Loop ((readBytes = bufferSize) AndAlso (offset <> fs.Length))
      fs.Close
End Sub
0
 
LVL 4

Accepted Solution

by:
vigrid earned 250 total points
ID: 12106499
Above code is a translation from C# made by .NET Reflector. Sorry, I'm not an expert in VB.NET, but as the .NET Framework is the same to all of the languages, I'm trying to help :)

The magic number 262144 is 256kbytes of memory - size of the buffer.
The magic number 10 is ASCII code for the newline character.

Above reads the file in 0.25MB chunks and searches for the newline characters. When you find enough of them, you must note the offset where it did occur (should be something like: seekOffset = offset - readBytes + counter) and use the FileStream.Seek method to point your stream at the beginning of the line you need.

HTH
0

Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Since .Net 2.0, Visual Basic has made it easy to create a splash screen and set it via the "Splash Screen" drop down in the Project Properties.  A splash screen set in this manner is automatically created, displayed and closed by the framework itsel…
Microsoft Reports are based on a report definition, which is an XML file that describes data and layout for the report, with a different extension. You can create a client-side report definition language (*.rdlc) file with Visual Studio, and build g…
This video shows how to quickly and easily add an email signature for all users on Exchange 2016. The resulting signature is applied on a server level by Exchange Online. The email signature template has been downloaded from: www.mail-signatures…
Finds all prime numbers in a range requested and places them in a public primes() array. I've demostrated a template size of 30 (2 * 3 * 5) but larger templates can be built such 210  (2 * 3 * 5 * 7) or 2310  (2 * 3 * 5 * 7 * 11). The larger templa…

856 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question