?
Solved

Start reading a File from a Specific Row or Position

Posted on 2004-09-17
6
Medium Priority
?
420 Views
Last Modified: 2010-05-18
Basically I know how to loop through the file with StreamReader (i.e. sr.ReadLine) but what I want to do is start reading a row from a specific point in the file.

My routine basically reads a file into StreamReader, loops through the starting set of lines to get to my Starting Point, and then it reads the lines I want extracted to another file.

Here is some example code:
//iStart is declared as int32, line is a String
//oWrite is a class that creates the new file

Private Function openFile(ByVal fileName As String, ByVal getCount As Int32)
            Try
                Dim sr As StreamReader = New StreamReader(fileName)
                Dim sLines As New System.Text.StringBuilder
                Do While sCount <> iStart
                    line = sr.ReadLine
                    sCount += 1
                Loop
                sCount = 0
                Do While sCount <> getCount
                    line = sr.ReadLine
                    sLines.Append(line & vbCrLf)
                    sCount += 1
                Loop
                oWrite.WriteParse(sLines.ToString(), sfileName)
                sr.Close()
                sCount = 0
            Catch ex As Exception
                Throw ex
            End Try
        End Function

The worst situations occur when I am loading a file that has 2 million rows and I need something that started after the 500,000th row.  It takes a while to loop to the 500,000th position in the file, and when it finally gets there I am having memory problems slowing down my computer.  Any suggestions on how to jump to the position without reading each line?
0
Comment
Question by:deadferret
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 2
6 Comments
 
LVL 4

Expert Comment

by:vigrid
ID: 12089456
In order to know (for sure) the current line number you are reading, you must read the whole file from the beginning to the current point and count the CR/LF occurences.

Overloads Overrides Public Function Read(Char(), Integer, Integer) As Integer

You could read a file fragment to a predefined buffer, say in 16kb chunks and analyze them manually to find CR/LFs. That would solve your problem with memory.

HTH
0
 
LVL 4

Expert Comment

by:vigrid
ID: 12089462
ms-help://MS.MSDNQTR.2004JAN.1033/cpref/html/frlrfSystemIOStreamReaderClassReadTopic.htm

MSDN help link
0
 

Author Comment

by:deadferret
ID: 12101892
How would I go about doing this?  "You could read a file fragment to a predefined buffer, say in 16kb chunks and analyze them manually to find CR/LFs. That would solve your problem with memory."
0
VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

 

Author Comment

by:deadferret
ID: 12102136
I also continue to get this message when trying to read for a Cr/Lf.

"Offset and length were out of bounds for the array or count is greater than the number of elements from index to the end of the source collection."

Do I have to give it the full length of the file?
0
 
LVL 4

Expert Comment

by:vigrid
ID: 12106447
Private Shared Sub Main(ByVal args As String())
      Dim fs As FileStream
      If (args.Length = 1) Then
            fs = New FileStream(args(0), FileMode.Open)
      Else
            Console.Write("Enter file name: ")
            Dim fileName As String = Console.ReadLine
            fs = New FileStream(fileName, FileMode.Open)
      End If
      Dim buffer As Byte() = New Byte(262144) {}
      Dim bufferSize As Integer = buffer.Length
      Dim offset As Integer = 0
      Dim readBytes As Integer = 0
      Dim lines As Integer = 0
      Do
            readBytes = fs.Read(buffer, 0, bufferSize)
            offset = (offset + readBytes)
            Console.Write("Read {0} bytes... ", offset)
            Dim counter As Integer = 0
            Do While (counter < readBytes)
                  lines = (lines + IIf((buffer(counter++) = 10), 1, 0))
            Loop
            Console.WriteLine("{0} lines found", lines)
      Loop ((readBytes = bufferSize) AndAlso (offset <> fs.Length))
      fs.Close
End Sub
0
 
LVL 4

Accepted Solution

by:
vigrid earned 750 total points
ID: 12106499
Above code is a translation from C# made by .NET Reflector. Sorry, I'm not an expert in VB.NET, but as the .NET Framework is the same to all of the languages, I'm trying to help :)

The magic number 262144 is 256kbytes of memory - size of the buffer.
The magic number 10 is ASCII code for the newline character.

Above reads the file in 0.25MB chunks and searches for the newline characters. When you find enough of them, you must note the offset where it did occur (should be something like: seekOffset = offset - readBytes + counter) and use the FileStream.Seek method to point your stream at the beginning of the line you need.

HTH
0

Featured Post

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Parsing a CSV file is a task that we are confronted with regularly, and although there are a vast number of means to do this, as a newbie, the field can be confusing and the tools can seem complex. A simple solution to parsing a customized CSV fi…
Calculating holidays and working days is a function that is often needed yet it is not one found within the Framework. This article presents one approach to building a working-day calculator for use in .NET.
This is my first video review of Microsoft Bookings, I will be doing a part two with a bit more information, but wanted to get this out to you folks.
In this video, Percona Solutions Engineer Barrett Chambers discusses some of the basic syntax differences between MySQL and MongoDB. To learn more check out our webinar on MongoDB administration for MySQL DBA: https://www.percona.com/resources/we…
Suggested Courses

765 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question