• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 257
  • Last Modified:

Text to Sql Server Process Question

I have an RFID timing system that returns race finishers via a line in a text file.  Lets call that text file "raw reads".  Each participant is read at least once but usually several times due to the fact that there are several finish line mats, each with it's own reader.

I have written code to parse that data into an array (basically pulling out their best time and their bib number), shown in a list box.  At this point I send that to a sql server database that collects the other information about each participant (name, gender, age, etc).  This works fine, but...

Since I refresh this array multiple times during the race, and some races get pretty big (several thousand runners), I would like to not have to go back and get all data from the "raw reads" text file.  I would like to just pick up where I left off.  How can I do that?

Here is the code I use to get the data from the "raw reads" text file:
    k = 0
    
    If Not sRFIDFile = vbNullString Then
        frmRFID2.lstRawData.Clear
        frmRFID2.lstRawData.AddItem "NO" & vbTab & "DATA"
        
        iFileNum = FreeFile()

        Open sRFIDFile For Input As iFileNum
        sFileData = Input(LOF(iFileNum), #iFileNum)
        Close iFileNum
        
        LineArr = Split(sFileData, vbCrLf)
        sFileData = vbNullString
        
        For i = 0 To UBound(LineArr) - 1
            strFields = Split(LineArr(i), ",")

            sThisLine = LineArr(i)
            
            sThisBib = strFields(1)
            sThisChip = strFields(2)
            sThisTime = strFields(3)
                
            If Len(sThisBib) <= 4 Then
                sThisTime = Replace(sThisTime, """", "")
                sngThisTime = ConvertToSeconds(sThisTime)
                iThisBib = CInt(sThisBib)
            
                If sngThisTime > sngStartTime + sngMinTime Then
                    RawRslts(0, k) = k + 1
                    RawRslts(1, k) = sThisBib
                    RawRslts(2, k) = sThisTime
                    RawRslts(3, k) = sThisChip

                    frmRFID2.lstRawData.AddItem k + 1 & vbTab & LineArr(i)

                    k = k + 1
                End If
            End If
        Next i
    End If

Open in new window


Any suggestions on how to better manage this process would be much appreciated.

Note:  I have set the RawRslts array dim at 11000 to avoid any ReDim Preserve issues.  Is there a better way to do that?
0
Bob Schneider
Asked:
Bob Schneider
  • 3
  • 3
3 Solutions
 
Jim P.Commented:
This may be a dumb question but is there anyway to make the files with a timestamp?

Such as Mat1_raw_reads_1404181132.txt then the next Mat1_raw_reads_1404181133.txt. That way you would have your app just do a continuous dir for the files to open and read in succession and if the file has been read you skip it on the next pass.

Rough code example:

PathName = "C:\timerfiles\"

FileName =  Dir("C:\timerfiles\" & "*raw_reads*.txt")

Do while FileName <> ""

If FileName Not in select ReadFiles FROM FileList WHERE ReadFiles <> FileName 

sRFIDFile  = PathName & FileName 
        Open sRFIDFile For Input As iFileNum
        sFileData = Input(LOF(iFileNum), #iFileNum)
        Close iFileNum
        
' The process code
FileName = Dir
LOOP

Open in new window

0
 
Bob SchneiderCo-OwnerAuthor Commented:
I might be able to append the files but I really want to keep that file intact since it comes right out of the timer...kind of sacred data.
0
 
Jim P.Commented:
I'm saying that file1 is a stand alone file from file2, from file3, and so on. Not a matter of appending them together.

So a directory would look like:
Mat1_raw_reads_1404181132.txt
Mat1_raw_reads_1404181133.txt
Mat1_raw_reads_1404181134.txt
Mat1_raw_reads_1404181135.txt
Mat1_raw_reads_1404181136.txt
Mat1_raw_reads_1404181137.txt
Mat1_raw_reads_1404181138.txt
Mat1_raw_reads_1404181139.txt
Mat1_raw_reads_1404181140.txt
Mat1_raw_reads_1404181141.txt
Mat1_raw_reads_1404181142.txt
Mat1_raw_reads_1404181143.txt
Mat1_raw_reads_1404181144.txt

Open in new window


Then you only process the new files, not every file again.
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 
Bob SchneiderCo-OwnerAuthor Commented:
Interesting idea.  I like the idea of creating successive text files...but I can't change the fact that the original text file will have all the reads.  It is created by the third party rfid timing system.  Is there a way to use your idea and keep the original text file intact?
0
 
Jim P.Commented:
So the system appends to the raw_reads.txt file from beginning to end?

In that case try this idea. Instead of reading the whole file as one thing do a line input row by row.

select LineNum = CurrLine from LineNumTbl

        Open sRFIDFile For Input shared  As iFileNum
       if I < LineNum Then
           Do While I < LineNum 
             Line Input #iFileNum, sFileData
             I=I+1
           Loop
       End If
Do Until EOF( &iFileNum) = True
     Line Input #iFileNum, sFileData
     ' rest of processing
      I=I+1
Loop

Close #iFileNum

SQL = "UPDATE LineNumTbl " & _
       "SET CurrLine = " & I 

RunSQL SQL

Open in new window


Then the next time you run it will chunk through the first 250 rows in seconds and then from there it will just import and process the new rows.
0
 
Bob SchneiderCo-OwnerAuthor Commented:
Very helpful.  Thank you!
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

  • 3
  • 3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now