Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

Text to Sql Server Process Question

Posted on 2014-04-08
6
Medium Priority
?
255 Views
Last Modified: 2014-04-14
I have an RFID timing system that returns race finishers via a line in a text file.  Lets call that text file "raw reads".  Each participant is read at least once but usually several times due to the fact that there are several finish line mats, each with it's own reader.

I have written code to parse that data into an array (basically pulling out their best time and their bib number), shown in a list box.  At this point I send that to a sql server database that collects the other information about each participant (name, gender, age, etc).  This works fine, but...

Since I refresh this array multiple times during the race, and some races get pretty big (several thousand runners), I would like to not have to go back and get all data from the "raw reads" text file.  I would like to just pick up where I left off.  How can I do that?

Here is the code I use to get the data from the "raw reads" text file:
    k = 0
    
    If Not sRFIDFile = vbNullString Then
        frmRFID2.lstRawData.Clear
        frmRFID2.lstRawData.AddItem "NO" & vbTab & "DATA"
        
        iFileNum = FreeFile()

        Open sRFIDFile For Input As iFileNum
        sFileData = Input(LOF(iFileNum), #iFileNum)
        Close iFileNum
        
        LineArr = Split(sFileData, vbCrLf)
        sFileData = vbNullString
        
        For i = 0 To UBound(LineArr) - 1
            strFields = Split(LineArr(i), ",")

            sThisLine = LineArr(i)
            
            sThisBib = strFields(1)
            sThisChip = strFields(2)
            sThisTime = strFields(3)
                
            If Len(sThisBib) <= 4 Then
                sThisTime = Replace(sThisTime, """", "")
                sngThisTime = ConvertToSeconds(sThisTime)
                iThisBib = CInt(sThisBib)
            
                If sngThisTime > sngStartTime + sngMinTime Then
                    RawRslts(0, k) = k + 1
                    RawRslts(1, k) = sThisBib
                    RawRslts(2, k) = sThisTime
                    RawRslts(3, k) = sThisChip

                    frmRFID2.lstRawData.AddItem k + 1 & vbTab & LineArr(i)

                    k = k + 1
                End If
            End If
        Next i
    End If

Open in new window


Any suggestions on how to better manage this process would be much appreciated.

Note:  I have set the RawRslts array dim at 11000 to avoid any ReDim Preserve issues.  Is there a better way to do that?
0
Comment
Question by:Bob Schneider
  • 3
  • 3
6 Comments
 
LVL 38

Assisted Solution

by:Jim P.
Jim P. earned 2000 total points
ID: 39987492
This may be a dumb question but is there anyway to make the files with a timestamp?

Such as Mat1_raw_reads_1404181132.txt then the next Mat1_raw_reads_1404181133.txt. That way you would have your app just do a continuous dir for the files to open and read in succession and if the file has been read you skip it on the next pass.

Rough code example:

PathName = "C:\timerfiles\"

FileName =  Dir("C:\timerfiles\" & "*raw_reads*.txt")

Do while FileName <> ""

If FileName Not in select ReadFiles FROM FileList WHERE ReadFiles <> FileName 

sRFIDFile  = PathName & FileName 
        Open sRFIDFile For Input As iFileNum
        sFileData = Input(LOF(iFileNum), #iFileNum)
        Close iFileNum
        
' The process code
FileName = Dir
LOOP

Open in new window

0
 

Author Comment

by:Bob Schneider
ID: 39987505
I might be able to append the files but I really want to keep that file intact since it comes right out of the timer...kind of sacred data.
0
 
LVL 38

Assisted Solution

by:Jim P.
Jim P. earned 2000 total points
ID: 39987560
I'm saying that file1 is a stand alone file from file2, from file3, and so on. Not a matter of appending them together.

So a directory would look like:
Mat1_raw_reads_1404181132.txt
Mat1_raw_reads_1404181133.txt
Mat1_raw_reads_1404181134.txt
Mat1_raw_reads_1404181135.txt
Mat1_raw_reads_1404181136.txt
Mat1_raw_reads_1404181137.txt
Mat1_raw_reads_1404181138.txt
Mat1_raw_reads_1404181139.txt
Mat1_raw_reads_1404181140.txt
Mat1_raw_reads_1404181141.txt
Mat1_raw_reads_1404181142.txt
Mat1_raw_reads_1404181143.txt
Mat1_raw_reads_1404181144.txt

Open in new window


Then you only process the new files, not every file again.
0
Efficient way to get backups off site to Azure

This user guide provides instructions on how to deploy and configure both a StoneFly Scale Out NAS Enterprise Cloud Drive virtual machine and Veeam Cloud Connect in the Microsoft Azure Cloud.

 

Author Comment

by:Bob Schneider
ID: 39987638
Interesting idea.  I like the idea of creating successive text files...but I can't change the fact that the original text file will have all the reads.  It is created by the third party rfid timing system.  Is there a way to use your idea and keep the original text file intact?
0
 
LVL 38

Accepted Solution

by:
Jim P. earned 2000 total points
ID: 39987733
So the system appends to the raw_reads.txt file from beginning to end?

In that case try this idea. Instead of reading the whole file as one thing do a line input row by row.

select LineNum = CurrLine from LineNumTbl

        Open sRFIDFile For Input shared  As iFileNum
       if I < LineNum Then
           Do While I < LineNum 
             Line Input #iFileNum, sFileData
             I=I+1
           Loop
       End If
Do Until EOF( &iFileNum) = True
     Line Input #iFileNum, sFileData
     ' rest of processing
      I=I+1
Loop

Close #iFileNum

SQL = "UPDATE LineNumTbl " & _
       "SET CurrLine = " & I 

RunSQL SQL

Open in new window


Then the next time you run it will chunk through the first 250 rows in seconds and then from there it will just import and process the new rows.
0
 

Author Closing Comment

by:Bob Schneider
ID: 39998617
Very helpful.  Thank you!
0

Featured Post

[Webinar] Cloud and Mobile-First Strategy

Maybe you’ve fully adopted the cloud since the beginning. Or maybe you started with on-prem resources but are pursuing a “cloud and mobile first” strategy. Getting to that end state has its challenges. Discover how to build out a 100% cloud and mobile IT strategy in this webinar.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Occasionally there is a need to clean table columns, especially if you have inherited legacy data. There are obviously many ways to accomplish that, including elaborate UPDATE queries with anywhere from one to numerous REPLACE functions (even within…
How to leverage one TLS certificate to encrypt Microsoft SQL traffic and Remote Desktop Services, versus creating multiple tickets for the same server.
Get people started with the process of using Access VBA to control Outlook using automation, Microsoft Access can control other applications. An example is the ability to programmatically talk to Microsoft Outlook. Using automation, an Access applic…
Get people started with the process of using Access VBA to control Excel using automation, Microsoft Access can control other applications. An example is the ability to programmatically talk to Excel. Using automation, an Access application can laun…

886 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question