Solved

Text to Sql Server Process Question

Posted on 2014-04-08
6
238 Views
Last Modified: 2014-04-14
I have an RFID timing system that returns race finishers via a line in a text file.  Lets call that text file "raw reads".  Each participant is read at least once but usually several times due to the fact that there are several finish line mats, each with it's own reader.

I have written code to parse that data into an array (basically pulling out their best time and their bib number), shown in a list box.  At this point I send that to a sql server database that collects the other information about each participant (name, gender, age, etc).  This works fine, but...

Since I refresh this array multiple times during the race, and some races get pretty big (several thousand runners), I would like to not have to go back and get all data from the "raw reads" text file.  I would like to just pick up where I left off.  How can I do that?

Here is the code I use to get the data from the "raw reads" text file:
    k = 0
    
    If Not sRFIDFile = vbNullString Then
        frmRFID2.lstRawData.Clear
        frmRFID2.lstRawData.AddItem "NO" & vbTab & "DATA"
        
        iFileNum = FreeFile()

        Open sRFIDFile For Input As iFileNum
        sFileData = Input(LOF(iFileNum), #iFileNum)
        Close iFileNum
        
        LineArr = Split(sFileData, vbCrLf)
        sFileData = vbNullString
        
        For i = 0 To UBound(LineArr) - 1
            strFields = Split(LineArr(i), ",")

            sThisLine = LineArr(i)
            
            sThisBib = strFields(1)
            sThisChip = strFields(2)
            sThisTime = strFields(3)
                
            If Len(sThisBib) <= 4 Then
                sThisTime = Replace(sThisTime, """", "")
                sngThisTime = ConvertToSeconds(sThisTime)
                iThisBib = CInt(sThisBib)
            
                If sngThisTime > sngStartTime + sngMinTime Then
                    RawRslts(0, k) = k + 1
                    RawRslts(1, k) = sThisBib
                    RawRslts(2, k) = sThisTime
                    RawRslts(3, k) = sThisChip

                    frmRFID2.lstRawData.AddItem k + 1 & vbTab & LineArr(i)

                    k = k + 1
                End If
            End If
        Next i
    End If

Open in new window


Any suggestions on how to better manage this process would be much appreciated.

Note:  I have set the RawRslts array dim at 11000 to avoid any ReDim Preserve issues.  Is there a better way to do that?
0
Comment
Question by:Bob Schneider
  • 3
  • 3
6 Comments
 
LVL 38

Assisted Solution

by:Jim P.
Jim P. earned 500 total points
ID: 39987492
This may be a dumb question but is there anyway to make the files with a timestamp?

Such as Mat1_raw_reads_1404181132.txt then the next Mat1_raw_reads_1404181133.txt. That way you would have your app just do a continuous dir for the files to open and read in succession and if the file has been read you skip it on the next pass.

Rough code example:

PathName = "C:\timerfiles\"

FileName =  Dir("C:\timerfiles\" & "*raw_reads*.txt")

Do while FileName <> ""

If FileName Not in select ReadFiles FROM FileList WHERE ReadFiles <> FileName 

sRFIDFile  = PathName & FileName 
        Open sRFIDFile For Input As iFileNum
        sFileData = Input(LOF(iFileNum), #iFileNum)
        Close iFileNum
        
' The process code
FileName = Dir
LOOP

Open in new window

0
 

Author Comment

by:Bob Schneider
ID: 39987505
I might be able to append the files but I really want to keep that file intact since it comes right out of the timer...kind of sacred data.
0
 
LVL 38

Assisted Solution

by:Jim P.
Jim P. earned 500 total points
ID: 39987560
I'm saying that file1 is a stand alone file from file2, from file3, and so on. Not a matter of appending them together.

So a directory would look like:
Mat1_raw_reads_1404181132.txt
Mat1_raw_reads_1404181133.txt
Mat1_raw_reads_1404181134.txt
Mat1_raw_reads_1404181135.txt
Mat1_raw_reads_1404181136.txt
Mat1_raw_reads_1404181137.txt
Mat1_raw_reads_1404181138.txt
Mat1_raw_reads_1404181139.txt
Mat1_raw_reads_1404181140.txt
Mat1_raw_reads_1404181141.txt
Mat1_raw_reads_1404181142.txt
Mat1_raw_reads_1404181143.txt
Mat1_raw_reads_1404181144.txt

Open in new window


Then you only process the new files, not every file again.
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 

Author Comment

by:Bob Schneider
ID: 39987638
Interesting idea.  I like the idea of creating successive text files...but I can't change the fact that the original text file will have all the reads.  It is created by the third party rfid timing system.  Is there a way to use your idea and keep the original text file intact?
0
 
LVL 38

Accepted Solution

by:
Jim P. earned 500 total points
ID: 39987733
So the system appends to the raw_reads.txt file from beginning to end?

In that case try this idea. Instead of reading the whole file as one thing do a line input row by row.

select LineNum = CurrLine from LineNumTbl

        Open sRFIDFile For Input shared  As iFileNum
       if I < LineNum Then
           Do While I < LineNum 
             Line Input #iFileNum, sFileData
             I=I+1
           Loop
       End If
Do Until EOF( &iFileNum) = True
     Line Input #iFileNum, sFileData
     ' rest of processing
      I=I+1
Loop

Close #iFileNum

SQL = "UPDATE LineNumTbl " & _
       "SET CurrLine = " & I 

RunSQL SQL

Open in new window


Then the next time you run it will chunk through the first 250 rows in seconds and then from there it will just import and process the new rows.
0
 

Author Closing Comment

by:Bob Schneider
ID: 39998617
Very helpful.  Thank you!
0

Featured Post

How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

Join & Write a Comment

If you have heard of RFC822 date formats, they can be quite a challenge in SQL Server. RFC822 is an Internet standard format for email message headers, including all dates within those headers. The RFC822 protocols are available in detail at:   ht…
SQL Server engine let you use a Windows account or a SQL Server account to connect to a SQL Server instance. This can be configured immediatly during the SQL Server installation or after in the Server Authentication section in the Server properties …
Get people started with the utilization of class modules. Class modules can be a powerful tool in Microsoft Access. They allow you to create self-contained objects that encapsulate functionality. They can easily hide the complexity of a process from…
This lesson covers basic error handling code in Microsoft Excel using VBA. This is the first lesson in a 3-part series that uses code to loop through an Excel spreadsheet in VBA and then fix errors, taking advantage of error handling code. This l…

706 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

16 Experts available now in Live!

Get 1:1 Help Now