Reading mixed data from a text file.

Posted on 2003-11-20
Last Modified: 2010-05-01
I wish to read data from a text file that can vary in size from a few bytes to a few hundred Megabytes in size.
The data is rather "mixed". There will be two types of lines.. data and occasional time stamps..

Each line of data will have 9 data values separated by a comma and ending with a CR/LF.
Each time stamp line will be just a series of bytes indicating Day of Week, Year, Month... etc...

So looking at the file, you will see a time stamp line, followed by several lines of data associated with that time stamp... followed by another time stamp, and more data.. etc..

Here is an example of what the file looks like:
4031119112557                                                          << time stamp
0,7233,16744,27259,33305,39699,41602,60837,1          << data
4031119114816                                                             << another time stamp
0,7162,19017,28989,37766,40992,41219,60246,1              << more data
... etc..
format for the time stamp is "DYYMMDDHHMMSS"  with the first D being the nth day of the week

This can go on for a long time... there may long intervals between time stamps or short.. it will vary..

I'm trying to do several things with this data... I need to read all the data (ignoring the time stamps) so I can just store the raw data.. I don't know how to just ignore time stamp lines...
I also would like (as an option) to somehow distinguish each piece of data with an appropriate day, or month.. (taken from the timestamp)..  so this would involve reading the time stamp, putting it into a group (group by day or by month).. then storing all the data lines associated with those times stamps..

The overal goal of all this is to have this data, which is taken at various times during a day..
I don't know if this is confusing, I hope it's clear.
I appreciate any help.

Question by:jscharpf
  • 4
  • 2

Author Comment

ID: 9788119
I wanted to add.. the data must be kept in the order it is shown, and sometimes there are timestamps without any data following. (you'll just see two time stamps one after the other) I can ignore these...

LVL 43

Accepted Solution

TimCottee earned 500 total points
ID: 9788169
Hi jscharpf,

Basically you need to scan it like this:

Dim intFHandle As Integer
intFHandle = FreeFile()
Dim strLine As String
Dim aryElements() As String
Open "Myfile.txt" For Input As #intFHandle
  Line Input #intFHandle,strLine
  If Instr(strLine,",") Then 'This is an element line and not a timestampe
    aryElements = Split(strLine,",") 'break up the line into an array we can then handle
    For i = 0 To 8
      txtDisplay(i).Text = aryElements(i) 'Show the result in a text box (one for each element)
    'It is a timestamp so do something else with it for example:
    Msgbox "Timestamp of : " & vbLf & Mid(strLine,6,2) & "/" & Mid(strLine,4,2) & "/" & Mid(strLine,2,2) 'dd/mm/yy format display
  End If
Loop Until Eof(intFHandle)
Close #intFHandle

This of course is just an example of how you could process the file, the exact operations on each line would be up to you.

Brainbench MVP for Visual Basic
LVL 85

Expert Comment

by:Mike Tomlinson
ID: 9788716
What is the end goal here?

Do you want an easier way to view data sets associated with a time stamp?
... or are you looking to do some calculations based upon the sets and only look at the results?
... or are you wanting to see timestamps with data sets and some kind of calculated results together?

Gigs: Get Your Project Delivered by an Expert

Select from freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely and get projects done right.


Author Comment

ID: 9788864
There won't be an calculations on the data. It will only be graphed. But the problem I have is that the data will be taken at various times during a day, and for undefined lengths of time. So, for example, A time stamp of 11 Am on a certain day wil be followed by a bunch of data.. I'm not sure how many. It could be zero, one, or 10000.... then, another time stamp, followed by more data.. these will be stamped at various times during the day (

To give you more info.. I'm installing a microcontroller in a power tool and letting the operator use it. Each time the operator turns it on and uses it, I time stamp it, then begin saving data until it's turned off. This can take place several times in a day. We are limited in our data saving options for now.. this is how we are getting the data..

So.. I will graph (I have a very good OCX for this) all 8 data points vs. time, and initially this will just be one long graph, ignoring the time stamps, and knowing the sample rate is XX ms, we will see a profile and we can look at all the data..
but as an added option, we will need to somehow ALSO (I can do two graphs and two sets of data if I need to) need to somehow associate what days each part of the graph correspond to.. The graphing OCX I have is very good and can graph an unlimited set of points, so I'm thinking of just storing a second set of data, that is associated with the time stamps, then giving the user the option to just go day by day and look at data for that day only.. (redrawing the graph with the new data each time)..

I am sorry if this is confusing.. :)


Author Comment

ID: 9788905
also.. lol..
suppose the operator turns the tool on and off 100 times in one day.. I will have alot of data and alot of time stamps to deal with. I'm just trying to find a good way to group these so that I don't have a million graphs (one for each time stamp would be crazy)
I realize that I have to compromise somewhere.. so I thought that maybe grouping the data by Day... and just keeping sets of data per day.. it might help.

BUt the MOST important part is to get all the data out and separate it from the timestamps.. so I can at least see nothing but data. I can worry about when it happened later.. Doing this at least gives us good info, we just don't know when it occured (day, etc.)

LVL 85

Expert Comment

by:Mike Tomlinson
ID: 9789123
One approach would be to read the file in just as TimCottee has shown you and add the arrays of data to a collection.

If you want to get tricky, then you could create a separate collection for each day, then have a collection of collections.

As you read each stamp, convert it to a date, without the time data in it, to be used as a key.  Then add all data you read into a collection until you reach a stamp that is from the next day.  Now add the collection you just made to the main collection using the date as a key, and then start a new collection that will have the new stamp key associated with it.

You could place all the date keys in a listbox so you can easily select a different day to view and use the selected item text to retrieve the correct collection containing the corresponding data.

I hope that wasn't too terribly confusing.



Author Comment

ID: 9796483
Thanks to both of you for your answers! I guess I can only give points to one person.


Featured Post

Gigs: Get Your Project Delivered by an Expert

Select from freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely and get projects done right.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
vb6 programatically bring window to front 8 70
Windows 10 start screen issues 9 54
Copy a row 12 61
How to make an ADE file by code? 11 83
Introduction While answering a recent question ( in the VB classic zone, I wrote some VB code in the (Office) VBA environment, rather than fire up my older PC.  I didn't post completely correct code o…
Most everyone who has done any programming in VB6 knows that you can do something in code like Debug.Print MyVar and that when the program runs from the IDE, the value of MyVar will be displayed in the Immediate Window. Less well known is Debug.Asse…
Get people started with the process of using Access VBA to control Excel using automation, Microsoft Access can control other applications. An example is the ability to programmatically talk to Excel. Using automation, an Access application can laun…
This lesson covers basic error handling code in Microsoft Excel using VBA. This is the first lesson in a 3-part series that uses code to loop through an Excel spreadsheet in VBA and then fix errors, taking advantage of error handling code. This l…

785 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question