Reading mixed data from a text file.

Posted on 2003-11-20
Last Modified: 2010-05-01
I wish to read data from a text file that can vary in size from a few bytes to a few hundred Megabytes in size.
The data is rather "mixed". There will be two types of lines.. data and occasional time stamps..

Each line of data will have 9 data values separated by a comma and ending with a CR/LF.
Each time stamp line will be just a series of bytes indicating Day of Week, Year, Month... etc...

So looking at the file, you will see a time stamp line, followed by several lines of data associated with that time stamp... followed by another time stamp, and more data.. etc..

Here is an example of what the file looks like:
4031119112557                                                          << time stamp
0,7233,16744,27259,33305,39699,41602,60837,1          << data
4031119114816                                                             << another time stamp
0,7162,19017,28989,37766,40992,41219,60246,1              << more data
... etc..
format for the time stamp is "DYYMMDDHHMMSS"  with the first D being the nth day of the week

This can go on for a long time... there may long intervals between time stamps or short.. it will vary..

I'm trying to do several things with this data... I need to read all the data (ignoring the time stamps) so I can just store the raw data.. I don't know how to just ignore time stamp lines...
I also would like (as an option) to somehow distinguish each piece of data with an appropriate day, or month.. (taken from the timestamp)..  so this would involve reading the time stamp, putting it into a group (group by day or by month).. then storing all the data lines associated with those times stamps..

The overal goal of all this is to have this data, which is taken at various times during a day..
I don't know if this is confusing, I hope it's clear.
I appreciate any help.

Question by:jscharpf
  • 4
  • 2

Author Comment

ID: 9788119
I wanted to add.. the data must be kept in the order it is shown, and sometimes there are timestamps without any data following. (you'll just see two time stamps one after the other) I can ignore these...

LVL 43

Accepted Solution

TimCottee earned 500 total points
ID: 9788169
Hi jscharpf,

Basically you need to scan it like this:

Dim intFHandle As Integer
intFHandle = FreeFile()
Dim strLine As String
Dim aryElements() As String
Open "Myfile.txt" For Input As #intFHandle
  Line Input #intFHandle,strLine
  If Instr(strLine,",") Then 'This is an element line and not a timestampe
    aryElements = Split(strLine,",") 'break up the line into an array we can then handle
    For i = 0 To 8
      txtDisplay(i).Text = aryElements(i) 'Show the result in a text box (one for each element)
    'It is a timestamp so do something else with it for example:
    Msgbox "Timestamp of : " & vbLf & Mid(strLine,6,2) & "/" & Mid(strLine,4,2) & "/" & Mid(strLine,2,2) 'dd/mm/yy format display
  End If
Loop Until Eof(intFHandle)
Close #intFHandle

This of course is just an example of how you could process the file, the exact operations on each line would be up to you.

Brainbench MVP for Visual Basic
LVL 85

Expert Comment

by:Mike Tomlinson
ID: 9788716
What is the end goal here?

Do you want an easier way to view data sets associated with a time stamp?
... or are you looking to do some calculations based upon the sets and only look at the results?
... or are you wanting to see timestamps with data sets and some kind of calculated results together?

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.


Author Comment

ID: 9788864
There won't be an calculations on the data. It will only be graphed. But the problem I have is that the data will be taken at various times during a day, and for undefined lengths of time. So, for example, A time stamp of 11 Am on a certain day wil be followed by a bunch of data.. I'm not sure how many. It could be zero, one, or 10000.... then, another time stamp, followed by more data.. these will be stamped at various times during the day (

To give you more info.. I'm installing a microcontroller in a power tool and letting the operator use it. Each time the operator turns it on and uses it, I time stamp it, then begin saving data until it's turned off. This can take place several times in a day. We are limited in our data saving options for now.. this is how we are getting the data..

So.. I will graph (I have a very good OCX for this) all 8 data points vs. time, and initially this will just be one long graph, ignoring the time stamps, and knowing the sample rate is XX ms, we will see a profile and we can look at all the data..
but as an added option, we will need to somehow ALSO (I can do two graphs and two sets of data if I need to) need to somehow associate what days each part of the graph correspond to.. The graphing OCX I have is very good and can graph an unlimited set of points, so I'm thinking of just storing a second set of data, that is associated with the time stamps, then giving the user the option to just go day by day and look at data for that day only.. (redrawing the graph with the new data each time)..

I am sorry if this is confusing.. :)


Author Comment

ID: 9788905
also.. lol..
suppose the operator turns the tool on and off 100 times in one day.. I will have alot of data and alot of time stamps to deal with. I'm just trying to find a good way to group these so that I don't have a million graphs (one for each time stamp would be crazy)
I realize that I have to compromise somewhere.. so I thought that maybe grouping the data by Day... and just keeping sets of data per day.. it might help.

BUt the MOST important part is to get all the data out and separate it from the timestamps.. so I can at least see nothing but data. I can worry about when it happened later.. Doing this at least gives us good info, we just don't know when it occured (day, etc.)

LVL 85

Expert Comment

by:Mike Tomlinson
ID: 9789123
One approach would be to read the file in just as TimCottee has shown you and add the arrays of data to a collection.

If you want to get tricky, then you could create a separate collection for each day, then have a collection of collections.

As you read each stamp, convert it to a date, without the time data in it, to be used as a key.  Then add all data you read into a collection until you reach a stamp that is from the next day.  Now add the collection you just made to the main collection using the date as a key, and then start a new collection that will have the new stamp key associated with it.

You could place all the date keys in a listbox so you can easily select a different day to view and use the selected item text to retrieve the correct collection containing the corresponding data.

I hope that wasn't too terribly confusing.



Author Comment

ID: 9796483
Thanks to both of you for your answers! I guess I can only give points to one person.


Featured Post

Announcing the Most Valuable Experts of 2016

MVEs are more concerned with the satisfaction of those they help than with the considerable points they can earn. They are the types of people you feel privileged to call colleagues. Join us in honoring this amazing group of Experts.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I’ve seen a number of people looking for examples of how to access web services from VB6.  I’ve been using a test harness I built in VB6 (using many resources I found online) that I use for small projects to work out how to communicate with web serv…
If you need to start windows update installation remotely or as a scheduled task you will find this very helpful.
As developers, we are not limited to the functions provided by the VBA language. In addition, we can call the functions that are part of the Windows operating system. These functions are part of the Windows API (Application Programming Interface). U…
Get people started with the utilization of class modules. Class modules can be a powerful tool in Microsoft Access. They allow you to create self-contained objects that encapsulate functionality. They can easily hide the complexity of a process from…

830 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question