Go Premium for a chance to win a PS4. Enter to Win

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 373
  • Last Modified:

Reading mixed data from a text file.

I wish to read data from a text file that can vary in size from a few bytes to a few hundred Megabytes in size.
The data is rather "mixed". There will be two types of lines.. data and occasional time stamps..

Each line of data will have 9 data values separated by a comma and ending with a CR/LF.
Each time stamp line will be just a series of bytes indicating Day of Week, Year, Month... etc...

So looking at the file, you will see a time stamp line, followed by several lines of data associated with that time stamp... followed by another time stamp, and more data.. etc..

Here is an example of what the file looks like:
----------------------------------------------------------
4031119112557                                                          << time stamp
0,7228,19942,29396,36720,40631,41293,60838,0
0,7233,16744,27259,33305,39699,41602,60837,1          << data
0,7222,17645,28492,35510,39312,38306,60848,2
0,7229,16677,27185,35742,37983,38328,60842,3
4031119114816                                                             << another time stamp
400,13692,25187,34470,37555,39239,40761,60224,0
0,7162,19017,28989,37766,40992,41219,60246,1              << more data
0,7143,24054,30181,36815,39571,40076,60251,2
0,7140,21018,29885,35846,38206,39768,60267,3
0,7153,23132,31717,37398,40756,41971,60227,4
0,7154,18958,29391,37196,39542,40636,60223,5
0,7152,20048,29692,37307,39751,41080,60224,6
0,7136,20733,32039,37401,40820,42348,60238,7
... etc..
---------------------------------------------------------------
format for the time stamp is "DYYMMDDHHMMSS"  with the first D being the nth day of the week

This can go on for a long time... there may long intervals between time stamps or short.. it will vary..

I'm trying to do several things with this data... I need to read all the data (ignoring the time stamps) so I can just store the raw data.. I don't know how to just ignore time stamp lines...
I also would like (as an option) to somehow distinguish each piece of data with an appropriate day, or month.. (taken from the timestamp)..  so this would involve reading the time stamp, putting it into a group (group by day or by month).. then storing all the data lines associated with those times stamps..

The overal goal of all this is to have this data, which is taken at various times during a day..
I don't know if this is confusing, I hope it's clear.
I appreciate any help.

Jeff
0
jscharpf
Asked:
jscharpf
  • 4
  • 2
1 Solution
 
jscharpfAuthor Commented:
I wanted to add.. the data must be kept in the order it is shown, and sometimes there are timestamps without any data following. (you'll just see two time stamps one after the other) I can ignore these...

Jeff
0
 
TimCotteeCommented:
Hi jscharpf,

Basically you need to scan it like this:

Dim intFHandle As Integer
intFHandle = FreeFile()
Dim strLine As String
Dim aryElements() As String
Open "Myfile.txt" For Input As #intFHandle
Do
  Line Input #intFHandle,strLine
  If Instr(strLine,",") Then 'This is an element line and not a timestampe
    aryElements = Split(strLine,",") 'break up the line into an array we can then handle
    For i = 0 To 8
      txtDisplay(i).Text = aryElements(i) 'Show the result in a text box (one for each element)
    Next
  Else
    'It is a timestamp so do something else with it for example:
    Msgbox "Timestamp of : " & vbLf & Mid(strLine,6,2) & "/" & Mid(strLine,4,2) & "/" & Mid(strLine,2,2) 'dd/mm/yy format display
  End If
Loop Until Eof(intFHandle)
Close #intFHandle

This of course is just an example of how you could process the file, the exact operations on each line would be up to you.

Tim Cottee MCSD, MCDBA, CPIM
Brainbench MVP for Visual Basic
http://www.brainbench.com
0
 
Mike TomlinsonMiddle School Assistant TeacherCommented:
What is the end goal here?

Do you want an easier way to view data sets associated with a time stamp?
... or are you looking to do some calculations based upon the sets and only look at the results?
... or are you wanting to see timestamps with data sets and some kind of calculated results together?

Idle_Mind
0
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

 
jscharpfAuthor Commented:
There won't be an calculations on the data. It will only be graphed. But the problem I have is that the data will be taken at various times during a day, and for undefined lengths of time. So, for example, A time stamp of 11 Am on a certain day wil be followed by a bunch of data.. I'm not sure how many. It could be zero, one, or 10000.... then, another time stamp, followed by more data.. these will be stamped at various times during the day (

To give you more info.. I'm installing a microcontroller in a power tool and letting the operator use it. Each time the operator turns it on and uses it, I time stamp it, then begin saving data until it's turned off. This can take place several times in a day. We are limited in our data saving options for now.. this is how we are getting the data..

So.. I will graph (I have a very good OCX for this) all 8 data points vs. time, and initially this will just be one long graph, ignoring the time stamps, and knowing the sample rate is XX ms, we will see a profile and we can look at all the data..
but as an added option, we will need to somehow ALSO (I can do two graphs and two sets of data if I need to) need to somehow associate what days each part of the graph correspond to.. The graphing OCX I have is very good and can graph an unlimited set of points, so I'm thinking of just storing a second set of data, that is associated with the time stamps, then giving the user the option to just go day by day and look at data for that day only.. (redrawing the graph with the new data each time)..

I am sorry if this is confusing.. :)

Jeff
0
 
jscharpfAuthor Commented:
also.. lol..
suppose the operator turns the tool on and off 100 times in one day.. I will have alot of data and alot of time stamps to deal with. I'm just trying to find a good way to group these so that I don't have a million graphs (one for each time stamp would be crazy)
 :)
I realize that I have to compromise somewhere.. so I thought that maybe grouping the data by Day... and just keeping sets of data per day.. it might help.

BUt the MOST important part is to get all the data out and separate it from the timestamps.. so I can at least see nothing but data. I can worry about when it happened later.. Doing this at least gives us good info, we just don't know when it occured (day, etc.)

Jeff
0
 
Mike TomlinsonMiddle School Assistant TeacherCommented:
One approach would be to read the file in just as TimCottee has shown you and add the arrays of data to a collection.

If you want to get tricky, then you could create a separate collection for each day, then have a collection of collections.

As you read each stamp, convert it to a date, without the time data in it, to be used as a key.  Then add all data you read into a collection until you reach a stamp that is from the next day.  Now add the collection you just made to the main collection using the date as a key, and then start a new collection that will have the new stamp key associated with it.

You could place all the date keys in a listbox so you can easily select a different day to view and use the selected item text to retrieve the correct collection containing the corresponding data.

I hope that wasn't too terribly confusing.

Idle_Mind

0
 
jscharpfAuthor Commented:
Thanks to both of you for your answers! I guess I can only give points to one person.


Jeff
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

  • 4
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now