Solved

Reading mixed data from a text file.

Posted on 2003-11-20
7
364 Views
Last Modified: 2010-05-01
I wish to read data from a text file that can vary in size from a few bytes to a few hundred Megabytes in size.
The data is rather "mixed". There will be two types of lines.. data and occasional time stamps..

Each line of data will have 9 data values separated by a comma and ending with a CR/LF.
Each time stamp line will be just a series of bytes indicating Day of Week, Year, Month... etc...

So looking at the file, you will see a time stamp line, followed by several lines of data associated with that time stamp... followed by another time stamp, and more data.. etc..

Here is an example of what the file looks like:
----------------------------------------------------------
4031119112557                                                          << time stamp
0,7228,19942,29396,36720,40631,41293,60838,0
0,7233,16744,27259,33305,39699,41602,60837,1          << data
0,7222,17645,28492,35510,39312,38306,60848,2
0,7229,16677,27185,35742,37983,38328,60842,3
4031119114816                                                             << another time stamp
400,13692,25187,34470,37555,39239,40761,60224,0
0,7162,19017,28989,37766,40992,41219,60246,1              << more data
0,7143,24054,30181,36815,39571,40076,60251,2
0,7140,21018,29885,35846,38206,39768,60267,3
0,7153,23132,31717,37398,40756,41971,60227,4
0,7154,18958,29391,37196,39542,40636,60223,5
0,7152,20048,29692,37307,39751,41080,60224,6
0,7136,20733,32039,37401,40820,42348,60238,7
... etc..
---------------------------------------------------------------
format for the time stamp is "DYYMMDDHHMMSS"  with the first D being the nth day of the week

This can go on for a long time... there may long intervals between time stamps or short.. it will vary..

I'm trying to do several things with this data... I need to read all the data (ignoring the time stamps) so I can just store the raw data.. I don't know how to just ignore time stamp lines...
I also would like (as an option) to somehow distinguish each piece of data with an appropriate day, or month.. (taken from the timestamp)..  so this would involve reading the time stamp, putting it into a group (group by day or by month).. then storing all the data lines associated with those times stamps..

The overal goal of all this is to have this data, which is taken at various times during a day..
I don't know if this is confusing, I hope it's clear.
I appreciate any help.

Jeff
0
Comment
Question by:jscharpf
  • 4
  • 2
7 Comments
 

Author Comment

by:jscharpf
ID: 9788119
I wanted to add.. the data must be kept in the order it is shown, and sometimes there are timestamps without any data following. (you'll just see two time stamps one after the other) I can ignore these...

Jeff
0
 
LVL 43

Accepted Solution

by:
TimCottee earned 500 total points
ID: 9788169
Hi jscharpf,

Basically you need to scan it like this:

Dim intFHandle As Integer
intFHandle = FreeFile()
Dim strLine As String
Dim aryElements() As String
Open "Myfile.txt" For Input As #intFHandle
Do
  Line Input #intFHandle,strLine
  If Instr(strLine,",") Then 'This is an element line and not a timestampe
    aryElements = Split(strLine,",") 'break up the line into an array we can then handle
    For i = 0 To 8
      txtDisplay(i).Text = aryElements(i) 'Show the result in a text box (one for each element)
    Next
  Else
    'It is a timestamp so do something else with it for example:
    Msgbox "Timestamp of : " & vbLf & Mid(strLine,6,2) & "/" & Mid(strLine,4,2) & "/" & Mid(strLine,2,2) 'dd/mm/yy format display
  End If
Loop Until Eof(intFHandle)
Close #intFHandle

This of course is just an example of how you could process the file, the exact operations on each line would be up to you.

Tim Cottee MCSD, MCDBA, CPIM
Brainbench MVP for Visual Basic
http://www.brainbench.com
0
 
LVL 85

Expert Comment

by:Mike Tomlinson
ID: 9788716
What is the end goal here?

Do you want an easier way to view data sets associated with a time stamp?
... or are you looking to do some calculations based upon the sets and only look at the results?
... or are you wanting to see timestamps with data sets and some kind of calculated results together?

Idle_Mind
0
Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

 

Author Comment

by:jscharpf
ID: 9788864
There won't be an calculations on the data. It will only be graphed. But the problem I have is that the data will be taken at various times during a day, and for undefined lengths of time. So, for example, A time stamp of 11 Am on a certain day wil be followed by a bunch of data.. I'm not sure how many. It could be zero, one, or 10000.... then, another time stamp, followed by more data.. these will be stamped at various times during the day (

To give you more info.. I'm installing a microcontroller in a power tool and letting the operator use it. Each time the operator turns it on and uses it, I time stamp it, then begin saving data until it's turned off. This can take place several times in a day. We are limited in our data saving options for now.. this is how we are getting the data..

So.. I will graph (I have a very good OCX for this) all 8 data points vs. time, and initially this will just be one long graph, ignoring the time stamps, and knowing the sample rate is XX ms, we will see a profile and we can look at all the data..
but as an added option, we will need to somehow ALSO (I can do two graphs and two sets of data if I need to) need to somehow associate what days each part of the graph correspond to.. The graphing OCX I have is very good and can graph an unlimited set of points, so I'm thinking of just storing a second set of data, that is associated with the time stamps, then giving the user the option to just go day by day and look at data for that day only.. (redrawing the graph with the new data each time)..

I am sorry if this is confusing.. :)

Jeff
0
 

Author Comment

by:jscharpf
ID: 9788905
also.. lol..
suppose the operator turns the tool on and off 100 times in one day.. I will have alot of data and alot of time stamps to deal with. I'm just trying to find a good way to group these so that I don't have a million graphs (one for each time stamp would be crazy)
 :)
I realize that I have to compromise somewhere.. so I thought that maybe grouping the data by Day... and just keeping sets of data per day.. it might help.

BUt the MOST important part is to get all the data out and separate it from the timestamps.. so I can at least see nothing but data. I can worry about when it happened later.. Doing this at least gives us good info, we just don't know when it occured (day, etc.)

Jeff
0
 
LVL 85

Expert Comment

by:Mike Tomlinson
ID: 9789123
One approach would be to read the file in just as TimCottee has shown you and add the arrays of data to a collection.

If you want to get tricky, then you could create a separate collection for each day, then have a collection of collections.

As you read each stamp, convert it to a date, without the time data in it, to be used as a key.  Then add all data you read into a collection until you reach a stamp that is from the next day.  Now add the collection you just made to the main collection using the date as a key, and then start a new collection that will have the new stamp key associated with it.

You could place all the date keys in a listbox so you can easily select a different day to view and use the selected item text to retrieve the correct collection containing the corresponding data.

I hope that wasn't too terribly confusing.

Idle_Mind

0
 

Author Comment

by:jscharpf
ID: 9796483
Thanks to both of you for your answers! I guess I can only give points to one person.


Jeff
0

Featured Post

What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

Suggested Solutions

When trying to find the cause of a problem in VBA or VB6 it's often valuable to know what procedures were executed prior to the error. You can use the Call Stack for that but it is often inadequate because it may show procedures you aren't intereste…
You can of course define an array to hold data that is of a particular type like an array of Strings to hold customer names or an array of Doubles to hold customer sales, but what do you do if you want to coordinate that data? This article describes…
Get people started with the process of using Access VBA to control Excel using automation, Microsoft Access can control other applications. An example is the ability to programmatically talk to Excel. Using automation, an Access application can laun…
Get people started with the utilization of class modules. Class modules can be a powerful tool in Microsoft Access. They allow you to create self-contained objects that encapsulate functionality. They can easily hide the complexity of a process from…

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now