How to retrieve information from a text file


I capture a string of data to a text file and would like to load it in memory to run to seek and display information.

I am thinking of using a dataset, unless there is a faster way to retrieve info. I am more than open to suggestions.

The Data file as the following items:

Some commentaries at the start of the file
time, date, string1, string2, string3
time, date, string1, string2, string3

each items are separated by commas. each line by cr + lf

what I would like is to load it in a dataset with the following format:
index, time,date,field1,field2,field3

this way I could efficiently search by any item and get the whole line of data.

I want to retrieve information every second and display the result to the user.

any idea?

many thanks
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

>> I want to retrieve information every second and display the result to the user.

is this re-creating that textfile every second and implicitly the dataset or accessing the dataset every second?

moreover, how is that textfile created? can you instead generate the dataset directly?

you can use embedded firebird as the dataset. see this paq:

if you're not afraid of getting your hands dirty, then you can use the TCLientDataSet as an inmemory database. see this tip for that:
more about client data set here:

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Geert GOracle dbaCommented:
>>I want to retrieve information every second and display the result to the user.
the user can't possible read that fast, why would you want to represent info each second ?

why not use something that loads all text files into a database
you can then more easily write queries on this database
to extract information

that fastest algorithms for searching are databases.  That's what they were designed for ...
Geert GOracle dbaCommented:
oops, didn't type fast enough
well looks like we're going the same way again, aren't we ciuly ?
CompTIA Cloud+

The CompTIA Cloud+ Basic training course will teach you about cloud concepts and models, data storage, networking, and network infrastructure.

yeah, and I'm still faster than you :P
etiennedemersAuthor Commented:
I am getting the data from a GPS. I have no problem in displaying the information live.

But on playback, it is different. I have to read the text file, put it in memory and search through it. Using a Dataset is my best guess. But I have no experience in manipulating a text file in Pascal. (I recently crossed over from VB)

If it take a couple of seconds to load the text file into a dataset or Listbox, it is not important. As long as I can read/query it fast enough for data to be displayed every second.

Each line represent a string of different component. Lets say Item 1 is altitude, 2 is latitude, 3 longitude, 4 heading, 5 temparature.

I want to display that information in a window in realtime where basically the altidute textbox will be updated every second...

The application I am working on will have a map with a point that moves according to lat/long, altitude, heading, temperature etc. This way you can playback your excursion just like it is live.

suppose you have the data in data.txt you will do something like this (based on the tip from scallabium. please do read the info from that page!)
 (note that I wrote this dirrectly in the browser so if something does not compile or doens't run as you wish, just let me know):

uses strutils;
procedure TForm1.loaddata;
var l,t:TStringList;
  table := TClientDataset.Create(Application);
  table.FieldDefs.Add('ID', ftInteger, 0, False);
  table.FieldDefs.Add('TTime', ftDate, 0, False);// not sure if "time" and "date" are reserved words here so better safe than sorry
  table.FieldDefs.Add('DDate', ftDate, 0, False);
  table.FieldDefs.Add('String1', ftSting, whatever size, False);
  table.FieldDefs.Add('String2', ftSting, whatever size, False);
  table.FieldDefs.Add('String3', ftSting, whatever size, False);
      for i:=1 to l.count do
        table.FieldByName('ID').AsInteger := i;
        table.FieldByName('TTime').Value := strtotime(t[0]);
        table.FieldByName('DDate').Value := strtodate(t[1]);
        table.FieldByName('String1').AsString := t[2];
        table.FieldByName('String2').AsString := t[3];
        table.FieldByName('String3').AsString := t[4];
then, just use table as you would normally use it.

Open in new window

etiennedemersAuthor Commented:

Read through and am very glad you suggested a TClientDataset. This will do the job perfectly provided that I can make it work...

The problem I am encountering is on the append because the table is not opened. I have tried to open it and get errors.

See attached code for more info.

procedure TForm1.Button1Click(Sender: TObject);
//Sample String:03/11/2008 02:36:41,Route1,02:36:37.21,03/11/08,174.8,263101.3,3972163.0,22.310,5.4,27.6,63.1
  table := TClientDataset.Create(Application);
  table.FieldDefs.Add('ID', ftInteger, 0, False);
  table.FieldDefs.Add('tdStamp', ftDateTime, 0, False);
  table.FieldDefs.Add('sIdent', ftString, 0, False);
  table.FieldDefs.Add('tStringTime', ftTime, 0, False);
  table.FieldDefs.Add('dStringDate', ftDate, 0, False);
  table.FieldDefs.Add('sHeading', ftString, 0, False);
  table.FieldDefs.Add('sEastings', ftString, 0, False);
  table.FieldDefs.Add('sNorthings', ftString, 0, False);
  table.FieldDefs.Add('sKP', ftString, 0, False);
  table.FieldDefs.Add('sDCC', ftString, 0, False);
  table.FieldDefs.Add('sAltitude', ftString, 0, False);
  table.FieldDefs.Add('sTemp', ftString, 0, False);
    table.CreateDataSet;   //raises exception class EDBClient with message "Invalid parameter"
    table.Open;  //raises exception class EDataBaseError with message "Missing data provider or data packet"
      for i:=1 to l.count do
        table.FieldByName('ID').AsInteger := i;
        table.FieldByName('tdStamp').Value := strtotime(t[0]);
        table.FieldByName('sIdent').Value := strtodate(t[1]);
        table.FieldByName('tStringTime').AsString := t[2];
        table.FieldByName('dStringDate').AsString := t[3];
        table.FieldByName('sHeading').AsString := t[4];
        table.FieldByName('sEastings').AsString := t[5];
        table.FieldByName('sNorthings').AsString := t[6];
        table.FieldByName('sKP').AsString := t[7];
        table.FieldByName('sDCC').AsString := t[8];
        table.FieldByName('sAltitude').AsString := t[9];
        table.FieldByName('sTemp').AsString := t[10];

Open in new window

Geert GOracle dbaCommented:
your ftString fieldtypes should have a length
are you sure you read it? because from my experience people don't read the resources I give them to read. they just get satisfied that the code works. Which is why lately I don't use the IDE anymore and just write the code in the browser. and this just prooves that you didn't read that article.

And here is the proof that you didn't read the article fully:
- there are 7 steps to be done mentioned in that article
- I have written the code for steps 1, 2 and 5.
- I have left out steps 3, 4 and 6 (and 7, but you don't need that)

You took my code and modified it to your needs and that's abuot it. Even though you got an error of table not open you did not remember that there was a step 4 saying "open a dataset". If you have indeed read that article you would have either remembered that OR at least you would have noticed that there is missing code compared to the article.

It's nothing perosnal but trust me: everybody does as you do. I'e been here long enough to say this. They say the've read everything and they understand it and all but they still can't get it to work. I can understand that; it's much more easier and less time consuming to just say "I did all that and it's not working. So you do it for me" then actually spending some time and learning all that. You might have read that article in between the lines but you did not read it fully.

You have a problem and you don't know how to solve it. For Gods sake, pay attention and read stuff fully, even read it more times. Don't assume that if you read between the lines you will understand what's there. Obviously that's not working for you. Because either you read that article between the lines or you didn't read it at all. Sure, reading carefully will take twice or maybe 3 times more time, but at least you'll get it right. Overall, you win if you do it right and you loose if you don't.

so, in conclusion, write the code for steps 3 and 4 and if you need indexes to sort the table then also consider step 6. then it will work.
etiennedemersAuthor Commented:

I am not an expert at Pascal which is why I use EE and pay a membership. I do not appreciate the lecture particually comming from a person that does not do what he accuses others of not doing...

Check out line 25 and 26 of my code snippet.

Also, your code did not work as you did not include DB and DBClient in your uses nor declare table as a variable.

Geert, thanks, this was my error as I missread the borland help
if you have looked at my code I wrote there "whatever size" specifically so that you will enter your desired size. now it's my fault that you written 0? ok, be it so.

>> Also, your code did not work as you did not [etc,etc,etc,]
my code has a tone of chances for not working. I did say "note that I wrote this dirrectly in the browse." so beside the 2 missing units from uses clause and the missing table declaration in form section, and you forgot to mention the ftsting typo, everything else compiles just fine.
you expected some code written directly in the browser to complie? (rethorical question)

the lecture is indeed my fault, because I didn't look at your code but went with the table is not open error you mentioned and which I was expecting you to say. It's like doing something and expecting the other party to behave in some manner, which they do and then saying "I knew you would do that".
ok, it's my experiment went bad because even though you said what I was expecting, you also went forward with the issue, which I wasn't expecting.
So this is my fault for which I apologise.
(for the record, that doesn't change the fact that most of the askers do it)
And as I said, it's nothing personal.

but don't through the "code doesn't work/compile" in my face as it would be my fault. I warned you about that upfront.

the rest I'll leave up to geert.
Geert GOracle dbaCommented:
I agree with ciuly that people on EE tend to think they will get an app or code thrown at them,
all ready, all perfect solution. They think they just need to copy paste and hey presto, all done.

I approve Ciuly really trying to *bang* these people back to reality and telling them,
"hey start thinking for yourself."

In University we had a professor.  At the start of the year, he said:"You will be getting 10 lessons from me,
and in every lesson there will be a lie.  I will start putting a lie in the lesson as from next lesson.  At the end of each lesson you can donate a paper noting the lie. When you come to the exam and have all 10 lies correct, you don't need to do the exam and will receive and A grade." This off course had the attention of every single student.  In the beginning the lies were easy, but the last 3 required the student to have studied the earlier lessons.
In the end, nobody got all lies. (The second sentence he said was a lie in itself and he didn't tell a lie in the last lesson)

Ciuly seems to be following this same direction, the only difference being the diplomacy approach.

If you take up each answer this way, you should get better at Delphi every time ... :)
etiennedemersAuthor Commented:

<<quote:but don't through the "code doesn't work/compile" in my face as it would be my fault. I warned you about that upfront.>>

I do not understand why you are being so agressive/defensive. I am not throwing anything at your face, you are helping me with a problem, why would I do that?

You accused me of 1, not reading your solution and 2 stupidly cut and pasting your code and expecting it to work.

I was only stating that your code didn't work because you told me it might not and I confirmed it. Also confirming that I did not just cut and pasted your code...

Truth is I appreciate your help and suggestion, not so much your attitude. Maybe, you have been doing this for too long...

By the way, I saw through your little test as I had noticed you cut and pasted part of your solution from Scalabium.

I provided a snipet of my code and phrased my response as best as possible for you to understand where my fault was occuring.

Seems you have time to waste playing games. I do not.

Thank you for your help
etiennedemersAuthor Commented:

quote"I approve Ciuly really trying to *bang* these people back to reality and telling them,
"hey start thinking for yourself.""

I understand, but this is not a university, its a help system. You come accross all kinds of different people.

My point is, do not go around lecturing people about not reading stuff when you do not read responses yourself.

Ciuly, god bless him, has just proven he is one of those people that he tries to "bang into reality".

I find in life that you hate about others what you hate the more about yourself.

Anyway, thanks for your help as well
etiennedemersAuthor Commented:
Thank you for your help. It has been appreciated. If not for the attitude, I would have given an "A".
>> It has been appreciated. If not for the attitude, I would have given an "A".

well, one thing you didn't read for sure, just like over 95% of the asker: the grading tips.
the grade is for the answer and not for the expert. you grade the answer not the expert. if the answer deserves an A but the expert deserves a C, you grade an A. it doesn't matter how rude and idiotic I am, if the answer solves your problem, you grade with an A. you're right, it's not a university so you don't grade as in a university, unless you accept being fed with lectures as in a university.
who's tricking who?
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.