Solved

Rendering over 47,000 Lines in a TAdvMemo - Reboots computer

Posted on 2006-07-09
22
282 Views
Last Modified: 2010-04-05
Hello All;

  I am using the following code to generate HTML lines into a Memo (TAdvMemo)
=======================================================
var
CID, Con : String;
I : Integer;
begin
  for I := 0 to Memo3.Lines.Count -1 do
  begin
  CID := StringReplace(Memo3.Lines[i], ' ', ' ',[rfReplaceAll]);
  Con := StringReplace(Memo2.Lines[i], ' ', ' ',[rfReplaceAll]);
  AdvMemo1.Lines.Add(' <loc>http://somesite.com/content_by_cat3.asp?contentid='+Con+'&#38;catid='+CID+'</loc>');

=======================================================

I have loaded:
47,000 Lines loaded in:  Memo3
And another 47,000 Lines loaded in the Memo2.
(These work together for the link above)

After about 35-Minutes of running, the computer Reboots itself.

Is there a way to make it do:

After Running for [5 minutes] take a rest for: 10000 <-- 10 Seconds.
(Like:    sleep(10000); )
And then resume.
After another [5 minutes] take a rest for: 10000 <-- 10 seconds.
Then resume process, and continue this routine until the job is done?

(or)
After every 1,000 Lines is processed, Take a [10 Second] break
Then resume until another 1,000 lines are hit, then [10 second] break
And continue this routine until the job is done?

(The above, are 2 examples of what might work good in this type of process.
I will leave it up to you all, on which is the better one to go for?)

Thanks All;
Carrzkiss
0
Comment
Question by:Wayne Barron
  • 12
  • 10
22 Comments
 
LVL 17

Expert Comment

by:TheRealLoki
ID: 17069787
you are looping though memo3.lines (as i)
but you access memo2.lines[i] also
this looks doom to failure unbless you are absolutely sure memo2 has at least the same number of lines as memo3 does

this piece of code will do both, sleep every 1000 lines or every 5 minutes, whichever happens first
var
  CID, Con : String;
  I : Integer;
  starttime: TDateTime;
begin
  i := 0;
  starttime := now;
  for I := 0 to Memo3.Lines.Count -1 do
  begin
      CID := StringReplace(Memo3.Lines[i], ' ', ' ',[rfReplaceAll]);
      Con := StringReplace(Memo2.Lines[i], ' ', ' ',[rfReplaceAll]);
      AdvMemo1.Lines.Add(' <loc>http://somesite.com/content_by_cat3.asp?contentid='+Con+'&#38;catid='+CID+'</loc>');
      if (i > 0) and (i mod 100) = 0 then
      begin // only bother checking the time every 100 lines
        if  ((starttime + encodetime(0, 5, 0, 0)) > now) or (i mod 1000) = 0 then
        begin
          sleep(10000);
          starttime := now; // reset the timer
        end;
      end;
  end;
0
 
LVL 17

Expert Comment

by:TheRealLoki
ID: 17069790
oops, obligatory typo
" if  ((starttime + encodetime(0, 5, 0, 0)) > now) or (i mod 1000) = 0 then"
should be
if  (
      ((starttime + encodetime(0, 5, 0, 0)) < now) or
      ((i mod 1000) = 0)
    ) then
0
 
LVL 17

Accepted Solution

by:
TheRealLoki earned 400 total points
ID: 17069806
oops, also should be
    if (i > 0) and ( (i mod 100) = 0) then

so the complete routine will be :-

var
  CID, Con : String;
  I : Integer;
  starttime: TDateTime;
begin
  i := 0;
  starttime := now;
  for I := 0 to Memo3.Lines.Count -1 do
  begin
    CID := StringReplace(Memo3.Lines[i], ' ', ' ',[rfReplaceAll]);
    Con := StringReplace(Memo2.Lines[i], ' ', ' ',[rfReplaceAll]);
    AdvMemo1.Lines.Add(' <loc>http://somesite.com/content_by_cat3.asp?contentid='+Con+'&#38;catid='+CID+'</loc>');
    if (i > 0) and ( (i mod 100) = 0) then
    begin // only bother checking the time every 100 lines
      if (
           ((starttime + encodetime(0, 5, 0, 0)) < now) or
           ((i mod 1000) = 0)
         ) then
      begin
        sleep(10000);
        starttime := now; // reset the timer
      end;
    end;
  end;
end;

sorry about that. That'll teach me for not testing it first :-)
0
 
LVL 30

Author Comment

by:Wayne Barron
ID: 17069819
Thanks,
Am starting up Delphi after another "Reboot" due to the program.
No problem on the code changes, as I am just now able to check it.

Fingers crossed.
It will be about an hour or so before I will be able to post the results of the code back.
As that is about how long it is going to take to run the 47,000+ lines.


Also.
Memo2 & Memo3
Both have the same amount of lines.
Thus the reason for them both using the [i].
0
 
LVL 30

Author Comment

by:Wayne Barron
ID: 17069853
It seems to be working pretty good so far.
-------
I have done this to let me know when it is "Sleeping" & "Processing"
======================================
      begin
        Label3.Caption := 'sleeping';
        Label3.Refresh;
        Panel1.Refresh;
        sleep(10000);
        starttime := now; // reset the timer
        Label3.Caption := 'Processing';
        Label3.Refresh;
        Panel1.Refresh;
      end;
======================================
Is there a way to count each time that the Label shows:  Sleeping

Basically would be something like this:

Sleeping    [1]
processing
Sleeping   [2]
Processing
Sleeping   [3]
Processing. And so on until it is completed.

This way I can keep track of "where" it is in the process.
As each number will stand for "Thousand" So: 1 = 1,000  2 = 2,000   3 = 3,000
And so on.
0
 
LVL 17

Assisted Solution

by:TheRealLoki
TheRealLoki earned 400 total points
ID: 17069996
either keep a counter :-

var
    timesslept: integer;

...
      begin
        inc(timesslept);
        Label3.Caption := 'sleeping  [' + IntToStr(timesslept) + ']';
        Label3.Refresh;
        Panel1.Refresh;
        sleep(10000);
        starttime := now; // reset the timer
        Label3.Caption := 'Processing';
        Label3.Refresh;
        Panel1.Refresh;
      end;


or if 1000 lines is occurring before 5 minutes, you can just use the value of the for loop
eg.

        Label3.Caption := 'sleeping  [' + IntToStr(i div 1000) + ']';
0
 
LVL 30

Author Comment

by:Wayne Barron
ID: 17070148
Great.
Thanks a bunch.

Strang though.
Once the process was complete. (I did the full process before you posted the code above for Count)
The computer Screen went blank, as though it went to "Sleep" itself.
But this time it did not "Reboot" like previous times. (Which is a GOOD thing).

I jumped the points from 250 - 400
I am going to do a split between you Working Code.
And then with the "Count" code.

Take Care and thank you for saving my computer for me :)

Wayne
0
 
LVL 17

Expert Comment

by:TheRealLoki
ID: 17070224
thanks!
0
 
LVL 30

Author Comment

by:Wayne Barron
ID: 17086885
Hello [TheRealLoki];

  I have another one for you. Of which I tried to use your code here to implement it, but
Would not work, as this code above looks at the lines that are "Inserted" into the Memo Field.
And what I am doing in this code is [Replacing] certain words in each Line that is in the Memo.

Here is the code. (And like this Topic here, we are rendering through a large number of
Lines, and replacing words in each line.
-----------------------------------------------
SearchEdit.Clear;
ReplaceEdit.Clear;
SearchEdit.Text := '[ID Here]';
ReplaceEdit.Text := '55555555';
AdvMemo1.Lines.Text:=FastReplace(AdvMemo1.Lines.Text, trim(SearchEdit.Text), trim(ReplaceEdit.Text));
-----------------------------------------------

If you have an idea for this one. It basically needs to be just like the code you provided above.
I need it to sleep every (in this case, I am dealing with over 500,000 lines) so needs to sleep every 2-3 minutes
For about 10-20 Seconds. And possibly freeup any memory or what you think that should be freed then
Start the process running again.

Thank you and let me know, I will create a [new post] for you, if you can do this.

Thanks
Wayne
0
 
LVL 17

Expert Comment

by:TheRealLoki
ID: 17087101
What Version of Delphi are you using, because you could do what you want by using Pos and PosEx
stringreplace without [rfReplaceAll] could do it, but it would needlessly search from teh beginning each time, so would be inefficient.
0
 
LVL 30

Author Comment

by:Wayne Barron
ID: 17087197
D6.02  (Win2k SP4)

You stated: (Search from the beginning each time).
If you are referring to after it does a replace, then it would start over again at the beginning
Then, Nope, cannot have that happen.

If you are referring to if you want to do a new :Search/Replace: and it would
Start at the Beginning, then , That would be fine.
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 17

Expert Comment

by:TheRealLoki
ID: 17087517
to speed things up (for this, and your first question) you should use BeginUpdate and EndUpdate while working with the strings/lines

eg.
AdvMemo1.Lines.BeginUpdate;
try
  AdvMemo1.Lines.Text:=FastReplace(AdvMemo1.Lines.Text, trim(SearchEdit.Text), trim(ReplaceEdit.Text));
finally
  AdvMemo1.Lines.EndUpdate;
end;
0
 
LVL 30

Author Comment

by:Wayne Barron
ID: 17087804
OK, using this.
Speed thing up?? <-- Do you mean that this would speed through the lines a lot faster then what I am
Using right now?

Also, the [BeginUpdate; & EndUpdate;] how can I do a "Sleep" every few minutes. (Like every 2-3 minutes)
As working with the file that I am dealing with at the moment, it is over 500,000 lines,
And you are looking at around an hour or so. Unless this code makes it faster?

Please let me know, before I test it, as it kills the computer after 35-minutes of processing.

Thanks a bunch.

Wayne
0
 
LVL 17

Expert Comment

by:TheRealLoki
ID: 17088649
beginupdate and endupdate, when used with a memo or similar means that it will not try to display it while you are working with it. this makes the process a _lot_ faster.
I'll do yo a demo for the 2nd problem tomorrow
0
 
LVL 30

Author Comment

by:Wayne Barron
ID: 17088653
ok Thank you.

Take Care
0
 
LVL 17

Expert Comment

by:TheRealLoki
ID: 17094933
This will do 1 line in the memo at a time, as above

procedure TForm1.Button2Click(Sender: TObject);
    var
        i: integer;
        starttime: TDateTime;
        searchword: string;
        newword: string;
        fulltext: string;
        timesslept: integer;
    begin
        memo1.lines.BeginUpdate;
        try
            searchword := 'fox';
            newword := 'cow';
            fulltext := '';
            timesslept := 0;
            starttime := now;
            for i := 0 to pred(memo1.lines.count) do
            begin
                fulltext := memo1.lines[i];
                fulltext := StringReplace(FullText, searchword, newword, [rfReplaceAll]);
                memo1.lines[i] := fulltext;
                if (i > 0) and ( (i mod 100) = 0) then
                begin // only bother checking the time every 100 "replaces"
                    if (
                       ((starttime + encodetime(0, 5, 0, 0)) < now) or
                       ((i mod 1000) = 0)
                     ) then
                    begin
                        inc(timesslept);
                        Label3.Caption := 'sleeping  [' + IntToStr(timesslept) + ']';
                        Label3.Refresh;
                        Panel1.Refresh;
                        sleep(10000);
                        starttime := now; // reset the timer
                        Label3.Caption := 'Processing';
                        Label3.Refresh;
                        Panel1.Refresh;
                    end;
                end;
            end;
        finally
            memo1.lines.endupdate;
        end;
    end;
0
 
LVL 30

Author Comment

by:Wayne Barron
ID: 17095781
That is pretty darn fast.
I repeated this over and over again for .. not sure how many lines.

this is a fox this is a fox this is a fox this is a fox this is a fox this is a fox this is a fox this is a fox this is a fox this is a fox

It took it about 1-1/2 minutes to replace all the instances of "fox" to "cow"

I just checked the lines.
Well over 6,000 + lines.
To fast.
Might just work good on the 500,000 Line job(s).

----------
I just had to reinstall my Laptop, due to too many long processes on these Feeds that I am dealing with.
So "hopefully" this code will help to save my computer a little longer then a "1 Month Install".
(Yep, it was 1 month ago since I last installed it, and just did it today).

OK.
Give me a little while, and I will post a new question for you and add the link in here.

Take Care now.
Wayne
0
 
LVL 30

Author Comment

by:Wayne Barron
ID: 17098371
I almost forgot about you :)
Na, just joking, have not been to sleep yet. (Cannot sleep, too much in my head to sleep)

Got a quick question. (Do not worry, I will take real good care of you on the Question I am going
To put up for you)

How would I implement your latest code (Quick Code) into the "Original Code"
That we did here?

Reason:
The original code took over 30-Minutes to process 47,000+ Lines.
Then it's wrapper, which endedded up being a total of 367,000+ lines.

OK. So, as you can tell, that it a LOT of lines.
I was thinking about your code here that you last gave to me.
If there was a way to implement it, getting rid of the:

    CID := StringReplace(Memo3.Lines[i], ' ', ' ',[rfReplaceAll]);
    Con := StringReplace(Memo2.Lines[i], ' ', ' ',[rfReplaceAll]);

And replacing it with something else. (As the   CID & Con   are being used in:)
AdvMemo1.Lines.Add(' <loc>http://somesite.com/content_by_cat3.asp?contentid='+Con+'&#38;catid='+CID+'</loc>');

Thanks a bunch, And as I promised, I will post you a question once I know you are online.

Take Care
Wayne
0
 
LVL 17

Expert Comment

by:TheRealLoki
ID: 17104146
not sure exactly what you mean, but here is what I think

    CID := StringReplace(Memo3.Lines[i], ' ', ' ',[rfReplaceAll]);
    Con := StringReplace(Memo2.Lines[i], ' ', ' ',[rfReplaceAll]);
this really does nothing at all, it looks like it replaces a space with a space
is the same as going
    CID := Memo3.Lines[i];
    Con := Memo2.Lines[i];
so it looks to me like you want to add a line in advmemo1 that has memo1, and memo2 in it
please let me know if I am wrong

var
  CID, Con : String;
  I : Integer;
  starttime: TDateTime;
begin
  i := 0;
  starttime := now;
  memo2.lines.beginupdate;
  memo3.lines.beginupdate;
  AdvMemo1.lines.beginupdate;
  try
    for I := 0 to Memo3.Lines.Count -1 do
    begin
      CID := StringReplace(Memo3.Lines[i], ' ', ' ',[rfReplaceAll]);
      Con := StringReplace(Memo2.Lines[i], ' ', ' ',[rfReplaceAll]);
      AdvMemo1.Lines.Add(' <loc>http://somesite.com/content_by_cat3.asp?contentid='+Memo2.Lines[i]+'&#38;catid='+Memo3.Lines[i]+'</loc>');
      if (i > 0) and ( (i mod 100) = 0) then
      begin // only bother checking the time every 100 lines
        if (
             ((starttime + encodetime(0, 5, 0, 0)) < now) or
             ((i mod 1000) = 0)
           ) then
        begin
          inc(timesslept);
          Label3.Caption := 'sleeping  [' + IntToStr(timesslept) + ']';
          Label3.Refresh;
          Panel1.Refresh;
          sleep(10000);
          starttime := now; // reset the timer
          Label3.Caption := 'Processing';
          Label3.Refresh;
          Panel1.Refresh;
        end;
      end;
    end;
  finally
    memo2.lines.endupdate;
    memo3.lines.endupdate;
    AdvMemo1.lines.endupdate;
  end;
end;
0
 
LVL 17

Expert Comment

by:TheRealLoki
ID: 17104152
oops, still asleep

var
  I : Integer;
  starttime: TDateTime;
begin
  i := 0;
  starttime := now;
  memo2.lines.beginupdate;
  memo3.lines.beginupdate;
  AdvMemo1.lines.beginupdate;
  try
    for I := 0 to Memo3.Lines.Count -1 do
    begin
      AdvMemo1.Lines.Add(' <loc>http://somesite.com/content_by_cat3.asp?contentid='+Memo2.Lines[i]+'&#38;catid='+Memo3.Lines[i]+'</loc>');
      if (i > 0) and ( (i mod 100) = 0) then
      begin // only bother checking the time every 100 lines
        if (
             ((starttime + encodetime(0, 5, 0, 0)) < now) or
             ((i mod 1000) = 0)
           ) then
        begin
          inc(timesslept);
          Label3.Caption := 'sleeping  [' + IntToStr(timesslept) + ']';
          Label3.Refresh;
          Panel1.Refresh;
          sleep(10000);
          starttime := now; // reset the timer
          Label3.Caption := 'Processing';
          Label3.Refresh;
          Panel1.Refresh;
        end;
      end;
    end;
  finally
    memo2.lines.endupdate;
    memo3.lines.endupdate;
    AdvMemo1.lines.endupdate;
  end;
end;
0
 
LVL 30

Author Comment

by:Wayne Barron
ID: 17104758
Processed 45,911 Lines
Time by Clock: 5 Minutes 20 Seconds
Time by CPU  : 1 Minute 18 Seconds
Cut time from 10-Second Sleep. -To- 5-Second Sleep.

Original Code:
Time by Clock: 1hr 25 Minutes.
Time by CPU  : 45 Minutes. (give or take a few seconds)

Outcome?
My computer is happier, and it is safer.
No more Rebooting itself, and hopefully will not have to reinstall again no time soon.

Just copy and paste your Last Comment [Code] into the following Post.
http:Q_21918923.html   500-Points
You deserve more, but that is all that it will allow me to give, as you very well know.

Thank you, and my computer(s) will thank you.
Maybe now, I will not burn up my processor, like I was afraid I would do.

Take Care Friend.
Wayne
0
 
LVL 17

Expert Comment

by:TheRealLoki
ID: 17117948
you're welcome
0

Featured Post

Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

Join & Write a Comment

The uses clause is one of those things that just tends to grow and grow. Most of the time this is in the main form, as it's from this form that all others are called. If you have a big application (including many forms), the uses clause in the in…
Creating an auto free TStringList The TStringList is a basic and frequently used object in Delphi. On many occasions, you may want to create a temporary list, process some items in the list and be done with the list. In such cases, you have to…
Illustrator's Shape Builder tool will let you combine shapes visually and interactively. This video shows the Mac version, but the tool works the same way in Windows. To follow along with this video, you can draw your own shapes or download the file…
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now