No speed increase using Threads

Hi

I just started playing with threads, so I'm a beginner and I'm not sure what I'm doing wrong in my first try.

So, my project reads huge amount of data from files, process it all in memory and creates html reports.
So, all data is stored in 50+ arrays of records defined in main unit. Report is generated through reading
all this data and producing html files. it can create from 500 - 10000 files - depending on type and volume of data.
As an example, a 7000 html files is created in around 3mins - in one long thread. And I thought I can speed it up using multiple threads.

The html files are created for each main object (7000 main objects) and they are just reports, so reading all data from memory and creating htmls.
So, I split into 5 threads, split main objects into 5 ranges and make sure each thread creates 1/5th of html files. It is working good, but it takes about the same time, 3 minutes. I set call-back procedure for each thread to show me with object number is processing and they are doing good job, all numbers are turning simultaneously, it seems, but close observing of numbers in label captions on form it seems a little like they might be waiting each other at some point - it is very brief freeze of numbers rotating at various points.

So, I assume they must be locking resource from each other, thus waiting and so the time is same as in one main thread.

I put all procedure that structure data to be ready to write (temporary arrays) and the actual procedure that do the writing into procedures defined in the Thread itself.
So, I assume this is how you make these process of writing separate from each thread and thus no locking should appear.

The procedures that read all the data from memory are all the same, not split with threads, so they do access same arrays, but different parts of arrays (as each is only locating data related to the main object they are creating html for).
So, they are accessing same array, but not the same data (memory parts) - in 80% cases, in 20% they access the same data, but only reading, no writing!

I now it is hard for you to imagine what I have in my code, but what other issues could I have, that the whole process is not taking 5x faster???
Maybe I'm missing on basic threading knowledge, but I thought this should be the result, maybe the whole process should run in 30seconds...?

So, here are parts of my code, that I think are importunate to show you:

Thread definition:

TMyWorkerThread = class(TThread)
private
  FProgressProc: TProgressProc;
  FFolderUpdate:TFolderUpdate;
  FProgressValue: integer;
...
  procedure SynchedProgress;
  procedure SynchedFolder;
protected
    procedure Progress(aProgress: integer); virtual;
    procedure UpFolder; virtual;
public
  constructor Create(aFolder:string; aProgressProc: TProgressProc; UpdateFolder: TFolderUpdate; CreateSuspended: Boolean = False); reintroduce; virtual;
  procedure Execute; override;
  procedure GenerateHTMLDoc;
...
  procedure GenerateSectionData_T(vData: TData; var vHtml: TextFile);
  procedure GenerateNavigationBar_T(var vHtml: TextFile; vPrg: integer);
  procedure GenerateInfo_T(vPrg, vTaks: integer; var vHtml: TextFile);
  procedure GenerateStructure_JavaScript_T(vPrg: integer; var vHtml: TextFile);
  procedure GenerateStructure_T(vPrg: integer; var html_programs: textfile);
  procedure GenerateProperties_T(vPrg, vTask, vIndex: integer; var vHtml: TextFile);
  procedure GenerateFlow_new_T(vPrg, vTask: integer; var vHtml: TextFile; vForSinglePageReport: boolean = false);
  procedure GenerateLocateInfo_T(vPrg, vTask, vTaskIndex: integer; var vHtml: TextFile);
  procedure GenerateSQLInfo_T(vPrg, vTask, vTaskIndex: integer; var vHtml: TextFile);
...

end;

Open in new window



thread creating from unit1:

 
if MyWorkerThread<>nil then
    raise Exception.Create('One thread have already been started!')
  else
  begin
    MyWorkerThread := TMyWorkerThread.Create(DestFolder, UpdateProgressbar,UpdateFolderInfo1);
    MyWorkerThread.OnTerminate := ThreadDone;
  end;

Open in new window




In threads, I run GenerateHTMLDoc and it reads data, creates structure and produces html files by writint into html and calling various procedure passing the TextFile handle.

This is how html files are used:
(the old style)


assignfile(html_file, DestFolder + 'object_'+IntToStr(ObjectID)+'.html');
  rewrite(html_file);
  writeln(html_file,...);
... 
...
close(html_file);

Open in new window



Part of the html is also creating images that are part of main object, but I disabled it for testing purposes as I read, that TCanvas is not thera-safe... so, for now I want to speed up the process that does the writing and then I will address the TBitmap, TCanvas usage in threads.

Delphi XE7

Thank you for any suggestion what to look for!
Delphi_developerAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Sinisa VukCommented:
Try to disable (comment) line where you open/read/write to files and check speed. Maybe accessing files can take to much...

Edit: without code which you do sync (where thread may block) between threads I cannot tell more...
0
Delphi_developerAuthor Commented:
When I disable all file handling, so no output, it drops down to 45 secs in 1 thread - but 1 min in 5 threads. This is a long and process a lot of code, and I don't know how to approach it to identify what could be the issue.
There is no sync between threads - each get instructions on which main objects to work on  and to produce htmls. I only sync back to main thread the number of object they are currently processing so I can see the numbers running on the form.
Here is how I snyc with main form:

I pass UpdateProgressbar to each thread (UpdateProgressbar2,UpdateProgressbar3,UpdateProgressbar4 and UpdateProgressbar5 to other threads)...

MyWorkerThread := TMyWorkerThread.Create(DestFolder, UpdateProgressbar,UpdateFolderInfo1);

procedure TForm1.UpdateProgressBar(aProgress: Integer);
begin
  ProgressLabel.Caption:=inttostr(aProgress);
end;

Open in new window

and in thread sends the number with:
constructor TMyWorkerThread.Create(aFolder:string; aProgressProc: TProgressProc; UpdateFolder: TFolderUpdate; CreateSuspended: Boolean = False);
begin
  inherited Create(CreateSuspended);
  FreeOnTerminate := True;
...
  FProgressProc := aProgressProc;
...
end;

procedure TMyWorkerThread.Progress(aProgress: Integer);
begin
  FProgressValue := aProgress;
  Synchronize(SynchedProgress);
end;

procedure TMyWorkerThread.SynchedProgress;
begin
  if Assigned(FProgressProc) then
    FProgressProc(FProgressValue);
end;

Open in new window


and in code I update progress with

Progress(i);

Open in new window

0
Sinisa VukCommented:
slowdowns are possible when threads need shared resource: code protected by Synchronize (even updatin something on main form), accesss to files on disk (disk is a one), access to database, ....
Try to disable first almost the whole Execute code, then step by step allow more and more code - measure it.... Maybe you'll find code (procedure)  which takes more time...
0
Cloud Class® Course: Certified Penetration Testing

This CPTE Certified Penetration Testing Engineer course covers everything you need to know about becoming a Certified Penetration Testing Engineer. Career Path: Professional roles include Ethical Hackers, Security Consultants, System Administrators, and Chief Security Officers.

Delphi_developerAuthor Commented:
Ok. So, just reading from arrays, that are filled before threads start running, should not lock, freeze threads, right? So, as long it's all read-only, I don't need to be looking into read-only parts of the code (100s of quick procedure), correct?
0
Sinisa VukCommented:
How big are your arrays? In one of my projects I pass bunch of arrays as static data into each thread as parameter.
0
Delphi_developerAuthor Commented:
1Gb in this case, with 7000+ objects, across 50+ arrays.
0
Delphi_developerAuthor Commented:
10M+ records across these arrays.
0
Delphi_developerAuthor Commented:
Just did some testing to see if threads lock each other when calling same procedures reading data from array. Either 1M calls of same procedure and simple array loop; or 1 call and 1M loops on array; or 1call and 1M reads/accessing same field in same record
Pos(vTmp, MyArrayOfStrings[185573].StringExpression)>0 //(185573 = longest string in array)... 

Open in new window

they all run the same time and seems to not lock/freeze each other. So, this is good to know.

Moving forward with dissecting the code.
0
Geert GOracle dbaCommented:
did you try switching off the feedback ?
feedback is invaluable during development and testing, but not necessary when actually running

an alternative is to use feedback into a file without using synchronize !
and even better is to have each thread write feedback into it's own logfile

writing threaded programs can be a nightmare

i'm assuming your next step would be to convert your program to a service ?

fwiw,
writing to an invisible TBitmap canvas is thread safe (inside 1 thread)
and you don't need the main thread to draw
0
Delphi_developerAuthor Commented:
I have been testing and testing and since it is hard to pint point what causes delays, if there are any locks, it is very hard to know what will work and what not. So, I managed to get work with 2 threads and reducing the execution time to almost half. With 3 threads it only reduces another 10%. It seems like heavy memory accessing (readonly) and writing files on disk may not really lock threads, but I assume memory, disk and processor resources are getting stuck at certain point and still handle everything OK, just not as simultaneously, as theory would expect: 5 threads, whole execution time in 20%!
The more threads the more system resource (CPU, memory) are used and they become bottlenecks.

So, I'm happy with results, so far. I do create bitmaps, but non-visible, so threads draw them on their on canvas, I assume there is no locking - it doesn't appear to be.

Also, first thing I learned with threads: don't look at processors running time, as with 2 threads 1 second becomes 2 seconds, So, using GetTickCount is the way to go!

I read numerous articles about threads and accessing memory and string manipulation - it makes sense that my process has same obstacles as it does just that, reading a lot of memory and manipulating strings.

So, no magic solution... making sure most of the methods are in thread class, limit same string manipulation from threads,
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Sinisa VukCommented:
Maybe you can spend some time to investigate BMDThread component too :-) This component allows to move away from Threads - just create running task ....
0
Delphi_developerAuthor Commented:
Thank you Sinisa, I'm not sure if I want a component for threads or replacement of them - I just have to learn about them and use them properly and learn how they work with my process.
0
Sinisa VukCommented:
It is good to know Threads, but using known framework (around Threads) can help you at this specific (and large) project.
1
Geert GOracle dbaCommented:
Trial and error is actually a very good description of multi threaded programming, but ...
"Nightmare and error" is even better !
1
Delphi_developerAuthor Commented:
Thank you for all the suggestions, but I had to learn on my own what works in my code and what not - I have a complicated process and used a lot of trial and error to get to these conclusions.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Delphi

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.