I just started playing with threads, so I'm a beginner and I'm not sure what I'm doing wrong in my first try.
So, my project reads huge amount of data from files, process it all in memory and creates html reports.
So, all data is stored in 50+ arrays of records defined in main unit. Report is generated through reading
all this data and producing html files. it can create from 500 - 10000 files - depending on type and volume of data.
As an example, a 7000 html files is created in around 3mins - in one long thread. And I thought I can speed it up using multiple threads.
The html files are created for each main object (7000 main objects) and they are just reports, so reading all data from memory and creating htmls.
So, I split into 5 threads, split main objects into 5 ranges and make sure each thread creates 1/5th of html files. It is working good, but it takes about the same time, 3 minutes. I set call-back procedure for each thread to show me with object number is processing and they are doing good job, all numbers are turning simultaneously, it seems, but close observing of numbers in label captions on form it seems a little like they might be waiting each other at some point - it is very brief freeze of numbers rotating at various points.
So, I assume they must be locking resource from each other, thus waiting and so the time is same as in one main thread.
I put all procedure that structure data to be ready to write (temporary arrays) and the actual procedure that do the writing into procedures defined in the Thread itself.
So, I assume this is how you make these process of writing separate from each thread and thus no locking should appear.
The procedures that read all the data from memory are all the same, not split with threads, so they do access same arrays, but different parts of arrays (as each is only locating data related to the main object they are creating html for).
So, they are accessing same array, but not the same data (memory parts) - in 80% cases, in 20% they access the same data, but only reading, no writing!
I now it is hard for you to imagine what I have in my code, but what other issues could I have, that the whole process is not taking 5x faster???
Maybe I'm missing on basic threading knowledge, but I thought this should be the result, maybe the whole process should run in 30seconds...?
So, here are parts of my code, that I think are importunate to show you:
TMyWorkerThread = class(TThread)
procedure Progress(aProgress: integer); virtual;
procedure UpFolder; virtual;
constructor Create(aFolder:string; aProgressProc: TProgressProc; UpdateFolder: TFolderUpdate; CreateSuspended: Boolean = False); reintroduce; virtual;
procedure Execute; override;
procedure GenerateSectionData_T(vData: TData; var vHtml: TextFile);
procedure GenerateNavigationBar_T(var vHtml: TextFile; vPrg: integer);
procedure GenerateInfo_T(vPrg, vTaks: integer; var vHtml: TextFile);
procedure GenerateStructure_T(vPrg: integer; var html_programs: textfile);
procedure GenerateProperties_T(vPrg, vTask, vIndex: integer; var vHtml: TextFile);
procedure GenerateFlow_new_T(vPrg, vTask: integer; var vHtml: TextFile; vForSinglePageReport: boolean = false);
procedure GenerateLocateInfo_T(vPrg, vTask, vTaskIndex: integer; var vHtml: TextFile);
procedure GenerateSQLInfo_T(vPrg, vTask, vTaskIndex: integer; var vHtml: TextFile);
thread creating from unit1:
if MyWorkerThread<>nil then
raise Exception.Create('One thread have already been started!')
MyWorkerThread := TMyWorkerThread.Create(DestFolder, UpdateProgressbar,UpdateFolderInfo1);
MyWorkerThread.OnTerminate := ThreadDone;
In threads, I run GenerateHTMLDoc and it reads data, creates structure and produces html files by writint into html and calling various procedure passing the TextFile handle.
This is how html files are used:
(the old style)
assignfile(html_file, DestFolder + 'object_'+IntToStr(ObjectID)+'.html');
Part of the html is also creating images that are part of main object, but I disabled it for testing purposes as I read, that TCanvas is not thera-safe... so, for now I want to speed up the process that does the writing and then I will address the TBitmap, TCanvas usage in threads.
Thank you for any suggestion what to look for!