We help IT Professionals succeed at work.

Delphi Divide a TStringList Contents Evenly into an array

nickdelphi777
on
Hey Guys,

I am attempting to divide a tstringlist of accounts evenly among 10 threads.

I am inserting the "batches" of accounts into different arrays.

So if i had 30 accounts, i would be dividing the list up into batches of 3 accounts and inserting them into their appropriate account array.. that way each thread can process 3 accounts and speed up things

problem I am running into.. is when the division is not even.. I've tried 3 different variations of code but when i change up the amount of accounts i get all funky results. I am hoping one of you have a very clean and simple way to do this without any errors.. regardless of how many accounts i use.

Remember i am only using 10 threads.. so essentially the list has to be split up between 10 threads evenly.
Comment
Watch Question

Author

Commented:
Please keep in mind.. i want it to always use 10 threads unless theres less than 10 accounts of course.

So if I have 15 accounts.. i do not want it to divide up by 2. and only use 7 threads.. I would like it to put say 1 account in most.. then 2 here.. and 2 there.

please help i have been stuck with this for hours
ste5anSenior Developer

Commented:
Just round-robin them.

Author

Commented:
Thats not really what I'm after...

Author

Commented:
Nevermind.. it popped in my head.. just loop through entire list and keep adding to the threads.. 1 to 10, 1 to 10:

imageamount := memo1.lines.count;
threads :=10;
icount := 0;
for i := 0 to imageamount-1 do begin
  memo2.lines.add('Insert '+memo1.lines[i]+ ' into '+inttostr(icount));
 if(icount = threads-1) then begin
  icount := 0;
 end else begin
  inc(icount);
 end;
end;

Open in new window

Top Expert 2011

Commented:
Compact code:
  imageamount := memo1.lines.count;
  threads :=10;
  for i := 0 to imageamount-1 do begin
    memo2.lines.add('Insert '+memo1.lines[i]+ ' into '+inttostr(i mod threads));
  end;

Open in new window

Top Expert 2011
Commented:
In case you want to know beforehand how many to cut from accounts, in one step to assign to the processing threads in same order, then:
function ImagesToProcess(thrdCount, thrdindx, imgs:Integer):integer;
begin
  Result:= 0;

  if thrdindx <= ((imgs) mod thrdCount) then
    begin
     Result:= ((imgs-1) div thrdCount)+1;
    end
  else
    begin
      Result:= ((imgs-1) div thrdCount);
    end;
end;

procedure TForm1.Button1Click(Sender: TObject);
var
  thrd, threads, imageamount:integer;
begin
  imageamount:= memo1.lines.count;
  threads:= 10;

  for thrd:= 1 to threads do
    begin
      memo2.lines.add('thread '+ inttostr(thrd)+' will get '+ IntTostr(ImagesToProcess(threads, thrd, imageamount)));
    end;
end;

Open in new window


If you want to see that working as you tested in your last comment then:
procedure TForm1.Button2Click(Sender: TObject);
var
  k, thrd, threads, img, imageamount:integer;
  str:String;
begin
  Memo2.Clear;
  imageamount := memo1.lines.count;
  threads :=10;
  img:= 0;

  for thrd:= 1 to threads do
    begin
      str:= '';

      if thrd <= ((imageamount) mod threads) then begin
        for k:= 0 to ((imageamount-1) div threads) do
          begin
            str:= str+ memo1.lines[img];
            inc(img);
          end;
      end
      else begin
        for k:= 0 to ((imageamount-1) div threads)-1 do
          begin
            str:= str+ memo1.lines[img];
            inc(img);
          end;
      end;

      memo2.lines.add('thread '+ inttostr(thrd)+' will have '+ Str);
    end;
end;

Open in new window

Geert GOracle dba
Top Expert 2009

Commented:
you don't even need that, just use Otl

use a pipeline from omnithreadlibrary ...
http://otl.17slon.com/index.htm

you only need 1 stage, this pipeline can buffer up to 10000 elements
optionally add an message proc
Numtasks indicates 10 threads

 
 fPipeLine :=
    Parallel.PipeLine
    .Throttle(10000)
    .Stage(
      TaskProcessItem, Parallel.TaskConfig.OnMessage(MsgProcessItem)
      ).NumTasks(10)
  .Run;

Open in new window


Loading the queue:
for I := 0 to Strings.Count-1
  fPipeLine.AddInput(Strings[I]);

Open in new window


the thread processing code
procedure TaskProcessItem(const input, output: IOmniBlockingCollection; const task: IOmniTask);
  Value: TOmniValue;
begin
  while not Task.Terminated do
  begin
    if input.TryTake(Value, 1000) then
    begin
        // Thread processing code: 
        // string for this is : Value.AsString

        // optionally send a message
        task.Comm.Send(0, 'Processed item : ' + Value.AsString);
    end;
  end;
end;

Open in new window


procedure MsgProcessItem(const task: IOmniTaskControl; const msg: TOmniMessage);
begin
   fNotifyEvent(Self, msg.MsgData.AsString);
end;

Open in new window

Geert GOracle dba
Top Expert 2009

Commented:
the 10 threads will take the next item when they have processed the previous
if no items are in the queue, they wait for 1 second and then retry

the next thing about this pipeline technique is that you don't need to divide anything yourself

the numtasks (15) would setup 15 threads.
numtasks(2) would only setup 2 threads
ste5anSenior Developer

Commented:
@nickdelphi777: Your code IS the round-robin algorithm.