asked on

Delphi Divide a TStringList Contents Evenly into an array

Hey Guys,

I am attempting to divide a tstringlist of accounts evenly among 10 threads.

I am inserting the "batches" of accounts into different arrays.

So if i had 30 accounts, i would be dividing the list up into batches of 3 accounts and inserting them into their appropriate account array.. that way each thread can process 3 accounts and speed up things

problem I am running into.. is when the division is not even.. I've tried 3 different variations of code but when i change up the amount of accounts i get all funky results. I am hoping one of you have a very clean and simple way to do this without any errors.. regardless of how many accounts i use.

Remember i am only using 10 threads.. so essentially the list has to be split up between 10 threads evenly.

nickdelphi777

ASKER

Please keep in mind.. i want it to always use 10 threads unless theres less than 10 accounts of course.

So if I have 15 accounts.. i do not want it to divide up by 2. and only use 7 threads.. I would like it to put say 1 account in most.. then 2 here.. and 2 there.

please help i have been stuck with this for hours

ste5an

Just round-robin them.

nickdelphi777

ASKER

Thats not really what I'm after...

nickdelphi777

ASKER

Nevermind.. it popped in my head.. just loop through entire list and keep adding to the threads.. 1 to 10, 1 to 10:

imageamount := memo1.lines.count;
threads :=10;
icount := 0;
for i := 0 to imageamount-1 do begin
  memo2.lines.add('Insert '+memo1.lines[i]+ ' into '+inttostr(icount));
 if(icount = threads-1) then begin
  icount := 0;
 end else begin
  inc(icount);
 end;
end;

Open in new window

jimyX

Compact code:

  imageamount := memo1.lines.count;
  threads :=10;
  for i := 0 to imageamount-1 do begin
    memo2.lines.add('Insert '+memo1.lines[i]+ ' into '+inttostr(i mod threads));
  end;

Open in new window

ASKER CERTIFIED SOLUTION

jimyX

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

Geert G

you don't even need that, just use Otl

use a pipeline from omnithreadlibrary ...
http://otl.17slon.com/index.htm

you only need 1 stage, this pipeline can buffer up to 10000 elements
optionally add an message proc
Numtasks indicates 10 threads

 fPipeLine :=
    Parallel.PipeLine
    .Throttle(10000)
    .Stage(
      TaskProcessItem, Parallel.TaskConfig.OnMessage(MsgProcessItem)
      ).NumTasks(10)
  .Run;

Open in new window

Loading the queue:

for I := 0 to Strings.Count-1
  fPipeLine.AddInput(Strings[I]);

Open in new window

the thread processing code

procedure TaskProcessItem(const input, output: IOmniBlockingCollection; const task: IOmniTask);
  Value: TOmniValue;
begin
  while not Task.Terminated do
  begin
    if input.TryTake(Value, 1000) then
    begin
        // Thread processing code: 
        // string for this is : Value.AsString

        // optionally send a message
        task.Comm.Send(0, 'Processed item : ' + Value.AsString);
    end;
  end;
end;

Open in new window

procedure MsgProcessItem(const task: IOmniTaskControl; const msg: TOmniMessage);
begin
   fNotifyEvent(Self, msg.MsgData.AsString);
end;

Open in new window

Geert G

the 10 threads will take the next item when they have processed the previous
if no items are in the queue, they wait for 1 second and then retry

the next thing about this pipeline technique is that you don't need to divide anything yourself

the numtasks (15) would setup 15 threads.
numtasks(2) would only setup 2 threads

ste5an

@nickdelphi777: Your code IS the round-robin algorithm.