Loops and Threads

I have an app which gets data from a network and waits for Ondata events. Each Ondata event passes a buffer of data which I then process. Problem is when the data arrives very fast the Ondata event is retriggered as soon as the proc for the last one finishes - so the app "locks up" because no other events are processed.
So, I thought, lets put the processing in a thread so it can process as fast as it can, but still give control back to the user window. That works EXCEPT that the thread consumes windows system & user resource as it goes - presumably because the threads message queue is filling?? If I suspend the thread for a few seconds and then resume it all the windows resources are restored but then start being eaten again. If I "throttle back" the machine which is delivering the data then there is no problem at all.
So my question is: How can I process the message queue (main window or thread) without processing the Ondata event until all the others are deailt with. So the flow would be:
1. Start data get
2. Wait for Ondata event and process data
3. Check for other messages and process them
4. Wait for next Ondata event & process data
5. Back to 3.
I have given this a "hard" rating - it's driving me nuts! Hopefully it will be "easy" for someone out there.

Thanks

John
LVL 1
jellisonAsked:
Who is Participating?

[Webinar] Streamline your web hosting managementRegister Today

x
 
rarigoConnect With a Mentor Commented:
Could you not implement something like this:

1 - Tell process to get data
2 - Wait for Ondata event
3 - Tell process to stop getting data
4 - Process data
5 - Back to 1?

Tchau
Reginaldo
0
 
interCommented:
Hi,
By system resource do you mean CPU time?
igor
0
 
dotanCommented:
Hi Jellison
What you need to do in order to avoid consumming windows system & user resource is:
1. Set the thread prioraty to tpLowest or tpLower.
2. Make sure the thread FreeOnTerminate property is set to true and that you exit the execute procedure after youre data was proccessed.
0
Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

 
intheCommented:
also you can call application.processmessages when your waiting for data..this allows windows some time to  process the other programs messages  etc
0
 
MadshiCommented:
Look at this function:

procedure ProcessMessages;
var msg : TMsg;
begin
  while PeekMessage(msg, 0, 0, 0, PM_REMOVE) do begin
    TranslateMessage(msg);
    DispatchMessage(msg);
  end;
end;

It handles all messages that are in the message queue in the moment. Look at the documentation about PeekMessage. You can give in a message range. So you could give in a range, that covers all important messages, but leaves out your net messages.

You could also try this:

procedure DummyPeekMessage;
var msg : TMsg;
begin
  PeekMessage(msg,0,WM_ENDSESSION,WM_ENDSESSION,PM_NOREMOVE);
end;

I noticed, when I call OpenPrinter/GetPrinter/SetPrinter very often without any pause, then the system resources go down (like you decribed). Then I added this DummyPeekMessage (which has no real effect at all). Now I've no problems with the resources anymore. Try it. Perhaps it solves your problems, too.

If both suggestions do not help: Please show us some code...

Regards, Madshi.
0
 
jellisonAuthor Commented:
Wow. Fast response! Thanks all for responding.

Igor:
By system resource I mean the resorces shown by the RSRCMTR program, System, User, and GDI. With my app running you can see System & User shrinking away.

Dotan:
I tried changing priority. It just eats resource a little more slowly because (I think) the thread is still doing exactly the same but gets less timescices to do it. I already do the other things you suggested.

Inthe:
Tried that. Because of the loop I have to do the Processmessages within the Ondata event proc. All it does (I think) is make the next Ondata just a bit quicker because it triggers within the previous one.

Madshi:
Sounds interesting. I did see Peekmsg but I don't know what my Ondata message looks like so I asked the question first. I will certainly try the dummy message & then experiment with the Translate/Dispatch if that doesn't work. Thanks.

I will post another update later.

John

0
 
MadshiCommented:
The message will be what you (or the component you're using) give in to WSAAsyncSelect. So probably a PeekMessage filter from 0 to WM_USER - 1 should fulfill your needs, I guess...

Regards, Madshi.
0
 
williams2Commented:
I'm sorry to say to you all guys:

You are very wrong about severel aspects in this thread:
1. To set the thread priority at lowest to reduce CPU overhed: No that has not anything to do with it. This will only work at timecritical levels. When a window is doing nothing, the Process priority will typically consume nothing as saying "You, there mr thread, go eat it all, I'm not busy!", so it might seem to work by setting the thread priority at a very low ratio, but the CPU Usage still seems to run at 100% (In the taskmanager window on WinNT systems) and the thread seems to stop if other thread loops are running (ie. a console process like Lotus Notes or similar, or just a simple SETI@Home client can make it hang)
2. By calling processmessages, it's like saying: "Hey, I know I'm eating all the time around, ok here you got some, but it's ME who decide when to give you some!" ..the system overhead STILL seems to eat up 100 % of the CPU Usage...

So WHY is it that I can claim, that you get this so wrong? Hmm... Borland and Microsoft LOVES to hide everything important around, so what are they scared of ?
I have been doing a lot of research into this topic lately, and found some small but interesting aspects in multitier programming at Borland and Microsoft's support sites.

So I claimed that the Delphi threads are useless. but...
You probably do not use the following very often, if you use it at all, but these methods are necessary to make multitier applications which doesn't make the system hang - like they sometimes do.

VOID Sleep(
    DWORD  cMilliseconds       // sleep time in milliseconds
);

Will make the thread sleeping and release all CPU resources in the meanwhile.

DWORD SleepEx(
    DWORD  dwTimeout,      // time-out interval in milliseconds  
    BOOL  fAlertable       // early completion flag
);

These commands are both very handy in a runing loop, as calling them periodically see to, that the thread is not consuming too many CPU resources.

The other very handy methods are:

WaitForSingleObject / WaitForMuyltipleObject

and similar:

WaitForSingleObjectEx / WaitForMuyltipleObjectEx

Which is very handy when it comes to waiting for a specific event to happen.

By waiting for an event to happen, you can allways make a
hEvent:= ChreateEvent(nil, False, False, nil);

...The event:

Procedure TTest.OnSomethingArrive(Sender: TObject);
Begin
  SetEvent(hEvent);
End;

...the loop

Procedure TTest.Execute;
Var
  Res: DWORD;
Begin
  While not Terminated do
  Begin
    Res:= WaitForSingleObjectEx(hEvent, 1000, True);
    //The thread wil return every second at all costs. Event if the thread is terminated it will return from it's waiting state.
    if Res = WAIT_OBJECT_0 = then
    Begin
      ResetEvent(hEvent);
      Synchronize(DoSomething);
    End;
  End;
End;

...the handler
Procedure TTest.DoSomething;
begin
  //..Do something..
End;


....BUT anyway, if you are not up to all this, and find it very hard to make this work at any costs, you can download my free components at:
http://www.torry.webnoth.com/vcl/system/twthreads.zip

...Or you might request the newest update from me, As my website is not up yet.

Happy processing.. F-)

Regards,
Williams
0
 
williams2Commented:
NO BODY'S PERFECT! :-)
Some error corrections:

WaitForMuyltipleObject, WaitForMuyltipleObjectEx

Should be..

WaitForMultipleObject ,WaitForMultipleObjectEx

In the code:

  if Res = WAIT_OBJECT_0 = then

Should be..

  if Res = WAIT_OBJECT_0 then

The URL:
  http://www.torry.webnorth.com/vcl/system/twthreads.zip

Regards,
Williams
0
 
MadshiCommented:
Williams, you're right in terms of CPU usage, but John was talking about user/gdi/system resources (that is the available percentage amount of handles), not about CPU usage. Read this extract from his original question:

"... the thread consumes windows system & user resource as it goes..."

That's a quite different story!!   :-)

Regards, Madshi.
0
 
jellisonAuthor Commented:
An update:

I have tried the various suggestions - none of them stopped anything working but they didn't make any difference either. There didn't seem to be any messages to process!

My latest theory. The resource is being gobbled by a stack. The Ondata event puts a return address on the stack, does its processing, gets to the end and goes straight back in again without "popping" the stack.

What do you think?  :)

PS I agree re the CPU issue. It's not that my PC slows down - it doesn't (too much!). It's the consuption of the Windows System & User resources which is the problem.

John
0
 
MadshiCommented:
Could you please show us some code?

The user/gdi/system resources simply show how many handles are free and how many are consumed. In win9x handles are a restricted resource (winNT doesn't have this restriction). There is no stack for these kind of resources (at least I don't know of a stack)...

Regards, Madshi.
0
 
williams2Commented:
Yes, Madshi

You are right about the resoruce-thing, but I forgot to mention, that when the thread is running at full speed, it sometimes "forget" to read the messagequeue. When in SleeEx mode, the messagequeue is able be handled. I have a wellfunctioning socket server using the above mentioned technology, and it has now been running without problems for the last two weeks (since last update).

But a problem I have had is: When trying to allocate more than 5 megs of memory for a uploaded file in a memorystream, the system gave me the mentioned error (Couldn't allocate resources...) forcing me to save the stream to disk instead - and that has been working since then.

So John, that might be the real problem !? ..but as Madshi says, it would help a lot, if you could show us the code. If that is not easy for you, then you can send it to:
williams90@hotmail.com

Regards,
Williams
0
 
aacrgCommented:
I agree with williams2
0
 
jellisonAuthor Commented:
Sorry about the delay - I have been experimenting with lots of different routines to see if I can find out what is happening.

I still think this is happening because the program is in a very tight loop when receiving data. I don't think it's worth sending all the code but the core is that have an Ondata procedure which just copies data from an input buffer, processes the data (it's actually a stream of jpeg images) and finishes. The problem (I think) is that as soon as the procedure finishes it gets another Ondata event and does it all again. This leads to sluggish or sometimes non-existant input response and a gradual reduction in Windows System and User resource.

If I slow down the delivery of the data at its source all the problems go away. But the whole idea of the app is to receive & display the data as fast as possible.

All this seems to happen whether the process is threaded or not.

I would still be grateful for any ideas or comments.

John Ellison
0
 
jellisonAuthor Commented:
I decide to give the points to rarigo as his suggestion was the closest to my final working solution. There are more comments in Q.10241656 in the Programming Windows section if anyone wants to see more info.

Thanks to all who contributed
0
All Courses

From novice to tech pro — start learning today.