sinister
asked on
Speeding up a multi-threaded program
Greetings,
I have a question about trying to speed up a program that is already multi-threaded. I have a program that may have up to 120 threads continuosly waiting on separate sockets for data from another process. When data is received some work is done and a reply message is sent back through the socket. Then the thread goes back to waiting on the socket. When examing the trace file for this operation I noticed that when a particular thread has completed its tasks and goes back into waiting on the socket, there is about 100 milliseconds before the context is switched to another thread.
I know that this amount of time doesn't seem much but I was wondering anything could be done to speed up the context switching between threads. Is there a way of telling Win NT that the thread is about to wait become idle and that it should now switch to another thread?
By the way, I remember a function called Yield that kind of did this?
Thanks
I have a question about trying to speed up a program that is already multi-threaded. I have a program that may have up to 120 threads continuosly waiting on separate sockets for data from another process. When data is received some work is done and a reply message is sent back through the socket. Then the thread goes back to waiting on the socket. When examing the trace file for this operation I noticed that when a particular thread has completed its tasks and goes back into waiting on the socket, there is about 100 milliseconds before the context is switched to another thread.
I know that this amount of time doesn't seem much but I was wondering anything could be done to speed up the context switching between threads. Is there a way of telling Win NT that the thread is about to wait become idle and that it should now switch to another thread?
By the way, I remember a function called Yield that kind of did this?
Thanks
snifong
Do a Sleep(0). It will "force" a context switch.
>> Do a Sleep(0).
Then the thread can't wait for the socket.
>> have a program that may have up to
>> 120 threads continuosly
You may wish to reconsider that design. Threads are very costly. Especially on non-NT windows. Even a suspended thread consumes resources. I've seen rules that recomend that you have no more than 10 at a single time threads in a single process.
>> Is there a way of telling Win NT that the thread is
>> about to wait become idle
Well how are you waiting at the moment? Are you doing overlapeedd I/O and oing a WaitForSingleObject() on an event handle from an OVERLAPPED structure?
Then the thread can't wait for the socket.
>> have a program that may have up to
>> 120 threads continuosly
You may wish to reconsider that design. Threads are very costly. Especially on non-NT windows. Even a suspended thread consumes resources. I've seen rules that recomend that you have no more than 10 at a single time threads in a single process.
>> Is there a way of telling Win NT that the thread is
>> about to wait become idle
Well how are you waiting at the moment? Are you doing overlapeedd I/O and oing a WaitForSingleObject() on an event handle from an OVERLAPPED structure?
Thanks, nietod.
ASKER
Nietod:
There is no overlapped io. The reason that we have so many threads is that this program is a service for a telephony system, so we want to have a thread dedicated to each telephone line. I have a feeling that a thread pool would be even slower. If I did a sleep(0) then my TCP Recv socket function would not get executed until the thread scheduler back to that thread. Is this correct? To be just off topic a little - What are fibers? I have never of them before but I know they are like threads? By the way, whoever wants the points just post another answer and you can have them. Thanks.
There is no overlapped io. The reason that we have so many threads is that this program is a service for a telephony system, so we want to have a thread dedicated to each telephone line. I have a feeling that a thread pool would be even slower. If I did a sleep(0) then my TCP Recv socket function would not get executed until the thread scheduler back to that thread. Is this correct? To be just off topic a little - What are fibers? I have never of them before but I know they are like threads? By the way, whoever wants the points just post another answer and you can have them. Thanks.
I sever line in use at a time? If not just create threads for the lines in use.
>> sleep(0) then my TCP Recv socket function
>> would not get executed until the thread
>> scheduler back to that thread. Is this correct?
correct.
>> What are fibers?
Sort of like threads within a thread. A thread can have multiple fibers running within it. When the thread is running, one of its fibers is actually running. This current fiber can be switched to another fiber. However unlike thread scheduling, which is preemptive, the thread and fibers themselves schedule the fibers cooperatively.
The real question is what is the thread doing? is it waiting for I/O? If so it should be using overlapped I/O and waiting efficiently. If not, what is it doing?
>> sleep(0) then my TCP Recv socket function
>> would not get executed until the thread
>> scheduler back to that thread. Is this correct?
correct.
>> What are fibers?
Sort of like threads within a thread. A thread can have multiple fibers running within it. When the thread is running, one of its fibers is actually running. This current fiber can be switched to another fiber. However unlike thread scheduling, which is preemptive, the thread and fibers themselves schedule the fibers cooperatively.
The real question is what is the thread doing? is it waiting for I/O? If so it should be using overlapped I/O and waiting efficiently. If not, what is it doing?
I'll just agree with the comments posted earlier, threads are not a good match to this problem. There is just too much overhead switching between all these threads that are doing little tiny bits of work. So you end up spending more time switching thread contexts than you do working.
The asynchronous WINSOCK calls are a much better fit for this and you can easily run 100s or even 1000s of socket connections using async sockets. Windows handles the messages generated by the sockets very well (Windows was designed to process messages not threads).
The asynchronous WINSOCK calls are a much better fit for this and you can easily run 100s or even 1000s of socket connections using async sockets. Windows handles the messages generated by the sockets very well (Windows was designed to process messages not threads).
Scheduling between lot of thread can be a major overhead as in windows NT schduling policy is system wide.
Have you tried using select().
In one thread you can wait of all open socket connections and whenever any new socket is ready for connection ,add it to list after doing accept().This way all active sockets can be serviced by a single thread.Whenever you need to do some long operation , process it in a child thread passing it the socket handle .
Using Windows messaging is also a good idea as Windows OS seems to be more optimised for messaging based architectures.
Have you tried using select().
In one thread you can wait of all open socket connections and whenever any new socket is ready for connection ,add it to list after doing accept().This way all active sockets can be serviced by a single thread.Whenever you need to do some long operation , process it in a child thread passing it the socket handle .
Using Windows messaging is also a good idea as Windows OS seems to be more optimised for messaging based architectures.
>> so we want to have a thread dedicated to each telephone line.
>> I have a feeling that a thread pool would be even slower
This is wrong. Managing many threads is expensive, a thread pool is better.
Another option (mentioned previously by nietod) is overlapped IO. Very efficient on an NT platform.
I tend to disagree with the suggestion to use Async IO. IMHO, message dispatching will increase the overhead.
One more issue - you can change the length of the time quantum on NT. This however can have an undesirable effect on the whole system.
Lastly, beware of "premature optimization". If your program is IO bound (network traffic), optimizing the CPU overhead will not help. A profiler is your friend (although, in nietod's case, it can be a very unreliable friend...)
>> I have a feeling that a thread pool would be even slower
This is wrong. Managing many threads is expensive, a thread pool is better.
Another option (mentioned previously by nietod) is overlapped IO. Very efficient on an NT platform.
I tend to disagree with the suggestion to use Async IO. IMHO, message dispatching will increase the overhead.
One more issue - you can change the length of the time quantum on NT. This however can have an undesirable effect on the whole system.
Lastly, beware of "premature optimization". If your program is IO bound (network traffic), optimizing the CPU overhead will not help. A profiler is your friend (although, in nietod's case, it can be a very unreliable friend...)
ASKER
Well, thanks everyone for your comments, I think that I will consider these options if our system ever gets redesigned. Who wants the points?
ASKER
Just too clarify things, I was looking for a quick solution but unfortunately all the suggestions would require a complete redesign. However, these are some good optimization subjects, can anybody refer a book or something about these topics so that can learn more. I find that most books never get into deep detail about multi-threaded programming etc.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Notice that you should not do a sleep(0) before asking for input on the socket, but why don't you check to see if there is data on the socket and if so, then get it, otherwise do a sleep(0)
In perl I use the vec and select statements to see if there is data available. If not then do not bother..
andy
In perl I use the vec and select statements to see if there is data available. If not then do not bother..
andy
>> but why don't you check
>> to see if there is data on the socket
>> and if so, then get it, otherwise do a sleep(0)
No, that is a bussy wait. You will coninually waste CPU time just looking to see if there is data and then going back to sleep. The code that that tests to see if there is data to read probably doesn't consume a lot of time (althought it is wasted time) but the code that switches to your thread and then almost immediately away from your thread consumes a lot of wasted time. Beter to use a technique that puts your thread to sleep so that it uses absolutely no CPU time until there is something for it do do.
>> to see if there is data on the socket
>> and if so, then get it, otherwise do a sleep(0)
No, that is a bussy wait. You will coninually waste CPU time just looking to see if there is data and then going back to sleep. The code that that tests to see if there is data to read probably doesn't consume a lot of time (althought it is wasted time) but the code that switches to your thread and then almost immediately away from your thread consumes a lot of wasted time. Beter to use a technique that puts your thread to sleep so that it uses absolutely no CPU time until there is something for it do do.
ASKER
Thanks