We have a COM component that internally uses several threads that interact with an external device. The component host (the front end app) creates a window to which the component threads send custom Windows messages (using SendMessage). The messages contain pointers to structures allocated inside the component.
When the host app starts to terminate, the WM_CLOSE message handler calls the component close function. This function sets some internal flags (to signal the internal threads to start terminating) and then waits for the threads to terminate by waiting on the thread handles. However, one or more of the threads may be blocked waiting in a SendMessage call - a message it has already placed on the app message queue, but which will never get serviced since the message queue thread is processing WM_CLOSE and now waiting for the threads to terminate. Classic deadlock problem.
I tried using SendMessageTimeout on the internal threads, and although this solves the shutdown problem it means that if the host brings up a modal dialog (i.e. stops processing the message queue during normal runtime operation) then messages from the component threads are lost.
Using PostMessage would require an allocated list for my data which not ideal.
How can I ensure clean thread shutdown in the component without ever losing any messages during runtime?