pthread_join on Linux crashes on segmentation fault

use the following command:
               ulimit -c unlimited
The core file will be created.
let us know the output of the following command:
               gdb binaryName coreFileName

gdb binaryFileName coreFileName
gdb) where

Answer to Genius:
typedef pthread_t OT_THREAD_HANDLE;

How about (b) and (c) ?

Generating the core file like murugesandins suggested will allow you to see the exact location of the segmentation fault, so that's a good idea too.

Core dump file is not created. The where command output is as follow:

Program received signal SIGSEGV, Segmentation fault.
0x007c777b in __deallocate_stack () from /lib/tls/libpthread.so.0
(gdb) where
#0  0x007c777b in __deallocate_stack () from /lib/tls/libpthread.so.0
#1  0x007c7d2b in __free_tcb () from /lib/tls/libpthread.so.0
#2  0x007c8d9f in pthread_join () from /lib/tls/libpthread.so.0
#3  0x0804b73a in OT_WaitForThreads (nCount=4, pHandles=0x81efef8, bWaitAll=1, dwMilliseconds=0) at OTThread.c:80
#4  0x0804a89e in EngineCleanUp () at Engine.c:106
#5  0x0804a962 in main () at Engine.c:138
(gdb)

That seems to indicate that your heap got corrupted. That could have happened anywhere and anytime. Check whether you can spot any obvious buffer overflows, or other write operations to unallocated (or simply wrong) memory. If you can't spot it immediately, I suggest using a memory debugger to find the problem for you.

That is, assuming you've already eliminated (b) and (c) I've mentioned earlier ... (you still haven't replied to that)

Answer for b:

There is no return value. The application crashes within the function.

Answer for c:

I derify that ThreadArray holds the number of the threads expected. the code you asked for is attached.

BTW - Since the general code in the application is crosspatform and since in Windows i seo problem with this application i'm pretty sure this is pure Linux / Unix pthread_join related issue.

>> There is no return value. The application crashes within the function.

Of course there is a return value ... It returns int.
The crash you mentioned was not in pthread_create, but in pthread_join. pthread_create happens before pthread_join (or at least it should), and my remark was about pthread_create's return value not being checked. As long as you don't do so, you can't know whether the creation of the thread failed or not (which could cause problems later, including the crash you experience).


>> I derify that ThreadArray holds the number of the threads expected.

And are all entries in the array valid ?


>> the code you asked for is attached.

You must have forgotten it, as I can't see it ;)


>> i'm pretty sure this is pure Linux / Unix pthread_join related issue.

I'm not ... There's a LOT of differences between a Windows and a Linux platform, so just because the problem occurs on one platform and not on the other, does not mean that you can immediately point to one function as the problem.

Hre is the code:
for(i = 0 ; i < MAX_THREAD ; i++)
      {
            if(ThreadArray[i] == NULL)
            {
                  waitForAllThreads = 0;
                  break;
            }
      }

      if(      0 != waitForAllThreads)
      {
            // DON'T wait for the listen thread it is block by the read command
            // Make sure Listen thread is always the last one on the handle array
            OT_WaitForThreads(MAX_THREAD-1, ThreadArray, 1 /*TRUE*/, INFINITE);
      }      

You're comparing a pthread_t with NULL here ... :

>>             if(ThreadArray[i] == NULL)

pthread_t is not a pointer.

The code you posted also doesn't show how ThreadArray is constructed and filled.

LoggerAddMessage( PROXY_LOG_LEVEL_ANALYSIS , FLUSH_TO_DISK,"EngineInit - Start ...\n" ) ;
      QueueInit(&messageQ);
      QueueInit(&writeQ);
      QueueInit(&transmitQ);
      
      strcpy(messageQ.qName , "MessageQ");
      messageQ.hMutex = OT_CreateMutex(messageQ.qName);

      strcpy(writeQ.qName , "WriteQ");
      writeQ.hMutex = OT_CreateMutex(writeQ.qName);

      strcpy(transmitQ.qName , "TransmitQ");
      transmitQ.hMutex = OT_CreateMutex(transmitQ.qName);

      ThreadArray[WORKER_THREAD] = OT_CreateThread( MessageQWorkerThread,            // thread function
                                                        &messageQ,                                          // argument to thread function                                     
                                                        &ThreadArray[WORKER_THREAD]);                  // The pointer that will store the thread handle - in use only in non windows systems

      ThreadArray[WRITER_THREAD] = OT_CreateThread(      MessageQWriterThread,      // thread function
                                                            &writeQ,                                          // argument to thread function
                                                            &ThreadArray[WRITER_THREAD]);            // The pointer that will store the thread handle - in use only in non windows systems

      ThreadArray[TRANSMIT_THREAD] = OT_CreateThread(      MessageQTransmitThread,      // thread function
                                                            &transmitQ,                                          // argument to thread function
                                                            &ThreadArray[TRANSMIT_THREAD]);            // The pointer that will store the thread handle - in use only in non windows systems

      ThreadArray[CMT_THREAD] = OT_CreateThread(      CMTThread,                              // thread function
                                                                        &messageQ,                                          // argument to thread function
                                                                        &ThreadArray[TRANSMIT_THREAD]);            // The pointer that will store the thread handle - in use only in non windows systems

      
      // Make sure Listen thread is always the last one on the handle array - refer to the comment in the EngineCleanup
      ThreadArray[LISTEN_THREAD] = OT_CreateThread(      ListeningThread,            // thread function
                                                                              &messageQ,                                          // argument to thread function
                                                                              &ThreadArray[LISTEN_THREAD]);            // The pointer that will store the thread handle - in use only in non windows systems

Just in case some one is interested in a formatted code snippet ...

   LoggerAddMessage( PROXY_LOG_LEVEL_ANALYSIS , FLUSH_TO_DISK,"EngineInit - Start ...\n" ) ;
   QueueInit(&messageQ);
 
   QueueInit(&writeQ);
   QueueInit(&transmitQ);
 
   strcpy(messageQ.qName , "MessageQ");
   messageQ.hMutex = OT_CreateMutex(messageQ.qName);
 
   strcpy(writeQ.qName , "WriteQ");
   writeQ.hMutex = OT_CreateMutex(writeQ.qName);
 
   strcpy(transmitQ.qName , "TransmitQ");
   transmitQ.hMutex = OT_CreateMutex(transmitQ.qName);
 
   ThreadArray[WORKER_THREAD] = 
      OT_CreateThread( MessageQWorkerThread,          
                      &messageQ,                      
                      &ThreadArray[WORKER_THREAD]);    
 
   ThreadArray[WRITER_THREAD] = 
      OT_CreateThread(MessageQWriterThread,           
                      &writeQ,                        
                      &ThreadArray[WRITER_THREAD]);    
 
   ThreadArray[TRANSMIT_THREAD] = 
      OT_CreateThread(MessageQTransmitThread,         
                      &transmitQ,                     
                      &ThreadArray[TRANSMIT_THREAD]);  
 
   ThreadArray[CMT_THREAD] = 
      OT_CreateThread(CMTThread,                      
                      &messageQ,                      
                      &ThreadArray[TRANSMIT_THREAD]);  
 
 
   ThreadArray[LISTEN_THREAD] = 
      OT_CreateThread(ListeningThread,                
                      &messageQ,                      
                      &ThreadArray[LISTEN_THREAD]);    
                                              

1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:

>>>> I can see a typo here though :

Good catch, Infinity  ;-)

Should be possible that

         ThreadArray[TRANSMIT_THREAD]);  

has the wrong handles and

         ThreadArray[CMT_THREAD]

is uninitialized.

Could be enough to make the system crash.

Good catch with the typo. I'm sure it solved some future problem, but It didn't help this issue.

I've added a line "res = pthread_create(...)" to see the result. They all get 0.
I can also see that ThreadArray has 5 consequtive values

I'll start working with a memory debugger then

// Number of threads in the application 
#define MAX_THREAD			5
// Assign ID's to the applicaion threads
#define WORKER_THREAD		0
#define WRITER_THREAD		1
#define TRANSMIT_THREAD		2
#define CMT_THREAD			3
#define	LISTEN_THREAD		4

                                              

1:
2:
3:
4:
5:
6:
7:
8:

>> I'll start working with a memory debugger then

Likely candidates to look for are buffer overflows.

I've run valgrind memory check, and it helped me see that the CreatedThreadHandle thread is actually lost (and used redundantly).
I fixed it and it stop falling on segmentation.