• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1131
  • Last Modified:

dump analysis

Hello
I see the following data after analysing dump using procdump windows tool
=====================================================================
Description:
Detected a serious critical section related problem in dsmcell_111028_093922.dmp
Lock at 0x00328fc0 owned by thread 16 is Deadlocked with lock at ntdll!LdrpLoaderLock owned by thread 7
Impact analysis
5.26% of threads blocked
(Threads 7)
The following functions are involved in the root cause
awscomm!AW_SimpleSyncGrabEx+a9
The following modules are involved in the root cause
C:\Program Files\CA\CA_APPSW\awscomm.dll
Recommendation:
The entry-point function for a dynamic link library (DLL) should perform only simple initialization or termination tasks, however this thread (7) is waiting on a critical section. Follow the guidance in the MSDN documentation for DllMain to avoid access violations and deadlocks while loading and unloading libraries.
Please follow up with the vendor for C:\Program Files\CA\CA_APPSW\awscomm.dll

Locked critical section report
Critical Section    0x00328fc0  
Lock State   Deadlocked
Lock Count   1
Recursion Count   1
Entry Count   0
Contention Count   1
Spin Count   0
Owner Thread   16
Owner Thread System ID   6400


Critical Section    ntdll!LdrpLoaderLock  
Lock State   Deadlocked
Lock Count   4
Recursion Count   1
Entry Count   0
Contention Count   9
Spin Count   0
Owner Thread   7
Owner Thread System ID   4512

Thread 7 - System ID 4512
This thread is waiting on critical section 0x00328fc0 owned by thread 16.
Thread 16 in turn is deadlocked with another thread.
Function   Source
ntdll!KiFastSystemCallRet    
ntdll!ZwWaitForSingleObject+c    
ntdll!RtlpWaitOnCriticalSection+1a3    
ntdll!RtlEnterCriticalSection+a8    
awscomm!AW_SimpleSyncGrabEx+a9    
awscomm!AW_SimpleSyncGrab+13    
awscomm!AWInetFreeTlsArray+25e    
awscomm!AWInetFreeTlsArray+1d2    
awscomm!AWInetFreeTlsArray+12    
awscomm!AW_Init+e7    
awscomm!inet_addr+3eb    
ntdll!LdrpCallInitRoutine+14    
ntdll!LdrShutdownThread+c8    
kernel32!ExitThread+2f    
kernel32!BaseExitThreadPoolThread+d    
ntdll!EtwpEventPump+335    
kernel32!BaseThreadStart+34

Thread 15 - System ID 1988
This thread is waiting on critical section ntdll!LdrpLoaderLock owned by thread 7.
Thread 7 in turn is deadlocked with another thread.
Function   Source
ntdll!KiFastSystemCallRet    
ntdll!ZwWaitForSingleObject+c    
ntdll!RtlpWaitOnCriticalSection+1a3    
ntdll!RtlEnterCriticalSection+a8    
ntdll!LdrShutdownThread+33    
kernel32!ExitThread+2f    
msvcrt!_endthreadex+29    
msvcrt!_endthreadex+a8    
kernel32!BaseThreadStart+34    

Thread 16 - System ID 6400
This thread is waiting on critical section ntdll!LdrpLoaderLock owned by thread 7.
Thread 7 in turn is deadlocked with another thread.

Function   Source
ntdll!KiFastSystemCallRet    
ntdll!ZwWaitForSingleObject+c    
ntdll!RtlpWaitOnCriticalSection+1a3    
ntdll!RtlEnterCriticalSection+a8    
ntdll!LdrLockLoaderLock+e4    
ntdll!LdrLoadDll+c9    
kernel32!LoadLibraryExW+1b2    
kernel32!LoadLibraryExA+1f    
kernel32!LoadLibraryA+b5    
iphlpapi!NotifyAddrChange+d1    
iphlpapi!NotifyAddrChange+153    
iphlpapi!GetAdaptersAddresses+455    
iphlpapi!GetIpAddrTableFromStack+dc    
iphlpapi!GetAdaptersAddresses+2e4    
iphlpapi!GetAdaptersAddresses+8f    
iphlpapi!GetAdaptersAddresses+5a    
awscomm!AWInetGetNameKey+1f0    
awscomm!AWInetNameResolve+179d    
awscomm!AWInetNameResolve+45a    
awscomm!AWInetNameResolve+2cd    
awscomm!AWM_Here+20


====================================================
My question:
1) I would like to understand, the meaning of Lock count/Recursion count/Entry count/contention count/spin count
2) What do you infer from the above dump? is this microsft problem or app problem?

Sham
0
mohet01
Asked:
mohet01
  • 13
  • 9
1 Solution
 
tampnicCommented:
Is this a dump from a crash or hang situation, or to investigate a performance issue?

Cheers,
   Chris
0
 
tampnicCommented:
Sometimes you have to be careful about reports that threads are deadlocked. Accidental locks can occur that will free themselves quickly, and you dumped the process during an accidental lock. These can create performance issues. The application needs to schedule its attempts to access resources in a better fashion usually.

For an explanation of the terms used in the dump you can look at the following Microsoft document, http://msdn.microsoft.com/en-us/magazine/cc164040.aspx (see the section titled "Digging In: the RTL_CRITICAL_SECTION Structure")

Which application/package provides awscomm.dll?  

Cheers,
  Chris
0
 
mohet01Author Commented:
this dump is from hang situation
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
mohet01Author Commented:
It is my application that provides awscomm.dll
0
 
tampnicCommented:
Almost 99.999% of times its the application fault.

Its probably better to redesign your DLL with a very simple DLLmain() and put your DLL initialisation code into another function in the DLL, to be called once the DLL has loaded. This puts initialisation responsibility onto the code loading the DLL but can simplify the picture when problems occur.  

Does the discussion in the following link pertain to your situation? I don't know what you have in the DLLmain() of awscomm.dll.
http://stackoverflow.com/questions/724560/initialize-critical-section-only-once-for-a-process

Cheers,
  Chris
0
 
mohet01Author Commented:
Hello chris
this is what we have in ddlmain() of awscomm.dll
BOOL APIENTRY DllMain(HANDLE hModule,
                      DWORD  ul_reason_for_call,
                      LPVOID lpReserved)
{
    switch( ul_reason_for_call )
    {
    case DLL_PROCESS_ATTACH:
        /* AW_ExceptionHandlerRegister(); */
        break;
    case DLL_THREAD_ATTACH:
        break;
    case DLL_THREAD_DETACH:
        AWInetFreeTlsArray();
        UTIL_ThreadGlobalRelease();
        break;
    case DLL_PROCESS_DETACH:
        break;
    }

    return TRUE;
}
Sham
0
 
mohet01Author Commented:
As per my analysis,
 ntdll!LdrpLoaderLock  is being hold by thread 7 and
thread 16, 17, 18 are waitign for this reaource.

I would like to know, how do i avoid thread 7 to not hold this resource.

Sham
0
 
mohet01Author Commented:
void AW_Init(void)
{

      AW_Init_AWSComm(NULL); /* lcl_unicode.c - check and initialize standalone SNMP SDK mode */

    ICF_DefineAttributeIntegerVariable( "Common",
                                        "Counters",      
                                        "HeapBlocks",                      
                                        &G_Malloc,
                                        NULL,
                                        0,
                                        0 );

    ICF_DefineAttributeIntegerVariable( "Common",
                                        "Counters",      
                                        "Heaps",                            
                                        &G_Heaps,
                                        NULL,
                                        0,
                                        0 );

    ICF_DefineAttributeIntegerVariable( "Common",
                                        "Options",        
                                        "GlobalExceptionHandlerEnabled",    
                                        &G_ExceptionHandlerEnabled,
                                        (aw_icfVarIntWrite) AW_ExceptionHandlerSet,
                                        0,
                                        1 );

    ICF_DefineAttributeIntegerVariable( "Common",
                                        "Options",        
                                        "MemMaxTestLen",                    
                                        &_aw_stringMaxLenForDebugTest,
                                        ICF_VariableWriteAlwaysInteger,
                                        0,
                                        (256*1024*1024) );

    ICF_DefineAttributeIntegerVariable( "Common",
                                        "Options",        
                                        "HeapCheckInterval",                
                                        &G_HeapCheck,
                                        ICF_VariableWriteAlwaysInteger,
                                        0,
                                        (256*1024*1024) );
}

0
 
tampnicCommented:
When does the application hang, on a DLL_THREAD_DETACH or DLL_THREAD_ATTACH?

http://blogs.msdn.com/b/harip/archive/2010/11/04/performance-tips-for-dllmains.aspx makes the point that on DLL_THREAD_ATTACH "The current process is creating a new thread. When this occurs, the system calls the entry-point function of all DLLs currently attached to the process." Are other DLL's attached to your main process and are they threads-aware?

I notice you are not allocating a thread local storage index when DLL_THREAD_ATTACH is processed, yet your function AWInetFreeTlsArray() suggests you are freeing some thread-local-storage when the calling thread exits. Is the TLS index set up elsewhere in your code? UTIL_ThreadGlobalRelease() is a function name which suggests releasing some global resource of one kind or another. DLL global variables were set up on DLL_PROCESS_ATTACH and are shared amongst threads of the process. How many threads are attached to the DLL and would releasing any of the globals be a source of synchronisation trouble? I'm guessing what UTIL_ThreadGlobalRelease() might be doing. Without knowing what AWInetFreeTlsArray() and UTIL_ThreadGlobalRelease() are doing its difficult to advise further.

Cheers,
  Chris
0
 
tampnicCommented:
ntdll!LdrpLoaderLock is a system lock used when loading/unloading/attaching/detaching a DLL.

Thread 16 has called AWInetGetNameKey, which eventuall tries loading a DLL further down the call stack. AWInetGetNameKey trying to load a DLL might cause the deadlock. What DLL is AWInetGetNameKey trying to load, and does it have any references to awscomm.dll.

Cheers,
  Chris
0
 
mohet01Author Commented:
"Without knowing what AWInetFreeTlsArray() and UTIL_ThreadGlobalRelease() are doing its difficult to advise further."
Please find the attache files
 Source1.c
0
 
mohet01Author Commented:
Hello Chris
I think the problem is with thread 7.
Why thread 7 is not releasing ntdll!LdrpLoaderLock ?
Sham
0
 
mohet01Author Commented:
Please find the attached dump analysis report

1639Thread640.htm
0
 
tampnicCommented:
>>"Why thread 7 is not releasing ntdll!LdrpLoaderLock ?"
That's the crux of the matter. Its acquiring the lock while the thread is exiting, when DllMain(DLL_THREAD_DETACH) is called, that points to either AWInetFreeTlsArray() or UTIL_ThreadGlobalRelease() as the problem.

I'm finding the functionality of UTIL_ThreadGlobalRelease() confusing. It seems strange to initialise TLS with the awtls_init() function while the thread is exiting (I'm assuming that's what the function does). The "_util_cloneTls" global variable appears, but I can't tell how it was set.

My recommendation would be to move the        
         AWInetFreeTlsArray();
        UTIL_ThreadGlobalRelease();
calls out of DllMain() and into a new function inside the DLL called something like "aws_ThreadDetach()". Make the thread call "aws_ThreadDetach()" just before it exits. If you are using LoadLIbrary() and FreeLibrary() to control when the thread loads the DLL, call ThreadDetach() just before you call FreeLibrary(). This way the two functions will be called before the ntdll!LdrpLoaderLock is aquired.

Cheers,
  Chris
0
 
mohet01Author Commented:
very valid point
Do i need to check now, where do ew exactly call loadlibrary() and freelibrary()?

0
 
tampnicCommented:
It depends on how the main process is loading the DLL.

The DLL can be compile-time linked to the main process when you compile your main application code, so that the application will be embedded with the knowledge of where to find the functions provided by the DLL. This is the most common scenario. In this case I would call "aws_ThreadDetach()" just before the exit/return of any thread procedure. I would also call it at the exit of the main process, though this is probably unneccessary.

The alternative to compile-time linking is run-time linking where you use LoadLibrary() to load the DLL, and then call GetProcAddress() to find the location for each function you use in the DLL. Once you are finished using the DLL you call FreeLibrary() to release it. In this case the DLL could be loaded either by the main process or by a thread procedure. If loaded and freed in a thread procedure, just call aws_ThreadDetach() before the FreeLibrary() call. If the DLL is loaded by the main process, then call "aws_ThreadDetach()" just before returning from any thread procedures.

By "thread procedure" I mean any function which is passed when CreateThread() is called.

Cheers,
  Chris
0
 
mohet01Author Commented:
But let me tell you
this functions
 AWInetFreeTlsArray();
        UTIL_ThreadGlobalRelease();

in different dll than awscomm.dll
0
 
mohet01Author Commented:
i agree with your point, "by moving those 2 functions,This way the two functions will be called before the ntdll!LdrpLoaderLock is aquired"

But what is the exact purpose of DLL_THREAD_DETACH/DLL_THREAD_ATTACH/DLL_PROCESS_ATTACH/.. in DllMain() if we are not suppose to make any calls under these macros?



0
 
tampnicCommented:
We can make calls under the DLL_THREAD_DETACH, DLL_THREAD_ATTACH etc macros. The code in DllMain() is just subject to rules which don't apply elsewhere - I put code in there all the time but for safety I keep it very very simple. For instance use of CRT functions is to be avoided. http://blog.barthe.ph/2009/07/30/no-stdlib-in-dllmai/ may well apply to the situation here, as it describes a deadlock caused by code in DllMain(). Thread 16 certainly made a LoadLibrary() call which might be the root cause of why your deadlock occurred.

Cheers,
  Chris
0
 
mohet01Author Commented:
may be am still not clear why those 2 functions should not be under detach macro.
Let me tell u those 2 functions are in different dll
0
 
tampnicCommented:
In DllMain() there are many potential pitfalls. I usually have nothing in there if possible, preferring to initialise/release in dedicated PAttach(),PDetach(),TAttach() and TDetach() functions. Of course I have to remember to call these functions at the appropriate times in any code using the DLL. I personally don't find trouble using this method which steps around potentially difficult bugs to trace.

Without access to your full source code I couldn't say what your functions are doing to cause the deadlock ... and even then it could be very difficult to track down the problem. Its a trivial exercise to move them out of the Dll entry point and into a specific detach function so that's what I recommend. This would usually get around a ntdll!LdrpLoaderLock problem.

http://blogs.msdn.com/b/oldnewthing/archive/2004/01/28/63880.aspx gives more advice on keeping DllMain() processing to the absolute minimum.
http://msdn.microsoft.com/en-us/windows/hardware/gg487379.aspx is a link to a Microsft best practice paper and discusses ntdll!LdrpLoaderLock deadlocking

Cheers,
   Chris

Cheers,
   Chris
0
 
mohet01Author Commented:
Perfect
0

Featured Post

Hire Technology Freelancers with Gigs

Work with freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely, and get projects done right.

  • 13
  • 9
Tackle projects and never again get stuck behind a technical roadblock.
Join Now