Solved

vc6.dll causes error when compiled with .NET

Posted on 2004-07-30
19
260 Views
Last Modified: 2012-06-27
We have several vc6 c++ non-MFC projects that have been converted to unmanaged .NET2003 solutions.  One project accesses another project, which is a dll.
Everything works fine when everything is compiled with vc6
when compiled with .NET our main app, within a class does:
....
fpSetSpeechSaveReturn = (LPFNSETSPEECHSAVERETURN)GetProcAddress(hHookDll,"SetSpeechSaveReturn");
if (fpSetSpeechSaveReturn) {
   // this pointer is fine here
   fpSetSpeechSaveReturn(hToNotify/*window handle within class */,WM_DRAGON_PAD_DONE);
  // this pointer is NULL here ??????
   bSetHook = true;
}
.....
all the dll is doing is setting some variables
.....
void CALLBACK SetSpeechSaveReturn(HWND hwnd,int iMessageToSend)
{
   g_hSpeechSaveReturn = hwnd; /** global to dll **/
   g_iSpeechSaveReturnMessage = iMessageToSend;
} // SetSpeechSaveReturn
.....


Any idea why the *this* pointer is getting whacked?
Thank You
0
Comment
Question by:PhilC
  • 10
  • 9
19 Comments
 
LVL 16

Expert Comment

by:nonubik
ID: 11677906
I see the dll is also compiled under .NET. And you see 'this' scrambled during debugging. Try a rebuild all on the projects, cause the .pdb files may be messed (between 2 VC++ versions). Or try to make different project configurations for .NET starting from old ones (output folder cahnged)
0
 

Author Comment

by:PhilC
ID: 11678374
after a full rebuild of all projects, the error still happens.  I also get an error when running Release build.
0
 
LVL 16

Expert Comment

by:nonubik
ID: 11678470
Is LPFNSETSPEECHSAVERETURN defined as void (*LPFNSETSPEECHSAVERETURN)(HWND, int) ?
If you step over the fpSetSpeechSaveReturn call, is 'this' still turning to NULL?
0
 

Author Comment

by:PhilC
ID: 11678514
typedef void (*LPFNSETSPEECHSAVERETURN)(HWND,int);
and the function def
void CALLBACK SetSpeechSaveReturn(HWND hwnd,int iMessageToSend);
if I comment out the load and call of the function *this* is fine
Thank You
0
 
LVL 16

Expert Comment

by:nonubik
ID: 11678593
Is SetSpeechSaveReturn doing anything else than posted above?
Cause it looks strange. You can try setting a breakpoint when 'this' value changes to see exactly who is guilty.
0
 

Author Comment

by:PhilC
ID: 11679696
nope, that is the full function
I've never used a breakpoint when a variable changes before...I set the breakpoint
get a message that it may make the application run very slowly, and let go for 10 minutes and nothing happens
my next line number breakpoint is not hit, seems as though devstudio is in space somewhere
could it be a calling convention type issue?
Thank You
0
 

Author Comment

by:PhilC
ID: 11679977
maybe a little more information on exactly what we are doing and why will help
We attempted to integrate ScanSoft Dragon Naturally speaking into our application, but found their SDK extremely buggy and unreliable
we added an interface such that the user click a button which will post keyboard events that launch the DragonPad (speech recoginition input) window
and enable the microphone
We then set a hook that will, when the user says "save" or clicks the save icon on the toolbar will get the text from the dragon editting window, close down the
dragon editing window, and send the text back to the window set in SetSpeechSaveReturn() using the message also specified in that call.
FARPROC fpSpeechHook=NULL;
fpSpeechHook = GetProcAddress(hHookDll,"SpeechSaveHookProc");
if (fpSpeechHook) {
      hHook = SetWindowsHookEx(WH_CALLWNDPROC,(HOOKPROC)fpSpeechHook,hHookDll,GetWindowThreadProcessId(hDragonPad,NULL));
... do processing above with SetSpeechSaveReturn...
where SpeechSaveHookProc is:
LRESULT CALLBACK SpeechSaveHookProc(int nCode, WORD wParam, DWORD lParam)
{
      CWPSTRUCT *lpCwp=(CWPSTRUCT *)lParam;

      if (lpCwp && (lpCwp->message == WM_COMMAND)) {
            if (LOWORD(lpCwp->wParam) == 57603) {
                  char szClassName[200]="";
                  
                  GetClassName(lpCwp->hwnd,szClassName,200);
                  if (!strcmp(szClassName,"TalkpadClass")) {
                        HWND hwndRichEdit=NULL;

                        hwndRichEdit = FindWindowEx(lpCwp->hwnd,NULL,"RICHEDIT",NULL);
                        if (hwndRichEdit && IsWindow(g_hSpeechSaveReturn)) {
                              PostMessage(g_hSpeechSaveReturn,g_iSpeechSaveReturnMessage,(WPARAM)hwndRichEdit,(LPARAM)lpCwp->hwnd);
                        } // if have rich edit control handle
                  } // if have talkpadclass
            } // if h ave the message we are looking for
      } // if have a wm command message
      return 0;
} // SpeechSaveHookProc


It's a lot for little I know, but our clients demand the ability...
Thank You
0
 
LVL 16

Expert Comment

by:nonubik
ID: 11691773
I don't know if these will solve your problem, but there are some requirements that a hook must meet, I'll try to help fixing :)

If you want a system-wide hook, you can just leave the last parameter in SetWindowsHookEx 0. But I presume that GetWindowThreadProcessId(hDragonPad,NULL) returns the thread id of the DragonPad window proc.

Every hook should pass the control to the next hook proc, otherwise, other applications that have installed WH_CALLWNDPROC hooks will not receive hook notifications and may behave incorrectly as a result. So I suggest to move 'return 0;' after the PostMessage call and to return CallNextHookEx(hHook, nCode, wParam, lParam) instead. But if your hook is the last one attached to that window, this modification will have no further effect.

Now to return to SetSpeechSaveReturn ..
If you comment the 2 lines inside the function (empty function), after retuning 'this' will be still NULL?
0
 

Author Comment

by:PhilC
ID: 11692561
Thank You, I have updated the hook, and commented out the 2 lines of SetSpeechSaveReturn() and this pointer still is NULL after return
0
Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

 
LVL 16

Expert Comment

by:nonubik
ID: 11692594
Still strange. How do you know 'this' is NULL? Watch the debugger, got crashes, etc..
I'm trying to track down the problem more deeply.
Overriding of pointers may accour if someone writes some unallocated memory area (like a char pointer that some writes in it more than allocated, can overwrite the 'this' memory area or so)
0
 

Author Comment

by:PhilC
ID: 11692875
I put a break point at SetSpeechSaveReturn(). I f11 into the function, f10 to the ending }, next f10 takes me back to calling module (this is still ok) f10 to next line and this is NULL in the watch window.  I will try to recreate this issue witha smaller program.  If I can, that might help you help me, If I can't then I'll look elsewere in our code.
0
 
LVL 16

Expert Comment

by:nonubik
ID: 11693008
>(this is still ok) f10 to next line and this is NULL

Could you step into assembly at this time? Maybe it will be more clear.
0
 

Author Comment

by:PhilC
ID: 11693111
Registers
EAX = CCCCCCCC EBX = 00000000 ECX = 00000000 EDX = 01E30608 ESI = 0045A1A0 EDI = 0012EC44 EIP = 00522912 ESP = 0012E6B8 EBP = 0012E6E4 EFL = 00000206


                        if (fpSetSpeechSaveReturn) {
0052289F  push        4    
005228A1  push        0C0000643h
005228A6  lea         eax,[fpSetSpeechSaveReturn]
005228A9  push        eax  
005228AA  call        dword ptr [_numega_finalcheck_C_110456 (85BA2Ch)]
005228B0  cmp         dword ptr [eax],0
005228B3  je          0052294A
                              fpSetSpeechSaveReturn(hToNotify,WM_DRAGON_PAD_DONE);
005228B9  push        427h
005228BE  push        0A4A1E030h
005228C3  push        4    
005228C5  push        80800643h
005228CA  push        4    
005228CC  push        80000643h
005228D1  push        10h  
005228D3  push        0C1002643h
005228D8  lea         eax,[this]
005228DB  push        eax  
005228DC  call        dword ptr [_numega_finalcheck_C_110456 (85BA2Ch)]
005228E2  mov         eax,dword ptr [eax]
005228E4  add         eax,10h
005228E7  push        eax  
005228E8  call        dword ptr [_numega_finalcheck_C_110456 (85BA2Ch)]
005228EE  push        dword ptr [eax]
005228F0  call        dword ptr [_numega_finalcheck_X_110456 (85BA6Ch)]
005228F6  push        eax  
005228F7  push        4    
005228F9  push        0C0000643h
005228FE  lea         eax,[fpSetSpeechSaveReturn]
00522901  push        eax  
00522902  call        dword ptr [_numega_finalcheck_C_110456 (85BA2Ch)]
00522908  push        dword ptr [eax]
0052290A  call        dword ptr [_numega_finalcheck_J_110456 (85BA3Ch)]
00522910  call        eax  
==> break here after return from function <=== 00522912  pop         ecx  
00522913  pop         ecx  
00522914  push        0    
00522916  push        0    
00522918  push        1    
0052291A  push        0A4A1E030h
0052291F  push        10000000h
00522924  call        dword ptr [_numega_finalcheck_XA_110456 (85BA78h)]
                              bSetHook = true;
0052292A  mov         byte ptr [ebp-25h],1
0052292E  lea         eax,[ebp-25h]
00522931  push        eax  
00522932  push        1    
00522934  push        40000212h
00522939  lea         eax,[bSetHook]
0052293C  push        eax  
0052293D  call        dword ptr [_numega_finalcheck_Y_110456 (85BA70h)]
00522943  mov         cl,byte ptr [ebp-25h]
00522946  mov         byte ptr [eax],cl
                        }
0
 
LVL 16

Expert Comment

by:nonubik
ID: 11693143
Seems you run boundschecker final instrumentation stuff. Could you turn it off, please?
0
 

Author Comment

by:PhilC
ID: 11693282
void CALLBACK SetSpeechSaveReturn(HWND hwnd,int iMessageToSend)
{
035010B0  push        ebp  
035010B1  mov         ebp,esp
035010B3  sub         esp,0C0h
035010B9  push        ebx  
035010BA  push        esi  
035010BB  push        edi  
035010BC  lea         edi,[ebp-0C0h]
035010C2  mov         ecx,30h
035010C7  mov         eax,0CCCCCCCCh
035010CC  rep stos    dword ptr [edi]
//      g_hSpeechSaveReturn = hwnd;
//      g_iSpeechSaveReturnMessage = iMessageToSend;
} // SetSpeechSaveReturn

                        if (fpSetSpeechSaveReturn) {
00521739  cmp         dword ptr [fpSetSpeechSaveReturn],0
0052173D  je          CDragonPadAccess::SetHook+0D7h (521757h)
                              fpSetSpeechSaveReturn(hToNotify,WM_DRAGON_PAD_DONE);
0052173F  push        427h
00521744  mov         ecx,dword ptr [this]
00521747  mov         edx,dword ptr [ecx+10h]
0052174A  push        edx  
0052174B  call        dword ptr [fpSetSpeechSaveReturn]
0052174E  add         esp,8
                              bSetHook = true;
00521751  mov         byte ptr [bSetHook],1
                        }

EAX = CCCCCCCC EBX = 00000000 ECX = 00000000 EDX = 003E0B4E ESI = 0045A1A0 EDI = 0012EC44 EIP = 0052174E ESP = 0012E708 EBP = 0012E718 EFL = 00000202
0
 

Author Comment

by:PhilC
ID: 11693318
wow! seems the debug build does not crash, and this is not NULL....I will double check he release build....$%*#$&%($
Will let you know, Thank You
0
 
LVL 16

Accepted Solution

by:
nonubik earned 500 total points
ID: 11693332
:) maybe it was related to boundschecker
0
 

Author Comment

by:PhilC
ID: 11693336
Well there you have it.....with bounds checker totally disabled all configurations run fine.
A very good, yet time consuming lesson.
Thank you very much for all of your help
0
 
LVL 16

Expert Comment

by:nonubik
ID: 11693356
Glad to help.
That's why sometimes you have to dig really deep for the source of the evil :)
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

The following diagram presents a diamond class hierarchy: As depicted, diamond inheritance denotes when two classes (e.g., CDerived1 and CDerived2), separately extending a common base class (e.g., CBase), are sub classed simultaneously by a fourt…
In Easy String Encryption Using CryptoAPI in C++ (http://www.experts-exchange.com/viewArticle.jsp?aid=1193) I described how to encrypt text and recommended that the encrypted text be stored as a series of hexadecimal digits -- because cyphertext may…
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…
This video shows how to remove a single email address from the Outlook 2010 Auto Suggestion memory. NOTE: For Outlook 2016 and 2013 perform the exact same steps. Open a new email: Click the New email button in Outlook. Start typing the address: …

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now