vc6.dll causes error when compiled with .NET

We have several vc6 c++ non-MFC projects that have been converted to unmanaged .NET2003 solutions.  One project accesses another project, which is a dll.
Everything works fine when everything is compiled with vc6
when compiled with .NET our main app, within a class does:
fpSetSpeechSaveReturn = (LPFNSETSPEECHSAVERETURN)GetProcAddress(hHookDll,"SetSpeechSaveReturn");
if (fpSetSpeechSaveReturn) {
   // this pointer is fine here
   fpSetSpeechSaveReturn(hToNotify/*window handle within class */,WM_DRAGON_PAD_DONE);
  // this pointer is NULL here ??????
   bSetHook = true;
all the dll is doing is setting some variables
void CALLBACK SetSpeechSaveReturn(HWND hwnd,int iMessageToSend)
   g_hSpeechSaveReturn = hwnd; /** global to dll **/
   g_iSpeechSaveReturnMessage = iMessageToSend;
} // SetSpeechSaveReturn

Any idea why the *this* pointer is getting whacked?
Thank You
Who is Participating?
:) maybe it was related to boundschecker
I see the dll is also compiled under .NET. And you see 'this' scrambled during debugging. Try a rebuild all on the projects, cause the .pdb files may be messed (between 2 VC++ versions). Or try to make different project configurations for .NET starting from old ones (output folder cahnged)
PhilCAuthor Commented:
after a full rebuild of all projects, the error still happens.  I also get an error when running Release build.
Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

If you step over the fpSetSpeechSaveReturn call, is 'this' still turning to NULL?
PhilCAuthor Commented:
and the function def
void CALLBACK SetSpeechSaveReturn(HWND hwnd,int iMessageToSend);
if I comment out the load and call of the function *this* is fine
Thank You
Is SetSpeechSaveReturn doing anything else than posted above?
Cause it looks strange. You can try setting a breakpoint when 'this' value changes to see exactly who is guilty.
PhilCAuthor Commented:
nope, that is the full function
I've never used a breakpoint when a variable changes before...I set the breakpoint
get a message that it may make the application run very slowly, and let go for 10 minutes and nothing happens
my next line number breakpoint is not hit, seems as though devstudio is in space somewhere
could it be a calling convention type issue?
Thank You
PhilCAuthor Commented:
maybe a little more information on exactly what we are doing and why will help
We attempted to integrate ScanSoft Dragon Naturally speaking into our application, but found their SDK extremely buggy and unreliable
we added an interface such that the user click a button which will post keyboard events that launch the DragonPad (speech recoginition input) window
and enable the microphone
We then set a hook that will, when the user says "save" or clicks the save icon on the toolbar will get the text from the dragon editting window, close down the
dragon editing window, and send the text back to the window set in SetSpeechSaveReturn() using the message also specified in that call.
FARPROC fpSpeechHook=NULL;
fpSpeechHook = GetProcAddress(hHookDll,"SpeechSaveHookProc");
if (fpSpeechHook) {
      hHook = SetWindowsHookEx(WH_CALLWNDPROC,(HOOKPROC)fpSpeechHook,hHookDll,GetWindowThreadProcessId(hDragonPad,NULL));
... do processing above with SetSpeechSaveReturn...
where SpeechSaveHookProc is:
LRESULT CALLBACK SpeechSaveHookProc(int nCode, WORD wParam, DWORD lParam)
      CWPSTRUCT *lpCwp=(CWPSTRUCT *)lParam;

      if (lpCwp && (lpCwp->message == WM_COMMAND)) {
            if (LOWORD(lpCwp->wParam) == 57603) {
                  char szClassName[200]="";
                  if (!strcmp(szClassName,"TalkpadClass")) {
                        HWND hwndRichEdit=NULL;

                        hwndRichEdit = FindWindowEx(lpCwp->hwnd,NULL,"RICHEDIT",NULL);
                        if (hwndRichEdit && IsWindow(g_hSpeechSaveReturn)) {
                        } // if have rich edit control handle
                  } // if have talkpadclass
            } // if h ave the message we are looking for
      } // if have a wm command message
      return 0;
} // SpeechSaveHookProc

It's a lot for little I know, but our clients demand the ability...
Thank You
I don't know if these will solve your problem, but there are some requirements that a hook must meet, I'll try to help fixing :)

If you want a system-wide hook, you can just leave the last parameter in SetWindowsHookEx 0. But I presume that GetWindowThreadProcessId(hDragonPad,NULL) returns the thread id of the DragonPad window proc.

Every hook should pass the control to the next hook proc, otherwise, other applications that have installed WH_CALLWNDPROC hooks will not receive hook notifications and may behave incorrectly as a result. So I suggest to move 'return 0;' after the PostMessage call and to return CallNextHookEx(hHook, nCode, wParam, lParam) instead. But if your hook is the last one attached to that window, this modification will have no further effect.

Now to return to SetSpeechSaveReturn ..
If you comment the 2 lines inside the function (empty function), after retuning 'this' will be still NULL?
PhilCAuthor Commented:
Thank You, I have updated the hook, and commented out the 2 lines of SetSpeechSaveReturn() and this pointer still is NULL after return
Still strange. How do you know 'this' is NULL? Watch the debugger, got crashes, etc..
I'm trying to track down the problem more deeply.
Overriding of pointers may accour if someone writes some unallocated memory area (like a char pointer that some writes in it more than allocated, can overwrite the 'this' memory area or so)
PhilCAuthor Commented:
I put a break point at SetSpeechSaveReturn(). I f11 into the function, f10 to the ending }, next f10 takes me back to calling module (this is still ok) f10 to next line and this is NULL in the watch window.  I will try to recreate this issue witha smaller program.  If I can, that might help you help me, If I can't then I'll look elsewere in our code.
>(this is still ok) f10 to next line and this is NULL

Could you step into assembly at this time? Maybe it will be more clear.
PhilCAuthor Commented:
EAX = CCCCCCCC EBX = 00000000 ECX = 00000000 EDX = 01E30608 ESI = 0045A1A0 EDI = 0012EC44 EIP = 00522912 ESP = 0012E6B8 EBP = 0012E6E4 EFL = 00000206

                        if (fpSetSpeechSaveReturn) {
0052289F  push        4    
005228A1  push        0C0000643h
005228A6  lea         eax,[fpSetSpeechSaveReturn]
005228A9  push        eax  
005228AA  call        dword ptr [_numega_finalcheck_C_110456 (85BA2Ch)]
005228B0  cmp         dword ptr [eax],0
005228B3  je          0052294A
005228B9  push        427h
005228BE  push        0A4A1E030h
005228C3  push        4    
005228C5  push        80800643h
005228CA  push        4    
005228CC  push        80000643h
005228D1  push        10h  
005228D3  push        0C1002643h
005228D8  lea         eax,[this]
005228DB  push        eax  
005228DC  call        dword ptr [_numega_finalcheck_C_110456 (85BA2Ch)]
005228E2  mov         eax,dword ptr [eax]
005228E4  add         eax,10h
005228E7  push        eax  
005228E8  call        dword ptr [_numega_finalcheck_C_110456 (85BA2Ch)]
005228EE  push        dword ptr [eax]
005228F0  call        dword ptr [_numega_finalcheck_X_110456 (85BA6Ch)]
005228F6  push        eax  
005228F7  push        4    
005228F9  push        0C0000643h
005228FE  lea         eax,[fpSetSpeechSaveReturn]
00522901  push        eax  
00522902  call        dword ptr [_numega_finalcheck_C_110456 (85BA2Ch)]
00522908  push        dword ptr [eax]
0052290A  call        dword ptr [_numega_finalcheck_J_110456 (85BA3Ch)]
00522910  call        eax  
==> break here after return from function <=== 00522912  pop         ecx  
00522913  pop         ecx  
00522914  push        0    
00522916  push        0    
00522918  push        1    
0052291A  push        0A4A1E030h
0052291F  push        10000000h
00522924  call        dword ptr [_numega_finalcheck_XA_110456 (85BA78h)]
                              bSetHook = true;
0052292A  mov         byte ptr [ebp-25h],1
0052292E  lea         eax,[ebp-25h]
00522931  push        eax  
00522932  push        1    
00522934  push        40000212h
00522939  lea         eax,[bSetHook]
0052293C  push        eax  
0052293D  call        dword ptr [_numega_finalcheck_Y_110456 (85BA70h)]
00522943  mov         cl,byte ptr [ebp-25h]
00522946  mov         byte ptr [eax],cl
Seems you run boundschecker final instrumentation stuff. Could you turn it off, please?
PhilCAuthor Commented:
void CALLBACK SetSpeechSaveReturn(HWND hwnd,int iMessageToSend)
035010B0  push        ebp  
035010B1  mov         ebp,esp
035010B3  sub         esp,0C0h
035010B9  push        ebx  
035010BA  push        esi  
035010BB  push        edi  
035010BC  lea         edi,[ebp-0C0h]
035010C2  mov         ecx,30h
035010C7  mov         eax,0CCCCCCCCh
035010CC  rep stos    dword ptr [edi]
//      g_hSpeechSaveReturn = hwnd;
//      g_iSpeechSaveReturnMessage = iMessageToSend;
} // SetSpeechSaveReturn

                        if (fpSetSpeechSaveReturn) {
00521739  cmp         dword ptr [fpSetSpeechSaveReturn],0
0052173D  je          CDragonPadAccess::SetHook+0D7h (521757h)
0052173F  push        427h
00521744  mov         ecx,dword ptr [this]
00521747  mov         edx,dword ptr [ecx+10h]
0052174A  push        edx  
0052174B  call        dword ptr [fpSetSpeechSaveReturn]
0052174E  add         esp,8
                              bSetHook = true;
00521751  mov         byte ptr [bSetHook],1

EAX = CCCCCCCC EBX = 00000000 ECX = 00000000 EDX = 003E0B4E ESI = 0045A1A0 EDI = 0012EC44 EIP = 0052174E ESP = 0012E708 EBP = 0012E718 EFL = 00000202
PhilCAuthor Commented:
wow! seems the debug build does not crash, and this is not NULL....I will double check he release build....$%*#$&%($
Will let you know, Thank You
PhilCAuthor Commented:
Well there you have it.....with bounds checker totally disabled all configurations run fine.
A very good, yet time consuming lesson.
Thank you very much for all of your help
Glad to help.
That's why sometimes you have to dig really deep for the source of the evil :)
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.