Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call.

jspies
jspies used Ask the Experts™
on
Hi there - I am getting the following error although my calling conventions on both my exe and my dll  is _cdecl.  Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call.  This is usually a result of calling a function declared with one calling convention with a function pointer declared with a different calling convention.

exe code:
HINSTANCE dllHandle;

typedef short (CALLBACK* iRDSToBLGType)(char *,char *,bool,char *);
iRDSToBLGType iRDSToBLGPtr;

void CrdstooutcallerDlg::OnBnClickedButton3()
{
      dllHandle = ::LoadLibrary("RDStoBLG.dll");
      iRDSToBLGPtr = (iRDSToBLGType)GetProcAddress(dllHandle,"iRDSToBLG");

      char m_rds[128000];
      memset(m_rds,0,sizeof(m_rds));
      char m_blgOut[128000];
      memset(m_blgOut,0,sizeof(m_blgOut));

      FILE* lFile = fopen("d:\\generalwork\\blg2\\rdsstring.txt","r+t");
      if(lFile)
      {
            fgets(m_rds,128000,lFile);
            fclose(lFile);
      }
      

      iRDSToBLGPtr(NULL,m_rds,false,m_blgOut);

      ::FreeLibrary(dllHandle);
}

dll code:

int  DLL_DECL _cdecl iRDSToBLG(char *pRetroDate, char *RDSData, bool bIDS, char *OutputData)
{
.
.
.
}
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
evilrixSenior Software Engineer (Avast)

Commented:
Well, the typedef for iRDSToBLGType doesn't match the function prototype (return types are different). Also, the typdef sets the calling convention to CALLBACK, which is __stdcall. Is this the same convention actually used? how is DLL_DECL _cdecl defined?

Commented:
Are there any optimisations at work here? For example, although you're using _cdecl is one part being compiled with, say, fastcall optimisations?
evilrixSenior Software Engineer (Avast)

Commented:
>>  how is DLL_DECL _cdecl defined?
Heh. Just noticed there is a space there :)

_cdecl is not the same calling conversion as __stdcall.

"This is the default calling convention for C and C++ programs. Because the stack is cleaned up by the caller, it can do vararg functions. The __cdecl calling convention creates larger executables than __stdcall, because it requires each function call to include stack cleanup code. The following list shows the implementation of this calling convention."

"The __stdcall calling convention is used to call Win32 API functions. The callee cleans the stack, so the compiler makes vararg functions __cdecl. Functions that use this calling convention require a function prototype."

http://msdn.microsoft.com/en-us/library/zkwh89ks.aspx
http://msdn.microsoft.com/en-us/library/zxk0tw93.aspx
Become a CompTIA Certified Healthcare IT Tech

This course will help prep you to earn the CompTIA Healthcare IT Technician certification showing that you have the knowledge and skills needed to succeed in installing, managing, and troubleshooting IT systems in medical and clinical settings.

Author

Commented:
I understand the difference between _stdcall and _cdecl, but what I don't understand is why I am getting this error while both the exe and dll that is written in C++ are compiled with the _cdecl flag.  This means that the calling convention must be the same - right?
Senior Software Engineer (Avast)
Commented:
>> while both the exe and dll that is written in C++ are compiled with the _cdecl flag.
But when you use the function you are using it with the CALLBACK specifier, which is stdcall not cdecl. This is resulting in a stack-frame mis-match between the caller and the callee, hence the error about the stack pointer.

Author

Commented:
Ok that is interesting.  I've tried making both my dll and my exe _stdcall.  I've also removed the _cdecl from the function call - Now looking like this:

int  DLL_DECL iRDSToBLG(char *pRetroDate, char *RDSData, bool bIDS, char *OutputData)
{...
}

, but now the exe crashes immediately when trying to call the function with the following error :
Unhandled exception at 0x00000000 in rdstooutcaller.exe: 0xC0000005: Access violation reading location 0x00000000.
evilrixSenior Software Engineer (Avast)

Commented:
>> the exe crashes immediately when trying to call the function
That's suggests the GetProcAddress() is failing and returning NULL. This should always be checked.
Did you recompile everything just to make sure everything is using the same convention?

Author

Commented:
I've changed the CALLBACK* to _cdecl* and everything works now!  Thanks so much!  I am accepting your solution.  Thank again!
evilrixSenior Software Engineer (Avast)

Commented:
>>  Thank again!
Heh. No worries.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial