Link to home
Create AccountLog in
Avatar of Mike Broderick
Mike BroderickFlag for United States of America

asked on

Visual Basic calls native C function, return string variable causes error.

I have a Visual Basic.Net application that calls a native C function in a DLL. It fails on 64 bit (heap error) but works on 32 bit:

VB:
    Declare Function Test1 Lib "xxx.dll" ( _
        ByVal SSLParm1 As Byte() _
           ) As String

        Dim ts As String = Test1(SSLParm1)

C:
LPSTR
WINAPI
Test1(st_Parm1 * pszParm1)
{
      return "000";
}

Both VB and C projects are defined as x64. I can debug into the native C to the return statement OK. The next step gives me (in the VB source):

Unhandled exception at 0x77c06df2 in VBtoC64Bit.exe: 0xC0000374: A heap has been corrupted.

Am I doing somethig wrong that caught up to me in 64 bit (or windows 7 or Visual Studio 2010, they all changed at the same time)? Is there a recommended way to return a string back to a VB app. It doesnt have to be a string, I could work with a bytearray if I had to.
Avatar of Subrat (C++ windows/Linux)
Subrat (C++ windows/Linux)
Flag of India image

try using dynamin allocation inside the fuction.(malloc/calloc) and caller needs to free the memory.

else try returning const string, LPCTSTR
Avatar of Mike Broderick

ASKER

The function actually returns a pointer to an offset within the passed parameter. I simplified the code shown. I have also tried passing LPSTR instead of char *. It doesnt work. It does work if the return type is integer.
SOLUTION
Avatar of HooKooDooKu
HooKooDooKu

Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
ASKER CERTIFIED SOLUTION
Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
Both worked well. Note that the c function return type couldnt be just BSTR, it had to be BSTR __declspec(dllexport) FAR PASCAL (it worked ok if the function returned "ABCDE" but gave a memory error if it returned "12345").  Declaring the function as returning IntPtr seems to be the best because the old C routine did not have to be changed, it could return type LPSTR or char*.

    Declare Function Test1 _
        Lib "xxx.dll" ( _
        ByVal SSLParm1 As Byte() _
           ) As IntPtr

        Dim ip As IntPtr
        Dim ts As String
        ip = Test1(SSLParm1)
        ts = Marshal.PtrToStringAnsi(ip)

Thanks for your help.