Solved

DLL Function Not Working Properly, Needs Debug

Posted on 2004-08-26
22
386 Views
Last Modified: 2008-01-09
Hello,

Me and a friend are making this DLL together, unfortunetly one of the functions in the DLL does not work, and since i have only recently learnt C/++ i am clueless to the answer, just wondering if anyone could help, thanks! :)

The DLL(RenFDSComm.dll) basically calls functions from another DLL (FDSTalk.dll) , but returns the return values, in a way that does not crash Visual Basic 6



char *_stdcall RenFDSMsg(char *msg,unsigned long ip,char *rpass,unsigned int rport)
{
Send_Message((char *) *rpass,ip,rport);
Send_Message((char *)*msg,ip,rport);

Service();

char response[100000 + 1]; //I Have No Idea How Long The Return Value Of Get_Responce Is Going To Be, it could even posibly be over the current array subscript =/

Get_Response((char *&) response);

Service();

Send_Message((char *) "Bye",ip,rport);

return responce;
}

i basically need to return a string as the return value

the following is the header file that comes with FDSTalk.dll


 #ifndef _FDSTALK_H
#define _FDSTALK_H

#ifdef FDSTALK_EXPORTS
#define FDSTALK_API __declspec(dllexport)
#else
#define FDSTALK_API __declspec(dllimport)
#endif

//Now its also posible to use the dll in MS Visual Basic and Borland Delphi.
#ifdef __cplusplus
extern "C" {
#endif // __cplusplus
     


     bool _stdcall Init(short port, char *password);
     void _stdcall Get_Response(char *&response);
     void _stdcall Shutdown (void);
     void _stdcall Send_Message(char *text, long ip, short port);
     void _stdcall Service(void);

}

#endif //_FDSTALK_H

Thanks in advanced guys :)
0
Comment
Question by:Nightma12
  • 12
  • 5
  • 3
  • +1
22 Comments
 
LVL 55

Expert Comment

by:Jaime Olivares
ID: 11902462
if this is correct:
Get_Response((char *&) response);

Then you don't need to declare a character array to respose, because it appear to be that Get_Response will allocate memory space and **assign** it to response (that why the & is there, I guess).
Also you cannot do it because when you return the array pointer you created, it will be destroyed before return.
0
 
LVL 1

Author Comment

by:Nightma12
ID: 11904274
if this is correct:
Get_Response((char *&) response);

i have no idea if that is correct, lol, but i know

     void _stdcall Get_Response(char *&response);

that shows in the header file is correct though





Then you don't need to declare a character array to respose, because it appear to be that Get_Response will allocate memory space and **assign** it to response (that why the & is there, I guess).


ok, i will try without assigning it :), but first....



Also you cannot do it because when you return the array pointer you created, it will be destroyed before return.


oh, damn, is it possible for me to copy the array to somewhere else in memory, where it wont get destroyed, and return a pointer to that?
0
 
LVL 55

Expert Comment

by:Jaime Olivares
ID: 11904581
a traditional method to return is this:

void ReturnMeAString(char *bufferPtr)
{
       strcpy(buffer, "this is a test");
}

to test:
char buf[200];   // Reserve enough space
ReturnMeAString(buffer);
puts(buffer);

0
 
LVL 19

Accepted Solution

by:
drichards earned 155 total points
ID: 11904880
To pass strings back from C++ to VB is problematic because of the memory allocation issue.  The easiest way to do it is to add a char** parameter to your C++:

    void _stdcall RenFDSMsg(char *msg,unsigned long ip,char *rpass,unsigned int rport, char **result)
    {
        ...
        GetResponse(*result);
    }

Then on VB side:

    Private Declare Sub RenFDSMsg Lib "..." (ByVal msg As String, ByVal ip As Long, ByVal rpass As String, ByRef result As String)

    Dim response As String
    response = String$(1024, vbNullChar)
    RenFDSMsg(..., response)

Of course, you'll probably also want to pass a size parameter to manage the potential buffer overrun problem.  If you don't preallocate the string in VB, you're stuck having to 'new' a string buffer to pass back, and now you've got memory leaking.
0
 
LVL 1

Author Comment

by:Nightma12
ID: 11905006
the problem is though, doing ByRef though a DLL is what appears to crash VB6

Get_Responce has a ByRef in it, which is what appears to crash VB6 (im useing SP6, my friend is useing no SP0, and he gets the same problem)

which is why we are creating the DLL in the first place, call the functions in FDSTalk.dll, and return the return values in a way that does not crash Visual Basic 6
0
 
LVL 19

Expert Comment

by:drichards
ID: 11905331
There are two separate problems as I look through this:

1) As jaime_olivares points out, is GetResponse allocating a buffer, and if so, who owns it?

2) You cannot pass strings from C++ to VB in that manner.  Basically there are two methods.  One is as I described - allocate a string in VB and pass it ByRef to a char** parameter in C++.  I have this code working in a program of mine.  This works but is not very elegant as you have allocated a fixed buffer for a string of unknown size.  Here's another way:

   http://support.microsoft.com/default.aspx?scid=kb;en-us;118643

It's a bit more complicated but it is more elegant since the C++ side is in control of allocatinf the string and gives control of it back to VB.
0
 
LVL 39

Expert Comment

by:itsmeandnobodyelse
ID: 11905590
>> the problem is though, doing ByRef though a DLL is what appears to crash VB6

If you run that code from above it will crash beside of any BYREF arguments.

jaime already told you the biggest bugs:

1. casting a local char array to char*& wwill crash immediately if the pointer gets
    assigned in GetResponse().

Replace
       
         char response[100000 + 1];  

by

         char* response = NULL;

However, that only will work if GetResponse() allocates dynamic memory (what leads to a leak as it never gets freed) or if GetResponse uses static (or global) memory like that:

        static char response[100000];

The only practicable way is what drichards has suggested. Allocate memory in VB and pass the variable using BYREF.

2. returning a pointer to a locally defined buffer that got freed when leaving the DLL function

As told before: if you don't get the buffer by reference yo HAVE to allocate it by 'new' or make it static or use a globally defined buffer.


But there are more bugs, i suppose:

3. Send_Message((char *) *rpass,ip,rport);

Here you have a cast on a dereferencered variable that is passed as a char* buffer. That's ok if you've passed it as BYREF rpass AS STRING. But as you told us that you don't use BYREF i assume you've passed it using BYVAL. Then, Send_Message most likely will crash as you are passing a char* pointer that isn't a pointer value but the character value of the first byte of the message buffer.  

Regards, Alex

BTW, who did tell you that you have to cast all arguments? E.g.

      Send_Message((char *) "Bye",ip,rport);

will compile also if you have

     Send_Message("Bye",ip,rport);

and you would give the compiler a chance to find invalid argument types. ;-)





0
 
LVL 55

Expert Comment

by:Jaime Olivares
ID: 11905688
You can make a test to see if the function is assigning any value to 'response'
0
 
LVL 1

Author Comment

by:Nightma12
ID: 11910583
>>>>>>Here's another way:

   http://support.microsoft.com/default.aspx?scid=kb;en-us;118643

It's a bit more complicated but it is more elegant since the C++ side is in control of allocatinf the string and gives control of it back to VB.<<<<<<<


this looks like it may work, but i dont exactly understand it that much =/ what is HLSTR?, would i be able to copy the return value from Get_Responce to that?


   HLSTR __far __pascal __export returnAString()
   {
       HLSTR temp;
       char *buff = {"This function returns a string from a DLL."};

      // Creates Hlstr from standard C string.
       temp = VBCreateTempHlstr(buff, lstrlen(buff));

      // Report error (if any).
      if (HIWORD(temp) == -1)
         VBRuntimeError(LOWORD(temp));

      // Return Hlstr to calling VB function.
      return temp;
   }





>>>>>>3. Send_Message((char *) *rpass,ip,rport);

Here you have a cast on a dereferencered variable that is passed as a char* buffer. That's ok if you've passed it as BYREF rpass AS STRING. But as you told us that you don't use BYREF i assume you've passed it using BYVAL. Then, Send_Message most likely will crash as you are passing a char* pointer that isn't a pointer value but the character value of the first byte of the message buffer.  

Regards, Alex

BTW, who did tell you that you have to cast all arguments? E.g.<<<<<<<<


it wasnt me that did that part of the DLL, that was the other guy thats making it with me

i just attempted to remove the cast and i got a compile error,

c:\Documents and Settings\Dedicated Server\My Documents\Visual Studio Projects\RenFDSComm\RenFDSComm.cpp(83): error C2664: 'Send_Message' : cannot convert parameter 1 from 'char' to 'char *'
        Conversion from integral type to pointer type requires reinterpret_cast, C-style cast or function-style cast

im useing VC++.NET
0
 
LVL 1

Author Comment

by:Nightma12
ID: 11912687
ok, when i include the header file vbapi.h, i get

c:\Documents and Settings\Dedicated Server\My Documents\Visual Studio Projects\RenFDSComm\stdafx.h(16): fatal error C1083: Cannot open include file: 'vbapi.h': No such file or directory


same with HLSTR,  i get c:\Documents and Settings\Dedicated Server\My Documents\Visual Studio Projects\RenFDSComm\RenFDSComm.cpp(80): error C2501: 'HLSTR' : missing storage-class or type specifiers



/*Send Messages To The FDS, And Return Get_Responce, in a way that dousnt crash VB6 programs

msg = Message To Send To The FDS
ip = IP address of the server turned into a long with IP2Long
rpass = The RenRem Password On The Server
rport = The RenRem Port On The Server
*/
HLSTR _stdcall RenFDSMsg(char *msg,unsigned long ip,char *rpass,unsigned int rport)
{
Send_Message(*rpass,ip,rport);
Send_Message(*msg,ip,rport);

Service();

Get_Response(response);

Service();

Send_Message("Bye",ip,rport);

HLSTR responce2;

responce2 = VBCreateTempHlstr(responce,strlen(responce))

      if (HIWORD(responce2) == -1)
         VBRuntimeError(LOWORD(responce2));

return responce2;
}


i also get those Cast errors in the above post
0
 
LVL 39

Expert Comment

by:itsmeandnobodyelse
ID: 11914421
>>>> error C2664: 'Send_Message' : cannot convert parameter 1 from 'char' to 'char *'

This error you'll get with those statements

   Send_Message((char *) *rpass,ip,rport);
   Send_Message((char *)*msg,ip,rport);

if you omit the cast to (char*)

   Send_Message( *rpass,ip,rport);
   Send_Message(*msg,ip,rport);

The problem is - and i already tried to tell you - that rpass is a char* and by *rpass you are dereferencing it to a char type (the first character of rpass char array). So, the correct statements - without any casts - are

  Send_Message(rpass,ip,rport);   // simply pass the argumenrt to SendMessage - no cast, no dereference
  Send_Message(msg,ip,rport);

And it is a good chance that your prog doesn't crash after you changed that together wiit passing a result string BYREF as drichards has suggested.

>>>> it wasnt me that did that part of the DLL, that was the other guy

I hope, you can fix his bugs inspite of that.

You should post GetResponse() code and the RenFDSComm declaration in VB.

Regards, Alex




0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 1

Author Comment

by:Nightma12
ID: 11916669
>>>>This error you'll get with those statements

   Send_Message((char *) *rpass,ip,rport);
   Send_Message((char *)*msg,ip,rport);

if you omit the cast to (char*)

   Send_Message( *rpass,ip,rport);
   Send_Message(*msg,ip,rport);

The problem is - and i already tried to tell you - that rpass is a char* and by *rpass you are dereferencing it to a char type (the first character of rpass char array). So, the correct statements - without any casts - are

  Send_Message(rpass,ip,rport);   // simply pass the argumenrt to SendMessage - no cast, no dereference
  Send_Message(msg,ip,rport);

And it is a good chance that your prog doesn't crash after you changed that together wiit passing a result string BYREF as drichards has suggested.<<<<


thanks,  i no longer get compile errors for Send_Message, ive g2g real soon, so i will try i with the By Ref tommrrow morning to see if it will crash (and i suspect that it will, since every over DLL i have used with byref has crashed =/)



the declareation for FDSTalk.dll and RenFDSComm in VB is:


Private Declare Function Connect Lib "RenFDSComm.dll" (ByVal lport As Integer, ByVal rpass As String) As Boolean
Private Declare Function RenFDSMsg Lib "RenFDSComm.dll" (ByVal msg As String, ByVal ip As Long, ByVal rpass As String, ByVal rport As Integer) As String
Private Declare Sub Disconnect Lib "RenFDSComm.dll" ()


Private Declare Sub Send_Message Lib "FDSTalk.dll" (ByVal strMessage As String, ByVal IP As Long, ByVal port As Integer)
Private Declare Sub Service Lib "FDSTalk.dll" ()
Private Declare Sub Shutdown Lib "FDSTalk.dll" ()
Private Declare Sub Get_Response Lib "FDSTalk.dll" (ByRef Response As String)
Private Declare Function Init Lib "FDSTalk.dll" (ByVal port As Integer, ByVal password As String) As Boolean
0
 
LVL 1

Author Comment

by:Nightma12
ID: 11920524
void _stdcall RenFDSMsg(char *msg,unsigned long ip,char *rpass,unsigned int rport, char **reply)
{
Send_Message(rpass,ip,rport);
Send_Message(msg,ip,rport);

Service();

Get_Response(response);

Service();

Send_Message("Bye",ip,rport);

**reply = *responce;

}


c:\Documents and Settings\Dedicated Server\My Documents\Visual Studio Projects\RenFDSComm\RenFDSComm.cpp(93): error C2065: 'responce' : undeclared identifier


i was told that Get_Responce declares it?
0
 
LVL 39

Assisted Solution

by:itsmeandnobodyelse
itsmeandnobodyelse earned 20 total points
ID: 11921295
>> **reply = *responce;

That is still wrong. First, it should be 'response' and not 'responce'. Second, with char** you get a buffer passed as an argument that is returned to caller, i. e. to the VB program. You have to use the sequence drichards has suggested above:

          Dim response As String
          response = String$(65536, vbNullChar)
           RenFDSMsg(..., response, 65536)

Note, i added the size of the string - also suggested by drichards.

     Private Declare Sub RenFDSMsg Lib "..."
          (ByVal msg As String, ByVal ip As Long, ByVal rpass As String,  ByVal rport As Long, ByRef response As String, ByVal siz As Long)

In C++ we have

  char *_stdcall RenFDSMsg(char *msg,unsigned long ip,char *rpass,unsigned int rport, char** reply, unsigned long siz)

and call GetResponse like that

    Get_Response(*reply, siz);

However, you should make sure that GetResponse function doesn't allocate any own memory but simply is using the buffer passed upto the number of bytes known by 'siz' argument.

Regards, Alex
0
 
LVL 1

Author Comment

by:Nightma12
ID: 11921855
c:\Documents and Settings\Dedicated Server\My Documents\Visual Studio Projects\RenFDSComm\RenFDSComm.cpp(87): error C2660: 'Get_Response' : function does not take 2 arguments


>>>>>However, you should make sure that GetResponse function doesn't allocate any own memory but simply is using the buffer passed upto the number of bytes known by 'siz' argument.<<<<

urrrm, how would i check? =/
0
 
LVL 1

Author Comment

by:Nightma12
ID: 11921861
Increased Points.
0
 
LVL 39

Expert Comment

by:itsmeandnobodyelse
ID: 11923815
>> 'Get_Response' : function does not take 2 arguments

Yeah, i hoped you - or that other guy - could update Get_Response to take 2 arguments instead of one. The second parameter is the size of the string that should be used in Get_Response to check the maximum of characters that could be copied to the buffer.

If you would post the implementation of Get_Response() we can make sure that your prog will compile and work.

Regards, Alex
0
 
LVL 1

Author Comment

by:Nightma12
ID: 11924769
ok, im not sure if you understand


FDSTalk.dll, was created by a company, to make remote communication between the C&C Renegade FDS easier, unfortunetly, Get_Response will crash a VB6 program, and the company was bought out (by EA) before the bug was fixed, therefore, there is no chance of Get_Response getting changed

RenFDSComm.dll is what us 2 are making, all it does is call the functions in FDSTalk.dll, and returns Get_Response in a way that does not crash VB6 programs

if i can find a way to call Get_Response from VB6, without it crashing then all the better :P
0
 
LVL 39

Expert Comment

by:itsmeandnobodyelse
ID: 11925009
OK, i got it.

So, simply remove the second argument of GetResponse... However, you should verify the documentation of GetResponse whether there is any remarks about buffer size. If so, you should provide an allocation (in VB) that is sufficient to meet all possible replies. If not, it's a poor interface... but maybe you could check the sender's site what's the maximum that could be received at one time.

Regards. Alex
0
 
LVL 1

Author Comment

by:Nightma12
ID: 11925938
ok, there is no documentation for FDSTalk.dll, although i could possible work out what the max returned from Get_Response is, unfortunetly, the server that the FDS connects to is down at the moment (damn EA >.<) so i need to wait for it to come back up so i can test it, could you let me know where i would put the max in the code, ty :)
0
 
LVL 1

Author Comment

by:Nightma12
ID: 11971386
ok, i am too lazy to work out,  an absolute number :P

but the MAX get_Response could EVER possibly return is 26601 bytes of data
0
 
LVL 1

Author Comment

by:Nightma12
ID: 12091295
:O found out how to get Get_Response working on VB6 :O

unfortunetly, due to more VB6 errors, when Get_Response is called, it makes Service() crash >.<

    "Dim response As String
    response = String$(1024, vbNullChar)
    RenFDSMsg(..., response)"

do that with Get_Response :P

i find it amusing that nobody knew about this b4, lmao :P

im working on an external C++ program to be called with Shell() to overcome these errors now :), cheers for the discovery drichards :P


and for itsmeandnobodyelse pointing it out over and over again to me :P
0

Featured Post

Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

Join & Write a Comment

Often, when implementing a feature, you won't know how certain events should be handled at the point where they occur and you'd rather defer to the user of your function or class. For example, a XML parser will extract a tag from the source code, wh…
  Included as part of the C++ Standard Template Library (STL) is a collection of generic containers. Each of these containers serves a different purpose and has different pros and cons. It is often difficult to decide which container to use and …
The goal of the tutorial is to teach the user how to use functions in C++. The video will cover how to define functions, how to call functions and how to create functions prototypes. Microsoft Visual C++ 2010 Express will be used as a text editor an…
The viewer will learn additional member functions of the vector class. Specifically, the capacity and swap member functions will be introduced.

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now