Solved

How do I debug a C++ classic dll 64 bit in VS 2008, called by a VB.net project ?

Posted on 2011-03-16
22
820 Views
Last Modified: 2012-05-11
Hi everybody,
I have converted a C++ 32 bit dll built in VS6 (not a COM object) to a 64 bit dll  in VS 2008. The solution rebuilding is OK.
That dll has functions called from a VB.Net project (Windows Form) which itself has been converted from a VB 6 application. The calling works OK (there are no longer problems of passing arguments, end so on). But I need to debug the dll in the VS 2008-64 bit environment.
In the VS6 environment, I defined a debugging command line in the project properties , which was the VB6.exe executable, so when I launched the execution of the dll, VB6 opened and let me choose the VB application which itself was launched and passed the debugger on the dll when calling a function.
In VS 2008 64 bit, I cannot find the equivalent of VB6.exe: what should I enter in the debugging properties of the dll project in order to have the equivalent of the VS6 environment debugging operation?
Thanks in advance for your help.

Verdan
0
Comment
Question by:verdan
  • 11
  • 9
  • 2
22 Comments
 
LVL 9

Accepted Solution

by:
Orcbighter earned 292 total points
Comment Utility
The way I would test a DLL is this
(1) Write a test harness that call a function in the DLL. You can use your converted forms program, but if the logic to get to the functions you want to test is too complicated, it may be simpler to write a console application and call the function dirdctly, hardcoding the required values. If your form is OK, then use that.
(2) Build it in Debug
(3) Build your DLL in Debug in a separate instance of VS.
(4) Set a break point in your DLL code at the entry point to the function you are going to call from the form.
(5) Run your test harness/Windoews Form
(5) In the DLL project, go to the Menu and select Debug\Attach To Process... and select the exe that is your test harness.
In your Windows Form press the button (or whatever) that makes the call to the function you want to test.
This call should activate the breakpoint in your DLL code, from this point you can step through the code.
 
0
 

Author Comment

by:verdan
Comment Utility
Thanks Orcbighter,
In fact I had  already written a console (exe) application which calls the DLL function with hardcoded args (actually it is not a form like I said in my question)
I've done according to the process you indicate, but the breakpoint does not function in the DLL: it changes color (from orange to pale yellow) but nothing happens when I press F10. The DLL remains however running.... but with no debugging function!

If it can help, here are some more details:
The DLL has two kind of entry functions: several APIs (like the one I want to test), and client functions which are accessed when the DLL runs as a service. In the 32 bit version, I could debug the service functions by setting the service exe in the command line in the Project settings, debugging tabs. But for the non-service API I had to  write VB6.exe in the command line and open the calling VB project in debug, then the breakpoints in the DLL worked OK.

In the present environnement (VS2008) the debug in service mode works OK, in the same way as in 32 bit. But in the case of the non-service API, I encounter the problem of my question.

Thanks for your help.

VerDan
0
 
LVL 21

Assisted Solution

by:mastoo
mastoo earned 58 total points
Comment Utility
Possibly another copy of the dll is being used.  You can also set the project's debug properties tab to list the test harness exe and then just hit F5 in VS with your DLL project.  It will start the test harness and hopefully hit breakpoints in the dll.
0
 
LVL 9

Expert Comment

by:Orcbighter
Comment Utility
How does the DLL run as a service? A service has to be an executable.
Do you mean your DLL is called from a Service?
0
 

Author Comment

by:verdan
Comment Utility
The same DLL may work in 2 different ways with 2 different kind of entry points

1) As a service it runs in the memory space of a service program which is an executable: in that case the service functions can be degugged OK when called with the service excutable set in the command line of the DLL project properties page-> debugging -> command line

2) As a simple API, the API functions are called by the harness program VB : in this case the debugging does not work, whatever way (attached process or debug properties tab)

The dll is copied in \Windows\system32
0
 
LVL 9

Expert Comment

by:Orcbighter
Comment Utility
THe fact that the breakpoint in the DLL code changes colour means that this dll is not the one being called.
Your VB app may be linked against a different version of the DLL (as pointed out by mastoo).
One further point of clarification: I take it your C++ DLL which was converted to VS 2008 is unmanaged C++?.
Yes, then when calling from a .NET exe it would have genereated some interop files when you built it. Can you confirm that these were copied to the DLL directory when you installed it?
0
 

Author Comment

by:verdan
Comment Utility
How can I identify interop files ? Are they named from the .NET project or the unmanaged DLL?
0
 
LVL 9

Assisted Solution

by:Orcbighter
Orcbighter earned 292 total points
Comment Utility
They are named from the .NET project. The name usually takes the form of:
                                                 interop.[mydll].dll
where [mydll] is the name of your c++ dll.
and can be found in the Debug or Release subfolder of the bin folder of the .NET project that calls the C++ dll, depending on which build type you are running.
0
 

Author Comment

by:verdan
Comment Utility
There are no interop files related to my c++ dll in the .NET folders, neither anywhere in the computer! That certainly means that I missed something in the .NET project?
0
 
LVL 21

Expert Comment

by:mastoo
Comment Utility
Were you going to try my suggestion?
0
 

Author Comment

by:verdan
Comment Utility
Yes, Mastoo, I tried your suggestion, but that did not work. In ay case, your remark about the dll not being the one which is called has to be investigated, I will do that. Also the last comment by Orcbighter means that I missed something in the VB.Net project.
0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 9

Expert Comment

by:Orcbighter
Comment Utility
Checking your project:
(1) Ensure the DLL was included as a reference in the project. It wouldn't compile without it but it is an easy check.
(2) In the properties for that reference, check the path and confirm it is pointing where you thought it was.
(3) Before building the .NET ptoject ensure that the DLL is properly registered and pointing to the version of the DLL you think it is.
(4) clean out the project completely, ie delete the bin and obj directories, then rebuild.

Remember: interop files are generated when a managed (.NET) program calls an unmanaged DLL. I am assuming from your comments that you just converted to a higher version of VS, you did not modify the DLL code to use managed C++?
0
 

Author Comment

by:verdan
Comment Utility
Thans Orcbighter for you last comment. But my C++ DLL is a classic dll, not a COM object, so it cannot be added as a reference in the .NET project properties. Anyway, this project compiled OK.

The objective of  the whole process is to convert my whole application  (VB + ASP + classical DLL) from 32 bits to 64 bits (i.e. ASP.NET + classical DLL in 64 bits). The first step is to test the new DLL from a VB.Net simplified project compiled in 64 bit mode (I should say that the classical 32 bit application worked perfectly when built up with VS 2008 in 32 bit mode).

To create this project, I upgraded a simplfied version of the VB6 project in order to have a harness application to call the DLL and debug it. I modified the code according to the upgrading report This simplified VB.NET project has a main function and a few lines of code, declaring the DLL function to test and calling it with the correct arguments. The calling of the DLL works OK until a point in the DLL, where it crashes with a memory problem (of course coming from the conversion 32->64 bits). That is the reason I definitely need to debug it (putting messages in the DLL helps a lot, but it is a very cumbersone process!)

0
 
LVL 9

Assisted Solution

by:Orcbighter
Orcbighter earned 292 total points
Comment Utility
I have a couple of suggestions on where to look for the problem:
(1) int and long. A long is 32 bits, regardless of platform. An int is platform specific, so it is 32 bits on a 32-bit machine ans is 64-bits on a 64-bit machine. Check your c++ code to ensure your not trying to write into memory you shouldn't
one example might be:
long myVar = 0;
memset( &myVar, 0, sizeof(int));
don't laugh! I have seen this done. Convert tfrom 32 bit to 64-bit and this function is trashing memory outside the space allocated to myVar.

a second example might be:
int myVar = 0;
...
...
void function(MESSAGE_ID id, void* param)
{
    if(id == FOO)
    {
        long real_param = (long)param;
        // ...
    }
}
Note: possible truncation of data leading to indeterminate behaviour.
To sum up; the assumption in code that sizeof(int) == sizeof(long) == sizeof(void*) is definitely FALSE.

(2) The size of pointers are different. They are 64-bit, not 32-bit, on 64-bit platforms.

(3) Here is avery good article that may help:
http://www.viva64.com/en/a/0004/
0
 

Author Comment

by:verdan
Comment Utility
Thanks for those suggestions and the reference to that very good article indeed.

I have already carefully looked at those questions of data and pointer sizes on 64 bit in my dll and I've resolved most of them, but obviously not all (I checked the sizeof of my variables and pointers, but  some functions are pretty sophisticated and there are still hidden errors).

So I need to debug to find which ones and I still do not know how to debug my dll with the VS2008 debugger... This is now the problem I have to solve before going ahead.
0
 
LVL 9

Expert Comment

by:Orcbighter
Comment Utility
Can you provide a few simple code snippets that show one of the functions you call in the dll, and the code you use in the VB.NET app that calls this function?
0
 

Author Comment

by:verdan
Comment Utility
Hi Orcbighter,

here attached are:
The code of the VB.NET calling project
a part of the code of the called DLL function.

However, I think the problem is in the IDE, not in the code (even though there is a bug well below in the code - that I need to debug!)
conv.txt
import.txt
0
 
LVL 9

Assisted Solution

by:Orcbighter
Orcbighter earned 292 total points
Comment Utility
I see a number of places in the code that may present potential areas for crashes:

(1) In the conv.txt file you create a number of strings that are constructed file paths. You then pass two of these into the ImportDico function as the first two parameters. In the import.txt file we see that the function maps the first two parameters to c++ string classes (probably STL from what I can see).
The VB passes the string ByVal so that it passes the data as a null-terminated string - all good.
One gotcha is that VB.NET strings are Unicode whereas the default type of C++ string is ASCII. You might want to check that.

(2) StrDup takes a char* as a parameter. Are you sure of the conversion that is taking place when you pass in an STL string? Again, the code assumes success and does no error checking.

(3) In import.txt you create a pointer to a logger object and then pass this into a RfxLogger_Logf function without checking if the create worked. That logger object could be null, and cause a crash. Try wrapping that code in a try-catch block.

(4) in import.txt we see the code does some proccessing on the two strings, which are imput and output file names, which you then use without checking that the path is valid.

(5) Although the older DECLARE syntax IS still supported, the preferred method is the DllImport syntax is the preferred method for VB.Net. You might want to investigate that for future support concerns.

Apart from that, I can't see any real problems with the code.
0
 

Author Comment

by:verdan
Comment Utility
Hi Orcbighter,
Thanks for your useful remarks.
However, I understand that you have no answer to my question about "how to debug the C++ dll called by a VB.Net project". This point is crucial for our development of the 64 bit version. Please confirme, so I could reformulate the question, if needed.

Your remarks:
(1) string arguments: OK for the warning
(2) StrDup (case sensitive) is a proprietary function (not strdup) which deals with pointers and copy of memory.
(3) Logger object : in the complete code, that object is created and its existence checked (the code in import.txt is only an extract).
(4) Paths are checked elsewhere, they are created at the installation of the software and checked at the startup of the service (here they are hard coded with existing paths values).
(5) OK, I will check DllImport
0
 
LVL 9

Assisted Solution

by:Orcbighter
Orcbighter earned 292 total points
Comment Utility
Well verdan, the problem is that your story keeps changing, which makes things hard to analyse.
Firstly you stated that you wished to know how to call the dll.
  "I cannot find the equivalent of VB6.exe: what should I enter in the debugging properties of the dll project in order to have the equivalent of the VS6 environment debugging operation?"

When solutions were offered by myself and mastoo you reportted that the dugger would not hit the breakpoints in your dll code.
  "I've done according to the process you indicate, but the breakpoint does not function in the DLL: it changes color (from orange to pale yellow) but nothing happens when I press F10. The DLL remains however running.... but with no debugging function!"

When pressed for further details you revealed that your program crashes:
 "simplified VB.NET project has a main function and a few lines of code, declaring the DLL function to test and calling it with the correct arguments. The calling of the DLL works OK until a point in the DLL, where it crashes with a memory problem (of course coming from the conversion 32->64 bits)."

Not trying to be mean. just outlining the course of this question.
I would suggest to you the the real problem is to sort out the crash. If your dll crashes when it is loaded, then, of course, the debugger will not stop at any breakpoints in that dll.

So there are two possible reasons provided for your problem.
Firstly, one cause might bee first is that you are not calling the dll you think you are. Check the project settings, Does the project for the c++ dll have any post-build steps, like self-registering?
In the VB.NET project, in the properties for the reference to the c++ dll, is CopyLocal set to true or false? After you rebuild the c++ dll do you re-register it before rebuilding the .NET project?

Secondly, the dll is crashing on load, which is why it never lasts long enough for the debugger to catch the breakpoint.

0
 

Author Comment

by:verdan
Comment Utility
Hi OrcBighter,

I will not argue about the problem I have, which is making the debugger work, stated and restated several times...
I have fully explained it and I think I have made clear that the dll crashes well below its entry point, not on load.
I said that I had put messages in the DLL, so I know approximately where it crashes, I just want to debug it in detail, and for that I need to make the debugger work!

The dll is not registrable (native C++ classic one) and has no reference properties.
I have checked all the possible dlls of the same name and deleted them, so I'm sure that the right one is called.

Anyway, thanks for your help
0
 

Author Closing Comment

by:verdan
Comment Utility
The experts' comment have been helpful on several partial points (potential problems in my code) but have not given a solution to the question which concerns using the debugger in the Visual Studio 2008 IDE.
However, I admit that my question might have been more explicitly formulated.
0

Featured Post

Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

Join & Write a Comment

In my previous article (http://www.experts-exchange.com/Programming/Languages/.NET/.NET_Framework_3.x/A_4362-Serialization-in-NET-1.html) we saw the basics of serialization and how types/objects can be serialized to Binary format. In this blog we wi…
A long time ago (May 2011), I have written an article showing you how to create a DLL using Visual Studio 2005 to be hosted in SQL Server 2005. That was valid at that time and it is still valid if you are still using these versions. You can still re…
This is Part 3 in a 3-part series on Experts Exchange to discuss error handling in VBA code written for Excel. Part 1 of this series discussed basic error handling code using VBA. http://www.experts-exchange.com/videos/1478/Excel-Error-Handlin…
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now