DbgHelp fails with large structures

Hi,

I'm trying to use dbghelp to know the exact contents of a binary file. Everything works fine expect in programs with very large structures. For example, using the following test program

DbgHelp behaves very strangly. It gets the address of the the member variables 0 to 3213 but then gets the address of var_0 and then the calls to SymGetTypeInfo fail.

Am I doing something wrong or this is a bug?

Any help would be welcomed.


typedef struct {
     float var_0;
     float var_1;
     ....
     float var_5000;
} stTest_t;
 
stTest_t stTest;
 
void main()
{
     stTest.var_1 = 2;
}

Open in new window

goloapAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

DanRollinsCommented:
What code are you using in the GetType parameter?  What are you passing in the pInfo parameter?
What is error code is returned by GetLastError ?
Have you tried using the SymGetTypeInfoEx function?
0
goloapAuthor Commented:
I'm using the same code as similar examples on the web (see bellow)

GetLastError returns a random value (really large negative value)

Not yet, looks too much of a pain the setup the structure correctly. But I may have to do it.

// If the symbol has children
if (FALSE == SymGetTypeInfo(SYMBOL_HANDLE, ModuleBase, pSymInfo->TypeIndex, TI_GET_CHILDRENCOUNT, &ChildCount))
{
	CString ErrString;
	ErrString.Format("ERROR: Unable to get type information for symbol: %s", BaseName );
	ThrowException(ErrString);
}
	
if (ChildCount != 0 )
{
	DWORD dwKind = 0;
	// If this is a User Defined Type (UDT) which is a structure
	SymGetTypeInfo(SYMBOL_HANDLE, ModuleBase, pSymInfo->TypeIndex, TI_GET_UDTKIND, &dwKind))
	if (dwKind == UdtStruct)
	{
		int FindChildrenSize = sizeof(TI_FINDCHILDREN_PARAMS) + ChildCount*sizeof(ULONG); 
		TI_FINDCHILDREN_PARAMS* pFC = (TI_FINDCHILDREN_PARAMS*)malloc( FindChildrenSize ); 
		memset( pFC, 0, FindChildrenSize ); 
		pFC->Count = ChildCount;
							
		// Get the info for the type
		SymGetTypeInfo( SYMBOL_HANDLE, ModuleBase, pSymInfo->TypeIndex, TI_FINDCHILDREN, pFC ) ) 
		DWORD i =0 ;
 
		for (i = 0; i < ChildCount; i++)
		{
			// Get Name 
			WCHAR *pName = 0;
			SymGetTypeInfo( SYMBOL_HANDLE, ModuleBase, pFC->ChildId[i], TI_GET_SYMNAME, &pName ); 
						
			// Offset relative to parent
			DWORD dwOffset = 0;
			SymGetTypeInfo( SYMBOL_HANDLE, ModuleBase, pFC->ChildId[i], TI_GET_OFFSET, &dwOffset );
			
...

Open in new window

0
DanRollinsCommented:
That "random number" from GetLastError might look less random when displayed in hexadecimal format.  Post it here, or use the "Error Lookup" tool to see if it provides additional information.
0
Cloud Class® Course: MCSA MCSE Windows Server 2012

This course teaches how to install and configure Windows Server 2012 R2.  It is the first step on your path to becoming a Microsoft Certified Solutions Expert (MCSE).

goloapAuthor Commented:
Made some changes, and now I end up with an error code of 1on the 3214th element of a structure. The weird bit is that element 3213 is fine, the next element returned however is element 0 and then I get the error. It seems that the pFC array gets corrupted.

Sorry for the slownest of my replies, I'm very busy on other projects also, but thanks for your help.
0
DanRollinsCommented:
What is the value of ChildCount in the above example?
0
goloapAuthor Commented:
5001. So everything should be alright.
0
DanRollinsCommented:
Just for fun, create an extra-large buffer to hold the response data.  Right now you are calculating the size in your FindChildrenSize variable.  If you increase that to, say 1MB (ridiculously large) do you still have the same error?
0
goloapAuthor Commented:
Valid idea. Change my malloc call to
TI_FINDCHILDREN_PARAMS* pFC = (TI_FINDCHILDREN_PARAMS*)malloc( 1024*1024 );

but still same error.

This is ridiculous, any good microsoft forum I could ask?
0
DanRollinsCommented:
I don't know where else to ask.
If you suspect that the data is getting corrupted, then check your suspicion. Use the memory viewer to look in the buffer before you start to process the data. It should be all zeros (except Count) right before the call and after the call, it should hav the Count and Start value followed by 5000 4-byte values. Look at the area starting at about 12,856 bytes from the start. I'd expect the data to be "eyeball uniform" right up to offset 20,000.
Another angle:
How does the VC++ debugger behave with such a large structure? Does it have any problem displaying all 5000 of the fields?
0
DanRollinsCommented:
Another note: In the only examples I could find, they all assumed that there would be a smallish number of fields. One expected at most 256, and the Mat Pietrek article source code:
    http://filezilla.cvs.sourceforge.net/viewvc/*checkout*/filezilla/FileZilla%20Server/source/misc/WheatyExceptionReport.cpp?revision=1.1
assumes a max of 1024. It doesn't even appear to attempt to handle the situation where then value could be larger than that. The same is true with this code that appear to assume there would never be more than 200 (and takes no precautions against that possibility):
    http://www.reactos.org/pipermail/ros-diffs/2008-August/025027.html
Perhaps the guys who wrote these know something that we don't. Or perhaps they think that a structure with 5000 named fields is so unlikely that it's not worth the hassle of supporting...
0
goloapAuthor Commented:
I've checked the memory corruption bit, and it is not the problem. Since right the memory the data stored in the pFC is wrong after the call to TI_FINDCHILDREN as shows the memory viewer pFC = 0x010A3280 (pFC->Count = 5002 = 0x138a) &pFC->ChildId[3214] = 0x010A3280. You can see that the childId resets to the childId of item 0 for no reasons.

Visual Studio does not have a problem with displaying the large structure.
0x010A0040  138a 0000 0000 0000 0174 0000 0175 0000  `.......t...u...
0x010A0050  0176 0000 0177 0000 0178 0000 0179 0000  v...w...x...y...
0x010A0060  017a 0000 017b 0000 017c 0000 017d 0000  z...{...|...}...
 
...
 
0x010A3270  0dfe 0000 0dff 0000 0e00 0000 0e01 0000  þ...ÿ...........
0x010A3280  0174 0000 0e02 0000 0e03 0000 0e04 0000  t...............
0x010A3290  0179 0000 0e05 0000 0e06 0000 0e07 0000  y...............
0x010A32A0  017e 0000 017f 0000 0180 0000 0181 0000  ~.......¬.......

Open in new window

0
DanRollinsCommented:
Good feedback. The more I look into this, the more it appears to be bug in the library code. I'm sure you've read this MSDN article, but I want to post it here as the definitive info on DbgHelp:
    Improved Error Reporting with DBGHELP 5.1 APIs / Matt Pietrek
    http://msdn.microsoft.com/en-us/magazine/cc301692.aspx
Another good article with source code here:
    How to use dbghelp to access type information
    http://www.debuginfo.com/articles/dbghelptypeinfo.html
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
I note that you are not checking the return code from the relevant SymGetTypeInfo call. If it is 0, then you could call GetLastError().
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Just more thrashing about (trying to narrow down the issue):
1) What if you decrease the size of your structure to 3200? Does it work fine then? Increase it and continue to examine the memory dump above. Does it malfunction when there are 3213 fields or does it only malf when you get higher? Does it malf at a different place?
2) What if you have 2000 fields, but each is a two-field UDT? That is, I wonder if the problem relates to the total of all fields (including grandchildren), or just the total children of the structure.
Again, I'm sorry that I don't have an answer, and frankly doing the above tests will not solve anything. But it might generate a lead or at least some info you can pass to Microsoft when you make an official bug report.
-- Dan
0
DanRollinsCommented:
As to solving this problem, about the only thing I can think of is to (again) suggest that you give a try to:
   SymGetTypeInfoEx Function
   http://msdn.microsoft.com/en-us/library/ms681350(VS.85).aspx
since it appears to have the same functionality, but goes about it in a different way -- a way which may not impose internal limitations...
Another alternative would be to delve into the complex DIASDK:
   Debug Interface Access SDK
   http://msdn.microsoft.com/en-us/library/x93ctkx8.aspx
which provides a COM interface way to access the debug symbols. For instance:
   http://msdn.microsoft.com/en-us/library/sdes2ybc.aspx
shows c++ source code to enumerate through a collection of fields in a UDT after calling IDiaSession::findChildren
0
goloapAuthor Commented:
Good eye for the missing GetLastError(). But I had removed it when I pasted my code in the window because it made for too many indentation. The last error code is still 1.

==============================
3200 items: ok
3213 items: ok
3214 items: ok
3215 items: executes ok but last item of the structure is shown to be the first one. I.e. element 3214 (0 based numbering) should be var_3214 but is displayed ad var_0 address 0x0
3216 item: fail on element 3215 with error code 1.

All the children count values are correct

===================================

if I declare the following

typedef struct{
      uint16_T a;
      uint16_T b;
}stMini_t;

typedef struct{
        stMini_t var_0;
        stMini_t var_1;
        ...
        stMini_t var_1999;
} stTest_t;

This works fine, but increasing the number of elements in stTest_t will result in the same behaviour.

==================================================

For the DIA SDK, i would really like not to have to redo my work. And in addition the original program is written in Visual C++ 6 so I doubt the DIA SDK would work with it. (I have a separate test program under Visual Studio 2005 EE with which I'm doing this debugging).

=================================================

I had quickly tried the SymGetTypeInfoEx function, but it is a monster function and hadn't gotten it to work. Maybe i'll give it another go since I am less busy.

Thanks for sticking with this Dan. Even if it is not fixed yet, your ideas help.
0
goloapAuthor Commented:
FINALLY I got it to work. The key was SymGetTypeInfoEx. I had to thinker to get it to work, but in the end it works. I post the magic code below in case somebody else is in the same situation.

Thanks for your help Dan, I'll give you the points because you helped me.
IMAGEHLP_GET_TYPE_INFO_PARAMS stTypeInfoEx;
ULONG TypeIds[] = {pSymInfo->TypeIndex};
IMAGEHLP_SYMBOL_TYPE_INFO ReqKinds[] = {TI_GET_OFFSET, TI_GET_SYMNAME};
ULONG_PTR ReqOffsets[] = {0, sizeof(DWORD)};
ULONG ReqSizes[] = {sizeof(DWORD), sizeof(WCHAR*)};
struct Record
{
	DWORD offset;
	WCHAR* pName;
};
struct Record *buffer;
buffer = (struct Record *)malloc(dwCount * sizeof(struct Record));
memset(buffer, 0, dwCount*sizeof(struct Record));
 
memset(&stTypeInfoEx, 0, sizeof(stTypeInfoEx));
stTypeInfoEx.SizeOfStruct = sizeof(stTypeInfoEx);
stTypeInfoEx.Flags = IMAGEHLP_GET_TYPE_INFO_CHILDREN;
stTypeInfoEx.NumIds = 1;
stTypeInfoEx.TypeIds = &TypeIds;
stTypeInfoEx.TagFilter = 0xFFFF; //0xFFFFFFFFFFFFFFFFul;
stTypeInfoEx.NumReqs = 2;
stTypeInfoEx.ReqKinds = ReqKinds;
stTypeInfoEx.ReqOffsets = ReqOffsets;
stTypeInfoEx.ReqSizes = ReqSizes;
stTypeInfoEx.ReqStride = sizeof(struct Record);
stTypeInfoEx.BufferSize = dwCount * sizeof(struct Record);
stTypeInfoEx.Buffer = (PVOID)buffer;
 
if (!SymGetTypeInfoEx(hBinFile, qwModuleBase, &stTypeInfoEx))
{
	printf("Error[%i]: SymGetTypeInfoEx failed\n", GetLastError());
}

Open in new window

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Editors IDEs

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.