Solved

Exception / stack problem (VC++ 6.0)

Posted on 2004-03-21
20
1,426 Views
Last Modified: 2007-12-19
I need to test what happens when I cause a stack overflow, so I wrote this tiny test:

  __try
  {
    void * pMem0 = _alloca(0xFFFFFFFF);
    memset(pMem0, 0, 0xFFFFFFFF);
  }
  __except(EXCEPTION_EXECUTE_HANDLER)
  {
    MessageBox(NULL, "Exception!", "Exception", MB_OK);
  }

From the theory, I cannot allocate 0xFFFFFFFF on the stack on a 32 Bit machine. But I get no exception in this line ! Without the "memset" operation, the program runs through without any problem. Strange.
So I took the "memset" operation after the alloca, and now I am told "Unhandled Exception in Test.exe". Even more strange! I have an exception handler, I tried alternatively with "normal try/catch", but to the same result. "Enable exception handling" is switched on. I even set the /F0x64 switch just to make sure that I have a small stack, but to the same result.

So I have two questions:
1) Why doesn't my _alloca call cause any exception ?
2) Why is the memset exception not caught by my handler ?
0
Comment
Question by:PC-Alex
  • 9
  • 7
  • 3
  • +1
20 Comments
 
LVL 86

Expert Comment

by:jkr
Comment Utility
>>But I get no exception in this line !

That is because you are most likely to get a NULL pointer back, that is why 'memset()' leads to a crash.
0
 
LVL 1

Author Comment

by:PC-Alex
Comment Utility
This is an excellent idea, I did not check this yet, because it is not documented in MSDN.

MSDN just says clearly: "A stack overflow exception is generated if the space cannot be allocated."

I'll check it at once!
0
 
LVL 1

Author Comment

by:PC-Alex
Comment Utility
Nope.

pMem0 is 0x0012ff14
0
 
LVL 86

Expert Comment

by:jkr
Comment Utility
Checked it, too. It seems that there is a lot going wrong here that should not happen. With or without the SEH handler, the program just terminates after 'memset()' - and the stack is completely blown away...
0
 
LVL 86

Expert Comment

by:jkr
Comment Utility
>>and the stack is completely blown away...

BTW, this is the reason the exception handler does not kick in, since SEH is stack-frame based.
0
 
LVL 1

Author Comment

by:PC-Alex
Comment Utility
Do you have any idea how I can make sure at runtime that my stack is big enough to handle my _alloca() - request ?
0
 
LVL 86

Expert Comment

by:jkr
Comment Utility
Well, it is difficult to determine the stack size left at runtime, but http://support.microsoft.com/default.aspx?scid=kb;en-us;315937#5 ("HOW TO: Trap Stack Overflow in a Visual C++ Application - Trapping the Exception with __try and __except (Full Solution)") might help.
0
 
LVL 1

Author Comment

by:PC-Alex
Comment Utility
jkr,

thanks for the link. This article helps me in enlarging the stack at runtime, but it does not help protect myself from a failed _alloca call; I need to get into the catch handler to handle the situation in any way, but I cannot reproduce such a stack-overflow situation caused by _alloca.
0
 
LVL 1

Author Comment

by:PC-Alex
Comment Utility
At the moment I think that the guard page is not tested effectively; if I look at the assembler code in chkstk.asm, I cannot see some write operation into the newly allocated area, so I wonder how _alloca will detect that at all ?

Even more strange: The ESP register is not changed at all by the _alloca call. The _alloca result is the ESP value.
0
 
LVL 86

Expert Comment

by:jkr
Comment Utility
>>The ESP register is not changed at all by the _alloca call. The _alloca result is the ESP value.

That is the way it should be - it will point at the base of the free stack.
0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 1

Author Comment

by:PC-Alex
Comment Utility
>>That is the way it should be

Only the second part of the sentence.
Calling _alloca with reasonable values always changes the ESP register (it MUST).
0
 
LVL 86

Expert Comment

by:jkr
Comment Utility
>>Calling _alloca with reasonable values always changes the ESP register (it MUST).

Yup, that's right.
0
 
LVL 2

Expert Comment

by:guntherothk
Comment Utility
There is a difference between C++ exceptions and Windows Structured Exceptions. You have to convert windows structured exceptions into C++ structured exceptions. There is a call called _set_se_translator() that does the job. You also have potentially to change a compiler flag to allow asynchronous exceptions. Converting structured exceptions arising from the hardware into C++ exceptions doesn't always work properly unless you've set asynchronous exception mode using the compiler flag /EHa, which is not the default.

How about calling a function recursively that allocates a local array of 10,000 characters. It won't take long for such a function to exhaust the stack, and you'll probably return more elegantly.

void crashme()
{
    char crasharray[10000];
    crashme();
}

This probably simulates more accurately what you are testing for anyway.
0
 
LVL 86

Expert Comment

by:jkr
Comment Utility
But, how would a SE translator help here?
0
 
LVL 1

Author Comment

by:PC-Alex
Comment Utility
guntherotk,

thanks for your comment. The crashme() example is exactly the same test as is proposed by jkr's link above; but this test does not help me further. I repeat my problem: I need to allocate memory on the stack at some places in my app, and don't use new/delete, for several reasons (performance, potential mem leaks, etc). Sometimes the application crashes randomly at the customer, and I need to find out if this is caused by "out-of-stackspace" situations, where the usage of _alloca is to blame.

So my task is simply: Generate an exception that occurs when _alloca fetches more memory, than is available on stack, and handle the situation.

But I cannot reproduce such a case. As I wrote above, I tried __try/__except as well as try/catch, and nothing was caught, so I do not think that any translation can help (if you still insist that it might help, could you please tell me how, in more detail ?).

I tried /EHa together with /GX, but with no change of behaviour. The combination /GX with /EHs is the same.

To be continued ...
0
 
LVL 1

Author Comment

by:PC-Alex
Comment Utility
... I continued experimentation, thought: I will watch the ESP register, and if it changed, I have just allocated successully.

The new prog looks like this:

void * pMem0 = _alloca(0x000000FF);
void * pMem1 = _alloca(0x0000FFFF);
void * pMem2 = _alloca(0x00FFFFFF);
void * pMem3 = _alloca(0xFFFFFFFF);

I watch it under the debugger, still with a 100-Byte-Stack ( /F0x64 ) . So what happens ?

Line "pMem0" changes ESP from 0012FEFC to 0012FDFC (this is one Byte too much, but roughly behaves as expected)
Next line: ESP changes from 0012FDFC to 0011FDFC (once again: roughly as expected, but one byte too much)
Next line, big surprise: ESP is unchanged, and ... I catch an exception ! First-chance exception in Test.exe: 0xC00000FD: Stack Overflow.

So I have some change in questions:
1) why doesn't _alloca(0xFFFFFFFF) do (or throw) anything ?
2) why can I successfully allocate more than my predefined stacksize (100 Bytes) ?
3) why is always one Byte more allocated than desired ?

Again I analyzed CHTSTK.ASM and think, question 2 might be answered with the fact that _PAGESIZE_ is hardcoded 0x1000, so the /F switch makes only sense with multiples of 0x1000.

Question 3, the one byte, might be the way ECX is handled, saved with a "push" and restored with a "MOV". But I do not exactly understand why this is done so.
0
 
LVL 2

Accepted Solution

by:
Bronek-K earned 250 total points
Comment Utility
You cannot allocate memory of size 0xFFFFFFFF . That's because anytime you call _alloca, number you pass as argument gets rounded up to nearest multiple of 4. If you pass anything more than 0xFFFFFFFC, integer overflow happens and effectively you will alloca(0) from stack, which obviously succeeds. About your second question: there is hard-coded minimum size of stack 4096 bytes (this is size of single memory page). If you specify smaller size in compiler options, it will get rounded up to 4096. You can see that with dumpbin tool (dumpbin /headers your.exe). And your third question: this is actually result of round-up to multiple of 4 I described in first point. If you do _alloca(0x000000FE) or _alloca(0x000000FD) you will see that 0x100 bytes got allocated from stack.
Two advices: if your program runs on recent version of Windows, you need to call _resetstkoflw inside __catch block in order to enable handling next stack overflow exception. If you do not do that, your program will fail with next stack overflow exception (instead of passing control to __catch block). You also should not create C++ objects on stack inside __try block, because SEH exception handling mechanism does not guarantee calling destructors for these objects.

0
 
LVL 2

Expert Comment

by:Bronek-K
Comment Utility
In case you wonder "why this roundup to multiple of 4" let me remind you that x86 is 32 bit CPU, which means that its most natural (and fastest) way of handling data is in 32-bit words, ie. 4 bytes.
0
 
LVL 1

Author Comment

by:PC-Alex
Comment Utility
Thank you Bronek-K for the answer. Would you please tell me just one thing more ?

I examined the _alloca assembler code in chktsk.asm and want to understand on what command exactly the 4-byte-alignment is done. Is it done implicitly on a call to push or sub or something like that ? Or where else is the 0xFF rounded to 0x100 ?
0
 
LVL 2

Expert Comment

by:Bronek-K
Comment Utility
If 0xFF is submitted as constant literal (ie. is known during compilation) it will happen during compilation, just before calling _chkstk. You will see in disassembly of your code that 0xFF got replaced by 0x100. If 0xFF is submitted as variable (not known during compilation), then simple calculation is performed before calling _chkstk
  mov  eax,dword ptr [bytes] ; bytes is your variable
  add  eax,3
  and  eax,0FFFFFFFCh
This will rund up number to nearest multiple of 4. BTW: there is no __catch block in SEH, it's called __except instead. Sorry for mistake.
0

Featured Post

Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

Join & Write a Comment

Often, when implementing a feature, you won't know how certain events should be handled at the point where they occur and you'd rather defer to the user of your function or class. For example, a XML parser will extract a tag from the source code, wh…
This article shows you how to optimize memory allocations in C++ using placement new. Applicable especially to usecases dealing with creation of large number of objects. A brief on problem: Lets take example problem for simplicity: - I have a G…
The viewer will learn how to use the return statement in functions in C++. The video will also teach the user how to pass data to a function and have the function return data back for further processing.
The viewer will learn additional member functions of the vector class. Specifically, the capacity and swap member functions will be introduced.

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now