Exception / stack problem (VC++ 6.0)

I need to test what happens when I cause a stack overflow, so I wrote this tiny test:

  __try
  {
    void * pMem0 = _alloca(0xFFFFFFFF);
    memset(pMem0, 0, 0xFFFFFFFF);
  }
  __except(EXCEPTION_EXECUTE_HANDLER)
  {
    MessageBox(NULL, "Exception!", "Exception", MB_OK);
  }

From the theory, I cannot allocate 0xFFFFFFFF on the stack on a 32 Bit machine. But I get no exception in this line ! Without the "memset" operation, the program runs through without any problem. Strange.
So I took the "memset" operation after the alloca, and now I am told "Unhandled Exception in Test.exe". Even more strange! I have an exception handler, I tried alternatively with "normal try/catch", but to the same result. "Enable exception handling" is switched on. I even set the /F0x64 switch just to make sure that I have a small stack, but to the same result.

So I have two questions:
1) Why doesn't my _alloca call cause any exception ?
2) Why is the memset exception not caught by my handler ?
LVL 1
PC-AlexAsked:
Who is Participating?

Improve company productivity with a Business Account.Sign Up

x
 
Bronek-KConnect With a Mentor Commented:
You cannot allocate memory of size 0xFFFFFFFF . That's because anytime you call _alloca, number you pass as argument gets rounded up to nearest multiple of 4. If you pass anything more than 0xFFFFFFFC, integer overflow happens and effectively you will alloca(0) from stack, which obviously succeeds. About your second question: there is hard-coded minimum size of stack 4096 bytes (this is size of single memory page). If you specify smaller size in compiler options, it will get rounded up to 4096. You can see that with dumpbin tool (dumpbin /headers your.exe). And your third question: this is actually result of round-up to multiple of 4 I described in first point. If you do _alloca(0x000000FE) or _alloca(0x000000FD) you will see that 0x100 bytes got allocated from stack.
Two advices: if your program runs on recent version of Windows, you need to call _resetstkoflw inside __catch block in order to enable handling next stack overflow exception. If you do not do that, your program will fail with next stack overflow exception (instead of passing control to __catch block). You also should not create C++ objects on stack inside __try block, because SEH exception handling mechanism does not guarantee calling destructors for these objects.

0
 
jkrCommented:
>>But I get no exception in this line !

That is because you are most likely to get a NULL pointer back, that is why 'memset()' leads to a crash.
0
 
PC-AlexAuthor Commented:
This is an excellent idea, I did not check this yet, because it is not documented in MSDN.

MSDN just says clearly: "A stack overflow exception is generated if the space cannot be allocated."

I'll check it at once!
0
Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

 
PC-AlexAuthor Commented:
Nope.

pMem0 is 0x0012ff14
0
 
jkrCommented:
Checked it, too. It seems that there is a lot going wrong here that should not happen. With or without the SEH handler, the program just terminates after 'memset()' - and the stack is completely blown away...
0
 
jkrCommented:
>>and the stack is completely blown away...

BTW, this is the reason the exception handler does not kick in, since SEH is stack-frame based.
0
 
PC-AlexAuthor Commented:
Do you have any idea how I can make sure at runtime that my stack is big enough to handle my _alloca() - request ?
0
 
jkrCommented:
Well, it is difficult to determine the stack size left at runtime, but http://support.microsoft.com/default.aspx?scid=kb;en-us;315937#5 ("HOW TO: Trap Stack Overflow in a Visual C++ Application - Trapping the Exception with __try and __except (Full Solution)") might help.
0
 
PC-AlexAuthor Commented:
jkr,

thanks for the link. This article helps me in enlarging the stack at runtime, but it does not help protect myself from a failed _alloca call; I need to get into the catch handler to handle the situation in any way, but I cannot reproduce such a stack-overflow situation caused by _alloca.
0
 
PC-AlexAuthor Commented:
At the moment I think that the guard page is not tested effectively; if I look at the assembler code in chkstk.asm, I cannot see some write operation into the newly allocated area, so I wonder how _alloca will detect that at all ?

Even more strange: The ESP register is not changed at all by the _alloca call. The _alloca result is the ESP value.
0
 
jkrCommented:
>>The ESP register is not changed at all by the _alloca call. The _alloca result is the ESP value.

That is the way it should be - it will point at the base of the free stack.
0
 
PC-AlexAuthor Commented:
>>That is the way it should be

Only the second part of the sentence.
Calling _alloca with reasonable values always changes the ESP register (it MUST).
0
 
jkrCommented:
>>Calling _alloca with reasonable values always changes the ESP register (it MUST).

Yup, that's right.
0
 
guntherothkCommented:
There is a difference between C++ exceptions and Windows Structured Exceptions. You have to convert windows structured exceptions into C++ structured exceptions. There is a call called _set_se_translator() that does the job. You also have potentially to change a compiler flag to allow asynchronous exceptions. Converting structured exceptions arising from the hardware into C++ exceptions doesn't always work properly unless you've set asynchronous exception mode using the compiler flag /EHa, which is not the default.

How about calling a function recursively that allocates a local array of 10,000 characters. It won't take long for such a function to exhaust the stack, and you'll probably return more elegantly.

void crashme()
{
    char crasharray[10000];
    crashme();
}

This probably simulates more accurately what you are testing for anyway.
0
 
jkrCommented:
But, how would a SE translator help here?
0
 
PC-AlexAuthor Commented:
guntherotk,

thanks for your comment. The crashme() example is exactly the same test as is proposed by jkr's link above; but this test does not help me further. I repeat my problem: I need to allocate memory on the stack at some places in my app, and don't use new/delete, for several reasons (performance, potential mem leaks, etc). Sometimes the application crashes randomly at the customer, and I need to find out if this is caused by "out-of-stackspace" situations, where the usage of _alloca is to blame.

So my task is simply: Generate an exception that occurs when _alloca fetches more memory, than is available on stack, and handle the situation.

But I cannot reproduce such a case. As I wrote above, I tried __try/__except as well as try/catch, and nothing was caught, so I do not think that any translation can help (if you still insist that it might help, could you please tell me how, in more detail ?).

I tried /EHa together with /GX, but with no change of behaviour. The combination /GX with /EHs is the same.

To be continued ...
0
 
PC-AlexAuthor Commented:
... I continued experimentation, thought: I will watch the ESP register, and if it changed, I have just allocated successully.

The new prog looks like this:

void * pMem0 = _alloca(0x000000FF);
void * pMem1 = _alloca(0x0000FFFF);
void * pMem2 = _alloca(0x00FFFFFF);
void * pMem3 = _alloca(0xFFFFFFFF);

I watch it under the debugger, still with a 100-Byte-Stack ( /F0x64 ) . So what happens ?

Line "pMem0" changes ESP from 0012FEFC to 0012FDFC (this is one Byte too much, but roughly behaves as expected)
Next line: ESP changes from 0012FDFC to 0011FDFC (once again: roughly as expected, but one byte too much)
Next line, big surprise: ESP is unchanged, and ... I catch an exception ! First-chance exception in Test.exe: 0xC00000FD: Stack Overflow.

So I have some change in questions:
1) why doesn't _alloca(0xFFFFFFFF) do (or throw) anything ?
2) why can I successfully allocate more than my predefined stacksize (100 Bytes) ?
3) why is always one Byte more allocated than desired ?

Again I analyzed CHTSTK.ASM and think, question 2 might be answered with the fact that _PAGESIZE_ is hardcoded 0x1000, so the /F switch makes only sense with multiples of 0x1000.

Question 3, the one byte, might be the way ECX is handled, saved with a "push" and restored with a "MOV". But I do not exactly understand why this is done so.
0
 
Bronek-KCommented:
In case you wonder "why this roundup to multiple of 4" let me remind you that x86 is 32 bit CPU, which means that its most natural (and fastest) way of handling data is in 32-bit words, ie. 4 bytes.
0
 
PC-AlexAuthor Commented:
Thank you Bronek-K for the answer. Would you please tell me just one thing more ?

I examined the _alloca assembler code in chktsk.asm and want to understand on what command exactly the 4-byte-alignment is done. Is it done implicitly on a call to push or sub or something like that ? Or where else is the 0xFF rounded to 0x100 ?
0
 
Bronek-KCommented:
If 0xFF is submitted as constant literal (ie. is known during compilation) it will happen during compilation, just before calling _chkstk. You will see in disassembly of your code that 0xFF got replaced by 0x100. If 0xFF is submitted as variable (not known during compilation), then simple calculation is performed before calling _chkstk
  mov  eax,dword ptr [bytes] ; bytes is your variable
  add  eax,3
  and  eax,0FFFFFFFCh
This will rund up number to nearest multiple of 4. BTW: there is no __catch block in SEH, it's called __except instead. Sorry for mistake.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.