asked on

How to detect a stack overflow ?

Hi all,
I am working in C++ - QNX. How to check a stack overflow within the program ? I guess there will be some means to detect stack overflow in UNIX and the same may be applicable to QNX also.

Pls be comprehensive.

Thnx in Advance,
Jayaprakash.

chris_calabrese

It is not easy to detect this within your program. There are mechanisms such as placing "canary" values on the stack and then checking if they've changed, but this check has to be inserted by the compiler into the function return code to be useful.

Another mechanism is for the kernel to enforce the stack as non-executable in the MMU. Several Unix flavors can do this.

ASKER CERTIFIED SOLUTION

griessh

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

JIEXA

Here is an idea. Find the direction of a stack (I beleive, on QNX it grows from up to down), and then [by using a very first local variable's address from the main()], add to it stack limit from getrlimit() [possibly, subtract one used by getrusage()], and you get an address.
Round it to backward (to stack) direction onto page size getpagesize(), and mprotect() it to be not readable and not writable. Thus, you've created "red zone".
Then, when you try to overflow stack for a process, you will get a SIGSEGV.

If you want to just add index bound checking, ether use gcc's bound checking patch, or Purify, or Insure++.