• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 690
  • Last Modified:

How to detect a stack overflow ?

Hi all,
  I am working in C++ - QNX. How to check a stack overflow within the program ?  I guess there will be some means to detect stack overflow in UNIX and the same may be applicable to QNX also.

Pls be comprehensive.

Thnx in Advance,
1 Solution
It is not easy to detect this within your program. There are mechanisms such as placing "canary" values on the stack and then checking if they've changed, but this check has to be inserted by the compiler into the function return code to be useful.

Another mechanism is for the kernel to enforce the stack as non-executable in the MMU. Several Unix flavors can do this.

Watcom C/C++ for QNX apparently supports runtime checking for stack overflow (http://www.qnx.com/products/watcom).

I haven't used Watcom for a while now, so I don't know how good it is ...

Here is an idea. Find the direction of a stack (I beleive, on QNX it grows from up to down), and then [by using a very first local variable's address from the main()], add to it stack limit from getrlimit() [possibly, subtract one used by getrusage()], and you get an address.
Round it to backward (to stack) direction onto page size getpagesize(), and mprotect() it to be not readable and not writable. Thus, you've created "red zone".
Then, when you try to overflow stack for a process, you will get a SIGSEGV.

If you want to just add index bound checking, ether use gcc's bound checking patch, or Purify, or Insure++.

Featured Post

Receive 1:1 tech help

Solve your biggest tech problems alongside global tech experts with 1:1 help.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now