How to detect a stack overflow ?

Posted on 2003-03-27
Medium Priority
Last Modified: 2008-03-17
Hi all,
  I am working in C++ - QNX. How to check a stack overflow within the program ?  I guess there will be some means to detect stack overflow in UNIX and the same may be applicable to QNX also.

Pls be comprehensive.

Thnx in Advance,
Question by:jayaprakashns
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
LVL 14

Expert Comment

ID: 8218548
It is not easy to detect this within your program. There are mechanisms such as placing "canary" values on the stack and then checking if they've changed, but this check has to be inserted by the compiler into the function return code to be useful.

Another mechanism is for the kernel to enforce the stack as non-executable in the MMU. Several Unix flavors can do this.
LVL 11

Accepted Solution

griessh earned 500 total points
ID: 8219918

Watcom C/C++ for QNX apparently supports runtime checking for stack overflow (http://www.qnx.com/products/watcom).

I haven't used Watcom for a while now, so I don't know how good it is ...


Expert Comment

ID: 8309350
Here is an idea. Find the direction of a stack (I beleive, on QNX it grows from up to down), and then [by using a very first local variable's address from the main()], add to it stack limit from getrlimit() [possibly, subtract one used by getrusage()], and you get an address.
Round it to backward (to stack) direction onto page size getpagesize(), and mprotect() it to be not readable and not writable. Thus, you've created "red zone".
Then, when you try to overflow stack for a process, you will get a SIGSEGV.

If you want to just add index bound checking, ether use gcc's bound checking patch, or Purify, or Insure++.

Featured Post

New benefit for Premium Members - Upgrade now!

Ready to get started with anonymous questions today? It's easy! Learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When you do backups in the Solaris Operating System, the file system must be inactive. Otherwise, the output may be inconsistent. A file system is inactive when it's unmounted or it's write-locked by the operating system. Although the fssnap utility…
A metadevice consists of one or more devices (slices). It can be expanded by adding slices. Then, it can be grown to fill a larger space while the file system is in use. However, not all UNIX file systems (UFS) can be expanded this way. The conca…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Suggested Courses
Course of the Month7 days, 23 hours left to enroll

765 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question