Handling stack overflow in Linux

Posted on 2004-10-15
Last Modified: 2012-08-13

I need to handle stack overflow in Linux from programs written in C/C++. I've surprisingly found that it's not not easy... In Windows 2000 I'm using structured exception handling for this purpose, but what mechanism should be used in Linux? Writing handlers for all signals that can be handled didn't help me. The result of program execution is always Segmentation fault.

Thanks in advance,
Question by:Foma
LVL 12

Expert Comment

Comment Utility
Hi Foma,
I'm not aware of any tools which prevent stack overflows. You can set the stack limit with "limit" or "ulimit", but you cannot really detect them as such.

What are you putting on the stack which causes an overflow? Normally it's better to revise the initial design than to chase after error symptoms later.



Author Comment

Comment Utility
Hi Stefan!
Actually, lots of program need to handle stack overflow. For example i'm writing interpreter for a programming language. And of course, a user can write a program in this language that will cause infinite number of function calls, which leads to stack overflow.
Hm, I'm sure Java virtual machine, for instance, handle this problem correctly on any platform, including Linux.

Best regards,
LVL 12

Expert Comment

Comment Utility
True, but I don't think the Java virtual machine uses the system's stack for calling java methods.

Your interpreter should probably not recursive itself. I'm quite sure it is possible to calculate the maximum stack size.

A simple method is to call
pmap $$

...and then you see the current process's stack size + location. You can then easily add a call to check the stack to each function invocation. Make sure you have a bit of stack to spare when you trigger the overflow exception.

That's a classic for a simple assembly subroutine, something like (GCC's AT&T syntax):
    cmp $_stack_limit,%esp
    ble _overflow

...whereas "overflow" is a C routine which handles the stack overflow itself. You could even have the ASM in a little macro which you call in all the critical functions, as the overhead is minimal.

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline


Accepted Solution

mtmike earned 125 total points
Comment Utility
Catching SIGSEGV should work, except that signal handlers are executed on the current stack. So, when the stack overflows the signal handler will cause a segmentation fault too. To avoid this, you will have to establish an alternate signal stack using sigaltstack() and pass the SA_ONSTACK flag to sigaction().

Assisted Solution

Sadrul earned 50 total points
Comment Utility
you can also give libsigsegv a try. here's the URL:

-- Adil
LVL 12

Expert Comment

Comment Utility
I strongly discourage catching segmentation faults. Chances are high that you get hundreds of them in a very short time. And you have no guarantee that they are related to your stack problem.

The worst case would be that you're missing a core dump which was created by a NULL reference or something similar, and pass on to the user a totally wrong "stack overflow message".

Author Comment

Comment Utility
Thank you very much for your responces!
To mtmike:
Thank you. Simple solution and it works. Unfortunately I cannot distinguish stack overflow from other segmentation violations…
To Sadrul:
Nice library. It allows handling exactly stack overflows. It distinguishes arbitrary segmentation fault and stack overflow using some heuristics (I saw it in its source code, but I didn’t manage to look into it yet). Unfortunately, this library is not suitable to me because I need to return from signal handler to program automatically, what is achieved by using sigaction with SA_SIGINFO. But the library uses sigaction w/o this flag for Linux…
To stefan73:
I understand your position, but I can’t agree with you. Yes, in general it’s better not to get segmentation faults at all and write programs in such a manner that you will never get access violation. But sometimes it’s very hard or impossible… Concerning my program, I do my own memory management and SIGSEGV signal (when the address in selected diapason) is normal behavior of the program. When SIGSEGV signal is got I perform mmap call to correct the problem. In other case it can be stack overflow or error in program. I can suspect that it is stack overflow and make an effort to release resources. If during this step SIGSEGV arises, that means error in program and I just terminate the program… I think it not an ideal solution, but it is satisfactory for me

Featured Post

6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

Join & Write a Comment

The purpose of this article is to fix the unknown display problem in Linux Mint operating system. After installing the OS if you see Display monitor is not recognized then we can install "MESA" utilities to fix this problem or we can install additio…
The purpose of this article is to demonstrate how we can upgrade Python from version 2.7.6 to Python 2.7.10 on the Linux Mint operating system. I am using an Oracle Virtual Box where I have installed Linux Mint operating system version 17.2. Once yo…
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…
This video gives you a great overview about bandwidth monitoring with SNMP and WMI with our network monitoring solution PRTG Network Monitor ( If you're looking for how to monitor bandwidth using netflow or packet s…

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now