Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium


Handling stack overflow in Linux

Posted on 2004-10-15
Medium Priority
Last Modified: 2012-08-13

I need to handle stack overflow in Linux from programs written in C/C++. I've surprisingly found that it's not not easy... In Windows 2000 I'm using structured exception handling for this purpose, but what mechanism should be used in Linux? Writing handlers for all signals that can be handled didn't help me. The result of program execution is always Segmentation fault.

Thanks in advance,
Question by:Foma
LVL 12

Expert Comment

ID: 12318014
Hi Foma,
I'm not aware of any tools which prevent stack overflows. You can set the stack limit with "limit" or "ulimit", but you cannot really detect them as such.

What are you putting on the stack which causes an overflow? Normally it's better to revise the initial design than to chase after error symptoms later.



Author Comment

ID: 12318177
Hi Stefan!
Actually, lots of program need to handle stack overflow. For example i'm writing interpreter for a programming language. And of course, a user can write a program in this language that will cause infinite number of function calls, which leads to stack overflow.
Hm, I'm sure Java virtual machine, for instance, handle this problem correctly on any platform, including Linux.

Best regards,
LVL 12

Expert Comment

ID: 12321389
True, but I don't think the Java virtual machine uses the system's stack for calling java methods.

Your interpreter should probably not recursive itself. I'm quite sure it is possible to calculate the maximum stack size.

A simple method is to call
pmap $$

...and then you see the current process's stack size + location. You can then easily add a call to check the stack to each function invocation. Make sure you have a bit of stack to spare when you trigger the overflow exception.

That's a classic for a simple assembly subroutine, something like (GCC's AT&T syntax):
    cmp $_stack_limit,%esp
    ble _overflow

...whereas "overflow" is a C routine which handles the stack overflow itself. You could even have the ASM in a little macro which you call in all the critical functions, as the overhead is minimal.

Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.


Accepted Solution

mtmike earned 375 total points
ID: 12374921
Catching SIGSEGV should work, except that signal handlers are executed on the current stack. So, when the stack overflows the signal handler will cause a segmentation fault too. To avoid this, you will have to establish an alternate signal stack using sigaltstack() and pass the SA_ONSTACK flag to sigaction().

Assisted Solution

Sadrul earned 150 total points
ID: 12377291
you can also give libsigsegv a try. here's the URL: http://libsigsegv.sourceforge.net

-- Adil
LVL 12

Expert Comment

ID: 12409108
I strongly discourage catching segmentation faults. Chances are high that you get hundreds of them in a very short time. And you have no guarantee that they are related to your stack problem.

The worst case would be that you're missing a core dump which was created by a NULL reference or something similar, and pass on to the user a totally wrong "stack overflow message".

Author Comment

ID: 12433426
Thank you very much for your responces!
To mtmike:
Thank you. Simple solution and it works. Unfortunately I cannot distinguish stack overflow from other segmentation violations…
To Sadrul:
Nice library. It allows handling exactly stack overflows. It distinguishes arbitrary segmentation fault and stack overflow using some heuristics (I saw it in its source code, but I didn’t manage to look into it yet). Unfortunately, this library is not suitable to me because I need to return from signal handler to program automatically, what is achieved by using sigaction with SA_SIGINFO. But the library uses sigaction w/o this flag for Linux…
To stefan73:
I understand your position, but I can’t agree with you. Yes, in general it’s better not to get segmentation faults at all and write programs in such a manner that you will never get access violation. But sometimes it’s very hard or impossible… Concerning my program, I do my own memory management and SIGSEGV signal (when the address in selected diapason) is normal behavior of the program. When SIGSEGV signal is got I perform mmap call to correct the problem. In other case it can be stack overflow or error in program. I can suspect that it is stack overflow and make an effort to release resources. If during this step SIGSEGV arises, that means error in program and I just terminate the program… I think it not an ideal solution, but it is satisfactory for me

Featured Post

Get expert help—faster!

Need expert help—fast? Use the Help Bell for personalized assistance getting answers to your important questions.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The purpose of this article is to fix the unknown display problem in Linux Mint operating system. After installing the OS if you see Display monitor is not recognized then we can install "MESA" utilities to fix this problem or we can install additio…
The purpose of this article is to demonstrate how we can upgrade Python from version 2.7.6 to Python 2.7.10 on the Linux Mint operating system. I am using an Oracle Virtual Box where I have installed Linux Mint operating system version 17.2. Once yo…
This video shows how to quickly and easily deploy an email signature for all users in Office 365 and prevent it from being added to replies and forwards. (the resulting signature is applied on the server level in Exchange Online) The email signat…
Integration Management Part 2
Suggested Courses

571 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question