Managing Daemon Core File(s)

Hello all,

I have an application that I'm running as a daemon. This application crashes from time to time, and I'd like to get the core file to debug it.
My questions are:

- In which script should I put the ulimt -c unlimited comand so it'll apply to the entire system (including daemons)?

- In which directory will the core file be located?

- Can I control the directory in which the core will be located?

Thanks,

-- Gilad
giligenAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

jlevieCommented:
> In which script should I put the ulimt -c unlimited comand so it'll apply to the entire system

Short of modifying /bin/bash & /bin/sh (which are usually the same executable on Linux) there's no single place that this can be done for the entire system. Each Bash shell that is started defaults to not producing core files since that's built in to the shell. Since the shell that starts a daemon determines whether the daemon will produce a core file on exception abort it is necessary to set 'ulimit -c unlimited' in that shell before starting the daemon. In the general case that's done within the /etc/rc.d/init script that starts the daemon at boot.

> In which directory will the core file be located?

In most cases that will be the root directory (/), but some init scripts may change dirs elsewhere. The core file will be placed in the current working directory of the shell that started the daemon.

> Can I control the directory in which the core will be located?

See above.
0
cjjcliffordCommented:
there is a file /etc/security/limits that can be used to specify different limits for different groups/users.
adding the following:

* soft core 10000
* hard core 100000

will set the core size limits, soft limit to 10000K, hard limit to 100000K - as far as I know, the soft limit is the default without changing using ulimit, and the hard is the maximum limit when changed using ulimit.

Note that a reboot is probably necessary to get these changes enforced...

As for the directory, yup, whatever directory the process is started, which is probably the root directory.
0
giligenAuthor Commented:
First of all thanks for trying to help.

I've tried both of the methods suggested:
- Performed ulimit -c unlimited and then ran my program.
I tried to produce core dump in 3 different ways (assert / Kill -3/ another way)
In all of them the program crashed but there was no core dump in neither the current directory (from which I've started the program) or in the root directory.

Then I tried the /etc/secutiry/limits.conf stuff, I've added to the file * soft core unlimited, and repeated the test(s) above but still I can not find the core dump.

My app, is using the daemon command to change from regaulr executable to daemon status.

Ideas?

Thanks!
0
Upgrade your Question Security!

Your question, your audience. Choose who sees your identity—and your question—with question security.

jlevieCommented:
I assume by that last comment that you mean that you have an init script in /etc/rc.d/init.d that contains something like:

...
    ulimit -c unlimited
    daemon /path-to/my-prog $my-args
...

You won't get a core file in that case because 'daemon' starts a subshell and it will default to no core files. You'd need to do:

...
    ulimit -c unlimited
    /path-to/my-prog $my-args
...

to have a core file produced.
0
giligenAuthor Commented:
Hi,

Upon execution, my program (being started as a regular executable), changes into daemon mode by using the interface int daemon(int nochdir, int noclose);

so what I do is as you've specified
ulimit -c unlimited
./myProg

and I can't get the core dump (in the root directory or elsewhere).

Thanks
0
cjjcliffordCommented:
does your OS user have write-permission to the root directory? If not, and daemon() is called with nochdir=0, then the core file will not get generated, as there is no permission to create the file...
0
giligenAuthor Commented:
It should have, anyway I've re-compiled it with nochdir=1 and it still doesn't work.
BUT - I don't know how and why, it created a core ONCE, and I can not re-produce it again.
0
cjjcliffordCommented:
What caused the core file to be created the one time it did?

I just noticed you say you are trying to cause core dump with a "kill -3" : Signal 3 is "Quit from keyboard" which the application might trap and hide (e.g. Sun's JVM uses "kill -3" to provide stack traces for all running threads!).

Try "kill -6" which is "Abort signal" - the same as calling abort() from within the code (although this should be equivalent to assert() if NDEBUG is not defined a the #include <assert.h> stage...

Cheers,
C.
0
giligenAuthor Commented:
Thanks for the Kill 3 comment.
The other to ways are:
using an assert (that causes a core dump when being run not as a daemon)
and a broken-pipe signal (14).

What I really don't understand is that in the signle time that I did get a core, nothing special was done, I just used the assert to crash it, and for a single time it worked. I know it sounds bad, and you probably say to yourself, "can't be - he must have been doing something else", but I really can't think of anything.

Thanks!
0
cjjcliffordCommented:
had a play... (on RedHat9)

// simple test application

#include <stdio.h>
#include <assert.h>
#include <unistd.h>

int main(void) {
    int rc;
    rc = daemon( 0, 0 );
    if( rc != 0 ) {
        perror( "daemon()" );
        exit(1);
    }
    assert( 0 );
    return 0;
}

compiled this, and ran it (after doing a "ulimit -c unlimited") - note the daemon(0,0)...

When run as non-root user, without permission in "/" no core file created.
when run using "sudo" a core file was created in "/"

then I changed to daemon(1,0)

when run as non-root user, in directory the user has permission, core file was created.


0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
cjjcliffordCommented:
reading "man assert":

       If  the  macro  NDEBUG  was  defined  at the moment <assert.h> was last
       included, the macro assert() generates no code, and hence does  nothing
       at all.

Can you change the assert() call to force coredump to abort() - the macro might not be generating code as described in the man pages....
0
giligenAuthor Commented:
I've tried your sample application and of-course it worked fine, then I started to look and search what are we doing wrong. And after a short research I found out that our applciation uses the SUID/GUID bits. Once these bits are used one must explicitly enable the creation of the core file using the folloing api call:
prctl(PR_SET_DUMPABLE, 1, 0, 0, 0)

Once I've added it, it started to work fine.
Thanks for all your support and help.
0
aathanCommented:
You might also want to check out "man initscript" ... /etc/initscript is used by init to start all processes and can therefore be used to set ulimits on items in inittab.
0
cjjcliffordCommented:
aathan, initscript is used when starting inittab programs (from man "When  the  shell script /etc/initscript is present, init will use it execute the commands from inittab.")
This is not necessarily relevant here as the author didn't state that the process that was being deamon was being run from inittab...
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Fonts Typography

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.