• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 792
  • Last Modified:

Using OS X launchd issue

We have a script that performs some simple work as part of a larger backup scheme. It has been running for years with no problems on a FreeBSD server. Now we are adapting it to run on a Mac. We have been having lot of problems trying to get it to work.

The key part of the script uses duplicity to create an encrypted file set of the backup. The problem that we are having is that the script runs find from a shell (Terminal), but will not run under launchd. We have pinned the problem down to an area described by our sysadmin as follows:

I discovered that the problem is in fact that the ulimit is still defaulting to a value that is too low.  I had thought we solved that by adding ulimit -n 1024 to bashrc, but it doesn't appear to be taking.  There is talk out there that setting limit maxfiles 1024 unlimited
in /etc/launchd.conf

While probably not essential to solving the problem, I am attaching a code snippet for the code that is failing. Please note: since this script is part of a larger script, it is not runnable in the form as shown.

""" backup.py

This script shuffles around and tests the backups on nas.BackupDrive.net


import os, sys, smtplib, time, MySQLdb, shutil

os.environ['HOME'] = "/Users/bkupuser"
os.environ['TMPDIR'] = "/Volumes/BackupDrive/tmp"
os.environ['GNUPGHOME'] = "/Users/bkupuser"
os.environ['PASSPHRASE'] = "Some_clever_password"
os.environ['PATH'] = "/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/local/bin:/opt/local/sbin"

DAYOFMONTH = int(time.strftime("%d"))

BACKUP_DIR = "/Volumes/BackupDrive/backups/bkupuser"
USB_DIR = "/Volumes/FLASH"

# Only copy files to flash drive every 3 days
TEST = 1
if TEST == 1:
    if os.system("diskutil mount /dev/disk2s1 | grep %s" % USB_DIR):
        ret = os.system("diskutil mount /dev/disk2s1")
        if ret:
            nonfatal_error("Couldn't mount %s" % USB_DIR)
            flash_size = 0
            flash_size = get_space()
            if not os.path.isdir(os.path.join(USB_DIR, DATE)):
                os.mkdir(os.path.join(USB_DIR, DATE))
        flash_size = get_space()

    if flash_size:  # A clever way to determine if we actually mounted /flash
        setsize = 0
        filelist = os.listdir(os.path.join(BACKUP_DIR, DATE))
        for file in filelist:
            setsize = setsize + os.stat(os.path.join(BACKUP_DIR , DATE, file)).st_size

        if setsize < flash_size:
            ret = os.system("/opt/local/bin/duplicity %s/%s 'file://%s/%s/'" % \
                           (BACKUP_DIR, DATE, USB_DIR, DATE))

Open in new window

  • 8
  • 8
2 Solutions
1) launchd not involved AT ALL
2) OSX locks open files just like windows - you have to stop MySQL to backup data files (or make a replica server which can be stopped without touching master database(and used to improve data retrieval performance))

jasimon9Author Commented:
I believe you are misreading the question.

Here is in a nutshell what is being done:

1. Big script when run standalone runs fine. Duplicity copies the files to the flash drive.

2. Big script when run from launchd fails. Duplicity does not copy files.

Mysql is not even involved in this question. It is duplicity from a script that works, but not when run from launchd.

I am wondering why you are referring the mysql, as it is not even in the snippet?
MySQL is in topic areas snippels (btw none of them is relevant to your question)

Assuming launchd is a genial rename of dinosaur age cron, script fails when it tries to wrote output.

Thus invoke
duplicity </dev/null >/dev/nukk 2>&1
Get 10% Off Your First Squarespace Website

Ready to showcase your work, publish content or promote your business online? With Squarespace’s award-winning templates and 24/7 customer service, getting started is simple. Head to Squarespace.com and use offer code ‘EXPERTS’ to get 10% off your first purchase.

jasimon9Author Commented:
Thanks for your response. We are not using cron, and launchd does not work like cron.
jasimon9Author Commented:
Additional clarification:

We think the issue is most likely that the ulimit is not getting set to 1024, which is what the error log shows as the problem.

We have added ulimit -n 1024 to bashrc, but it does not seem to be having the desired effect.

That's where we are stuck.

sudo launchctl limit maxfiles 65536 65536
jasimon9Author Commented:
We may have solved the problem as it seems to have worked the last few times.

I am not sure whether this required the help from the last responder until I talk to our sysadmin. Of course, if that suggestion (to use "sudo launchctl limit maxfiles 65536 65536") was the key, points will be awarded. Plus points split to any other contributors whose answer provided assistance.

That of course depends upon what the solution actually was.
probaly 100x more file descriptors led to the situation when backup emitted actual root cause error.

It should not be an essence of a solution. OSX system (Leopard 10.5 PPC without extra products) contains ~100000 files and i cannot imagine what backup algorythm would like to open half of them at any given time (when normal user never hits default 1000 file limit or system boots reading ~2000 files).

Maybe duplicity is broken (the particular duplicity is built on more recent OSX than you have for example)?
(or it is a pre-universal binary for PPC and runs in interpreter for example)

i.e i would like to find more about how it got there.
jasimon9Author Commented:
The backup is not of "all the files on the Mac". Instead it is a specific set of about 30 files from a production server. However, these files are tarballs, with lots of files within them.

However, it seems to me that as far as duplicity is concerned, it is just 30 files.

By the way, it appears we may have a solution.  We ended up correcting the python script to include "ulimit -n 1024; duplicity..."

So far it appears to be a solution. However, it might be that the suggestion of "sudo launchctl limit maxfiles 65536 65536" may also be effective. However, the ulimit approach seems more specific to our need, and in fact how to implement the ulimit was part of the original question.
ulimit does not let common user (non-root) to rise limit, thus suggestion to set default to enormous, so that you can adjust as you want.
jasimon9Author Commented:
Possibly a good suggestion in some cases, but not necessary in our case.

Here is the working solution, which is the line of the python script that we have been using:

    ret = os.system("ulimit -n 1024;/opt/local/bin/duplicity %s/%s \
                           'file://%s/%s/'" % \
                           (BACKUP_DIR, DATE, USB_DIR, DATE))

So this is working. Without the ulimit in the command it fails, and with ulimit in the command it works. And this solves the original question, which effectively asks how to get ulimit to be effective.

Our sysadmin says that "anything LaunchDaemon runs is a root." And further that

Yeah…there are kind of two ways to go after it…have Launchd raise the limit for the whole script or within the script raise the ulimit at the time duplicity is run.  We opted for raising the ulimit from within the script.

jasimon9Author Commented:
Awarding some points for an alternate solution that probably would work although we did not test it.

The actual solution we found on our own is simpler and effective, as stated.
i would gzip and split and then copy parts to USB "if there still is some space"
uses like 20 file descriptors in total...
jasimon9Author Commented:
gheist: I awarded 100 points for the untested but potentially workable solution you provided.

We have a working solution. I am not sure why you are pursuing this further, except that our posts probably "crossed".

duplicity is a program that archives and encrypts. I creates about 30 files. I think we are done with this question.

Hope this helps.
I did not cancel the closure. I just suggested some alternative if .py script gets stuck again.
(no objections against such closure, still I would suggest to recompile failing software on particular system)

tar xf - $BACKUP_DIR | gzip -1 |  split -b 100MB - /tmp/.$?/.....
and then use your logic to move chunks to USBs
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Build your data science skills into a career

Are you ready to take your data science career to the next step, or break into data science? With Springboard’s Data Science Career Track, you’ll master data science topics, have personalized career guidance, weekly calls with a data science expert, and a job guarantee.

  • 8
  • 8
Tackle projects and never again get stuck behind a technical roadblock.
Join Now