We help IT Professionals succeed at work.

Setting that could cause Apache to grab tape drive?

denbosse
denbosse asked
on
259 Views
Last Modified: 2011-10-19
I have a problem where my backup software and the operating system are no longer able to access a tape drive attached to the server. The OS is SLES 10 x64 - no service pack.

I've narrowed down the problem to the use of pre/post job commands in the backup software.
As soon as I shut down Apache in the pre-job command and start it again in the post-job command, the next backup job will fail to access the tape drive.
lsof shows that httpd2 is accessing the corresponding /dev/sg device. tar will also fail to write to the tape drive at the point in time. tar will fail with "Invalid Argument".

The main question I have is the following: the backup vendor claims there is a setting in the Apache configuration that can be changed to avoid this behaviour. However, at the moment they cannot tell me what that setting is. I've not found anything after extensive searching. Does somebody know about such a setting?
Comment
Watch Question

Duncan RoeSoftware Developer
CERTIFIED EXPERT

Commented:
Why are you using /dev/sg and not /dev/[n]st? I am not at all surprised that tar gets an error writing to sg
cjl7freelance for hire

Commented:
It sounds very strange that Apache should access that device?!

Have you grep:ed for /dev/sg in the startup file and in the /etc/apache (or /etc/httpd) directories?

If you start apache manually will the same error occur?

Which backup application are you running?

//jonas

Author

Commented:
duncan_roe, the application is Yosemite Backup and it interacts with sg devices. The tar command was tested with the corresponding st device.

cjl7, using the pre/post scripts outside of the backup software was the next thing I was going to test. I will test and also check with grep. I'll report back.

Author

Commented:
I've done the tests. Grep for /dev/sg in /etc/apache2 does not return any results.

When I run the stop and start scripts on their own from the shell, none of the processes have /dev/sg1 open - i.e. lsof show no access to /dev/sg1.

But when I execute the same script as pre/post scripts with Yosemite Backup, lsof shows that almost all of the processes I stop&restart have access to /dev/sg1  (mysqld, nmbd, smbd, httpd2-prefork).

If I then run the stop and start scripts from the shell, all processes - except httpd2-prefork - no longer access the device. But httpd2-prefork, keeps accessing the device.
It looks like Apache no longer stops properly at this point. See attachment.

Any advise? Any conclusions you would draw?


output.txt
Duncan RoeSoftware Developer
CERTIFIED EXPERT

Commented:
"almost all of the processes I stop&restart have access to /dev/sg1" - it looks to me like the process that starts these processes opens /dev/sg1 (not sg0?) and doesn't set "close on exec" flag, so all forked processes have the device open too. If you are unlucky, the forking process then exited, but you may be luckier than that, so concentrate on the process with the lowest pid that has sg1 open.
(Forked processes always inherit all the parent's file units, but usually the first thing a forked process does is exec() some other application - this is where "close on exec" comes in. The flag is off on first opening a file).
Duncan RoeSoftware Developer
CERTIFIED EXPERT

Commented:
Just noticed output.txt - pid 11309 has /dev/sg1 open so all its forked processes will have it as well (they don't exec()). You have to somehow get that top-level process to close sg1 before spawning children else kill it completely and then restart. You only show httpd2-prefork processes with /dev/sg1, not mysqld, nmbd & smbd. Under what circumstances do these latter processes get it as well?

Author

Commented:
lsof shows that all processes (nmbd, smb, mysqld and httpd2-prefork) have access to /dev/sg1 after starting all those processes through a Yosemite Backup post-job command. See output2.txt attached.

But if you then stop and restart nmbd, smb, mysqld from the OS, they no longer access /dev/sg1.

However it looks like when you try to stop Apache after it was started with Yosemite Backup, it can't gracefully stop. So it also keeps the access to /dev/sg1 open.

When you mentioned the "close on exec" flag and the forking process, I suppose that one of the Yosemite Backup processes was the forking process, but that it has exited in the meantime. And I would assume that this Yosemite Backup process does not have the "close on exec" flag set.

Would you agree?
output2.txt
Duncan RoeSoftware Developer
CERTIFIED EXPERT

Commented:
Yes I would say that looks very likely. Assuming that Yosemite Backup ran as root, and also that the name of the executable was different from any shown in output2.txt.
Software Developer
CERTIFIED EXPERT
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION
Unlock the solution to this question.
Join our community and discover your potential

Experts Exchange is the only place where you can interact directly with leading experts in the technology field. Become a member today and access the collective knowledge of thousands of technology experts.

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.