Solved

Locking a process remotely as well as locally

Posted on 2004-04-21
9
305 Views
Last Modified: 2008-02-01
I have a web-based application that uses Perl CGI to provide an interface to execute unix commands remotely. Each set of command has got a name and a given set of commands always runs on a given set of remote machines. I use ssh to connect to the remote machines.

A set of commands cannot be executed by two persons simultaneously. So, two persons accessing the same instance of the web-application cannot execute the same command-set simultaneously. Suppose I have two instances of my web-application running on two different servers, both the servers should not run a given command-set simultaneously.

So, I am using a locking mechanism prevent such simultaneous opertation. I am creating a file called <command-set-name>.lck in a common location. Whenever a new user asks to execute the same command set, I first check if this file exists before allowing the user to execute it.

This mechanism has been working fine so far, but as you can see this is not atomic and so I am scared that the system will not scale. Moreever, we have only one server hosting the application so far. I am afraid the method doesn't work when we have simultaneous servers hostsing it.

I will be grateful to have some advice on how to proceed.
0
Comment
Question by:PerlKing
  • 4
  • 3
  • 2
9 Comments
 
LVL 20

Accepted Solution

by:
jmcg earned 125 total points
ID: 10885738
It sounds like you're looking for a "distributed mutex" mechanism, which is a difficult problem to solve in general.

It's unclear if your boundary for exclusion is to be the entire set of remote UNIX machines, or if it's sufficient to provide exclusion just on each remote UNIX machine as a single unit.

Within the UNIX file system, you use the fact that a 'link' operation is atomic. Create a candidate lock file containing an identifier for the webserver/user combination that is attempting to set the lock. Link that file (a hard link) to the name for the lock file. If the link succeeds AND the contents of the lock file match what you wrote in the candidate lock file, you can proceed, you have obtained the lock. When finished, check the lock file contents again and unlink the file if its contents still match what you wrote. If the contents do not match, report a failure (these failures will most likely come from human or time-based attempts to remove a perceived deadlock -- someone other than the lock owner removed a lock file -- but having the failure report may help sort out what went wrong).
0
 
LVL 3

Author Comment

by:PerlKing
ID: 10885923
Thanks jmcg. Thats Great!! I never thought of a hard link as possible locking mechanism.

>> It's unclear if your boundary for exclusion is to be the entire set of remote UNIX machines, or if it's sufficient to provide exclusion just on each remote UNIX machine as a single unit.

I think the best way to explain is through an example. Suppose I have two web-servers S1 and S2. I have a command-set C1 which is meant to be executed on the set of machines (m1, m2).

Now, when S1 is asked to execute C1, it wil open two parallel SSH connections to m1 and m2 and executes the commands.
If S1 and S2 were simultaneously asked to execute C1, then both of them will connect to both m1 and m2 simultaneously and try to execute the same commands.

Since S1 --> m1 and S1 --> m2 are parallel connections, if S1 --> m1 fails to aquire lock, S1 --> m2 will still try to get the lock because it is not aware that its friend S1 --> m1 failed to get the lock. Similarly what could happen is that S2 --> m1 succeeded and S2 --> m2 failed. In such a scenario both S1 and S2 are only partially successful. However, both will report it as failure since neither of them could complete the entire task successfully. The users who invoked S1 and S2 will be asked to try again later.

I hope these scenarios help answer your question.
0
 
LVL 3

Author Comment

by:PerlKing
ID: 10885932
Adding another 125 points! (unfortunately, I can't afford to be more generous than this)
0
 
LVL 1

Expert Comment

by:joesp
ID: 10886340
the trick is to set the system up so that it doesn't happen that:

"Similarly what could happen is that S2 --> m1 succeeded and S2 --> m2 failed. In such a scenario both S1 and S2 are only partially successful. However, both will report it as failure since neither of them could complete the entire task successfully."

that's why you need to protect your "critical section" wisely.

The old days of using non-atomic file-locking are now over!  and, you don't have to figure out the mess of protecting the critical section becuase the Thread module is here!  

http://www.unix.org.ua/orelly/perl/prog3/ch17_02.htm

No amount of points will save you from having to do a little growing as a programmer by self-teaching.  All I or anybody else can do is point you in the right direction. Without using an existing threads module/library it is hard work to protect critical sections.  I recommend using the existing thread module.

(done preaching)

0
Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

 
LVL 1

Expert Comment

by:joesp
ID: 10886365
Thread::Semaphore

You need some kind of semaphores working along with threads.  I guess in perl threads are unreliable and take extra effort.  Semaphores are used to protect critical sections, ie non-atomic code sections which need be made atomic.  My bad link says there aren't any mutexes in thread perl, so that's why I'm prodding you now :( to look into using ... semaphores on the file lock-ed file.
0
 
LVL 3

Author Comment

by:PerlKing
ID: 10886424
But joesp, this is not a problem that can be solved entirely using Perl.

It involves some locking within the perl application, but at least some part of it has to be done outside the perl application. Whatever is happening on S1 or S2 can be handled using perl but what about the S1 --> m1 and S2 --> m1 connections, for example?
0
 
LVL 1

Assisted Solution

by:joesp
joesp earned 125 total points
ID: 10889544
king of perl,

I am rereading your text:
"Since S1 --> m1 and S1 --> m2 are parallel connections, if S1 --> m1 fails to aquire lock, S1 --> m2 will still try to get the lock because it is not aware that its friend S1 --> m1 failed to get the lock. Similarly what could happen is that S2 --> m1 succeeded and S2 --> m2 failed. In such a scenario both S1 and S2 are only partially successful. However, both will report it as failure since neither of them could complete the entire task successfully. The users who invoked S1 and S2 will be asked to try again later."

what I meant to say above ("you need to protect your critical section wisely") was that it should never happen that S1 --> m1 fails and S2 --> m2 completes.  If one fails to "get the lock", the process should start doing non-blocking IO, or even blocking IO, and poll the locks periodically if it can't get a lock the first time.  This will ensure that the previous actions have completed (in database terms, the dbas would say 'the transaction was committed') in unison.  At your discretion, you may abort after some amount of time.

S1 would hang until S2 was finished, then would execute atomically (by protecting the critical section) its operations on m1 and m2.

I apologize for not having a clearer idea of how to do that.  I mean, I don't have a perl module and corresponding object method calls at my fingertips. However, I can point out that semaphores share a universal, operating system-level global common key among all proceses irrespective of proces lifetime.  I assume that S1 and S2

The project is doable, the logic will be more complex because you will have to coordinate access to both semaphores.  What about something like:

process 1 executes:

     get_resource(m1)
     get_resource(m2)
     -- modify m1 and m2
     release(m1)
     release(m2)


process 2 executes:

     get_resource(m2)
     get_resource(m1)
     -- modify m1 and m2
     release(m2)
     release(m1)


Or, as jmcg writes, as it is a "distributed mutex" problem, maybe a common database or another process which acts as a controller will be required.
0
 
LVL 3

Author Comment

by:PerlKing
ID: 10896458
Thanks jmcg and joesp! Although joesp's solutions is more sophisticated, I may not be able to take up such a complex implementation as of now. For the time being, I think I can live with the "hard link" solution suggested by jcmg. I will keep the distributed mutex implementation for a future release.
0
 
LVL 20

Expert Comment

by:jmcg
ID: 10905797
Sorry, I've been traveling and could not give this question the attention it deserves.

By "boundary", I was wondering whether you needed for S1 to exclude S2 from all of the machines in the set m1, m2, ... or if S1 could work on the machines m1, m2 sequentially, and S2 would be allowed to work on, say, m1 while S1 had moved on to working on m2. Achieving the mutex on each target machine individually is fairly easy, the hard link mechanism I outlined is just one way to do it. If, however, you need to do the exclusion on all of the machines in the set -- and the sets can possibly be different for S1 than for S2 -- or if you have to make things work in the presence of temporary outages of connectivity to the target machines (say m2 is out-of-service for a while, as seen by S1, but comes back into service before S2 tries do do a sequence of commands) ----- then things can get pretty hairy.

You've probably heard of "clusters", whether it is VMS clusters, Tru64 TruClusters, or Linux Beowulf clusters. One of the really hard things about cluster software is creating the distrubuted mechanisms used for mutexes and sequences,  both getting them to be algorithmically correct and getting them to scale reasonably as the number of cluster members goes up.

0

Featured Post

Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
Perl program to obtain a machine's memory usage 6 19
Perl strange behaviour 5 63
syslog unix file 20 61
Perl File::Find alternative 1 25
Many time we need to work with multiple files all together. If its windows system then we can use some GUI based editor to accomplish our task. But what if you are on putty or have only CLI(Command Line Interface) as an option to  edit your files. I…
Email validation in proper way is  very important validation required in any web pages. This code is self explainable except that Regular Expression which I used for pattern matching. I originally published as a thread on my website : http://www…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
This video shows how to remove a single email address from the Outlook 2010 Auto Suggestion memory. NOTE: For Outlook 2016 and 2013 perform the exact same steps. Open a new email: Click the New email button in Outlook. Start typing the address: …

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

16 Experts available now in Live!

Get 1:1 Help Now