Solved

Concurrent directory writes

Posted on 2004-10-14
2
213 Views
Last Modified: 2010-04-17
Hi,

I have a few computers running my software, which writes files on the server. Sometimes, it writes new files to the same directories, and some times tries to create new directories in the same directory.

I get locks, errors, and all kind of strange problems, such as new File(dir).exists() returning false for a directory which really does exist. The language I am using is Java, and the server is Windows 2003.

I wonder if I would have the same problems in Linux?

Thank you.
0
Comment
Question by:kerzner
2 Comments
 
LVL 23

Accepted Solution

by:
brettmjohnson earned 500 total points
ID: 12314508
Network file systems must make a tradeoff between performance and accuracy
on these issues.  Network file systems cache significant amounts of data on the
client to provide acceptable performance, and those caches may contain stale
information.  Non-atomic transactions such as:

test something;
if (something is true)
  do_this;
else
  do_that;

In concurrent systems, the underlying state of "something" might very well
change between the testing and the doing steps.  These problems can be helped
by doing the following:
1) Use "Atomic Network Operations":
      Rather than have separate network messages for "Check if file exists",
      "Create File" (if not exists), and "Open File"; use an atomic message like:
      "Open File - Create it if non-existant".  This removes the network latency
      from the window between "test" and "do", and the server OS and filesystem
      is structured to push that atomicity down to the low-level disk transactions.
      Consider atomic library calls like tmpfile(), rather than tmpnam().

2) Use file locks and lock files (they are different things) to coordinate transactions:
      - "file locking" is used to coordinate concurrent access to a single file or a
      range of bytes within that file.  This helps avoid problems such as two separate
      processes attempting to update the same record in a file.
      - "lock files" contain little-or-no user data.  They are used as coordination
      primitives - where their presence/absence and/or lockability is used to coordinate
      access to OTHER files.

3) Allocate ranges of filenames among the concurrent processes:
      If all the filenames a process creates have some embedded unique sequence,
      (the process PID or some other GUID), then you won't have filename collisions.
      Similarly, allow each process to create a separate subdirectory (with a GUID
      name), to which only that process writes new files.

0
 
LVL 1

Author Comment

by:kerzner
ID: 12321071
Very good explanation of what is going on inside. My solution at the moment is to have all processing computers form results locally, and then the organizing machine delivers the final results to the server. Seems reasonable in view of the above.

thank you.
0

Featured Post

Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Does the idea of dealing with bits scare or confuse you? Does it seem like a waste of time in an age where we all have terabytes of storage? If so, you're missing out on one of the core tools in every professional programmer's toolbox. Learn how to …
Although it can be difficult to imagine, someday your child will have a career of his or her own. He or she will likely start a family, buy a home and start having their own children. So, while being a kid is still extremely important, it’s also …
An introduction to basic programming syntax in Java by creating a simple program. Viewers can follow the tutorial as they create their first class in Java. Definitions and explanations about each element are given to help prepare viewers for future …
Viewers will learn how to properly install Eclipse with the necessary JDK, and will take a look at an introductory Java program. Download Eclipse installation zip file: Extract files from zip file: Download and install JDK 8: Open Eclipse and …

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now