?
Solved

Concurrent directory writes

Posted on 2004-10-14
2
Medium Priority
?
228 Views
Last Modified: 2010-04-17
Hi,

I have a few computers running my software, which writes files on the server. Sometimes, it writes new files to the same directories, and some times tries to create new directories in the same directory.

I get locks, errors, and all kind of strange problems, such as new File(dir).exists() returning false for a directory which really does exist. The language I am using is Java, and the server is Windows 2003.

I wonder if I would have the same problems in Linux?

Thank you.
0
Comment
Question by:kerzner
2 Comments
 
LVL 23

Accepted Solution

by:
brettmjohnson earned 2000 total points
ID: 12314508
Network file systems must make a tradeoff between performance and accuracy
on these issues.  Network file systems cache significant amounts of data on the
client to provide acceptable performance, and those caches may contain stale
information.  Non-atomic transactions such as:

test something;
if (something is true)
  do_this;
else
  do_that;

In concurrent systems, the underlying state of "something" might very well
change between the testing and the doing steps.  These problems can be helped
by doing the following:
1) Use "Atomic Network Operations":
      Rather than have separate network messages for "Check if file exists",
      "Create File" (if not exists), and "Open File"; use an atomic message like:
      "Open File - Create it if non-existant".  This removes the network latency
      from the window between "test" and "do", and the server OS and filesystem
      is structured to push that atomicity down to the low-level disk transactions.
      Consider atomic library calls like tmpfile(), rather than tmpnam().

2) Use file locks and lock files (they are different things) to coordinate transactions:
      - "file locking" is used to coordinate concurrent access to a single file or a
      range of bytes within that file.  This helps avoid problems such as two separate
      processes attempting to update the same record in a file.
      - "lock files" contain little-or-no user data.  They are used as coordination
      primitives - where their presence/absence and/or lockability is used to coordinate
      access to OTHER files.

3) Allocate ranges of filenames among the concurrent processes:
      If all the filenames a process creates have some embedded unique sequence,
      (the process PID or some other GUID), then you won't have filename collisions.
      Similarly, allow each process to create a separate subdirectory (with a GUID
      name), to which only that process writes new files.

0
 
LVL 1

Author Comment

by:kerzner
ID: 12321071
Very good explanation of what is going on inside. My solution at the moment is to have all processing computers form results locally, and then the organizing machine delivers the final results to the server. Seems reasonable in view of the above.

thank you.
0

Featured Post

How to Use the Help Bell

Need to boost the visibility of your question for solutions? Use the Experts Exchange Help Bell to confirm priority levels and contact subject-matter experts for question attention.  Check out this how-to article for more information.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Computer science students often experience many of the same frustrations when going through their engineering courses. This article presents seven tips I found useful when completing a bachelors and masters degree in computing which I believe may he…
We live in a world of interfaces like the one in the title picture. VBA also allows to use interfaces which offers a lot of possibilities. This article describes how to use interfaces in VBA and how to work around their bugs.
With the power of JIRA, there's an unlimited number of ways you can customize it, use it and benefit from it. With that in mind, there's bound to be things that I wasn't able to cover in this course. With this summary we'll look at some places to go…
Introduction to Processes

840 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question