• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 230
  • Last Modified:

Concurrent directory writes


I have a few computers running my software, which writes files on the server. Sometimes, it writes new files to the same directories, and some times tries to create new directories in the same directory.

I get locks, errors, and all kind of strange problems, such as new File(dir).exists() returning false for a directory which really does exist. The language I am using is Java, and the server is Windows 2003.

I wonder if I would have the same problems in Linux?

Thank you.
1 Solution
Network file systems must make a tradeoff between performance and accuracy
on these issues.  Network file systems cache significant amounts of data on the
client to provide acceptable performance, and those caches may contain stale
information.  Non-atomic transactions such as:

test something;
if (something is true)

In concurrent systems, the underlying state of "something" might very well
change between the testing and the doing steps.  These problems can be helped
by doing the following:
1) Use "Atomic Network Operations":
      Rather than have separate network messages for "Check if file exists",
      "Create File" (if not exists), and "Open File"; use an atomic message like:
      "Open File - Create it if non-existant".  This removes the network latency
      from the window between "test" and "do", and the server OS and filesystem
      is structured to push that atomicity down to the low-level disk transactions.
      Consider atomic library calls like tmpfile(), rather than tmpnam().

2) Use file locks and lock files (they are different things) to coordinate transactions:
      - "file locking" is used to coordinate concurrent access to a single file or a
      range of bytes within that file.  This helps avoid problems such as two separate
      processes attempting to update the same record in a file.
      - "lock files" contain little-or-no user data.  They are used as coordination
      primitives - where their presence/absence and/or lockability is used to coordinate
      access to OTHER files.

3) Allocate ranges of filenames among the concurrent processes:
      If all the filenames a process creates have some embedded unique sequence,
      (the process PID or some other GUID), then you won't have filename collisions.
      Similarly, allow each process to create a separate subdirectory (with a GUID
      name), to which only that process writes new files.

kerznerAuthor Commented:
Very good explanation of what is going on inside. My solution at the moment is to have all processing computers form results locally, and then the organizing machine delivers the final results to the server. Seems reasonable in view of the above.

thank you.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Keep up with what's happening at Experts Exchange!

Sign up to receive Decoded, a new monthly digest with product updates, feature release info, continuing education opportunities, and more.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now