Solved

Concurrent directory writes

Posted on 2004-10-14
2
218 Views
Last Modified: 2010-04-17
Hi,

I have a few computers running my software, which writes files on the server. Sometimes, it writes new files to the same directories, and some times tries to create new directories in the same directory.

I get locks, errors, and all kind of strange problems, such as new File(dir).exists() returning false for a directory which really does exist. The language I am using is Java, and the server is Windows 2003.

I wonder if I would have the same problems in Linux?

Thank you.
0
Comment
Question by:kerzner
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
2 Comments
 
LVL 23

Accepted Solution

by:
brettmjohnson earned 500 total points
ID: 12314508
Network file systems must make a tradeoff between performance and accuracy
on these issues.  Network file systems cache significant amounts of data on the
client to provide acceptable performance, and those caches may contain stale
information.  Non-atomic transactions such as:

test something;
if (something is true)
  do_this;
else
  do_that;

In concurrent systems, the underlying state of "something" might very well
change between the testing and the doing steps.  These problems can be helped
by doing the following:
1) Use "Atomic Network Operations":
      Rather than have separate network messages for "Check if file exists",
      "Create File" (if not exists), and "Open File"; use an atomic message like:
      "Open File - Create it if non-existant".  This removes the network latency
      from the window between "test" and "do", and the server OS and filesystem
      is structured to push that atomicity down to the low-level disk transactions.
      Consider atomic library calls like tmpfile(), rather than tmpnam().

2) Use file locks and lock files (they are different things) to coordinate transactions:
      - "file locking" is used to coordinate concurrent access to a single file or a
      range of bytes within that file.  This helps avoid problems such as two separate
      processes attempting to update the same record in a file.
      - "lock files" contain little-or-no user data.  They are used as coordination
      primitives - where their presence/absence and/or lockability is used to coordinate
      access to OTHER files.

3) Allocate ranges of filenames among the concurrent processes:
      If all the filenames a process creates have some embedded unique sequence,
      (the process PID or some other GUID), then you won't have filename collisions.
      Similarly, allow each process to create a separate subdirectory (with a GUID
      name), to which only that process writes new files.

0
 
LVL 1

Author Comment

by:kerzner
ID: 12321071
Very good explanation of what is going on inside. My solution at the moment is to have all processing computers form results locally, and then the organizing machine delivers the final results to the server. Seems reasonable in view of the above.

thank you.
0

Featured Post

Free Tool: Postgres Monitoring System

A PHP and Perl based system to collect and display usage statistics from PostgreSQL databases.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
C# Error - Add Failed 12 99
Removing line numbers from left column in config file 7 73
Help Required 2 46
CSV file copy field 1 to field 2 2 42
A short article about problems I had with the new location API and permissions in Marshmallow
Today, the web development industry is booming, and many people consider it to be their vocation. The question you may be asking yourself is – how do I become a web developer?
Viewers will learn how to properly install Eclipse with the necessary JDK, and will take a look at an introductory Java program. Download Eclipse installation zip file: Extract files from zip file: Download and install JDK 8: Open Eclipse and …

726 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question