Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

RandomAccessFile Read optimization

Posted on 2005-04-11
10
Medium Priority
?
900 Views
Last Modified: 2012-08-13
Hi All,

 
I am using a test case where I am opening a RandomAccessFile , read some bytes and

close the File. Output of this testcase gives different result on each plateform.

 
This test case simply observes the performance at DB layer.

On Window machine after reading n no of messages suddenly output time is increased. Same behavior is
found on solaris and On linux  this behavior is competely different. No slow down is observed on linux.


Ouput on windows machine as follows:
==================================

Time Taken by thread main Chunk messages 100 Total Mesages 1300 Time is --->63
Time Taken by thread main Chunk messages 100 Total Mesages 1400 Time is --->31
Time Taken by thread main Chunk messages 100 Total Mesages 1500 Time is --->47
Time Taken by thread main Chunk messages 100 Total Mesages 1600 Time is --->47
Time Taken by thread main Chunk messages 100 Total Mesages 1700 Time is --->46
Time Taken by thread main Chunk messages 100 Total Mesages 1800 Time is --->32
Time Taken by thread main Chunk messages 100 Total Mesages 1900 Time is --->47
Time Taken by thread main Chunk messages 100 Total Mesages 2000 Time is --->625
Time Taken by thread main Chunk messages 100 Total Mesages 2100 Time is --->1797
Time Taken by thread main Chunk messages 100 Total Mesages 2200 Time is --->2547
Time Taken by thread main Chunk messages 100 Total Mesages 2300 Time is --->2203
Time Taken by thread main Chunk messages 100 Total Mesages 2400 Time is --->2359
Time Taken by thread main Chunk messages 100 Total Mesages 2500 Time is --->3047
Time Taken by thread main Chunk messages 100 Total Mesages 2600 Time is --->1734
Time Taken by thread main Chunk messages 100 Total Mesages 2700 Time is --->2953
Time Taken by thread main Chunk messages 100 Total Mesages 2800 Time is --->3454
Time Taken by thread main Chunk messages 100 Total Mesages 2900 Time is --->3859
Time Taken by thread main Chunk messages 100 Total Mesages 3000 Time is --->2547
 

After a lot of analysis it is deducted that this behavior is based on disk behavior. As each randomAccessFile operation directly
involves Disk IO operaion .

If some body knows how we can solve this problem  Or Is there any other Disk optimization paramters by which we can minimize
this time for RandomAccessFile,.let me know about it. Such solution might be helpful in increaing the performance at DB layer ??????


*********Code snippet*********

                    randomFile = new RandomAccessFile (fileName, "rw");

                    byte[] buffer = new byte[1024];

                    randomFile.read(buffer);

                    randomFile.close();
*********************************

Thanks
-Rajesh
0
Comment
Question by:rajesh042
  • 5
  • 2
8 Comments
 
LVL 15

Expert Comment

by:JakobA
ID: 13751894
Have you tried running your test in reverse ? from 3000 meassages first down to 1300 last.I am not convinced that the behaviour really depend on nr of messages.
0
 
LVL 15

Expert Comment

by:JakobA
ID: 13752003
Do you release the memory used by old messages or do they accumulate in some list ?  If they accumulate you could get an effect like that when the java process runs out of some resource and have to reconfigure itself. fx the heap run out of free space and java have to allocate and move memory to a new and bigger heap.
0
 

Author Comment

by:rajesh042
ID: 13752244
This is my test code:- let me know if there is something I am doinf wrong?

public static void main(String[] args)
    {
        int rowIndex = 1;
        RandomAccessFile randomFile = null;
        int m_nDisplayRateAfter= 100;
        String fileName = null;
            boolean first = true;

        long startTime = System.currentTimeMillis();
        while(true)       {
            fileName =       "test" + rowIndex + ".dat";
            File file = new File(fileName);
            if(file.exists())   {
                try  {
                    randomFile = new RandomAccessFile (fileName, "rw");
                    byte[] buffer = new byte[1024];
                    randomFile.read(buffer);
                    randomFile.close();
                }
                catch (Exception ex)                {
                    System.out.println("Exception is generated in randomFile");
                    ex.printStackTrace();
                }
                rowIndex++;

                try      {
                    if (true && rowIndex % m_nDisplayRateAfter == 0)    {
                        _showRate (startTime, rowIndex);
                        startTime = System.currentTimeMillis ();
                    }
                }
                catch (Exception ex1)               {
                    ex1.printStackTrace ();
                }
            }
            else            {
                System.out.println("File is not existing "+ fileName);
                rowIndex = 1;
                System.out.println("Read all Files Starting again");
                startTime = System.currentTimeMillis ();
            }
        }
    }

    private static void _showRate(long startTime, long msgCount)
        throws Exception    
   {
        long diff = System.currentTimeMillis () - startTime;
        System.out.println ("Time Taken by thread " +
                            Thread.currentThread ().getName () +
                            " Chunk messages 100 " + "Total Mesages " +
                            msgCount + " Time is --->" + diff);
    }
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 15

Accepted Solution

by:
JakobA earned 500 total points
ID: 13757211
You are not doing anything wrong, but then again you are not really using random access files. All you do is read the first 1024 bytes from a lot of different files.

I see no reason for the very different timings.

At one time the micrsoft filesystem had a limit on the number of files you could have in a directory.
So I would still recommend that you try the test in 'reverse', starting rowIndex at 3000 and counting down to 0. It could be that the files with large numbers are more difficult to get at because they were added past those old limits and are accessed through some workaround. (but it should not give such very drastic diffences).

If you are in control of the program generating all those little files, You should change to actually using a 'random access file' with each message assigned to a diffent index in that single file.
0
 
LVL 15

Expert Comment

by:JakobA
ID: 13758623
Here is a small class for administering a random access files with fixed size records. it is untested so expect syntax errors and misspellings.

file structure is:
    <4 byte blocklength> <any nr of blocks>
block structure is
    <4 byte blockdata length>  <that many bytes of data> <unspecified bytes to fill up the block>

class RandomAccessBlockFile {

  private RandomAccessFile theFile;
  private String                   filePath;          // filename plus path info as needed
  private int                        blockLength;    // nr of bytes reserved for each record
  private boolean                writeable;        // true if opened for writing

  public RandomAccessBlockFile ( String filePath, String rwCode, int blockLength ) {
    if ( rwCode.equals("r") || rwCode.equals("rw") ) {
        File file = new File( filePath );
        if ( file.exists() ) {
            theFile = new RandomAccessFile( filePath, rwCode );
            int fileBlockLength = theFile.readInt();
            if ( blockLength <= 0 ) {
                blockLength = fileBlockLength;
            }
            if ( blockLength != fileBlockLength ) {
                    // ERROR situation. existing file have another blocklength
                throw new Exception( "wrong block length given for block file"
                                              +"\nfile " +filePath
                                              +" blockLength is " +fileBlockLength );
            }
        } else {
            if ( rwCode.equals("rw") ) {
                if ( blockLength <= 0 ) {
                    throw new Exception( "Illegal block length specified for new Block file" );
                }
                theFile = new RandomAccessFile( filePath, rwCode );
                theFile.writeInt( blockLength );
            } else {
                throw new Exception( "Block file requested does not exist." );
            }
        }
        if ( rwCode.equals("r") ) {
            File file = new File( filePath );
            if ( file.exists()
        }
        this.filePath = filePath;
        this.blocklength = blockLength;
        writeable = rwCode.equals("rw");
    } else {
            // ERROR situation. illegal mode specified.
        throw new Exception( "Illegal mode for Block file. only \"r\" and \"rw\" permitted." );
    }
  }//endconstructor

  private void pointTo ( int blockNr ) {
     while ( theFile.length() < ( 4 + blockNr*(4+blockLength)  )  ) {
         theFile.seek( theFile.length() );    // set pointer at end
         byte[] bs = new byte[4+blockLength];
         theFile.write( bs );                 // write a block of bytes
     }
     theFile.seek( 4 + blockNr*(4+blockLength) );  set pointer at desired block
  }

  public void writeBlock( String message, int blockNr ) {
    if ( ! writeable ) {
        throw new Exception( "file " +filePath
                            +" have not been opened for writing." );
    }
    byte[] bs = message.getBytes();
    if ( bs.length > blockLength ) {
            // ERROR situation you must decide how to handle:
            // you can throw an exception
            // or you can remove characters from the end of the string as I do below:
        message = message.subString( 0, message.length() - (bs.length-blockLength) );
            // this can go wrong. you may have to remove one character at a time.
        bs = message.getBytes();        
    }
    this.pointTo( blockNr );
    theFile.writeInt( bs.length );
    theFile.write( bs );
  }

  public String readBlock( int blockNr ) {
    if ( theFile.length() >= ( 4 + 4 + blockNr*(4+blockLength) ) ) {
                                                                                  // not really a safe test. sorry
        this.pointTo( blockNr );
        int lgt = theFile.readInt();
        byte[] bs = new byte[lgt];
        theFile.read( bs );
        return new String( bs );
    } else {
            // ERROR situation. no such block have been written
            // you can throw an exception
            // or return an enpty string ""
            // or be lazy and just return NULL as I do below
        return null;
    }
  }

} //endclass RandomAccessBlockFile

regards JakobA
0
 
LVL 15

Expert Comment

by:JakobA
ID: 13758647
Oops. 4 lines of garbage:
        if ( rwCode.equals("r") ) {
            File file = new File( filePath );
            if ( file.exists()
        }
should have been removed.  (near end of constructor)
0
 
LVL 13

Assisted Solution

by:Webstorm
Webstorm earned 500 total points
ID: 13760471
Hi rajesh042,

You can try to move the buffer definition to avoid heap problem.

public static void main(String[] args)
    {
        byte[] buffer = new byte[1024]; // <------------ moved here
        int rowIndex = 1;
        RandomAccessFile randomFile = null;
        int m_nDisplayRateAfter= 100;
        String fileName = null;
          boolean first = true;

        long startTime = System.currentTimeMillis();
        while(true)       {
            fileName =       "test" + rowIndex + ".dat";
            File file = new File(fileName);
            if(file.exists())   {
                try  {
                    randomFile = new RandomAccessFile (fileName, "rw");
                    // byte[] buffer = new byte[1024]; // move this line at the begining of the method
                    randomFile.read(buffer);
 
0
 
LVL 13

Expert Comment

by:Webstorm
ID: 13760493
>> if (true && rowIndex % m_nDisplayRateAfter == 0)
you can simplify this :
   if (rowIndex % m_nDisplayRateAfter == 0)

also
>> randomFile = new RandomAccessFile (fileName, "rw");
should be replaced by
    randomFile = new RandomAccessFile (fileName, "r");
because you don't write to the file.
0

Featured Post

Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction This article is the last of three articles that explain why and how the Experts Exchange QA Team does test automation for our web site. This article covers our test design approach and then goes through a simple test case example, how …
In this post we will learn how to make Android Gesture Tutorial and give different functionality whenever a user Touch or Scroll android screen.
Viewers learn how to read error messages and identify possible mistakes that could cause hours of frustration. Coding is as much about debugging your code as it is about writing it. Define Error Message: Line Numbers: Type of Error: Break Down…
The viewer will learn how to implement Singleton Design Pattern in Java.
Suggested Courses
Course of the Month21 days, 1 hour left to enroll

810 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question