Solved

Database vs reading files

Posted on 2004-10-16
10
207 Views
Last Modified: 2010-03-31
I've posted this in Java, because it uses Java brain cells rather than JSP brain cells.

I have a few thousand items of static content that are assembled into dynamic pages in a JSP. Typically, there are 20 items of static content to display on each page.

Currently, the static content is loaded into a database and the dynamic pages are assembled using the results of a database query, but the query itself is generally simply a list of item numbers. (e.g. "SELECT description FROM description_table WHERE item_number = 19 OR item_number = 77 OR item_number = 2076 OR..." with 20 specified item numbers).

Bearing in mind the descriptions are static and bearing in mind there is only the one server in my system, acting as the database, web server and application server, I was wondering if I could reduce the load on my server by publishing the static content into files (e.g. 20.txt...39.txt for this query) and doing something like...

<%
for (item_number_index = 0;item_number_index < 20;++item_number_index) {
    int item_number = results[offset+item_number_index]; // Get the item number from an array of results
    String str1 = "";
    try {
        BufferedReader in = new BufferedReader(new FileReader("/xxx/"+item_number+".txt"));
        String str2;
        while((str2 = in.readLine()) != null){
            str1 += str2;
        }
    }
    catch (IOException e) {
    }
    // ... output the str1 string in the page
}
%>

Typically, is it less expensive to open and close 20 files than to fetch read through 20 records in a result set?

Does it make any difference if the description fields in the database are BLOB fields?
0
Comment
Question by:rstaveley
  • 4
  • 2
  • 2
  • +1
10 Comments
 
LVL 24

Assisted Solution

by:sciuriware
sciuriware earned 155 total points
ID: 12326740
Of course plain files are faster than a database, but how about (future) retrieval and concurrent updates.
;JOOP!
0
 
LVL 9

Assisted Solution

by:Venci75
Venci75 earned 145 total points
ID: 12326844
It also depends on the data volume. The databases have many additional features like archiving.
Also, could it hapen that one day your application runs on more than one machine? In that case you will need a central data storage.
0
 
LVL 17

Author Comment

by:rstaveley
ID: 12329883
You say of course, sciuriware, but there are 20 file opens with the plain files, whereas the database just has one open and 20 fetches. I'm not too worried about updates,  because the static content is only updated once in a blue moon and it is quite acceptable to have doowntime during the publish.

I take the point about scaling, Venci. I guess you could put the files onto a shared mount.

So far, I get the impression that I'm onto something good for my little site, using the published plain files....
0
Optimizing Cloud Backup for Low Bandwidth

With cloud storage prices going down a growing number of SMBs start to use it for backup storage. Unfortunately, business data volume rarely fits the average Internet speed. This article provides an overview of main Internet speed challenges and reveals backup best practices.

 
LVL 92

Accepted Solution

by:
objects earned 200 total points
ID: 12330346
Why don't you read the data from the database once and store it in memory (in the app context).
0
 
LVL 17

Author Comment

by:rstaveley
ID: 12331397
Nice idea, objects, ....There are typically three thousand short descriptions (suitable for links) - let's say 3000x500 Bytes = 15 MB. What's a good rule of thumb for OK usage of the app context memory on a server? Is 15MB outrageous on an average system?
0
 
LVL 17

Author Comment

by:rstaveley
ID: 12331398
...I should add that the system is going to be pretty much dedicated to this application.
0
 
LVL 24

Assisted Solution

by:sciuriware
sciuriware earned 155 total points
ID: 12331426
rstaveley,
1) file open is much cheaper than database fetches,
2) loading some 1Gb of files into memory is no problem, if only you run with the option -Xmx1024M  or higher.
The load time might be obscured by putting the loading in a thread, while you do other things.
;JOOP!
0
 
LVL 9

Assisted Solution

by:Venci75
Venci75 earned 145 total points
ID: 12331656
>> Is 15MB outrageous on an average system

It depends on how much memory you need to process an average request and how many caches you will have. If this is the only one - then it shoud not be a problem. If the time for accessing the data from this cashe is critical for the performance of your applicatin - then definately you should cache it. If the memory consumed by this cache is too much for your system then you can improve this by using something like a 'last recently used' cache and keep in the memory only part of the data.
0
 
LVL 17

Author Comment

by:rstaveley
ID: 12331662
I've got my solution :-)

Many thanks, all!
0
 
LVL 92

Expert Comment

by:objects
ID: 12335183
:)
0

Featured Post

Is Your AD Toolbox Looking More Like a Toybox?

Managing Active Directory can get complicated.  Often, the native tools for managing AD are just not up to the task.  The largest Active Directory installations in the world have relied on one tool to manage their day-to-day administration tasks: Hyena. Start your trial today.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

For customizing the look of your lightweight component and making it look lucid like it was made of glass. Or: how to make your component more Apple-ish ;) This tip assumes your component to be of rectangular shape and completely opaque. (COD…
An old method to applying the Singleton pattern in your Java code is to check if a static instance, defined in the same class that needs to be instantiated once and only once, is null and then create a new instance; otherwise, the pre-existing insta…
Video by: Michael
Viewers learn about how to reduce the potential repetitiveness of coding in main by developing methods to perform specific tasks for their program. Additionally, objects are introduced for the purpose of learning how to call methods in Java. Define …
This theoretical tutorial explains exceptions, reasons for exceptions, different categories of exception and exception hierarchy.

778 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question