Solved

Database vs reading files

Posted on 2004-10-16
10
211 Views
Last Modified: 2010-03-31
I've posted this in Java, because it uses Java brain cells rather than JSP brain cells.

I have a few thousand items of static content that are assembled into dynamic pages in a JSP. Typically, there are 20 items of static content to display on each page.

Currently, the static content is loaded into a database and the dynamic pages are assembled using the results of a database query, but the query itself is generally simply a list of item numbers. (e.g. "SELECT description FROM description_table WHERE item_number = 19 OR item_number = 77 OR item_number = 2076 OR..." with 20 specified item numbers).

Bearing in mind the descriptions are static and bearing in mind there is only the one server in my system, acting as the database, web server and application server, I was wondering if I could reduce the load on my server by publishing the static content into files (e.g. 20.txt...39.txt for this query) and doing something like...

<%
for (item_number_index = 0;item_number_index < 20;++item_number_index) {
    int item_number = results[offset+item_number_index]; // Get the item number from an array of results
    String str1 = "";
    try {
        BufferedReader in = new BufferedReader(new FileReader("/xxx/"+item_number+".txt"));
        String str2;
        while((str2 = in.readLine()) != null){
            str1 += str2;
        }
    }
    catch (IOException e) {
    }
    // ... output the str1 string in the page
}
%>

Typically, is it less expensive to open and close 20 files than to fetch read through 20 records in a result set?

Does it make any difference if the description fields in the database are BLOB fields?
0
Comment
Question by:rstaveley
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 2
  • 2
  • +1
10 Comments
 
LVL 24

Assisted Solution

by:sciuriware
sciuriware earned 155 total points
ID: 12326740
Of course plain files are faster than a database, but how about (future) retrieval and concurrent updates.
;JOOP!
0
 
LVL 9

Assisted Solution

by:Venci75
Venci75 earned 145 total points
ID: 12326844
It also depends on the data volume. The databases have many additional features like archiving.
Also, could it hapen that one day your application runs on more than one machine? In that case you will need a central data storage.
0
 
LVL 17

Author Comment

by:rstaveley
ID: 12329883
You say of course, sciuriware, but there are 20 file opens with the plain files, whereas the database just has one open and 20 fetches. I'm not too worried about updates,  because the static content is only updated once in a blue moon and it is quite acceptable to have doowntime during the publish.

I take the point about scaling, Venci. I guess you could put the files onto a shared mount.

So far, I get the impression that I'm onto something good for my little site, using the published plain files....
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 92

Accepted Solution

by:
objects earned 200 total points
ID: 12330346
Why don't you read the data from the database once and store it in memory (in the app context).
0
 
LVL 17

Author Comment

by:rstaveley
ID: 12331397
Nice idea, objects, ....There are typically three thousand short descriptions (suitable for links) - let's say 3000x500 Bytes = 15 MB. What's a good rule of thumb for OK usage of the app context memory on a server? Is 15MB outrageous on an average system?
0
 
LVL 17

Author Comment

by:rstaveley
ID: 12331398
...I should add that the system is going to be pretty much dedicated to this application.
0
 
LVL 24

Assisted Solution

by:sciuriware
sciuriware earned 155 total points
ID: 12331426
rstaveley,
1) file open is much cheaper than database fetches,
2) loading some 1Gb of files into memory is no problem, if only you run with the option -Xmx1024M  or higher.
The load time might be obscured by putting the loading in a thread, while you do other things.
;JOOP!
0
 
LVL 9

Assisted Solution

by:Venci75
Venci75 earned 145 total points
ID: 12331656
>> Is 15MB outrageous on an average system

It depends on how much memory you need to process an average request and how many caches you will have. If this is the only one - then it shoud not be a problem. If the time for accessing the data from this cashe is critical for the performance of your applicatin - then definately you should cache it. If the memory consumed by this cache is too much for your system then you can improve this by using something like a 'last recently used' cache and keep in the memory only part of the data.
0
 
LVL 17

Author Comment

by:rstaveley
ID: 12331662
I've got my solution :-)

Many thanks, all!
0
 
LVL 92

Expert Comment

by:objects
ID: 12335183
:)
0

Featured Post

Get 15 Days FREE Full-Featured Trial

Benefit from a mission critical IT monitoring with Monitis Premium or get it FREE for your entry level monitoring needs.
-Over 200,000 users
-More than 300,000 websites monitored
-Used in 197 countries
-Recommended by 98% of users

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Are you developing a Java application and want to create Excel Spreadsheets? You have come to the right place, this article will describe how you can create Excel Spreadsheets from a Java Application. For the purposes of this article, I will be u…
Java Flight Recorder and Java Mission Control together create a complete tool chain to continuously collect low level and detailed runtime information enabling after-the-fact incident analysis. Java Flight Recorder is a profiling and event collectio…
Viewers will learn about arithmetic and Boolean expressions in Java and the logical operators used to create Boolean expressions. We will cover the symbols used for arithmetic expressions and define each logical operator and how to use them in Boole…
This tutorial covers a step-by-step guide to install VisualVM launcher in eclipse.
Suggested Courses

621 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question