Solved

Database vs reading files

Posted on 2004-10-16
10
208 Views
Last Modified: 2010-03-31
I've posted this in Java, because it uses Java brain cells rather than JSP brain cells.

I have a few thousand items of static content that are assembled into dynamic pages in a JSP. Typically, there are 20 items of static content to display on each page.

Currently, the static content is loaded into a database and the dynamic pages are assembled using the results of a database query, but the query itself is generally simply a list of item numbers. (e.g. "SELECT description FROM description_table WHERE item_number = 19 OR item_number = 77 OR item_number = 2076 OR..." with 20 specified item numbers).

Bearing in mind the descriptions are static and bearing in mind there is only the one server in my system, acting as the database, web server and application server, I was wondering if I could reduce the load on my server by publishing the static content into files (e.g. 20.txt...39.txt for this query) and doing something like...

<%
for (item_number_index = 0;item_number_index < 20;++item_number_index) {
    int item_number = results[offset+item_number_index]; // Get the item number from an array of results
    String str1 = "";
    try {
        BufferedReader in = new BufferedReader(new FileReader("/xxx/"+item_number+".txt"));
        String str2;
        while((str2 = in.readLine()) != null){
            str1 += str2;
        }
    }
    catch (IOException e) {
    }
    // ... output the str1 string in the page
}
%>

Typically, is it less expensive to open and close 20 files than to fetch read through 20 records in a result set?

Does it make any difference if the description fields in the database are BLOB fields?
0
Comment
Question by:rstaveley
  • 4
  • 2
  • 2
  • +1
10 Comments
 
LVL 24

Assisted Solution

by:sciuriware
sciuriware earned 155 total points
ID: 12326740
Of course plain files are faster than a database, but how about (future) retrieval and concurrent updates.
;JOOP!
0
 
LVL 9

Assisted Solution

by:Venci75
Venci75 earned 145 total points
ID: 12326844
It also depends on the data volume. The databases have many additional features like archiving.
Also, could it hapen that one day your application runs on more than one machine? In that case you will need a central data storage.
0
 
LVL 17

Author Comment

by:rstaveley
ID: 12329883
You say of course, sciuriware, but there are 20 file opens with the plain files, whereas the database just has one open and 20 fetches. I'm not too worried about updates,  because the static content is only updated once in a blue moon and it is quite acceptable to have doowntime during the publish.

I take the point about scaling, Venci. I guess you could put the files onto a shared mount.

So far, I get the impression that I'm onto something good for my little site, using the published plain files....
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 92

Accepted Solution

by:
objects earned 200 total points
ID: 12330346
Why don't you read the data from the database once and store it in memory (in the app context).
0
 
LVL 17

Author Comment

by:rstaveley
ID: 12331397
Nice idea, objects, ....There are typically three thousand short descriptions (suitable for links) - let's say 3000x500 Bytes = 15 MB. What's a good rule of thumb for OK usage of the app context memory on a server? Is 15MB outrageous on an average system?
0
 
LVL 17

Author Comment

by:rstaveley
ID: 12331398
...I should add that the system is going to be pretty much dedicated to this application.
0
 
LVL 24

Assisted Solution

by:sciuriware
sciuriware earned 155 total points
ID: 12331426
rstaveley,
1) file open is much cheaper than database fetches,
2) loading some 1Gb of files into memory is no problem, if only you run with the option -Xmx1024M  or higher.
The load time might be obscured by putting the loading in a thread, while you do other things.
;JOOP!
0
 
LVL 9

Assisted Solution

by:Venci75
Venci75 earned 145 total points
ID: 12331656
>> Is 15MB outrageous on an average system

It depends on how much memory you need to process an average request and how many caches you will have. If this is the only one - then it shoud not be a problem. If the time for accessing the data from this cashe is critical for the performance of your applicatin - then definately you should cache it. If the memory consumed by this cache is too much for your system then you can improve this by using something like a 'last recently used' cache and keep in the memory only part of the data.
0
 
LVL 17

Author Comment

by:rstaveley
ID: 12331662
I've got my solution :-)

Many thanks, all!
0
 
LVL 92

Expert Comment

by:objects
ID: 12335183
:)
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
SHA2 certs for IIS AND Java? 2 122
tomcat startup error 5 135
jmss example java 2 32
Java 8 to Java 6 8 32
Introduction Java can be integrated with native programs using an interface called JNI(Java Native Interface). Native programs are programs which can directly run on the processor. JNI is simply a naming and calling convention so that the JVM (Java…
Basic understanding on "OO- Object Orientation" is needed for designing a logical solution to solve a problem. Basic OOAD is a prerequisite for a coder to ensure that they follow the basic design of OO. This would help developers to understand the b…
Video by: Michael
Viewers learn about how to reduce the potential repetitiveness of coding in main by developing methods to perform specific tasks for their program. Additionally, objects are introduced for the purpose of learning how to call methods in Java. Define …
Viewers will learn about if statements in Java and their use The if statement: The condition required to create an if statement: Variations of if statements: An example using if statements:

749 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question