Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Database vs reading files

Posted on 2004-10-16
10
Medium Priority
?
212 Views
Last Modified: 2010-03-31
I've posted this in Java, because it uses Java brain cells rather than JSP brain cells.

I have a few thousand items of static content that are assembled into dynamic pages in a JSP. Typically, there are 20 items of static content to display on each page.

Currently, the static content is loaded into a database and the dynamic pages are assembled using the results of a database query, but the query itself is generally simply a list of item numbers. (e.g. "SELECT description FROM description_table WHERE item_number = 19 OR item_number = 77 OR item_number = 2076 OR..." with 20 specified item numbers).

Bearing in mind the descriptions are static and bearing in mind there is only the one server in my system, acting as the database, web server and application server, I was wondering if I could reduce the load on my server by publishing the static content into files (e.g. 20.txt...39.txt for this query) and doing something like...

<%
for (item_number_index = 0;item_number_index < 20;++item_number_index) {
    int item_number = results[offset+item_number_index]; // Get the item number from an array of results
    String str1 = "";
    try {
        BufferedReader in = new BufferedReader(new FileReader("/xxx/"+item_number+".txt"));
        String str2;
        while((str2 = in.readLine()) != null){
            str1 += str2;
        }
    }
    catch (IOException e) {
    }
    // ... output the str1 string in the page
}
%>

Typically, is it less expensive to open and close 20 files than to fetch read through 20 records in a result set?

Does it make any difference if the description fields in the database are BLOB fields?
0
Comment
Question by:rstaveley
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 2
  • 2
  • +1
10 Comments
 
LVL 24

Assisted Solution

by:sciuriware
sciuriware earned 620 total points
ID: 12326740
Of course plain files are faster than a database, but how about (future) retrieval and concurrent updates.
;JOOP!
0
 
LVL 9

Assisted Solution

by:Venci75
Venci75 earned 580 total points
ID: 12326844
It also depends on the data volume. The databases have many additional features like archiving.
Also, could it hapen that one day your application runs on more than one machine? In that case you will need a central data storage.
0
 
LVL 17

Author Comment

by:rstaveley
ID: 12329883
You say of course, sciuriware, but there are 20 file opens with the plain files, whereas the database just has one open and 20 fetches. I'm not too worried about updates,  because the static content is only updated once in a blue moon and it is quite acceptable to have doowntime during the publish.

I take the point about scaling, Venci. I guess you could put the files onto a shared mount.

So far, I get the impression that I'm onto something good for my little site, using the published plain files....
0
The top UI technologies you need to be aware of

An important part of the job as a front-end developer is to stay up to date and in contact with new tools, trends and workflows. That’s why you cannot miss this upcoming webinar to explore the latest trends in UI technologies!

 
LVL 92

Accepted Solution

by:
objects earned 800 total points
ID: 12330346
Why don't you read the data from the database once and store it in memory (in the app context).
0
 
LVL 17

Author Comment

by:rstaveley
ID: 12331397
Nice idea, objects, ....There are typically three thousand short descriptions (suitable for links) - let's say 3000x500 Bytes = 15 MB. What's a good rule of thumb for OK usage of the app context memory on a server? Is 15MB outrageous on an average system?
0
 
LVL 17

Author Comment

by:rstaveley
ID: 12331398
...I should add that the system is going to be pretty much dedicated to this application.
0
 
LVL 24

Assisted Solution

by:sciuriware
sciuriware earned 620 total points
ID: 12331426
rstaveley,
1) file open is much cheaper than database fetches,
2) loading some 1Gb of files into memory is no problem, if only you run with the option -Xmx1024M  or higher.
The load time might be obscured by putting the loading in a thread, while you do other things.
;JOOP!
0
 
LVL 9

Assisted Solution

by:Venci75
Venci75 earned 580 total points
ID: 12331656
>> Is 15MB outrageous on an average system

It depends on how much memory you need to process an average request and how many caches you will have. If this is the only one - then it shoud not be a problem. If the time for accessing the data from this cashe is critical for the performance of your applicatin - then definately you should cache it. If the memory consumed by this cache is too much for your system then you can improve this by using something like a 'last recently used' cache and keep in the memory only part of the data.
0
 
LVL 17

Author Comment

by:rstaveley
ID: 12331662
I've got my solution :-)

Many thanks, all!
0
 
LVL 92

Expert Comment

by:objects
ID: 12335183
:)
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

After being asked a question last year, I went into one of my moods where I did some research and code just for the fun and learning of it all.  Subsequently, from this journey, I put together this article on "Range Searching Using Visual Basic.NET …
Introduction This article is the second of three articles that explain why and how the Experts Exchange QA Team does test automation for our web site. This article covers the basic installation and configuration of the test automation tools used by…
Viewers learn about the scanner class in this video and are introduced to receiving user input for their programs. Additionally, objects, conditional statements, and loops are used to help reinforce the concepts. Introduce Scanner class: Importing…
Viewers will learn about arithmetic and Boolean expressions in Java and the logical operators used to create Boolean expressions. We will cover the symbols used for arithmetic expressions and define each logical operator and how to use them in Boole…
Suggested Courses

688 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question