Improve company productivity with a Business Account.Sign Up

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 215
  • Last Modified:

Database vs reading files

I've posted this in Java, because it uses Java brain cells rather than JSP brain cells.

I have a few thousand items of static content that are assembled into dynamic pages in a JSP. Typically, there are 20 items of static content to display on each page.

Currently, the static content is loaded into a database and the dynamic pages are assembled using the results of a database query, but the query itself is generally simply a list of item numbers. (e.g. "SELECT description FROM description_table WHERE item_number = 19 OR item_number = 77 OR item_number = 2076 OR..." with 20 specified item numbers).

Bearing in mind the descriptions are static and bearing in mind there is only the one server in my system, acting as the database, web server and application server, I was wondering if I could reduce the load on my server by publishing the static content into files (e.g. 20.txt...39.txt for this query) and doing something like...

<%
for (item_number_index = 0;item_number_index < 20;++item_number_index) {
    int item_number = results[offset+item_number_index]; // Get the item number from an array of results
    String str1 = "";
    try {
        BufferedReader in = new BufferedReader(new FileReader("/xxx/"+item_number+".txt"));
        String str2;
        while((str2 = in.readLine()) != null){
            str1 += str2;
        }
    }
    catch (IOException e) {
    }
    // ... output the str1 string in the page
}
%>

Typically, is it less expensive to open and close 20 files than to fetch read through 20 records in a result set?

Does it make any difference if the description fields in the database are BLOB fields?
0
rstaveley
Asked:
rstaveley
  • 4
  • 2
  • 2
  • +1
5 Solutions
 
sciuriwareCommented:
Of course plain files are faster than a database, but how about (future) retrieval and concurrent updates.
;JOOP!
0
 
Venci75Commented:
It also depends on the data volume. The databases have many additional features like archiving.
Also, could it hapen that one day your application runs on more than one machine? In that case you will need a central data storage.
0
 
rstaveleyAuthor Commented:
You say of course, sciuriware, but there are 20 file opens with the plain files, whereas the database just has one open and 20 fetches. I'm not too worried about updates,  because the static content is only updated once in a blue moon and it is quite acceptable to have doowntime during the publish.

I take the point about scaling, Venci. I guess you could put the files onto a shared mount.

So far, I get the impression that I'm onto something good for my little site, using the published plain files....
0
Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
objectsCommented:
Why don't you read the data from the database once and store it in memory (in the app context).
0
 
rstaveleyAuthor Commented:
Nice idea, objects, ....There are typically three thousand short descriptions (suitable for links) - let's say 3000x500 Bytes = 15 MB. What's a good rule of thumb for OK usage of the app context memory on a server? Is 15MB outrageous on an average system?
0
 
rstaveleyAuthor Commented:
...I should add that the system is going to be pretty much dedicated to this application.
0
 
sciuriwareCommented:
rstaveley,
1) file open is much cheaper than database fetches,
2) loading some 1Gb of files into memory is no problem, if only you run with the option -Xmx1024M  or higher.
The load time might be obscured by putting the loading in a thread, while you do other things.
;JOOP!
0
 
Venci75Commented:
>> Is 15MB outrageous on an average system

It depends on how much memory you need to process an average request and how many caches you will have. If this is the only one - then it shoud not be a problem. If the time for accessing the data from this cashe is critical for the performance of your applicatin - then definately you should cache it. If the memory consumed by this cache is too much for your system then you can improve this by using something like a 'last recently used' cache and keep in the memory only part of the data.
0
 
rstaveleyAuthor Commented:
I've got my solution :-)

Many thanks, all!
0
 
objectsCommented:
:)
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

  • 4
  • 2
  • 2
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now