Solved

Memory leaks

Posted on 2000-03-09
2
942 Views
Last Modified: 2013-12-14
My whole program is causing my machines resources to get tied up even after i finished running it. every time i keep running my machine gets worse and worse(slower) which it eventually leads it to being sapped and crashes, i think the memory leak is coming from my use of pointers(not sure but think it is),what would cause this to happen to my machine and how do i clear it up!!
also how do i do the garbabge disposal myself as i think for some reason which (i dont know)that its not doing it for me!!
0
Comment
Question by:j_baxter99
2 Comments
 
LVL 7

Accepted Solution

by:
Ravindra76 earned 50 total points
ID: 2601486

The Best question in this site

about memory leaks:-


Home    About Us    Help Desk    Jobs@EE    Earn Points    Advertise    Experts Only



Ask a Question Now!
ravindra76
Available Points: 1585
Expert Points: 22154



Previously Asked Java Programming Question
Ask a Question    Other Questions    Reload Question
Home : Computers : Programming : Languages : Java : Q.10235790

From: cheekycj
Title: "Java Memory Issues...(Immediate Attention)"

Points: 20
Date: Saturday, November 20 1999 - 12:20AM CST

I am working on a GUI Intensive (using all swing components and a couple of AWT
components) program
and it is finished but keeps running out of memory after running for a day.  It is a real-time
system so it
needs to keep running.  I am running System.gc as a separate thread that kicks off when the
system has
nothing to do.  It is reporting 60% free memory but I still get OutofMemory Errors.  Any
ideas/help?

I am running on a Solaris Ultra Machine with 128 MB RAM.
The Swap space is set to 256 MB and I have JDK 1.2.2 installed on it.

I run the program with the following command:
java -Xms64M -Xmx128M programname

Again it says I have 60% free mem but still get the outofMemory errors and for some reason
am using very
little of the swap space.

I am looking for ways of Memory management, freeing memory, using the VM to the MAX,
what components
are memory intensive etc.

Thanx,
CJ
PS: I will increase points as needed.


Accepted Answer

From: Jod
Date: Sunday, November 21 1999 - 01:50PM CST


Text below...

Question History

Comment

From: vladi21
Date: Friday, November 19 1999 - 11:34PM CST

http://www-4.ibm.com/software/developer/library/tip-heap-size.html

http://developer.java.sun.com/developer/bugParade/bugs/4065018.html




Comment

From: fontaine
Date: Saturday, November 20 1999 - 12:04AM CST

Go to the KL Group Web site (www.klgroup.com) and download the
trial version of JProbe. This tool
will help you to determine where
memory is consumed.


Comment

From: mbormann
Date: Saturday, November 20 1999 - 12:34AM CST

CJ,
can u pls look at this question so that u can have some pointer?

http://www.experts-exchange.com/bin/Q.10216695

Quoting wrostek
"In my special case I frequently added a new list model to a JList. It
seems that the old model is not garbadge collected. Now I reuse the
model which solves my problem."

end quote
well the thing to do is to recycle components.


Comment

From: mbormann
Date: Saturday, November 20 1999 - 12:40AM CST

http://forum.java.sun.com/forum?13@133.CDKpacmqbtm^0@.ee9b676/0

http://forum.java.sun.com/forum?13@133.CDKpacmqbtm^0@.ee791f6/0

www.optimizeit.com


Comment

From: Jod
Date: Saturday, November 20 1999 - 12:59AM CST

Have you tried with larger Xms values?

From specific experience I can give you some info regarding GUI components.

Even simple text components can eat large chunks of memory if there are enough of them on
screen or
created at any one time.

Try creating a large number of AWT components and watch the application grind to a halt as
they appear on
screen or are updated.

(for example...

http://forum.java.sun.com/forum?13@133.CDKpacmqbtm^0@.ee794b0/0

)

Lightweight components can be just as bad if they are double buffered (which swing
components are) and
do not effectively return resources after repainting.

In other words they can hang on to their graphics double buffer memory instead of releasing it
after each
repaint. This is often done to keep the speed up but means that even a text box can eat up
several K just
by being put on screen. Larger components, larger buffers...how far this applies to JDK 122
may need
investigating but it is has been a factor for us previously.

Swing components can have specific problems returning resources when being garbage
collected, even if
they are disposed specifically. For example, check out this bug:

http://developer.java.sun.com/developer/bugParade/bugs/4229318.html

or

http://developer.java.sun.com/developer/bugParade/bugs/4156232.html

Swing eats resources and has a large standard footprint.

Be aware that basically, neither Swing nor AWT doesn't disposal down the component
hierarchy. The result
is that any component that has registered itself as a listener doesn't get a chance to deregister
itself,
whatever it is listening to keeps a live pointer to it, and indirectly, all of the components in the
window
remain reachable, and not garbage collectable.

In Swing, if you create a button with an action (using the add( Action ) function of JMenu or
JToolBar), the
button registers itself. If the action is a singleton (presumable a frequent occurance), you have a
memory
leak.


So, this sort of memory leak is a common occurrence in Java - if you have a listener registered
on a GUI
component, the component may not be garbage collected until all the listeners are gone (or
deregistered
from the component) as well.
In general:

(see http://forum.java.sun.com/forum?13@133.CDKpacmqbtm^0@.ee78b38/0 for original info
on this)


In general:

Does your system allocate separate resources for different users/sessions or is it intrinsically a
single user
system with one set of resources?

Also does it have multiple screens or windows?


Another area is SQL and database resources. Do you use prepared statements that are kept
active for long
periods of time for example or maintain large data sets in memeory?

Hope this helps...



Comment

From: mbormann
Date: Saturday, November 20 1999 - 01:01AM CST

http://www.javaworld.com/javaworld/javatips/jw-javatip79.html

http://forum.java.sun.com/forum?13@133.CDKpacmqbtm^0@.ee78b38/0

CJ,
it would be better if u outline atleast some thing abt ur code so that others can help u out.


Comment

From: vladi21
Date: Saturday, November 20 1999 - 01:12AM CST

http://forum.java.sun.com/forum?13@133.CDKpacmqbtm^0@.ee78b38/0

mbormann  and Jod - you use same search engine :)


Comment

From: mbormann
Date: Saturday, November 20 1999 - 01:16AM CST

Sorry all for butting in but we r using KL group's Jclass BWT ( licensed and with our extended
widgets ) and
it does not SEEM to have these sort of problems.Atleast i haven't heard of it at our place
,our bosses have taken the decision not to move to Swing in the near future.

Also 128MB is way too low ,program will be spending way too much time in the swap
mode,have a
recommended 256 Megs atleast(?)

Jod has explained the things in detail and beautifully in sequence


Comment

From: Jod
Date: Saturday, November 20 1999 - 01:20AM CST

All roads lead to the same destination here I think (your link looks familiar as well...:).

It is a problem we have had before.


Comment

From: vladi21
Date: Saturday, November 20 1999 - 01:23AM CST

we also use KL group's products
and now we don't use  Dialogs recycling - its so buggly under Win32


Comment

From: Jod
Date: Saturday, November 20 1999 - 01:37AM CST

Win32 definately appears worst hit with memory leaks.

We are using iMacs for deployment as they are so easy to setup and configure to work with
Java.


Comment

From: captainkirk
Date: Saturday, November 20 1999 - 01:42AM CST

Try calling the system garbage collector directly when you're done with objects:

// I believe the call is (it's been a while)--
System.gc();

this may or may not force the GC to run right away, but it may be worth a try...


Comment

From: cheekycj
Date: Saturday, November 20 1999 - 01:45AM CST

OK, now that I am caught up on my reading :-)

Here are some details of the program for you guys.

It is a JFC Application with one Main JFrame.
The JFrame has 6 tabs (all the code for gui in the tabs are also in this class- not my design)
Besides having two configuration popups and a splash screen there are no extra windows.

Now the tabs,
Tab 1: JPanel w/ a 150 KB image on it (the image extends JPanel)
and has a mouselistner.
Tab 2: JPanel w/ dozens of labels, 10 text boxes, pulldown list, radio
buttons, 1 textarea and 20 buttons
Tab 3: JPanel w/ about the same controls as Tab2 but no pulldown list and no textarea but it has
buttons
thatmove a value between 1-140 for
different data to display according to that number.
Tab 4: JPanel w/ Tidestone's Formula One for Java (spreadsheet bean) there.  No data displayed
in the
spreadsheet.  Should be relatively small.
Has 4 buttons.
Tab 5: JPanel w/ 1 TextArea that logs all events.
Tab 6: JPanel w/ Tidestone's Formula One for Java (spreadsheet bean) there is a lot of data here
but
according to Tidestone we are not even beginning to push the bean's upperlimit.  The sheet is
set up to
load 30 days worth of values.
Tab 7: JPanel that is blank. (we will remove it .. just haven't gotten around to it)

The application has your standard menu bar with 4 small menu items.
And has a status bar with 10 values displayed.

The operation;
We awake at a certain time.  FTP files from a Server. Read them in and parse ints out of the
strings that are
read in.  Load the data in our
spreadsheet and run 4 forumulaes on the data.

The program is supposed to run forever but gets memory leaks after 1 day. There is a lot of use
of
StringTokenizer.  How bad is that.

Hope this is enough info.

Thanks for the help so far.
Improvements made so far:
Use Stringbuffer for any concatenations.
Set vars to null after they are not used.
Consistent Calls to System.gc() for cleanup.

Still doesn't help.

Thanx,
CJ


Comment

From: cheekycj
Date: Saturday, November 20 1999 - 01:49AM CST

For some reason I do get a warning of
running 1.1 version of SystemEventQueueUtilities
(because gui developed in Symantec VCafe 3.0 which only has JDK 1.1.8 with Swing 1.0.3 and
we are
delploying on JDK 1.2.2)
I hope that isn't causing problems.
I am going to keep the points the same for now (200) but
will give out more to all individuals who help out as seperate Questions.

CJ


Comment

From: vladi21
Date: Saturday, November 20 1999 - 02:21AM CST

- Optimize String operations
http://www.codeguru.com/java/articles/511.shtml

Maybe u can use StreamTokenizer?




Comment

From: mbormann
Date: Saturday, November 20 1999 - 02:24AM CST

>>>>>There is a lot of use of StringTokenizer.  How bad is that

Bad ,null it and CJ if ur StringBuffer's r huge then pre allocate them as i had a particular
problem of sending
500KB string so

StringBuffer buffForSending=new
StringBuffer(dataToSave.length()+((dataToSave.length())/4));
I was then adding some protocol data to String

>>>>>Tab 5: JPanel w/ 1 TextArea that logs all events.

From Linden FAQ
(Sect. 9) How much text can be put in a TextArea?

[*] TextArea just uses the corresponding widget of the             underlying   window system. It
will be
bounded by the limit             imposed in the native  window system. In Windows 95 TextAreas
can
hold about 28Kb. The native  widget allows 32Kb, but there is              some overhead which
reduces the
amount available to the programmer. The limit is removed in JTextComponent in  Swing (JDK
1.2) which
dispenses with peer controls.

so pls utilise JTextArea et al,i think u must be using that and meant to say it in the first place.


Comment

From: cheekycj
Date: Saturday, November 20 1999 - 02:40AM CST

mbormann:Yup, we are Using all Swing Components...
So its JComboBox, JButton, JTextArea,
The StringBuffers are actually quites small. All less than 80 chars.
Should I still set a sizelimit.  Also should I declare a new Stringbuffer and use it locally in a
method or use a
global one?

vladi21:  I am using StringTokenizer quite a bit.

I will try setting those to null when I done using them.

Any more ideas are welcome.

CJ


Comment

From: cheekycj
Date: Saturday, November 20 1999 - 02:44AM CST

Check this code:
file load code:
===============================
while ((line = inFile.readLine()) != null) {
StringTokenizer st = new StringTokenizer(line,",");
if (Util.compareDate(st.nextToken())) {
list.addElement(line);
dataFound = true;
}
st = null; // just added this to free up memory
}
=============================
the file that reads in is about 700 lines long
and we read in on avg 6 of these files.

CJ


Comment

From: cheekycj
Date: Saturday, November 20 1999 - 02:46AM CST

vladi21:  I apologize.. I read your comment wrong.  What are the advantages of using
StreamTokenizer
over StringTokenizer and can it replace StringTokenizer in all situations?

CJ


Comment

From: vladi21
Date: Saturday, November 20 1999 - 02:57AM CST

what better 1 StreamTokenizer per file or StringTokenizer per line? :)
check

http://developer.javasoft.com/developer/technicalArticles/Programming/PerfTuning/index.html#Tokenizati
on




Comment

From: mbormann
Date: Saturday, November 20 1999 - 03:00AM CST

Tip 1:
-----
I think u can simplify this and not go into StreamTokenizer

while ((line = inFile.readLine()) != null)
{
//see if this chek applies in ur case
if( (line=line.trim()).length() == 0)
{
line=null;
continue;
}

String st = line.substring(0,line.indexOf(',')).trim();
while (Util.compareDate(st))
{
list.addElement(line);
dataFound = true;
}
line=null;
st = null; // just added this to free up memory
}

Tip 2:
-----
have ALL Strings like "," etc which run in loops mentioned up top in class like say.

private static final String protocolSeparator = ",";

Tip 3:
------
seems liej u do a ReadLine ,well see what ur OS reads chunks of Files in exactly what sizes ,i
think it maybe
around 16KB?i am not sure,but pass the BufferedReader / DataInputStream whatever u use
Buffer size of
8192 bytes /16384 buytes whatever is applicable


Comment

From: mbormann
Date: Saturday, November 20 1999 - 03:05AM CST

this ensures that ur Porgram does not use hard disk much and slow down I/O
performance.Restrict ur
attack then to GUI's only.
and yes please when u close the BufferedReader / DataInputStream ,explicitly null that object
reference
out.close();out=null; internally it is set to null after sometime.but be on safe side.


Comment

From: cheekycj
Date: Saturday, November 20 1999 - 06:05AM CST

mbormann:
I have

LineNumberReader inFile = new LineNumberReader(new FileReader(readFile));

as my declaration for reading in the file.
I incorperated most of you changes.
the file read part.
Every possible Variable is being set to null now.
I am using StringBuffer whenever Possible
Most string literals have been removed unless I log an error- which is rare.

====================

I appreciate all the help so far guys... I will continue this on Monday.
I have incorperated a lot of changes into the code and will
run the program over the weekend to see how it does.
===================

A couple of questions...
1.  Can StreamTokenizer be used in situation without I/O Streams?
didn't seem like it to me.
2.  Anyone know anything about this stupid version 1.1 of systemEventQueueUtilities
warning??
3.  Is there a way to manage the memory that the components on the tabbed panes such that
when they
lose focus they release resource or something?

===================
You guys are gonna get a lot of points from me  :-)
Thanx a lot.
CJ


Comment

From: vladi21
Date: Saturday, November 20 1999 - 06:39AM CST

StreamTokenizer st = new StreamTokenizer(new StringReader("abc"));


Comment

From: ravindra76
Date: Saturday, November 20 1999 - 04:32PM CST

What a nice discussion among four worthty experts ( Above 20,000 points ).

It is the first time for me to see all in same question in the two months


Comment

From: ravindra76
Date: Saturday, November 20 1999 - 04:53PM CST


Hi experts,

Be ready for arriving question of the millenium which is coming soon


Accepted Answer

From: Jod
Date: Sunday, November 21 1999 - 01:50PM CST

>> Be ready for arriving question of the millenium which is coming soon

sounds ominous...


Something to identify is does the app run out of memory just running on it's own or is it the
reading in of
files that is causing the problem?

Try flicking between the tabs in your app without doing anything else for instance and see if
memory usage
goes up.

Just a hunch that, regarding redrawing of components.

Also play with the value and data in the fields on the tabs to see if this causes a memory usage
problem
here also. Keep pressing a button for example and see if it keeps on eating memory endlessly.

Every time you read a file you are creating a large number of temporary string objects that are
then being
left hanging almost immediately. These should be garbage collected but to be honest I am yet to
really trust
the garbage collecter.

Perhaps even pathological localisation of variables might help the GC remove references. For
example:

while ((String line = inFile.readLine()) != null) {
StringTokenizer st = new StringTokenizer(line,",");
if (Util.compareDate(st.nextToken())) {
list.addElement(line);
dataFound = true;
}
st = null; // just added this to free up memory
line = null;
// EXTREME......
System.gc();
}
//NORMAL
System.gc();

Localising the String Line variable might cleaner identification of when it's resources are out of
context.

Or even move the file reading code to another class to allow easy identification of the resources
allocated to
it.

The definition of readLine you are using is:

public String readLine() throws IOException {
synchronized (lock) {
String l = super.readLine(skipLF);
skipLF = false;
if (l != null)
lineNumber++;
return l;
}
}

The default buffer size is 8192 bytes.

OUt of interest, is it possible for you to read into a byte/character buffer instead of reading
strings and
whole lines? Reading bytes from a file is much more efficient and more predictable in a real
time system that
is running all the time, rather than using these convenience methods to do the character
processing for you.


It may be worth considering as then you will have complete control over the long term
efficiency of the code.


So what format are the files you are reading in and are they predictable in this manner?



I am presuming that the data read in is displayed in the Formula One grid? How is it processed
to get in and
what sort of data model does the grid use (how efficient is it at memory management?)

>> 2.  Anyone know anything about this stupid version 1.1 of systemEventQueueUtilities
warning??
>> 3.  Is there a way to manage the memory that the components on the tabbed panes such that
when
they lose focus they release resource or something?

Regarding the warning, I would try downloading some of the following tools to see what
objects are
hanging around and whether your components have any issues with this:

http://java.sun.com/products/jfc/jaccess-1.0/doc/examples.html

The warning sounds like it is related to the Java Accesibility API and the way that in JDK 1.1
the event queue
is replaced to retro fit this feature.

If the utilities do not work, then the event queue is not being replaced with the one relevant for
Java
Accessibility API. I would need to look into this in greater detail to tell you more about any
problems this
might be causing.

Enough for now...



Comment

From: Jod
Date: Sunday, November 21 1999 - 02:06PM CST

The definition of readLine given is as in linebufferreader.

The readline code in bufferedreader is also possibly not very efficient:

/**
* Read a line of text.  A line is considered to be terminated by any one
* of a line feed ('\n'), a carriage return ('\r'), or a carriage return
* followed immediately by a linefeed.
*
* @param      skipLF  If true, the next '\n' will be skipped
*
* @return     A String containing the contents of the line, not including
*             any line-termination characters, or null if the end of the
*             stream has been reached
*
* @see        java.io.LineNumberReader#readLine()
*
* @exception  IOException  If an I/O error occurs
*/
String readLine(boolean skipLF) throws IOException {
StringBuffer s = new StringBuffer(defaultExpectedLineLength);
synchronized (lock) {
ensureOpen();

bufferLoop:
for (;;) {

if (nextChar >= nChars)
fill();
if (nextChar >= nChars) { /* EOF */
if (s.length() > 0)
return s.toString();
else
return null;
}
boolean eol = false;
char c = 0;
int i;

/* Skip a leftover '\n' */
if (skipLF && (cb[nextChar] == '\n'))
nextChar++;

charLoop:
for (i = nextChar; i < nChars; i++) {
c = cb[i];
if ((c == '\n') || (c == '\r')) {
eol = true;
break charLoop;
}
}
s.append(cb, nextChar, i - nextChar);
nextChar = i;

if (eol) {
nextChar++;
if (c == '\r') {
if (nextChar >= nChars)
fill();
if ((nextChar < nChars) && (cb[nextChar] == '\n'))
nextChar++;
}
break bufferLoop;
}
skipLF = false;
}
}

return s.toString();
}

This is called by bufferedreader.readLine to do the read you want. The byte array readers are
generally a
bit 'cleaner'. Readline code is a convenience.




Comment

From: cheekycj
Date: Tuesday, November 23 1999 - 01:12AM CST

OK... I see the overhead problem w/ Readline.  I don't have to use
that.. it was just convenient.

The files are flat text files that are in comma delimited format.
Each line in the file is a record that is timestamped.
I need to read in each record(line) compare the timestamp to the
current time period and if it falls in the current time period add it to my table.  That is why I
used readline
and then I need to tokenize
the String to get its first element and compare its timestamp.
then I add it to a vector list.

but... would this be the equivalent:
====================================
FileInputStream fis = new FileInputStream(args[0]);
byte buf[] = new byte[1024];
int n;
while ((n = fis.read(buf)) != -1) {
for (int i = 0; i < n; i++) {
if (buf[i] == '\n')
String line = buff;
String st = line.substring(0,line.indexOf(Const.COMMA)).trim();
if (Util.compareDate(st)) {
list.addElement(line);
dataFound = true;
}
line=null;
st = null; // just added this to free up memory
}
}
fis.close();
======================================
If not how would I extract the each line into a string to get its first
element and compare the datestamp?

Thanx,
CJ


Comment

From: vladi21
Date: Tuesday, November 23 1999 - 09:15PM CST

also if u use ASCII text look at

/**
* Reads the next line of text from this data input stream. This
* method successively reads bytes from the underlying input stream
* until it reaches the end of a line of text.
* <p>
* A line of text is terminated by a carriage return character
* (<code>'&#92;r'</code>), a newline character (<code>'&#92;n'</code>), a
* carriage return character immediately followed by a newline
* character, or the end of the input stream. The line-terminating
* character(s), if any, are not returned as part of the string that
* is returned.
* <p>
* This method blocks until a newline character is read, a carriage
* return and the byte following it are read (to see if it is a
* newline), the end of the stream is detected, or an exception is
* thrown.
*
* @deprecated This method does not properly convert bytes to characters.
* As of JDK&nbsp;1.1, the preferred way to read lines of text is via the
* <code>BufferedReader.readLine()</code> method.  Programs that use the
* <code>DataInputStream</code> class to read lines can be converted to use
* the <code>BufferedReader</code> class by replacing code of the form
* <ul>
*     <code>DataInputStream d =&nbsp;new&nbsp;DataInputStream(in);</code>
* </ul>
* with
* <ul>
*     <code>BufferedReader d
*          =&nbsp;new&nbsp;BufferedReader(new&nbsp;InputStreamReader(in));
*      </code>
* </ul>
*
* @return     the next line of text from this input stream, or
*             <tt>null</tt> if no bytes are read before end-of-file
*             is reached.
* @exception  IOException  if an I/O error occurs.
* @see        java.io.BufferedReader#readLine()
* @see        java.io.FilterInputStream#in
*/
public final String readLine() throws IOException {
InputStream in = this.in;
char buf[] = lineBuffer;

if (buf == null) {
buf = lineBuffer = new char[128];
}

int room = buf.length;
int offset = 0;
int c;

loop: while (true) {
switch (c = in.read()) {
case -1:
case '\n':
break loop;

case '\r':
int c2 = in.read();
if (c2 != '\n') {
if (!(in instanceof PushbackInputStream)) {
in = this.in = new PushbackInputStream(in);
}
((PushbackInputStream)in).unread(c2);
}
break loop;

default:
if (--room < 0) {
buf = new char[offset + 128];
room = buf.length - offset - 1;
System.arraycopy(lineBuffer, 0, buf, 0, offset);
lineBuffer = buf;
}
buf[offset++] = (char) c;
break;
}
}
if ((c == -1) && (offset == 0)) {
return null;
}
return String.copyValueOf(buf, 0, offset);
}




maybe some modification needed? :)



Comment

From: Jod
Date: Tuesday, November 23 1999 - 10:20PM CST

cheekycj:

yep along the right lines - avoiding all the temporary strings is the main problem, because I am
not sure
how efficiently these are being garbage collected (in the real world that is)

What you really need is a sort of rolling buffer that always stores the next one or more lines at a
time. You
can then refill this buffer when it runs out and extract the current data from it as neccessary.

How to avoid extracting the data into a string unless the date at the start of this buffer indicates
it is to be
retained is not straightforward (possibly unavoidable and there are a few complications here)

Before heading down this route, have you been able to determine if it is  definately the file
reading that is
causing the application to run out of memory over time?


Comment

From: cheekycj
Date: Thursday, November 25 1999 - 02:45AM CST

well, the good news is that the system hasn't run out of memory.. though it is at 68% after only
2 days.  We
are going to let it run over the holidays and see at what point we run out of memory.  I think at
that point it
may become a issue of buy more memory rather than better memory management.  Does
anyone know if
there is harm in specifying a swap space that is 2x or 3x your physical memory (RAM) ?

CJ


Comment

From: vladi21
Date: Thursday, November 25 1999 - 03:23AM CST

swap =2x RAM


Comment

From: vladi21
Date: Thursday, November 25 1999 - 03:46AM CST

Q. How big and where should my Pagefile be?

A. Below are things to consider.

Do not have 2 pagefiles on the same physical disk.
The pagefile should be size of memory +11MB (or 2.5 times size of SAM file, which ever is
larger).
If possible have multiple page files on separate physical disks
There are some performance gains to be had by placing on a stripe set, however not as much as
on
separate disks.
Defragment disk pagefile is on
Try and make drive NTFS
Minimum pagefile size is 2MB
To enhance performance, one can create a second pagefile on another physical disk. MOVING,
however, is
never advised, since it disables the option to create a Memory dump file at a crash (System
Properties,
Startup-tab). In order to be able to dump the RAM content to the pagefile (saved i.e. as
MEMORY.DMP), the
pagefile MUST be located (as well) on the boot partition.

http://www.ntfaq.com/ntfaq/perform3.html


Comment

From: Jod
Date: Thursday, November 25 1999 - 04:06AM CST

The day that I have to look through a MEMORY.DMP file is the day I get a new job.

vladi21's advice is sound - however I think cheekycj is running under Solaris...:-)

(As an aside I have to say though, that I always try and move the pagefile to a disk other than
the system
disk to avoid thrashing and provide better dual access to the system and pagefile concurrently
under NT.)

NOTE:

This may be true under solaris as well, but under NT we try to specify a pagefile with a
minimum and
maximum size the same so that the operating system can always allocate the whole page file as
a single
block at startup rather than having to increase the size incrementally whilst running.

I think your max swap size is the right size at 2 X ram (as you have it...)


Comment

From: cheekycj
Date: Thursday, November 25 1999 - 04:21AM CST

Jod: Yup, I am on Solaris.
I use the command:
java -Xms128M -Xmx128M programname

but was thinking what if... I used:
java -Xms256M -Xmx256M programname
or even...
java -Xms512M -Xmx512M programname

The Sun Ultra 5 has 128MB physical RAM.
(I will increase the swapdisk size on unix of course)

CJ





Comment

From: mbormann
Date: Thursday, November 25 1999 - 02:29PM CST

CJ,
i think i mentioned in the beginning that please increase RAM to 256MB,that is the size we
recommend to
people who buy our product and advisably pls tell them to have that machine Only for that,this
figure was
arrived at after lotsa research ,our GUI is very fat around 1.8MB jar files size.
Remember that it is a necessary but not a sufficient condition(where did we hear that before?
Yup
Deadlocks. :-))

I have been after them for obfuscating as this reduces jar size and the memory fottprint,i
wonder why
nobody talked abt this before?The reduction is I have heard from another friend anyhting from
30 -60 %

Regards


Comment

From: mbormann
Date: Thursday, November 25 1999 - 02:32PM CST

another comment blaberring along ,why docha get those new fangled Java Stations running
Java from
ground up?ignore this if u feel it's Ho hum


Comment

From: vladi21
Date: Thursday, November 25 1999 - 09:21PM CST

yes
try
JAX reads class files, analyzes them, and generates a compressed ZIP file as output
http://www.alphaworks.ibm.com/formula/JAX

and

DashO
http://www.preemptive.com/index.html

PreEmptive is proud to announce the upcoming DashO-Embedded Edition! This product will
not only include
the power of DashO-Pro for maximum class level size reduction, but also the DashO bytecode
optimizer for
the smallest/fastest bytecode available!


Comment

From: mbormann
Date: Friday, November 26 1999 - 06:54PM CST

i saw this right now
http://www.sun.com/workshop/java/wp-javaio/


Comment

From: vladi21
Date: Sunday, November 28 1999 - 08:27PM CST

and
Tuning JavaTM I/O Performance
http://developer.javasoft.com/developer/technicalArticles/Programming/PerfTuning/index.html

Programming with JavaTM I/O Streams
http://developer.javasoft.com/developer/technicalArticles/Streams/ProgIOStreams/index.html

Writing Your Own JavaTM I/O Stream Classes
http://developer.javasoft.com/developer/technicalArticles/Streams/WritingIOSC/index.html


Comment

From: Jod
Date: Sunday, November 28 1999 - 11:10PM CST

Yet another thing that has been mentioned elsewhere but not here is the use of string.intern()

Strings computed at run time are newly created and therefore created as distinct objects.

However, as the docs say, the result of explicitly interning a computed string is the same string
as any
pre-existing literal string with the same contents.

In other words, you can reuse an existing string in the global string pool if an identical one
exists.

This is only useful if you have a lot of repetitive use of the same strings in your file that you
read in, which is
quite possible.


Comment

From: Jod
Date: Tuesday, November 30 1999 - 09:02AM CST

I saw this and thought of you cheekycj...

Am I not nice to you or what:

http://www.cs.cmu.edu/~jch/java/size.html




Comment

From: cheekycj
Date: Saturday, December 04 1999 - 12:04AM CST

After running more and more tests are some results ... for you guys...

StringTokenizer uses less memory than
the String.substring() command (go figure)

BufferedReader gave me the best performance when it came to the reading of the text file.

Direct Buffering was the fastest but when you wanted to extract and manipulate as you read in
the file (like
I was doing in BufferedReader) it became magnitudes slower.  Direct Buffering was also
inconsistent with
my Data.. sometimes pulling in a line at a time and sometimes not ... so I had to stick with
Readline cmd.

Using Const.COMMA instead of "," everywhere saved a few bytes but probably more after
being run for a
while.

We were running at -Xmx128M but increased it to -Xmx256M and will be
going to -Xmx512M as we are bumping up the RAM in the machine to 256 MB.

I am reading in a 860 KB TEXT file
in 1 Sec. including the String compares
and storing in a Vector.  I knew the size of the array before hand but when I used an array of
Strings the
memory usage was a lot more than with the Vector... another one I couldn't figure out.

I will post more as I learn more.

I am hoping to wrap up this question in a couple of weeks.  I appreciate all the info and support.
The points
will be given out to all active participants.

Thanx,
CJ




Comment

From: mbormann
Date: Saturday, December 04 1999 - 12:14AM CST

>>>>>bumping up the RAM in the machine to 256 MB.
good news!

>>>>>and storing in a Vector

dont use the default constructor of that ,control the memory space by allocating something like

public Vector(int initialCapacity,int capacityIncrement)

i used it for something like
new Vector(1021,193);

both r very important here ,chk out ur owbn requirement

>>>>>memory usage was a lot more than

can u post some rough approximate code?so others can see it?


Comment

From: mbormann
Date: Saturday, December 04 1999 - 12:16AM CST

forgot to add reading Core Java by gary cornell &Cay horstmann,
if i am not jumbled up they say that new Vector(int ,int) both shuld ideally be prime nos
between power of 2


is this right guys?


Comment

From: Jod
Date: Monday, December 06 1999 - 10:57PM CST

CJ, some points from the above link on optimising Java:

Regarding your experience with Vectors...

Don't initialize big arrays:

Although array initialization is a single statement in Java, the generated bytecode inserts one
element at a
time. If you've got a big array, this requires a lot of bytecode. You can save space by storing
your data in a
String and then parsing it into your array at runtime (tip from Andrew Scherpbier).

(So as vectors are just growable arrays, bearing this in mind they are very likely to be far more
code
efficient than creating large initialised arrays pre defined to the relevant size.)


Perhaps of significance...

Dates are big (not just broken)

Date objects take a surprising amount of space for something with such limited functionality. If
you're
storing a lot of them, considering using longs instead, and recreating Date objects from them as
necessary
(tip from Jean-Marc Lugrin).


Finally I imagine that Const.COMMA is more efficient than "," for the same reason that
String.intern() is more
emory efficient, because you are not creating a temporary string for "," every time it is needed

Did you look into using string.intern() at all?


mb:

Not sure about the Vector thing. What reason do they give?

You want to keep the initial vector size to be close to the minimum expected in this case so that
you only
allocate more space where neccessary - the amount of time it takes to increase the Vectors
memory space
is probably not an issue here.


Sounds like you are getting there...


Comment

From: mbormann
Date: Wednesday, December 15 1999 - 12:39AM CST

Sorry Jod,
I was away for a while but if u look up the JDK Docs for Vector

protected int capacityIncrement

The amount by which the capacity of the vector is automatically incremented when its size
becomes greater
than its capacity. If the capacity is 0, the capacity of the vector is doubled each time it needs to
grow.

Now if u give a default new Vector() this doubles every time the addElement() goes on adding
elements to
Vector.

This grwos enormously as u can see after calculating,so u should control the addition to the
capacity of the
vector.

More docs

the vector's
storage increases in chunks the size of capacityIncrement. An application can increase the
capacity of a
vector before
inserting a large number of components; this reduces the amount of incremental reallocation.


Also if u know that u are going to have say atleast 50 elems then better alloc it at start say

new Vector(64,64);

Remember that Vector and Hashtable is NOT MEANT for large stuff in the range of above
1000 or so,thats a
Rule of Thumb,as it EATS memory very much.

Regards


Comment

From: Jod
Date: Monday, December 20 1999 - 09:14PM CST

Here we go mb...

The Memory Management Reference
http://www.harlequin.com/products/st/mm/bib/full.html

What I actually meant in the above was, why prime nos between power of 2?


I would just make the vector large enough to accomodate the minimum expected items and then
if it was
too small only increment with enough space to store one additional item each time. that's what I
meant be
the trade off, "the amount of time it takes to increase the Vectors memory space is probably not
an issue
here" - it's the memory conservation that is more important.










Comment

From: cheekycj
Date: Tuesday, December 21 1999 - 12:08AM CST

we are storing 140 elements (max) each time... using a string array was an option but I think the
memory
used was less w/ Vectors.

So the code should look like what?

new Vector (128,256)

??

CJ


Comment

From: mbormann
Date: Tuesday, December 21 1999 - 10:37PM CST

CJ,
new Vector (140,23) ;

Jod,
the prime number between power of 2 is I believe a old hashing concept
as it is proved here
http://www-compsci.swan.ac.uk/~csmark/download.html
and
http://wwwinfo.cern.ch/asd/lhc++/Objectivity/V5.2/Java/api/com/objy/db/util/ooMap.html

http://hissa.ncsl.nist.gov/~black/CRCDict/HTML/hashtab.html
http://www.ee.uwa.edu.au/~plsd210/ds/hash_tables.html

I think this same thingy applies to Vectors


Comment

From: cheekycj
Date: Wednesday, December 22 1999 - 01:20AM CST

After reading some of the above (again)
I noticed Jod stated that do not initialize large arrays...
first, what is considered a large array?
( > 140, what about 2D- 140x30)

The Date Object's memory usage:
Could someone expand on this.

CJ



Comment

From: cheekycj
Date: Wednesday, December 22 1999 - 01:27AM CST

This project is coming to a close soon and unfortunately so will this discussion (it has been one
of the best
threads I have been involved in at EE)

I am going to give each of you 300 pts for participating.

The project is ending in about 3 weeks or so. This memory issue is the only outstanding issue
left.

This week we upgraded the Physical RAM on the machine to 384 MB and it has a swap space
of 2GB. so we
are now using
the java command:
java -Xms384M -Xmx768M programname

We still believe that the heap isn't being fully used.  I guess this does lead to the question...
Does java use
the heap for everything?  How can we be sure of its usage?  the command line option of -
verbosegc does
not provide that info.

CJ


Comment

From: mbormann
Date: Wednesday, December 22 1999 - 01:32AM CST

Yeas CJ, u r right Jod can u tell us where u got those info I mean from which URL?


Comment

From: mbormann
Date: Wednesday, December 22 1999 - 01:41AM CST

CJ,
that is the exact reason I gave that URL Memory management ref.

>>>>>(it has been one of the best threads I have been involved in at EE)

same goes with me,I love  anything to do with Memory issues count me in always

and fyi see this
http://www.sun.com/smi/Press/sunflash/9909/sunflash.990928.3.html
and
http://java.sun.com/aboutJava/communityprocess/jsr/jsr_001_real_time.html

see this too
http://www.ddj.com/articles/1999/9911/9911h/9911h.htm

see if u can implement this though my bosses take on new stuff very very slowly


Comment

From: Jod
Date: Wednesday, December 22 1999 - 03:51AM CST

The link was:

http://www.cs.cmu.edu/~jch/java/size.html

I guess large is relative, but anything over a 1000 I would consider to be getting large.

The point is about saving space though, so any array which takes up more space to initialise
using the
normal methods as opposed to using less byte code by parsing manually is the trade off so it
depends on
how important an issue this is in saving space.


Comment

From: Jod
Date: Wednesday, December 22 1999 - 04:26AM CST

mb, the thing is that technique is one for optimising hash table load factors and trying to deal
with key
collisions and other problems in hash tables and indexes in databases for example.

I'm not sure how it is relevant in a linear structure like a vector, which is basically a growable
array but don't
have any time to think about this right now...

Are you adding another 140 to the vector on each run max?

Why not

new Vector (140,[minimum number of elements you are going to add]);

Then you make only one or two additions to the vector capacity on each run never using more
space than is
necessary.


Comment

From: mbormann
Date: Wednesday, December 22 1999 - 04:05PM CST

Yes Jod ,
mebbe u r right I was overlooking that but somehow somewhere I feel that if u let the Vector
grow in Prime
number fashion the access methods would be faster.

This is just a hunch ,thats all ,since both r dynamic growing thingies.

Guys I am going on a hoilday for 10 days starting tommorrow ,have a Merry Christmas and a
happy new
year!


Comment

From: mbormann
Date: Wednesday, December 22 1999 - 10:32PM CST

http://c2.com/cgi/wiki?JavaIdioms

these r very good ,I came across this from THE starting page of patterns
http://hillside.net/patterns/


Comment

From: vladi21
Date: Sunday, December 26 1999 - 01:24AM CST

Merry Christmas :)


Comment

From: linda
Date: Friday, January 28 2000 - 11:22PM CST

This question has been undeleted.  The points for this question were originally 200 but have
been reduced
to 0


Comment

From: linda
Date: Friday, January 28 2000 - 11:24PM CST

cheekycj has requested I undelete this question.  I reset the points to 200.

Linda Gardner
Community Support @ Experts Exchange


Comment

From: mbormann
Date: Saturday, January 29 2000 - 12:14AM CST

hope it stays that way,Linda,
:-)
what is your policy when deleting these questions? Though if u dont answer here will definitely
look it up in
Customer service later ...

Thanks


Comment

From: linda
Date: Saturday, January 29 2000 - 12:23AM CST

We have an auto-delete function which deletes questions if there has been no activity for 20
days.  It looks
like this question may have gotten deleted by that feature.

Linda


Comment

From: cheekycj
Date: Saturday, January 29 2000 - 12:23AM CST

We are back and alive!!! I don't know if all of you noticed but the question disappeared after
the new year
and we just got back.

My memory issues were mostly caused by a Java Bean called Formula1.

It is a spreadsheet bean.  We were
loading 403K rows and 38 cols in it.

After extensive tech. discussion with their engineers we found out that the bean allocates 44
Bytes per cell
w/out any data-- Thats is 643 MB of RAM w/out any DATA!!! so when we were loading about
4 Bytes on
average but we allocated 6 Bytes to be conservative and the min required memory was 731 for
just the
BEAN.  So then adding on the application and other functions and iterations we were going
through the new
target machine needed about 1GB of memory to run safely.  Needless to say this was not an
option for us
so we ripped our code apart and wrote code that would break up the history into segments and
load
segment at a time (partitions of 5-7 days) instead of the whole 30 days of history.  We are still
implementing this but it is *supposed* to fix our memory issues.

I really enjoyed this thread and if we want to continue it ... sure.. I will start to give out points
next week.

I am starting a new project in Java :-)
that is supposed to have streamlined swing GUI ... something to look forward to.

Again appreciate all the help and
NOTE: you have been forewarned on
Tidestone's Formula1 for Java Spreadsheet Bean.

CJ


Comment

From: vladi21
Date: Saturday, January 29 2000 - 01:10AM CST

:)


Comment

From: Jod
Date: Saturday, January 29 2000 - 01:35AM CST

Ahhhh...At last you have the identity of your tormentor.

Out of interest, was it due to double buffering or graphics resources for each cell of the
spreadsheet grid?

I was thinking of what I wrote above...

"Lightweight components can be just as bad if they are double buffered (which swing
components are) and
do not effectively return resources after repainting.

In other words they can hang on to their graphics double buffer memory instead of releasing it
after each
repaint. This is often done to keep the speed up but means that even a text box can eat up
several K just
by being put on screen. Larger components, larger buffers...how far this applies to JDK 122
may need
investigating but it is has been a factor for us previously. "

As I said, this happened to us as well and took a bit of delving around to track down.




Comment

From: mbormann
Date: Saturday, February 05 2000 - 09:27PM CST

CJ,
I think that if u dont make this a PAQ it will again get deleted
:-)


Comment

From: cheekycj
Date: Monday, February 07 2000 - 11:00PM CST

I am waiting for a reply from the TECHIES at Tidestone to answer Jod's question.  I will give
them one more
day.  If not then I will start close this question and give out the rest of the points.

CJ


Comment

From: vladi21
Date: Monday, February 07 2000 - 11:22PM CST

waiting ...
:)


Comment

From: cheekycj
Date: Tuesday, February 08 2000 - 02:23AM CST

I heard back from them...

Here is their response:
>I consulted with some of the
>developers and found that part of the
>reason a cell takes about 35 to 44
>bytes per cell, is caused by the
>loading of Formula One objects that
>handle behaviors like fonts, event
>handling, etc.  The objects are
>allocated when data and/or formatting
>is put into a cell.

Not much huh?

Well, we will put this issue to rest.

I thoroughly enjoyed this thread.. and learned a lot about Java's Performance and memory
issues.

Thanx to all who participated.

I awarding the points for this to Jod
and I will post 200 pt questions for
vladi21 and mbormann

CJ


Comment

From: cheekycj
Date: Tuesday, February 08 2000 - 02:26AM CST

Thanks Jod, vladi21 and mbormann

Cheers,
CJ


Comment

From: Jod
Date: Tuesday, February 08 2000 - 08:58PM CST

Cheers CJ - I think we all learnt something here...









0
 

Author Comment

by:j_baxter99
ID: 2670510
sorry i was so late in awarding pnts, only getting round to it now
0

Featured Post

Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

For beginner Java programmers or at least those new to the Eclipse IDE, the following tutorial will show some (four) ways in which you can import your Java projects to your Eclipse workbench. Introduction While learning Java can be done with…
Introduction This article is the first of three articles that explain why and how the Experts Exchange QA Team does test automation for our web site. This article explains our test automation goals. Then rationale is given for the tools we use to a…
This tutorial will introduce the viewer to VisualVM for the Java platform application. This video explains an example program and covers the Overview, Monitor, and Heap Dump tabs.
This tutorial explains how to use the VisualVM tool for the Java platform application. This video goes into detail on the Threads, Sampler, and Profiler tabs.

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now