Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

Java Memory Issues...(Immediate Attention)

Posted on 1999-11-19
73
Medium Priority
?
5,277 Views
Last Modified: 2013-12-29
I am working on a GUI Intensive (using all swing components and a couple of AWT components) program and it is finished but keeps running out of memory after running for a day.  It is a real-time system so it needs to keep running.  I am running System.gc as a separate thread that kicks off when the system has nothing to do.  It is reporting 60% free memory but I still get OutofMemory Errors.  Any ideas/help?

I am running on a Solaris Ultra Machine with 128 MB RAM.
The Swap space is set to 256 MB and I have JDK 1.2.2 installed on it.

I run the program with the following command:
java -Xms64M -Xmx128M programname

Again it says I have 60% free mem but still get the outofMemory errors and for some reason am using very little of the swap space.

I am looking for ways of Memory management, freeing memory, using the VM to the MAX, what components are memory intensive etc.

Thanx,
CJ
PS: I will increase points as needed.
0
Comment
Question by:cheekycj
  • 20
  • 17
  • 15
  • +5
73 Comments
 
LVL 5

Expert Comment

by:fontaine
ID: 2219909
Go to the KL Group Web site (www.klgroup.com) and download the
trial version of JProbe. This tool
will help you to determine where
memory is consumed.
0
 
LVL 5

Expert Comment

by:mbormann
ID: 2220004
CJ,
can u pls look at this question so that u can have some pointer?

http://www.experts-exchange.com/jsp/qShow.jsp?ta=java&qid=10216695 

Quoting wrostek
"In my special case I frequently added a new list model to a JList. It
seems that the old model is not garbadge collected. Now I reuse the
model which solves my problem."

end quote
well the thing to do is to recycle components.
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 5

Expert Comment

by:Jod
ID: 2220102
Have you tried with larger Xms values?

From specific experience I can give you some info regarding GUI components.

Even simple text components can eat large chunks of memory if there are enough of them on screen or created at any one time.

Try creating a large number of AWT components and watch the application grind to a halt as they appear on screen or are updated.

(for example...

http://forum.java.sun.com/forum?13@133.CDKpacmqbtm^0@.ee794b0/0

)

Lightweight components can be just as bad if they are double buffered (which swing components are) and do not effectively return resources after repainting.

In other words they can hang on to their graphics double buffer memory instead of releasing it after each repaint. This is often done to keep the speed up but means that even a text box can eat up several K just by being put on screen. Larger components, larger buffers...how far this applies to JDK 122 may need investigating but it is has been a factor for us previously.

Swing components can have specific problems returning resources when being garbage collected, even if they are disposed specifically. For example, check out this bug:

http://developer.java.sun.com/developer/bugParade/bugs/4229318.html 

or

http://developer.java.sun.com/developer/bugParade/bugs/4156232.html

Swing eats resources and has a large standard footprint.

Be aware that basically, neither Swing nor AWT doesn't disposal down the component hierarchy. The result is that any component that has registered itself as a listener doesn't get a chance to deregister itself, whatever it is listening to keeps a live pointer to it, and indirectly, all of the components in the window remain reachable, and not garbage collectable.

In Swing, if you create a button with an action (using the add( Action ) function of JMenu or JToolBar), the button registers itself. If the action is a singleton (presumable a frequent occurance), you have a memory leak.


So, this sort of memory leak is a common occurrence in Java - if you have a listener registered on a GUI component, the component may not be garbage collected until all the listeners are gone (or deregistered from the component) as well.
In general:

(see http://forum.java.sun.com/forum?13@133.CDKpacmqbtm^0@.ee78b38/0 for original info on this)


In general:

Does your system allocate separate resources for different users/sessions or is it intrinsically a single user system with one set of resources?

Also does it have multiple screens or windows?


Another area is SQL and database resources. Do you use prepared statements that are kept active for long periods of time for example or maintain large data sets in memeory?

Hope this helps...
0
 
LVL 5

Expert Comment

by:mbormann
ID: 2220113
http://www.javaworld.com/javaworld/javatips/jw-javatip79.html

http://forum.java.sun.com/forum?13@133.CDKpacmqbtm^0@.ee78b38/0

CJ,
it would be better if u outline atleast some thing abt ur code so that others can help u out.
0
 
LVL 7

Expert Comment

by:vladi21
ID: 2220155
http://forum.java.sun.com/forum?13@133.CDKpacmqbtm^0@.ee78b38/0

mbormann  and Jod - you use same search engine :)
0
 
LVL 5

Expert Comment

by:mbormann
ID: 2220175
Sorry all for butting in but we r using KL group's Jclass BWT ( licensed and with our extended widgets ) and it does not SEEM to have these sort of problems.Atleast i haven't heard of it at our place
,our bosses have taken the decision not to move to Swing in the near future.

Also 128MB is way too low ,program will be spending way too much time in the swap mode,have a recommended 256 Megs atleast(?)

Jod has explained the things in detail and beautifully in sequence
0
 
LVL 5

Expert Comment

by:Jod
ID: 2220187
All roads lead to the same destination here I think (your link looks familiar as well...:).

It is a problem we have had before.
0
 
LVL 7

Expert Comment

by:vladi21
ID: 2220194
we also use KL group's products
and now we don't use  Dialogs recycling - its so buggly under Win32
0
 
LVL 5

Expert Comment

by:Jod
ID: 2220248
Win32 definately appears worst hit with memory leaks.

We are using iMacs for deployment as they are so easy to setup and configure to work with Java.
0
 
LVL 4

Expert Comment

by:captainkirk
ID: 2220271
Try calling the system garbage collector directly when you're done with objects:

// I believe the call is (it's been a while)--
System.gc();

this may or may not force the GC to run right away, but it may be worth a try...
0
 
LVL 19

Author Comment

by:cheekycj
ID: 2220283
OK, now that I am caught up on my reading :-)

Here are some details of the program for you guys.

It is a JFC Application with one Main JFrame.
The JFrame has 6 tabs (all the code for gui in the tabs are also in this class- not my design)
Besides having two configuration popups and a splash screen there are no extra windows.

Now the tabs,
Tab 1: JPanel w/ a 150 KB image on it (the image extends JPanel)
and has a mouselistner.
Tab 2: JPanel w/ dozens of labels, 10 text boxes, pulldown list, radio
buttons, 1 textarea and 20 buttons
Tab 3: JPanel w/ about the same controls as Tab2 but no pulldown list and no textarea but it has buttons thatmove a value between 1-140 for
different data to display according to that number.
Tab 4: JPanel w/ Tidestone's Formula One for Java (spreadsheet bean) there.  No data displayed in the spreadsheet.  Should be relatively small.
Has 4 buttons.
Tab 5: JPanel w/ 1 TextArea that logs all events.
Tab 6: JPanel w/ Tidestone's Formula One for Java (spreadsheet bean) there is a lot of data here but according to Tidestone we are not even beginning to push the bean's upperlimit.  The sheet is set up to load 30 days worth of values.
Tab 7: JPanel that is blank. (we will remove it .. just haven't gotten around to it)

The application has your standard menu bar with 4 small menu items.
And has a status bar with 10 values displayed.

The operation;
We awake at a certain time.  FTP files from a Server. Read them in and parse ints out of the strings that are read in.  Load the data in our
spreadsheet and run 4 forumulaes on the data.

The program is supposed to run forever but gets memory leaks after 1 day. There is a lot of use of StringTokenizer.  How bad is that.

Hope this is enough info.

Thanks for the help so far.
Improvements made so far:
Use Stringbuffer for any concatenations.
Set vars to null after they are not used.
Consistent Calls to System.gc() for cleanup.

Still doesn't help.

Thanx,
CJ
0
 
LVL 19

Author Comment

by:cheekycj
ID: 2220295
For some reason I do get a warning of
running 1.1 version of SystemEventQueueUtilities
(because gui developed in Symantec VCafe 3.0 which only has JDK 1.1.8 with Swing 1.0.3 and we are delploying on JDK 1.2.2)
I hope that isn't causing problems.
I am going to keep the points the same for now (200) but
will give out more to all individuals who help out as seperate Questions.

CJ
0
 
LVL 7

Expert Comment

by:vladi21
ID: 2220386
- Optimize String operations
http://www.codeguru.com/java/articles/511.shtml

Maybe u can use StreamTokenizer?

0
 
LVL 5

Expert Comment

by:mbormann
ID: 2220394
>>>>>There is a lot of use of StringTokenizer.  How bad is that

Bad ,null it and CJ if ur StringBuffer's r huge then pre allocate them as i had a particular problem of sending 500KB string so

StringBuffer buffForSending=new StringBuffer(dataToSave.length()+((dataToSave.length())/4));
I was then adding some protocol data to String

>>>>>Tab 5: JPanel w/ 1 TextArea that logs all events.

From Linden FAQ
(Sect. 9) How much text can be put in a TextArea?

[*] TextArea just uses the corresponding widget of the             underlying   window system. It will be bounded by the limit             imposed in the native  window system. In Windows 95 TextAreas              can hold about 28Kb. The native  widget allows 32Kb, but there is              some overhead which reduces the amount available to the programmer. The limit is removed in JTextComponent in  Swing (JDK 1.2) which dispenses with peer controls.

so pls utilise JTextArea et al,i think u must be using that and meant to say it in the first place.
0
 
LVL 19

Author Comment

by:cheekycj
ID: 2220461
mbormann:Yup, we are Using all Swing Components...
So its JComboBox, JButton, JTextArea,
The StringBuffers are actually quites small. All less than 80 chars.
Should I still set a sizelimit.  Also should I declare a new Stringbuffer and use it locally in a method or use a global one?

vladi21:  I am using StringTokenizer quite a bit.

I will try setting those to null when I done using them.

Any more ideas are welcome.

CJ
0
 
LVL 19

Author Comment

by:cheekycj
ID: 2220470
Check this code:
file load code:
===============================
            while ((line = inFile.readLine()) != null) {
                StringTokenizer st = new StringTokenizer(line,",");
                if (Util.compareDate(st.nextToken())) {
                    list.addElement(line);
                    dataFound = true;
                }
                st = null; // just added this to free up memory
            }
=============================
the file that reads in is about 700 lines long
and we read in on avg 6 of these files.

CJ
0
 
LVL 19

Author Comment

by:cheekycj
ID: 2220477
vladi21:  I apologize.. I read your comment wrong.  What are the advantages of using StreamTokenizer over StringTokenizer and can it replace StringTokenizer in all situations?

CJ
0
 
LVL 7

Expert Comment

by:vladi21
ID: 2220510
what better 1 StreamTokenizer per file or StringTokenizer per line? :)
check http://developer.javasoft.com/developer/technicalArticles/Programming/PerfTuning/index.html#Tokenization

0
 
LVL 5

Expert Comment

by:mbormann
ID: 2220518
Tip 1:
-----
I think u can simplify this and not go into StreamTokenizer

while ((line = inFile.readLine()) != null)
{
//see if this chek applies in ur case
      if( (line=line.trim()).length() == 0)
      {
            line=null;
            continue;
      }
                  
      String st = line.substring(0,line.indexOf(',')).trim();
      while (Util.compareDate(st))
      {
            list.addElement(line);
            dataFound = true;
      }
      line=null;
      st = null; // just added this to free up memory
}             

Tip 2:
-----
have ALL Strings like "," etc which run in loops mentioned up top in class like say.

private static final String protocolSeparator = ",";

Tip 3:
------
seems liej u do a ReadLine ,well see what ur OS reads chunks of Files in exactly what sizes ,i think it maybe around 16KB?i am not sure,but pass the BufferedReader / DataInputStream whatever u use Buffer size of 8192 bytes /16384 buytes whatever is applicable
0
 
LVL 5

Expert Comment

by:mbormann
ID: 2220529
this ensures that ur Porgram does not use hard disk much and slow down I/O performance.Restrict ur attack then to GUI's only.
and yes please when u close the BufferedReader / DataInputStream ,explicitly null that object reference
out.close();out=null; internally it is set to null after sometime.but be on safe side.
0
 
LVL 19

Author Comment

by:cheekycj
ID: 2220998
mbormann:
I have

LineNumberReader inFile = new LineNumberReader(new FileReader(readFile));

as my declaration for reading in the file.
I incorperated most of you changes.
the file read part.
Every possible Variable is being set to null now.
I am using StringBuffer whenever Possible
Most string literals have been removed unless I log an error- which is rare.

====================

I appreciate all the help so far guys... I will continue this on Monday.
I have incorperated a lot of changes into the code and will
run the program over the weekend to see how it does.
===================

A couple of questions...
1.  Can StreamTokenizer be used in situation without I/O Streams?
didn't seem like it to me.
2.  Anyone know anything about this stupid version 1.1 of systemEventQueueUtilities warning??
3.  Is there a way to manage the memory that the components on the tabbed panes such that when they lose focus they release resource or something?

===================
You guys are gonna get a lot of points from me  :-)
Thanx a lot.
CJ
0
 
LVL 7

Expert Comment

by:vladi21
ID: 2221068
StreamTokenizer st = new StreamTokenizer(new StringReader("abc"));
0
 
LVL 7

Expert Comment

by:Ravindra76
ID: 2221708
What a nice discussion among four worthty experts ( Above 20,000 points ).
 
It is the first time for me to see all in same question in the two months
0
 
LVL 7

Expert Comment

by:Ravindra76
ID: 2221720

Hi experts,

  Be ready for arriving question of the millenium which is coming soon
0
 
LVL 5

Accepted Solution

by:
Jod earned 800 total points
ID: 2222735
>> Be ready for arriving question of the millenium which is coming soon

sounds ominous...


Something to identify is does the app run out of memory just running on it's own or is it the reading in of files that is causing the problem?

Try flicking between the tabs in your app without doing anything else for instance and see if memory usage goes up.

Just a hunch that, regarding redrawing of components.

Also play with the value and data in the fields on the tabs to see if this causes a memory usage problem here also. Keep pressing a button for example and see if it keeps on eating memory endlessly.

Every time you read a file you are creating a large number of temporary string objects that are then being left hanging almost immediately. These should be garbage collected but to be honest I am yet to really trust the garbage collecter.

Perhaps even pathological localisation of variables might help the GC remove references. For example:

            while ((String line = inFile.readLine()) != null) {
                StringTokenizer st = new StringTokenizer(line,",");
                if (Util.compareDate(st.nextToken())) {
                    list.addElement(line);
                    dataFound = true;
                }
                st = null; // just added this to free up memory
                line = null;
                // EXTREME......
                System.gc();
            }
            //NORMAL
            System.gc();

Localising the String Line variable might cleaner identification of when it's resources are out of context.

Or even move the file reading code to another class to allow easy identification of the resources allocated to it.

The definition of readLine you are using is:

    public String readLine() throws IOException {
      synchronized (lock) {
          String l = super.readLine(skipLF);
            skipLF = false;
          if (l != null)
            lineNumber++;
          return l;
      }
    }

The default buffer size is 8192 bytes.

OUt of interest, is it possible for you to read into a byte/character buffer instead of reading strings and whole lines? Reading bytes from a file is much more efficient and more predictable in a real time system that is running all the time, rather than using these convenience methods to do the character processing for you.

It may be worth considering as then you will have complete control over the long term efficiency of the code.

So what format are the files you are reading in and are they predictable in this manner?



I am presuming that the data read in is displayed in the Formula One grid? How is it processed to get in and what sort of data model does the grid use (how efficient is it at memory management?)

>> 2.  Anyone know anything about this stupid version 1.1 of systemEventQueueUtilities warning??
>> 3.  Is there a way to manage the memory that the components on the tabbed panes such that when they lose focus they release resource or something?

Regarding the warning, I would try downloading some of the following tools to see what objects are hanging around and whether your components have any issues with this:

http://java.sun.com/products/jfc/jaccess-1.0/doc/examples.html

The warning sounds like it is related to the Java Accesibility API and the way that in JDK 1.1 the event queue is replaced to retro fit this feature.

If the utilities do not work, then the event queue is not being replaced with the one relevant for Java Accessibility API. I would need to look into this in greater detail to tell you more about any problems this might be causing.

Enough for now...
0
 
LVL 5

Expert Comment

by:Jod
ID: 2222741
The definition of readLine given is as in linebufferreader.

The readline code in bufferedreader is also possibly not very efficient:

    /**
     * Read a line of text.  A line is considered to be terminated by any one
     * of a line feed ('\n'), a carriage return ('\r'), or a carriage return
     * followed immediately by a linefeed.
     *
     * @param      skipLF  If true, the next '\n' will be skipped
     *
     * @return     A String containing the contents of the line, not including
     *             any line-termination characters, or null if the end of the
     *             stream has been reached
     *
     * @see        java.io.LineNumberReader#readLine()
     *
     * @exception  IOException  If an I/O error occurs
     */
    String readLine(boolean skipLF) throws IOException {
      StringBuffer s = new StringBuffer(defaultExpectedLineLength);
      synchronized (lock) {
          ensureOpen();

      bufferLoop:
          for (;;) {

            if (nextChar >= nChars)
                fill();
            if (nextChar >= nChars) { /* EOF */
                if (s.length() > 0)
                  return s.toString();
                else
                  return null;
            }
            boolean eol = false;
            char c = 0;
            int i;

                /* Skip a leftover '\n' */
            if (skipLF && (cb[nextChar] == '\n'))
                    nextChar++;

          charLoop:
            for (i = nextChar; i < nChars; i++) {
                c = cb[i];
                if ((c == '\n') || (c == '\r')) {
                  eol = true;
                  break charLoop;
                }
            }
            s.append(cb, nextChar, i - nextChar);
            nextChar = i;

            if (eol) {
                nextChar++;
                if (c == '\r') {
                  if (nextChar >= nChars)
                      fill();
                  if ((nextChar < nChars) && (cb[nextChar] == '\n'))
                      nextChar++;
                }
                break bufferLoop;
            }
                skipLF = false;
            }
      }

      return s.toString();
    }

This is called by bufferedreader.readLine to do the read you want. The byte array readers are generally a bit 'cleaner'. Readline code is a convenience.

0
 
LVL 19

Author Comment

by:cheekycj
ID: 2225465
OK... I see the overhead problem w/ Readline.  I don't have to use
that.. it was just convenient.

The files are flat text files that are in comma delimited format.
Each line in the file is a record that is timestamped.
I need to read in each record(line) compare the timestamp to the
current time period and if it falls in the current time period add it to my table.  That is why I used readline and then I need to tokenize
the String to get its first element and compare its timestamp.
then I add it to a vector list.

but... would this be the equivalent:
====================================
            FileInputStream fis = new FileInputStream(args[0]);
            byte buf[] = new byte[1024];
            int n;
            while ((n = fis.read(buf)) != -1) {
                for (int i = 0; i < n; i++) {
                    if (buf[i] == '\n')
                        String line = buff;
                        String st = line.substring(0,line.indexOf(Const.COMMA)).trim();
                if (Util.compareDate(st)) {
                    list.addElement(line);
                    dataFound = true;
                }
                line=null;
                st = null; // just added this to free up memory
                }
            }
            fis.close();
======================================
If not how would I extract the each line into a string to get its first
element and compare the datestamp?

Thanx,
CJ
0
 
LVL 7

Expert Comment

by:vladi21
ID: 2227873
also if u use ASCII text look at

    /**
     * Reads the next line of text from this data input stream. This
     * method successively reads bytes from the underlying input stream
     * until it reaches the end of a line of text.
     * <p>
     * A line of text is terminated by a carriage return character
     * (<code>'&#92;r'</code>), a newline character (<code>'&#92;n'</code>), a
     * carriage return character immediately followed by a newline
     * character, or the end of the input stream. The line-terminating
     * character(s), if any, are not returned as part of the string that
     * is returned.
     * <p>
     * This method blocks until a newline character is read, a carriage
     * return and the byte following it are read (to see if it is a
     * newline), the end of the stream is detected, or an exception is
     * thrown.
     *
     * @deprecated This method does not properly convert bytes to characters.
     * As of JDK&nbsp;1.1, the preferred way to read lines of text is via the
     * <code>BufferedReader.readLine()</code> method.  Programs that use the
     * <code>DataInputStream</code> class to read lines can be converted to use
     * the <code>BufferedReader</code> class by replacing code of the form
     * <ul>
     *     <code>DataInputStream d =&nbsp;new&nbsp;DataInputStream(in);</code>
     * </ul>
     * with
     * <ul>
     *     <code>BufferedReader d
     *          =&nbsp;new&nbsp;BufferedReader(new&nbsp;InputStreamReader(in));
     *      </code>
     * </ul>
     *
     * @return     the next line of text from this input stream, or
     *             <tt>null</tt> if no bytes are read before end-of-file
     *             is reached.
     * @exception  IOException  if an I/O error occurs.
     * @see        java.io.BufferedReader#readLine()
     * @see        java.io.FilterInputStream#in
     */
    public final String readLine() throws IOException {
      InputStream in = this.in;
      char buf[] = lineBuffer;

      if (buf == null) {
          buf = lineBuffer = new char[128];
      }

      int room = buf.length;
      int offset = 0;
      int c;

loop:      while (true) {
          switch (c = in.read()) {
            case -1:
            case '\n':
            break loop;

            case '\r':
            int c2 = in.read();
            if (c2 != '\n') {
                if (!(in instanceof PushbackInputStream)) {
                  in = this.in = new PushbackInputStream(in);
                }
                ((PushbackInputStream)in).unread(c2);
            }
            break loop;

            default:
            if (--room < 0) {
                buf = new char[offset + 128];
                room = buf.length - offset - 1;
                System.arraycopy(lineBuffer, 0, buf, 0, offset);
                lineBuffer = buf;
            }
            buf[offset++] = (char) c;
            break;
          }
      }
      if ((c == -1) && (offset == 0)) {
          return null;
      }
      return String.copyValueOf(buf, 0, offset);
    }




maybe some modification needed? :)
0
 
LVL 5

Expert Comment

by:Jod
ID: 2228013
cheekycj:

yep along the right lines - avoiding all the temporary strings is the main problem, because I am not sure how efficiently these are being garbage collected (in the real world that is)

What you really need is a sort of rolling buffer that always stores the next one or more lines at a time. You can then refill this buffer when it runs out and extract the current data from it as neccessary.

How to avoid extracting the data into a string unless the date at the start of this buffer indicates it is to be retained is not straightforward (possibly unavoidable and there are a few complications here)

Before heading down this route, have you been able to determine if it is  definately the file reading that is causing the application to run out of memory over time?
0
 
LVL 19

Author Comment

by:cheekycj
ID: 2232028
well, the good news is that the system hasn't run out of memory.. though it is at 68% after only 2 days.  We are going to let it run over the holidays and see at what point we run out of memory.  I think at that point it may become a issue of buy more memory rather than better memory management.  Does anyone know if there is harm in specifying a swap space that is 2x or 3x your physical memory (RAM) ?

CJ
0
 
LVL 7

Expert Comment

by:vladi21
ID: 2232171
swap =2x RAM
0
 
LVL 7

Expert Comment

by:vladi21
ID: 2232226
Q. How big and where should my Pagefile be?

A. Below are things to consider.

Do not have 2 pagefiles on the same physical disk.
The pagefile should be size of memory +11MB (or 2.5 times size of SAM file, which ever is larger).
If possible have multiple page files on separate physical disks
There are some performance gains to be had by placing on a stripe set, however not as much as on separate disks.
Defragment disk pagefile is on
Try and make drive NTFS
Minimum pagefile size is 2MB
To enhance performance, one can create a second pagefile on another physical disk. MOVING, however, is never advised, since it disables the option to create a Memory dump file at a crash (System Properties, Startup-tab). In order to be able to dump the RAM content to the pagefile (saved i.e. as MEMORY.DMP), the pagefile MUST be located (as well) on the boot partition.

http://www.ntfaq.com/ntfaq/perform3.html
0
 
LVL 5

Expert Comment

by:Jod
ID: 2232290
The day that I have to look through a MEMORY.DMP file is the day I get a new job.

vladi21's advice is sound - however I think cheekycj is running under Solaris...:-)

(As an aside I have to say though, that I always try and move the pagefile to a disk other than the system disk to avoid thrashing and provide better dual access to the system and pagefile concurrently under NT.)

NOTE:

This may be true under solaris as well, but under NT we try to specify a pagefile with a minimum and maximum size the same so that the operating system can always allocate the whole page file as a single block at startup rather than having to increase the size incrementally whilst running.

I think your max swap size is the right size at 2 X ram (as you have it...)
0
 
LVL 19

Author Comment

by:cheekycj
ID: 2232321
Jod: Yup, I am on Solaris.
I use the command:
java -Xms128M -Xmx128M programname

but was thinking what if... I used:
java -Xms256M -Xmx256M programname
or even...
java -Xms512M -Xmx512M programname

The Sun Ultra 5 has 128MB physical RAM.
(I will increase the swapdisk size on unix of course)

CJ


0
 
LVL 5

Expert Comment

by:mbormann
ID: 2233395
CJ,
i think i mentioned in the beginning that please increase RAM to 256MB,that is the size we recommend to people who buy our product and advisably pls tell them to have that machine Only for that,this figure was arrived at after lotsa research ,our GUI is very fat around 1.8MB jar files size.
Remember that it is a necessary but not a sufficient condition(where did we hear that before? Yup Deadlocks. :-))

I have been after them for obfuscating as this reduces jar size and the memory fottprint,i wonder why nobody talked abt this before?The reduction is I have heard from another friend anyhting from 30 -60 %

Regards
0
 
LVL 5

Expert Comment

by:mbormann
ID: 2233396
another comment blaberring along ,why docha get those new fangled Java Stations running Java from ground up?ignore this if u feel it's Ho hum
0
 
LVL 7

Expert Comment

by:vladi21
ID: 2234020
yes
try
JAX reads class files, analyzes them, and generates a compressed ZIP file as output
http://www.alphaworks.ibm.com/formula/JAX

and

DashO
http://www.preemptive.com/index.html

PreEmptive is proud to announce the upcoming DashO-Embedded Edition! This product will not only include the power of DashO-Pro for maximum class level size reduction, but also the DashO bytecode optimizer for the smallest/fastest bytecode available!
0
 
LVL 5

Expert Comment

by:mbormann
ID: 2235681
0
 
LVL 5

Expert Comment

by:Jod
ID: 2238723
Yet another thing that has been mentioned elsewhere but not here is the use of string.intern()

Strings computed at run time are newly created and therefore created as distinct objects.

However, as the docs say, the result of explicitly interning a computed string is the same string as any pre-existing literal string with the same contents.

In other words, you can reuse an existing string in the global string pool if an identical one exists.

This is only useful if you have a lot of repetitive use of the same strings in your file that you read in, which is quite possible.
0
 
LVL 5

Expert Comment

by:Jod
ID: 2242183
I saw this and thought of you cheekycj...

Am I not nice to you or what:

http://www.cs.cmu.edu/~jch/java/size.html

0
 
LVL 19

Author Comment

by:cheekycj
ID: 2252685
After running more and more tests are some results ... for you guys...

StringTokenizer uses less memory than
the String.substring() command (go figure)

BufferedReader gave me the best performance when it came to the reading of the text file.

Direct Buffering was the fastest but when you wanted to extract and manipulate as you read in the file (like I was doing in BufferedReader) it became magnitudes slower.  Direct Buffering was also inconsistent with my Data.. sometimes pulling in a line at a time and sometimes not ... so I had to stick with Readline cmd.

Using Const.COMMA instead of "," everywhere saved a few bytes but probably more after being run for a while.

We were running at -Xmx128M but increased it to -Xmx256M and will be
going to -Xmx512M as we are bumping up the RAM in the machine to 256 MB.

I am reading in a 860 KB TEXT file
in 1 Sec. including the String compares
and storing in a Vector.  I knew the size of the array before hand but when I used an array of Strings the memory usage was a lot more than with the Vector... another one I couldn't figure out.

I will post more as I learn more.

I am hoping to wrap up this question in a couple of weeks.  I appreciate all the info and support.  The points will be given out to all active participants.

Thanx,
CJ

0
 
LVL 5

Expert Comment

by:mbormann
ID: 2252721
>>>>>bumping up the RAM in the machine to 256 MB.
good news!

>>>>>and storing in a Vector

dont use the default constructor of that ,control the memory space by allocating something like

public Vector(int initialCapacity,int capacityIncrement)

i used it for something like
new Vector(1021,193);

both r very important here ,chk out ur owbn requirement

>>>>>memory usage was a lot more than

can u post some rough approximate code?so others can see it?
0
 
LVL 5

Expert Comment

by:mbormann
ID: 2252727
forgot to add reading Core Java by gary cornell &Cay horstmann,
if i am not jumbled up they say that new Vector(int ,int) both shuld ideally be prime nos between power of 2

is this right guys?
0
 
LVL 5

Expert Comment

by:Jod
ID: 2258470
CJ, some points from the above link on optimising Java:

Regarding your experience with Vectors...

Don't initialize big arrays:

Although array initialization is a single statement in Java, the generated bytecode inserts one element at a time. If you've got a big array, this requires a lot of bytecode. You can save space by storing your data in a String and then parsing it into your array at runtime (tip from Andrew Scherpbier).

(So as vectors are just growable arrays, bearing this in mind they are very likely to be far more code efficient than creating large initialised arrays pre defined to the relevant size.)


Perhaps of significance...

Dates are big (not just broken)

Date objects take a surprising amount of space for something with such limited functionality. If you're storing a lot of them, considering using longs instead, and recreating Date objects from them as necessary (tip from Jean-Marc Lugrin).


Finally I imagine that Const.COMMA is more efficient than "," for the same reason that String.intern() is more emory efficient, because you are not creating a temporary string for "," every time it is needed

Did you look into using string.intern() at all?


mb:

Not sure about the Vector thing. What reason do they give?

You want to keep the initial vector size to be close to the minimum expected in this case so that you only allocate more space where neccessary - the amount of time it takes to increase the Vectors memory space is probably not an issue here.


Sounds like you are getting there...
0
 
LVL 5

Expert Comment

by:mbormann
ID: 2280193
Sorry Jod,
I was away for a while but if u look up the JDK Docs for Vector

protected int capacityIncrement

The amount by which the capacity of the vector is automatically incremented when its size becomes greater than its capacity. If the capacity is 0, the capacity of the vector is doubled each time it needs to grow.

Now if u give a default new Vector() this doubles every time the addElement() goes on adding elements to Vector.

This grwos enormously as u can see after calculating,so u should control the addition to the capacity of the vector.

More docs

the vector's
storage increases in chunks the size of capacityIncrement. An application can increase the capacity of a vector before
inserting a large number of components; this reduces the amount of incremental reallocation.


Also if u know that u are going to have say atleast 50 elems then better alloc it at start say

new Vector(64,64);

Remember that Vector and Hashtable is NOT MEANT for large stuff in the range of above 1000 or so,thats a Rule of Thumb,as it EATS memory very much.

Regards
0
 
LVL 5

Expert Comment

by:Jod
ID: 2294639
Here we go mb...

The Memory Management Reference
http://www.harlequin.com/products/st/mm/bib/full.html 

What I actually meant in the above was, why prime nos between power of 2?


I would just make the vector large enough to accomodate the minimum expected items and then if it was too small only increment with enough space to store one additional item each time. that's what I meant be the trade off, "the amount of time it takes to increase the Vectors memory space is probably not an issue here" - it's the memory conservation that is more important.







0
 
LVL 19

Author Comment

by:cheekycj
ID: 2295030
we are storing 140 elements (max) each time... using a string array was an option but I think the memory used was less w/ Vectors.

So the code should look like what?

new Vector (128,256)

??

CJ
0
 
LVL 5

Expert Comment

by:mbormann
ID: 2297590
CJ,
new Vector (140,23) ;

Jod,
the prime number between power of 2 is I believe a old hashing concept
as it is proved here
http://www-compsci.swan.ac.uk/~csmark/download.html
and
http://wwwinfo.cern.ch/asd/lhc++/Objectivity/V5.2/Java/api/com/objy/db/util/ooMap.html

http://hissa.ncsl.nist.gov/~black/CRCDict/HTML/hashtab.html
http://www.ee.uwa.edu.au/~plsd210/ds/hash_tables.html

I think this same thingy applies to Vectors
0
 
LVL 19

Author Comment

by:cheekycj
ID: 2298033
After reading some of the above (again)
I noticed Jod stated that do not initialize large arrays...
first, what is considered a large array?
( > 140, what about 2D- 140x30)

The Date Object's memory usage:
Could someone expand on this.

CJ
0
 
LVL 19

Author Comment

by:cheekycj
ID: 2298052
This project is coming to a close soon and unfortunately so will this discussion (it has been one of the best threads I have been involved in at EE)

I am going to give each of you 300 pts for participating.

The project is ending in about 3 weeks or so. This memory issue is the only outstanding issue left.

This week we upgraded the Physical RAM on the machine to 384 MB and it has a swap space of 2GB. so we are now using
the java command:
java -Xms384M -Xmx768M programname

We still believe that the heap isn't being fully used.  I guess this does lead to the question... Does java use the heap for everything?  How can we be sure of its usage?  the command line option of -verbosegc does not provide that info.

CJ
0
 
LVL 5

Expert Comment

by:mbormann
ID: 2298066
Yeas CJ, u r right Jod can u tell us where u got those info I mean from which URL?
0
 
LVL 5

Expert Comment

by:mbormann
ID: 2298093
CJ,
that is the exact reason I gave that URL Memory management ref.

>>>>>(it has been one of the best threads I have been involved in at EE)

same goes with me,I love  anything to do with Memory issues count me in always

and fyi see this
http://www.sun.com/smi/Press/sunflash/9909/sunflash.990928.3.html
and
http://java.sun.com/aboutJava/communityprocess/jsr/jsr_001_real_time.html

see this too
http://www.ddj.com/articles/1999/9911/9911h/9911h.htm

see if u can implement this though my bosses take on new stuff very very slowly
0
 
LVL 5

Expert Comment

by:Jod
ID: 2298407
The link was:

http://www.cs.cmu.edu/~jch/java/size.html

I guess large is relative, but anything over a 1000 I would consider to be getting large.

The point is about saving space though, so any array which takes up more space to initialise using the normal methods as opposed to using less byte code by parsing manually is the trade off so it depends on how important an issue this is in saving space.
0
 
LVL 5

Expert Comment

by:Jod
ID: 2298522
mb, the thing is that technique is one for optimising hash table load factors and trying to deal with key collisions and other problems in hash tables and indexes in databases for example.

I'm not sure how it is relevant in a linear structure like a vector, which is basically a growable array but don't have any time to think about this right now...

Are you adding another 140 to the vector on each run max?

Why not

new Vector (140,[minimum number of elements you are going to add]);

Then you make only one or two additions to the vector capacity on each run never using more space than is necessary.
0
 
LVL 5

Expert Comment

by:mbormann
ID: 2299720
Yes Jod ,
mebbe u r right I was overlooking that but somehow somewhere I feel that if u let the Vector grow in Prime number fashion the access methods would be faster.

This is just a hunch ,thats all ,since both r dynamic growing thingies.

Guys I am going on a hoilday for 10 days starting tommorrow ,have a Merry Christmas and a happy new year!
0
 
LVL 5

Expert Comment

by:mbormann
ID: 2300323
http://c2.com/cgi/wiki?JavaIdioms

these r very good ,I came across this from THE starting page of patterns
http://hillside.net/patterns/
0
 
LVL 7

Expert Comment

by:vladi21
ID: 2305940
Merry Christmas :)
0
 
LVL 7

Expert Comment

by:linda101698
ID: 2396656
This question has been undeleted.  The points for this question were originally 200 but have been reduced to 0
0
 
LVL 7

Expert Comment

by:linda101698
ID: 2396669
cheekycj has requested I undelete this question.  I reset the points to 200.

Linda Gardner
Community Support @ Experts Exchange
0
 
LVL 5

Expert Comment

by:mbormann
ID: 2396862
hope it stays that way,Linda,
:-)
what is your policy when deleting these questions? Though if u dont answer here will definitely look it up in Customer service later ...

Thanks
0
 
LVL 7

Expert Comment

by:linda101698
ID: 2396898
We have an auto-delete function which deletes questions if there has been no activity for 20 days.  It looks like this question may have gotten deleted by that feature.  

Linda
0
 
LVL 19

Author Comment

by:cheekycj
ID: 2396900
We are back and alive!!! I don't know if all of you noticed but the question disappeared after the new year and we just got back.

My memory issues were mostly caused by a Java Bean called Formula1.

It is a spreadsheet bean.  We were
loading 403K rows and 38 cols in it.

After extensive tech. discussion with their engineers we found out that the bean allocates 44 Bytes per cell w/out any data-- Thats is 643 MB of RAM w/out any DATA!!! so when we were loading about 4 Bytes on average but we allocated 6 Bytes to be conservative and the min required memory was 731 for just the BEAN.  So then adding on the application and other functions and iterations we were going through the new target machine needed about 1GB of memory to run safely.  Needless to say this was not an option for us so we ripped our code apart and wrote code that would break up the history into segments and load segment at a time (partitions of 5-7 days) instead of the whole 30 days of history.  We are still implementing this but it is *supposed* to fix our memory issues.

I really enjoyed this thread and if we want to continue it ... sure.. I will start to give out points next week.

I am starting a new project in Java :-)
that is supposed to have streamlined swing GUI ... something to look forward to.

Again appreciate all the help and
NOTE: you have been forewarned on
Tidestone's Formula1 for Java Spreadsheet Bean.

CJ
0
 
LVL 7

Expert Comment

by:vladi21
ID: 2397079
:)
0
 
LVL 5

Expert Comment

by:Jod
ID: 2397166
Ahhhh...At last you have the identity of your tormentor.

Out of interest, was it due to double buffering or graphics resources for each cell of the spreadsheet grid?

I was thinking of what I wrote above...

"Lightweight components can be just as bad if they are double buffered (which swing components are) and do not effectively return resources after repainting.

In other words they can hang on to their graphics double buffer memory instead of releasing it after each repaint. This is often done to keep the speed up but means that even a text box can eat up several K just by being put on screen. Larger components, larger buffers...how far this applies to JDK 122 may need investigating but it is has been a factor for us previously. "

As I said, this happened to us as well and took a bit of delving around to track down.

0
 
LVL 5

Expert Comment

by:mbormann
ID: 2492489
CJ,
I think that if u dont make this a PAQ it will again get deleted
:-)
0
 
LVL 19

Author Comment

by:cheekycj
ID: 2496235
I am waiting for a reply from the TECHIES at Tidestone to answer Jod's question.  I will give them one more day.  If not then I will start close this question and give out the rest of the points.

CJ
0
 
LVL 7

Expert Comment

by:vladi21
ID: 2496294
waiting ...
:)
0
 
LVL 19

Author Comment

by:cheekycj
ID: 2497063
I heard back from them...

Here is their response:
>I consulted with some of the
>developers and found that part of the
>reason a cell takes about 35 to 44
>bytes per cell, is caused by the
>loading of Formula One objects that
>handle behaviors like fonts, event
>handling, etc.  The objects are
>allocated when data and/or formatting
>is put into a cell.

Not much huh?

Well, we will put this issue to rest.

I thoroughly enjoyed this thread.. and learned a lot about Java's Performance and memory issues.

Thanx to all who participated.

I awarding the points for this to Jod
and I will post 200 pt questions for
vladi21 and mbormann

CJ
0
 
LVL 19

Author Comment

by:cheekycj
ID: 2497071
Thanks Jod, vladi21 and mbormann

Cheers,
CJ
0
 
LVL 5

Expert Comment

by:Jod
ID: 2499631
Cheers CJ - I think we all learnt something here...
0

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

After being asked a question last year, I went into one of my moods where I did some research and code just for the fun and learning of it all.  Subsequently, from this journey, I put together this article on "Range Searching Using Visual Basic.NET …
Java had always been an easily readable and understandable language.  Some relatively recent changes in the language seem to be changing this pretty fast, and anyone that had not seen any Java code for the last 5 years will possibly have issues unde…
Viewers learn about the “for” loop and how it works in Java. By comparing it to the while loop learned before, viewers can make the transition easily. You will learn about the formatting of the for loop as we write a program that prints even numbers…
Viewers will learn about if statements in Java and their use The if statement: The condition required to create an if statement: Variations of if statements: An example using if statements:
Suggested Courses

916 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question