Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Memory Leak due to ConcurentHashMap

Posted on 2014-03-13
6
Medium Priority
?
2,269 Views
Last Modified: 2014-03-20
import java.util.WeakHashMap;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.ScheduledFuture;
import java.util.concurrent.TimeUnit;

public class SchedulerExecutorService {

	TaskConfig taskConfig;

	final static WeakHashMap<Integer, TaskConfig> taskList = new WeakHashMap<Integer, TaskConfig>();
	final static ConcurrentMap<Integer, ScheduledFuture<?>> futuresMap = new ConcurrentHashMap<Integer, ScheduledFuture<?>>();

	final static ConcurrentMap<Integer, ScheduledExecutorService> executorMap = new ConcurrentHashMap<Integer, ScheduledExecutorService>();
	private int id;

	private SchedulerExecutorService(Integer id, TaskConfig taskConfig) {
		this.id = id;
		this.taskConfig = taskConfig;
	}

	public static SchedulerExecutorService getInstance(Integer id,
			TaskConfig taskConfig) {

		return new SchedulerExecutorService(id, taskConfig);

	}

	public void executor() {

		taskList.put(id, taskConfig);

		ScheduledExecutorService executor = Executors.newScheduledThreadPool(2);
		futuresMap.put(id, executor.scheduleAtFixedRate(SchedulerTaskRunner
				.getInstance(taskList.get(id).getSchedulableClass()), taskList
				.get(id).getRepeatInterval(), taskList.get(id)
				.getRepeatInterval(), TimeUnit.SECONDS));

		executorMap.put(id, executor);

	}

	public void stop() {

		if (taskList.get(id) != null) {

			taskList.remove(id);

			taskList.remove(id);

			futuresMap.get(id).cancel(true);

			if (futuresMap.get(id).isCancelled()) {

				futuresMap.remove(id);
				executorMap.get(id).shutdown();

				if (executorMap.get(id).isShutdown()) {
					try {

						executorMap.get(id).awaitTermination(1,
								TimeUnit.MILLISECONDS);

					} catch (InterruptedException e) {

				
						e.printStackTrace();
					}

				}
			}
			executorMap.remove(id);
		}
	}

}

Open in new window



I'm using the above code to schedule multiple task. But my problem is after scheduling any task for 10 minutes, there is an outofmemory exception java heap space. After using yourkit memory analyzer, the retained size of ConcurentHashMap and DefaultListableBeanFactory is high.

It seems that there is memory leak.

How can I control the leak?
0
Comment
Question by:spectrumsofttech
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
6 Comments
 
LVL 28

Expert Comment

by:dpearson
ID: 39926977
Nothing leaps out at me from this that looks wrong.

What I'd suggest first is giving yourself better insight into what is happening during the task execution.  You can do this by switching the type that you are storing from
ScheduledExectutorService to the more specific ScheduledThreadPoolExecutor

It means just small changes here:

	final static ConcurrentMap<Integer, ScheduledThreadPoolExecutor> executorMap = new ConcurrentHashMap<Integer, ScheduledThreadPoolExecutor>();

	public void executor() {
                 ...
		ScheduledThreadPoolExecutor executor = (ScheduledThreadPoolExecutor)Executors.newScheduledThreadPool(2);
                 ...
	}

Open in new window


This will then allow you to add a new method which reports the current queue size, which you can call periodically - something like this:

	public void reportStatus() {
		ScheduledThreadPoolExecutor executor = executorMap.get(id) ;
		if (executor != null) {
			int tasksInQueue = executor.getQueue().size() ;
			System.out.println("Executor for task " + id + " contains " + tasksInQueue + " tasks") ;
		}
	}

Open in new window


Use this to check on how the queue is growing during the course of the 10 minutes of execution.  The obvious thing that will cause an out of memory exception is if you are submitting new tasks at a higher rate than the tasks complete.  In that case this queue will always grow without bound and eventually you'd run out of memory.

That's a function of the values returned by "getRepeatInterval()" from the task runner and how long the "run()" methods actually take to complete inside the TaskRunner.  If they block or fail to complete for any reason (or just are slower to execute than the rate they are scheduled), you'll get this sort of "leak".

Doug
0
 

Author Comment

by:spectrumsofttech
ID: 39928713
I've tried the above mentioned code. Using the above code, I submitted two different task with the delay of 10 minutes. Using the  reportStatus(), I get the following result:

Executor for task 1 contains 1 tasks
Executor for task 2 contains 1 tasks

It seems that for each id, a unique task is submitted.

The following class "SchedulerTaskRunner" is invoked from the above code.

import java.lang.ref.WeakReference;
import java.util.Hashtable;

public class SchedulerTaskRunner implements Runnable {

	WeakReference<String> clzz = null;

	private Hashtable classes = new Hashtable();

	public static SchedulerTaskRunner getInstance(String clzz) {

		return new SchedulerTaskRunner(clzz);

	}

	private SchedulerTaskRunner(String clzz) {

		this.clzz = new WeakReference<String>(clzz);
	}

	@SuppressWarnings("unchecked")
	@Override
	public void run() {

		try {

			if (classes.containsKey(clzz.get())) {
				
				Class<?> c = (Class) classes.get(clzz.get());
				c.newInstance();

				c = null;

			} else {

				Class<?> c = JarFinder.getInstance(clzz.get()).jarFinder();
				c.newInstance();
				classes.put(clzz.get(), c);

				c = null;
				
			}
		} catch (Exception e) {
			e.printStackTrace();
		}

	}


}

Open in new window

0
 
LVL 28

Expert Comment

by:dpearson
ID: 39933558
When I try with similar code it all seems to run just fine and memory usage remains flat - which is what I'd expect.

Can you post a full example that can be run that shows the problem?

When I try setting up the Task like this:

	public static class TaskConfig {
	}

	private static class Task implements Runnable {
		@Override
		public void run() {
				try {
					Thread.sleep(500);
					System.out.println("Task finished") ;
				} catch (Exception ex) {
				}
		}
	}

Open in new window


I'm not seeing any obvious memory issue.

Your current code is close to being runnable but seems to be newing up classes that are passed in by name, which of course could do a lot of things depending on the classes involved.  It also calls out to a JarFinder library.

I assume those aren't critical to reproing this?

Doug
0
Stressed Out?

Watch some penguins on the livecam!

 

Author Comment

by:spectrumsofttech
ID: 39936257
This is the JarFinder  class. It search for a jar file inside a folder and load the jar.
import java.io.File;
import java.net.URL;
import java.util.Enumeration;
import java.util.jar.JarEntry;
import java.util.jar.JarFile;

public class JarFinder {

	private String clzzName;

	public static JarFinder getInstance(String clzzName) {

		return new JarFinder(clzzName);

	}

	private JarFinder(String clzzName) {

		this.clzzName = clzzName;
	}

	public Class<?> jarFinder() {

		Class<?> c = null;
		try {

			boolean flag = false;

			File jarDir = new File(System.getProperty("user.home")
					+ "/.JarFolder");

			for (File file : jarDir.listFiles()) {

				if (file.isFile() && file.getName().endsWith(".jar")) {

					URL fileURL = file.toURI().toURL();
					String jarURL = "jar:" + fileURL + "!/";
					URL urls[] = { new URL(jarURL) };

					JarFile jarFile = new JarFile(file);

					Enumeration<JarEntry> jarEntries = jarFile.entries();

					while (jarEntries.hasMoreElements()) {

						JarEntry jarEntry = jarEntries.nextElement();

						String checkClass = clzzName.replaceAll("\\.", "/")
								+ ".class";
						if (jarEntry.getName().endsWith(".class")
								&& checkClass.equals(jarEntry.getName())) {

							c = JarClassLoader.getInstance(urls).loadClass(
									clzzName);
							JarClassLoader.getInstance(urls).close();
							flag = true;

						}

						if (flag) {

							jarFile.close();
							break;
						}

					}

				}

				if (flag) {
		
					break;
				}
			}
		} catch (Exception e) {

		}
		return c;

	}

	@Override
	protected void finalize() throws Throwable {
		super.finalize();
	}
}

Open in new window


This is the classloader I use to load the jar file.

import java.net.URL;
import java.net.URLClassLoader;

public class JarClassLoader extends URLClassLoader {

	ClassLoader currentThreadClassLoader = Thread.currentThread()
			.getContextClassLoader();

	public static JarClassLoader getInstance(URL[] urls) {

		return new JarClassLoader(urls);

	}

	private JarClassLoader(URL[] urls) {
		super(urls, JarClassLoader.class.getClassLoader());
		URLClassLoader ucl = new URLClassLoader(urls, currentThreadClassLoader);
		Thread.currentThread().setContextClassLoader(ucl);
		currentThreadClassLoader = Thread.currentThread()
				.getContextClassLoader();
	}

	@Override
	protected Class<?> findClass(String qualifiedClassName)
			throws ClassNotFoundException {


			synchronized (this) {
				Class<?> c = super.findClass(qualifiedClassName);

				return c;
		
		}
	}

	@Override
	public Class<?> loadClass(String name) throws ClassNotFoundException {

	
			synchronized (this) {

				Class<?> c = currentThreadClassLoader.loadClass(name);
				
				return c;
		}
	}

	@Override
	protected void finalize() throws Throwable {
		super.finalize();
	}
]

Open in new window


The jar being loaded is a Stand alone Spring Application.


Meanwhile, I have changed my code a bit. Now, I don't get any Out of memory error  due to Java heap.

But, now I get perm size Out of memory error.

I analyzed the dump file using Eclipse MAT. Under the leak suspect, It shows the following two:
 

The classloader/component "org.apache.catalina.loader.WebappClassLoader @ 0xe9159228" occupies 2,546,176 (11.87%) bytes. The memory is accumulated in classloader/component "org.apache.catalina.loader.WebappClassLoader @ 0xe9159228".

Keywords
org.apache.catalina.loader.WebappClassLoader @ 0xe9159228



4,838 instances of "java.lang.Class", loaded by "<system class loader>" occupy 4,741,584 (22.10%) bytes.

Biggest instances:

•class org.springframework.beans.CachedIntrospectionResults @ 0xe93e92a0 - 290,832 (1.36%) bytes.


What can be the problem for getting this error?
0
 
LVL 28

Accepted Solution

by:
dpearson earned 1500 total points
ID: 39941463
You get Perm Size errors when you are loading too many classes.  Classes (not instances of classes, the actual classes themselves) are loaded into the "PermGen" space - which means they last forever and don't get garbage collected.

This is usually no problem, but if you have your own class loader and it doesn't work quite right, you can end up loading a class over and over and eventually you run out of Perm Gen space.

Unfortunately debugging what your jar finder and class loader are doing, esp when you throw something like Spring into the mix isn't something I can really help with.  It's a long way from the original question about executors and threading, which I do know something about :)

Doug
0
 

Author Closing Comment

by:spectrumsofttech
ID: 39941627
Thanks for your help and advice, Doug.
0

Featured Post

Build and deliver software with DevOps

A digital transformation requires faster time to market, shorter software development lifecycles, and the ability to adapt rapidly to changing customer demands. DevOps provides the solution.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

A publishing tool, a Version Control System, or a Collaboration Platform! These can be some of the defining words for the two very famous web-hosting Git repositories: Bitbucket and Github. Git is widely used amongst the programmers and developers f…
Go is an acronym of golang, is a programming language developed Google in 2007. Go is a new language that is mostly in the C family, with significant input from Pascal/Modula/Oberon family. Hence Go arisen as low-level language with fast compilation…
Viewers will learn about arithmetic and Boolean expressions in Java and the logical operators used to create Boolean expressions. We will cover the symbols used for arithmetic expressions and define each logical operator and how to use them in Boole…
This tutorial covers a step-by-step guide to install VisualVM launcher in eclipse.
Suggested Courses

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question