Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

Memory Leak due to ConcurentHashMap

Posted on 2014-03-13
6
Medium Priority
?
2,373 Views
Last Modified: 2014-03-20
import java.util.WeakHashMap;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.ScheduledFuture;
import java.util.concurrent.TimeUnit;

public class SchedulerExecutorService {

	TaskConfig taskConfig;

	final static WeakHashMap<Integer, TaskConfig> taskList = new WeakHashMap<Integer, TaskConfig>();
	final static ConcurrentMap<Integer, ScheduledFuture<?>> futuresMap = new ConcurrentHashMap<Integer, ScheduledFuture<?>>();

	final static ConcurrentMap<Integer, ScheduledExecutorService> executorMap = new ConcurrentHashMap<Integer, ScheduledExecutorService>();
	private int id;

	private SchedulerExecutorService(Integer id, TaskConfig taskConfig) {
		this.id = id;
		this.taskConfig = taskConfig;
	}

	public static SchedulerExecutorService getInstance(Integer id,
			TaskConfig taskConfig) {

		return new SchedulerExecutorService(id, taskConfig);

	}

	public void executor() {

		taskList.put(id, taskConfig);

		ScheduledExecutorService executor = Executors.newScheduledThreadPool(2);
		futuresMap.put(id, executor.scheduleAtFixedRate(SchedulerTaskRunner
				.getInstance(taskList.get(id).getSchedulableClass()), taskList
				.get(id).getRepeatInterval(), taskList.get(id)
				.getRepeatInterval(), TimeUnit.SECONDS));

		executorMap.put(id, executor);

	}

	public void stop() {

		if (taskList.get(id) != null) {

			taskList.remove(id);

			taskList.remove(id);

			futuresMap.get(id).cancel(true);

			if (futuresMap.get(id).isCancelled()) {

				futuresMap.remove(id);
				executorMap.get(id).shutdown();

				if (executorMap.get(id).isShutdown()) {
					try {

						executorMap.get(id).awaitTermination(1,
								TimeUnit.MILLISECONDS);

					} catch (InterruptedException e) {

				
						e.printStackTrace();
					}

				}
			}
			executorMap.remove(id);
		}
	}

}

Open in new window



I'm using the above code to schedule multiple task. But my problem is after scheduling any task for 10 minutes, there is an outofmemory exception java heap space. After using yourkit memory analyzer, the retained size of ConcurentHashMap and DefaultListableBeanFactory is high.

It seems that there is memory leak.

How can I control the leak?
0
Comment
Question by:spectrumsofttech
  • 3
  • 3
6 Comments
 
LVL 28

Expert Comment

by:dpearson
ID: 39926977
Nothing leaps out at me from this that looks wrong.

What I'd suggest first is giving yourself better insight into what is happening during the task execution.  You can do this by switching the type that you are storing from
ScheduledExectutorService to the more specific ScheduledThreadPoolExecutor

It means just small changes here:

	final static ConcurrentMap<Integer, ScheduledThreadPoolExecutor> executorMap = new ConcurrentHashMap<Integer, ScheduledThreadPoolExecutor>();

	public void executor() {
                 ...
		ScheduledThreadPoolExecutor executor = (ScheduledThreadPoolExecutor)Executors.newScheduledThreadPool(2);
                 ...
	}

Open in new window


This will then allow you to add a new method which reports the current queue size, which you can call periodically - something like this:

	public void reportStatus() {
		ScheduledThreadPoolExecutor executor = executorMap.get(id) ;
		if (executor != null) {
			int tasksInQueue = executor.getQueue().size() ;
			System.out.println("Executor for task " + id + " contains " + tasksInQueue + " tasks") ;
		}
	}

Open in new window


Use this to check on how the queue is growing during the course of the 10 minutes of execution.  The obvious thing that will cause an out of memory exception is if you are submitting new tasks at a higher rate than the tasks complete.  In that case this queue will always grow without bound and eventually you'd run out of memory.

That's a function of the values returned by "getRepeatInterval()" from the task runner and how long the "run()" methods actually take to complete inside the TaskRunner.  If they block or fail to complete for any reason (or just are slower to execute than the rate they are scheduled), you'll get this sort of "leak".

Doug
0
 

Author Comment

by:spectrumsofttech
ID: 39928713
I've tried the above mentioned code. Using the above code, I submitted two different task with the delay of 10 minutes. Using the  reportStatus(), I get the following result:

Executor for task 1 contains 1 tasks
Executor for task 2 contains 1 tasks

It seems that for each id, a unique task is submitted.

The following class "SchedulerTaskRunner" is invoked from the above code.

import java.lang.ref.WeakReference;
import java.util.Hashtable;

public class SchedulerTaskRunner implements Runnable {

	WeakReference<String> clzz = null;

	private Hashtable classes = new Hashtable();

	public static SchedulerTaskRunner getInstance(String clzz) {

		return new SchedulerTaskRunner(clzz);

	}

	private SchedulerTaskRunner(String clzz) {

		this.clzz = new WeakReference<String>(clzz);
	}

	@SuppressWarnings("unchecked")
	@Override
	public void run() {

		try {

			if (classes.containsKey(clzz.get())) {
				
				Class<?> c = (Class) classes.get(clzz.get());
				c.newInstance();

				c = null;

			} else {

				Class<?> c = JarFinder.getInstance(clzz.get()).jarFinder();
				c.newInstance();
				classes.put(clzz.get(), c);

				c = null;
				
			}
		} catch (Exception e) {
			e.printStackTrace();
		}

	}


}

Open in new window

0
 
LVL 28

Expert Comment

by:dpearson
ID: 39933558
When I try with similar code it all seems to run just fine and memory usage remains flat - which is what I'd expect.

Can you post a full example that can be run that shows the problem?

When I try setting up the Task like this:

	public static class TaskConfig {
	}

	private static class Task implements Runnable {
		@Override
		public void run() {
				try {
					Thread.sleep(500);
					System.out.println("Task finished") ;
				} catch (Exception ex) {
				}
		}
	}

Open in new window


I'm not seeing any obvious memory issue.

Your current code is close to being runnable but seems to be newing up classes that are passed in by name, which of course could do a lot of things depending on the classes involved.  It also calls out to a JarFinder library.

I assume those aren't critical to reproing this?

Doug
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 

Author Comment

by:spectrumsofttech
ID: 39936257
This is the JarFinder  class. It search for a jar file inside a folder and load the jar.
import java.io.File;
import java.net.URL;
import java.util.Enumeration;
import java.util.jar.JarEntry;
import java.util.jar.JarFile;

public class JarFinder {

	private String clzzName;

	public static JarFinder getInstance(String clzzName) {

		return new JarFinder(clzzName);

	}

	private JarFinder(String clzzName) {

		this.clzzName = clzzName;
	}

	public Class<?> jarFinder() {

		Class<?> c = null;
		try {

			boolean flag = false;

			File jarDir = new File(System.getProperty("user.home")
					+ "/.JarFolder");

			for (File file : jarDir.listFiles()) {

				if (file.isFile() && file.getName().endsWith(".jar")) {

					URL fileURL = file.toURI().toURL();
					String jarURL = "jar:" + fileURL + "!/";
					URL urls[] = { new URL(jarURL) };

					JarFile jarFile = new JarFile(file);

					Enumeration<JarEntry> jarEntries = jarFile.entries();

					while (jarEntries.hasMoreElements()) {

						JarEntry jarEntry = jarEntries.nextElement();

						String checkClass = clzzName.replaceAll("\\.", "/")
								+ ".class";
						if (jarEntry.getName().endsWith(".class")
								&& checkClass.equals(jarEntry.getName())) {

							c = JarClassLoader.getInstance(urls).loadClass(
									clzzName);
							JarClassLoader.getInstance(urls).close();
							flag = true;

						}

						if (flag) {

							jarFile.close();
							break;
						}

					}

				}

				if (flag) {
		
					break;
				}
			}
		} catch (Exception e) {

		}
		return c;

	}

	@Override
	protected void finalize() throws Throwable {
		super.finalize();
	}
}

Open in new window


This is the classloader I use to load the jar file.

import java.net.URL;
import java.net.URLClassLoader;

public class JarClassLoader extends URLClassLoader {

	ClassLoader currentThreadClassLoader = Thread.currentThread()
			.getContextClassLoader();

	public static JarClassLoader getInstance(URL[] urls) {

		return new JarClassLoader(urls);

	}

	private JarClassLoader(URL[] urls) {
		super(urls, JarClassLoader.class.getClassLoader());
		URLClassLoader ucl = new URLClassLoader(urls, currentThreadClassLoader);
		Thread.currentThread().setContextClassLoader(ucl);
		currentThreadClassLoader = Thread.currentThread()
				.getContextClassLoader();
	}

	@Override
	protected Class<?> findClass(String qualifiedClassName)
			throws ClassNotFoundException {


			synchronized (this) {
				Class<?> c = super.findClass(qualifiedClassName);

				return c;
		
		}
	}

	@Override
	public Class<?> loadClass(String name) throws ClassNotFoundException {

	
			synchronized (this) {

				Class<?> c = currentThreadClassLoader.loadClass(name);
				
				return c;
		}
	}

	@Override
	protected void finalize() throws Throwable {
		super.finalize();
	}
]

Open in new window


The jar being loaded is a Stand alone Spring Application.


Meanwhile, I have changed my code a bit. Now, I don't get any Out of memory error  due to Java heap.

But, now I get perm size Out of memory error.

I analyzed the dump file using Eclipse MAT. Under the leak suspect, It shows the following two:
 

The classloader/component "org.apache.catalina.loader.WebappClassLoader @ 0xe9159228" occupies 2,546,176 (11.87%) bytes. The memory is accumulated in classloader/component "org.apache.catalina.loader.WebappClassLoader @ 0xe9159228".

Keywords
org.apache.catalina.loader.WebappClassLoader @ 0xe9159228



4,838 instances of "java.lang.Class", loaded by "<system class loader>" occupy 4,741,584 (22.10%) bytes.

Biggest instances:

•class org.springframework.beans.CachedIntrospectionResults @ 0xe93e92a0 - 290,832 (1.36%) bytes.


What can be the problem for getting this error?
0
 
LVL 28

Accepted Solution

by:
dpearson earned 1500 total points
ID: 39941463
You get Perm Size errors when you are loading too many classes.  Classes (not instances of classes, the actual classes themselves) are loaded into the "PermGen" space - which means they last forever and don't get garbage collected.

This is usually no problem, but if you have your own class loader and it doesn't work quite right, you can end up loading a class over and over and eventually you run out of Perm Gen space.

Unfortunately debugging what your jar finder and class loader are doing, esp when you throw something like Spring into the mix isn't something I can really help with.  It's a long way from the original question about executors and threading, which I do know something about :)

Doug
0
 

Author Closing Comment

by:spectrumsofttech
ID: 39941627
Thanks for your help and advice, Doug.
0

Featured Post

Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

What is Node.js? Node.js is a server side scripting language much like PHP or ASP but is used to implement the complete package of HTTP webserver and application framework. The difference is that Node.js’s execution engine is asynchronous and event…
International Data Corporation (IDC) prognosticates that before the current the year gets over disbursing on IT framework products to be sent in cloud environs will be $37.1B.
Viewers will learn about if statements in Java and their use The if statement: The condition required to create an if statement: Variations of if statements: An example using if statements:
This theoretical tutorial explains exceptions, reasons for exceptions, different categories of exception and exception hierarchy.
Suggested Courses

972 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question