Solved

Memory Leak due to ConcurentHashMap

Posted on 2014-03-13
6
1,983 Views
Last Modified: 2014-03-20
import java.util.WeakHashMap;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.ScheduledFuture;
import java.util.concurrent.TimeUnit;

public class SchedulerExecutorService {

	TaskConfig taskConfig;

	final static WeakHashMap<Integer, TaskConfig> taskList = new WeakHashMap<Integer, TaskConfig>();
	final static ConcurrentMap<Integer, ScheduledFuture<?>> futuresMap = new ConcurrentHashMap<Integer, ScheduledFuture<?>>();

	final static ConcurrentMap<Integer, ScheduledExecutorService> executorMap = new ConcurrentHashMap<Integer, ScheduledExecutorService>();
	private int id;

	private SchedulerExecutorService(Integer id, TaskConfig taskConfig) {
		this.id = id;
		this.taskConfig = taskConfig;
	}

	public static SchedulerExecutorService getInstance(Integer id,
			TaskConfig taskConfig) {

		return new SchedulerExecutorService(id, taskConfig);

	}

	public void executor() {

		taskList.put(id, taskConfig);

		ScheduledExecutorService executor = Executors.newScheduledThreadPool(2);
		futuresMap.put(id, executor.scheduleAtFixedRate(SchedulerTaskRunner
				.getInstance(taskList.get(id).getSchedulableClass()), taskList
				.get(id).getRepeatInterval(), taskList.get(id)
				.getRepeatInterval(), TimeUnit.SECONDS));

		executorMap.put(id, executor);

	}

	public void stop() {

		if (taskList.get(id) != null) {

			taskList.remove(id);

			taskList.remove(id);

			futuresMap.get(id).cancel(true);

			if (futuresMap.get(id).isCancelled()) {

				futuresMap.remove(id);
				executorMap.get(id).shutdown();

				if (executorMap.get(id).isShutdown()) {
					try {

						executorMap.get(id).awaitTermination(1,
								TimeUnit.MILLISECONDS);

					} catch (InterruptedException e) {

				
						e.printStackTrace();
					}

				}
			}
			executorMap.remove(id);
		}
	}

}

Open in new window



I'm using the above code to schedule multiple task. But my problem is after scheduling any task for 10 minutes, there is an outofmemory exception java heap space. After using yourkit memory analyzer, the retained size of ConcurentHashMap and DefaultListableBeanFactory is high.

It seems that there is memory leak.

How can I control the leak?
0
Comment
Question by:spectrumsofttech
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
6 Comments
 
LVL 27

Expert Comment

by:dpearson
ID: 39926977
Nothing leaps out at me from this that looks wrong.

What I'd suggest first is giving yourself better insight into what is happening during the task execution.  You can do this by switching the type that you are storing from
ScheduledExectutorService to the more specific ScheduledThreadPoolExecutor

It means just small changes here:

	final static ConcurrentMap<Integer, ScheduledThreadPoolExecutor> executorMap = new ConcurrentHashMap<Integer, ScheduledThreadPoolExecutor>();

	public void executor() {
                 ...
		ScheduledThreadPoolExecutor executor = (ScheduledThreadPoolExecutor)Executors.newScheduledThreadPool(2);
                 ...
	}

Open in new window


This will then allow you to add a new method which reports the current queue size, which you can call periodically - something like this:

	public void reportStatus() {
		ScheduledThreadPoolExecutor executor = executorMap.get(id) ;
		if (executor != null) {
			int tasksInQueue = executor.getQueue().size() ;
			System.out.println("Executor for task " + id + " contains " + tasksInQueue + " tasks") ;
		}
	}

Open in new window


Use this to check on how the queue is growing during the course of the 10 minutes of execution.  The obvious thing that will cause an out of memory exception is if you are submitting new tasks at a higher rate than the tasks complete.  In that case this queue will always grow without bound and eventually you'd run out of memory.

That's a function of the values returned by "getRepeatInterval()" from the task runner and how long the "run()" methods actually take to complete inside the TaskRunner.  If they block or fail to complete for any reason (or just are slower to execute than the rate they are scheduled), you'll get this sort of "leak".

Doug
0
 

Author Comment

by:spectrumsofttech
ID: 39928713
I've tried the above mentioned code. Using the above code, I submitted two different task with the delay of 10 minutes. Using the  reportStatus(), I get the following result:

Executor for task 1 contains 1 tasks
Executor for task 2 contains 1 tasks

It seems that for each id, a unique task is submitted.

The following class "SchedulerTaskRunner" is invoked from the above code.

import java.lang.ref.WeakReference;
import java.util.Hashtable;

public class SchedulerTaskRunner implements Runnable {

	WeakReference<String> clzz = null;

	private Hashtable classes = new Hashtable();

	public static SchedulerTaskRunner getInstance(String clzz) {

		return new SchedulerTaskRunner(clzz);

	}

	private SchedulerTaskRunner(String clzz) {

		this.clzz = new WeakReference<String>(clzz);
	}

	@SuppressWarnings("unchecked")
	@Override
	public void run() {

		try {

			if (classes.containsKey(clzz.get())) {
				
				Class<?> c = (Class) classes.get(clzz.get());
				c.newInstance();

				c = null;

			} else {

				Class<?> c = JarFinder.getInstance(clzz.get()).jarFinder();
				c.newInstance();
				classes.put(clzz.get(), c);

				c = null;
				
			}
		} catch (Exception e) {
			e.printStackTrace();
		}

	}


}

Open in new window

0
 
LVL 27

Expert Comment

by:dpearson
ID: 39933558
When I try with similar code it all seems to run just fine and memory usage remains flat - which is what I'd expect.

Can you post a full example that can be run that shows the problem?

When I try setting up the Task like this:

	public static class TaskConfig {
	}

	private static class Task implements Runnable {
		@Override
		public void run() {
				try {
					Thread.sleep(500);
					System.out.println("Task finished") ;
				} catch (Exception ex) {
				}
		}
	}

Open in new window


I'm not seeing any obvious memory issue.

Your current code is close to being runnable but seems to be newing up classes that are passed in by name, which of course could do a lot of things depending on the classes involved.  It also calls out to a JarFinder library.

I assume those aren't critical to reproing this?

Doug
0
Is Your DevOps Pipeline Leaking?

Is your CI/CD pipeline a hodge-podge of randomly connected tools? You’ve likely got a tool to fix one problem & then a different tool to fix another, resulting in a cluster of tools with overlapping functionality. Learn how to optimize your pipeline with Gartner's recommendations

 

Author Comment

by:spectrumsofttech
ID: 39936257
This is the JarFinder  class. It search for a jar file inside a folder and load the jar.
import java.io.File;
import java.net.URL;
import java.util.Enumeration;
import java.util.jar.JarEntry;
import java.util.jar.JarFile;

public class JarFinder {

	private String clzzName;

	public static JarFinder getInstance(String clzzName) {

		return new JarFinder(clzzName);

	}

	private JarFinder(String clzzName) {

		this.clzzName = clzzName;
	}

	public Class<?> jarFinder() {

		Class<?> c = null;
		try {

			boolean flag = false;

			File jarDir = new File(System.getProperty("user.home")
					+ "/.JarFolder");

			for (File file : jarDir.listFiles()) {

				if (file.isFile() && file.getName().endsWith(".jar")) {

					URL fileURL = file.toURI().toURL();
					String jarURL = "jar:" + fileURL + "!/";
					URL urls[] = { new URL(jarURL) };

					JarFile jarFile = new JarFile(file);

					Enumeration<JarEntry> jarEntries = jarFile.entries();

					while (jarEntries.hasMoreElements()) {

						JarEntry jarEntry = jarEntries.nextElement();

						String checkClass = clzzName.replaceAll("\\.", "/")
								+ ".class";
						if (jarEntry.getName().endsWith(".class")
								&& checkClass.equals(jarEntry.getName())) {

							c = JarClassLoader.getInstance(urls).loadClass(
									clzzName);
							JarClassLoader.getInstance(urls).close();
							flag = true;

						}

						if (flag) {

							jarFile.close();
							break;
						}

					}

				}

				if (flag) {
		
					break;
				}
			}
		} catch (Exception e) {

		}
		return c;

	}

	@Override
	protected void finalize() throws Throwable {
		super.finalize();
	}
}

Open in new window


This is the classloader I use to load the jar file.

import java.net.URL;
import java.net.URLClassLoader;

public class JarClassLoader extends URLClassLoader {

	ClassLoader currentThreadClassLoader = Thread.currentThread()
			.getContextClassLoader();

	public static JarClassLoader getInstance(URL[] urls) {

		return new JarClassLoader(urls);

	}

	private JarClassLoader(URL[] urls) {
		super(urls, JarClassLoader.class.getClassLoader());
		URLClassLoader ucl = new URLClassLoader(urls, currentThreadClassLoader);
		Thread.currentThread().setContextClassLoader(ucl);
		currentThreadClassLoader = Thread.currentThread()
				.getContextClassLoader();
	}

	@Override
	protected Class<?> findClass(String qualifiedClassName)
			throws ClassNotFoundException {


			synchronized (this) {
				Class<?> c = super.findClass(qualifiedClassName);

				return c;
		
		}
	}

	@Override
	public Class<?> loadClass(String name) throws ClassNotFoundException {

	
			synchronized (this) {

				Class<?> c = currentThreadClassLoader.loadClass(name);
				
				return c;
		}
	}

	@Override
	protected void finalize() throws Throwable {
		super.finalize();
	}
]

Open in new window


The jar being loaded is a Stand alone Spring Application.


Meanwhile, I have changed my code a bit. Now, I don't get any Out of memory error  due to Java heap.

But, now I get perm size Out of memory error.

I analyzed the dump file using Eclipse MAT. Under the leak suspect, It shows the following two:
 

The classloader/component "org.apache.catalina.loader.WebappClassLoader @ 0xe9159228" occupies 2,546,176 (11.87%) bytes. The memory is accumulated in classloader/component "org.apache.catalina.loader.WebappClassLoader @ 0xe9159228".

Keywords
org.apache.catalina.loader.WebappClassLoader @ 0xe9159228



4,838 instances of "java.lang.Class", loaded by "<system class loader>" occupy 4,741,584 (22.10%) bytes.

Biggest instances:

•class org.springframework.beans.CachedIntrospectionResults @ 0xe93e92a0 - 290,832 (1.36%) bytes.


What can be the problem for getting this error?
0
 
LVL 27

Accepted Solution

by:
dpearson earned 500 total points
ID: 39941463
You get Perm Size errors when you are loading too many classes.  Classes (not instances of classes, the actual classes themselves) are loaded into the "PermGen" space - which means they last forever and don't get garbage collected.

This is usually no problem, but if you have your own class loader and it doesn't work quite right, you can end up loading a class over and over and eventually you run out of Perm Gen space.

Unfortunately debugging what your jar finder and class loader are doing, esp when you throw something like Spring into the mix isn't something I can really help with.  It's a long way from the original question about executors and threading, which I do know something about :)

Doug
0
 

Author Closing Comment

by:spectrumsofttech
ID: 39941627
Thanks for your help and advice, Doug.
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
eclipse console opening separately 2 50
Grunt script for Build Process 1 90
Magento: different theme for Pc and for mobile devices 4 122
jsp error 6 72
Developer tools in browsers have been around for a while, yet they are still heavily underused by developers. Developers still fix html or CSS then refresh page to see effect, or they put alert or debugger in JavaScript and then try again and again …
Go is an acronym of golang, is a programming language developed Google in 2007. Go is a new language that is mostly in the C family, with significant input from Pascal/Modula/Oberon family. Hence Go arisen as low-level language with fast compilation…
This theoretical tutorial explains exceptions, reasons for exceptions, different categories of exception and exception hierarchy.
Viewers will learn how to properly install Eclipse with the necessary JDK, and will take a look at an introductory Java program. Download Eclipse installation zip file: Extract files from zip file: Download and install JDK 8: Open Eclipse and …
Suggested Courses

751 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question