Solved

Memory Leak due to ConcurentHashMap

Posted on 2014-03-13
6
1,650 Views
Last Modified: 2014-03-20
import java.util.WeakHashMap;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.ScheduledFuture;
import java.util.concurrent.TimeUnit;

public class SchedulerExecutorService {

	TaskConfig taskConfig;

	final static WeakHashMap<Integer, TaskConfig> taskList = new WeakHashMap<Integer, TaskConfig>();
	final static ConcurrentMap<Integer, ScheduledFuture<?>> futuresMap = new ConcurrentHashMap<Integer, ScheduledFuture<?>>();

	final static ConcurrentMap<Integer, ScheduledExecutorService> executorMap = new ConcurrentHashMap<Integer, ScheduledExecutorService>();
	private int id;

	private SchedulerExecutorService(Integer id, TaskConfig taskConfig) {
		this.id = id;
		this.taskConfig = taskConfig;
	}

	public static SchedulerExecutorService getInstance(Integer id,
			TaskConfig taskConfig) {

		return new SchedulerExecutorService(id, taskConfig);

	}

	public void executor() {

		taskList.put(id, taskConfig);

		ScheduledExecutorService executor = Executors.newScheduledThreadPool(2);
		futuresMap.put(id, executor.scheduleAtFixedRate(SchedulerTaskRunner
				.getInstance(taskList.get(id).getSchedulableClass()), taskList
				.get(id).getRepeatInterval(), taskList.get(id)
				.getRepeatInterval(), TimeUnit.SECONDS));

		executorMap.put(id, executor);

	}

	public void stop() {

		if (taskList.get(id) != null) {

			taskList.remove(id);

			taskList.remove(id);

			futuresMap.get(id).cancel(true);

			if (futuresMap.get(id).isCancelled()) {

				futuresMap.remove(id);
				executorMap.get(id).shutdown();

				if (executorMap.get(id).isShutdown()) {
					try {

						executorMap.get(id).awaitTermination(1,
								TimeUnit.MILLISECONDS);

					} catch (InterruptedException e) {

				
						e.printStackTrace();
					}

				}
			}
			executorMap.remove(id);
		}
	}

}

Open in new window



I'm using the above code to schedule multiple task. But my problem is after scheduling any task for 10 minutes, there is an outofmemory exception java heap space. After using yourkit memory analyzer, the retained size of ConcurentHashMap and DefaultListableBeanFactory is high.

It seems that there is memory leak.

How can I control the leak?
0
Comment
Question by:spectrumsofttech
  • 3
  • 3
6 Comments
 
LVL 26

Expert Comment

by:dpearson
Comment Utility
Nothing leaps out at me from this that looks wrong.

What I'd suggest first is giving yourself better insight into what is happening during the task execution.  You can do this by switching the type that you are storing from
ScheduledExectutorService to the more specific ScheduledThreadPoolExecutor

It means just small changes here:

	final static ConcurrentMap<Integer, ScheduledThreadPoolExecutor> executorMap = new ConcurrentHashMap<Integer, ScheduledThreadPoolExecutor>();

	public void executor() {
                 ...
		ScheduledThreadPoolExecutor executor = (ScheduledThreadPoolExecutor)Executors.newScheduledThreadPool(2);
                 ...
	}

Open in new window


This will then allow you to add a new method which reports the current queue size, which you can call periodically - something like this:

	public void reportStatus() {
		ScheduledThreadPoolExecutor executor = executorMap.get(id) ;
		if (executor != null) {
			int tasksInQueue = executor.getQueue().size() ;
			System.out.println("Executor for task " + id + " contains " + tasksInQueue + " tasks") ;
		}
	}

Open in new window


Use this to check on how the queue is growing during the course of the 10 minutes of execution.  The obvious thing that will cause an out of memory exception is if you are submitting new tasks at a higher rate than the tasks complete.  In that case this queue will always grow without bound and eventually you'd run out of memory.

That's a function of the values returned by "getRepeatInterval()" from the task runner and how long the "run()" methods actually take to complete inside the TaskRunner.  If they block or fail to complete for any reason (or just are slower to execute than the rate they are scheduled), you'll get this sort of "leak".

Doug
0
 

Author Comment

by:spectrumsofttech
Comment Utility
I've tried the above mentioned code. Using the above code, I submitted two different task with the delay of 10 minutes. Using the  reportStatus(), I get the following result:

Executor for task 1 contains 1 tasks
Executor for task 2 contains 1 tasks

It seems that for each id, a unique task is submitted.

The following class "SchedulerTaskRunner" is invoked from the above code.

import java.lang.ref.WeakReference;
import java.util.Hashtable;

public class SchedulerTaskRunner implements Runnable {

	WeakReference<String> clzz = null;

	private Hashtable classes = new Hashtable();

	public static SchedulerTaskRunner getInstance(String clzz) {

		return new SchedulerTaskRunner(clzz);

	}

	private SchedulerTaskRunner(String clzz) {

		this.clzz = new WeakReference<String>(clzz);
	}

	@SuppressWarnings("unchecked")
	@Override
	public void run() {

		try {

			if (classes.containsKey(clzz.get())) {
				
				Class<?> c = (Class) classes.get(clzz.get());
				c.newInstance();

				c = null;

			} else {

				Class<?> c = JarFinder.getInstance(clzz.get()).jarFinder();
				c.newInstance();
				classes.put(clzz.get(), c);

				c = null;
				
			}
		} catch (Exception e) {
			e.printStackTrace();
		}

	}


}

Open in new window

0
 
LVL 26

Expert Comment

by:dpearson
Comment Utility
When I try with similar code it all seems to run just fine and memory usage remains flat - which is what I'd expect.

Can you post a full example that can be run that shows the problem?

When I try setting up the Task like this:

	public static class TaskConfig {
	}

	private static class Task implements Runnable {
		@Override
		public void run() {
				try {
					Thread.sleep(500);
					System.out.println("Task finished") ;
				} catch (Exception ex) {
				}
		}
	}

Open in new window


I'm not seeing any obvious memory issue.

Your current code is close to being runnable but seems to be newing up classes that are passed in by name, which of course could do a lot of things depending on the classes involved.  It also calls out to a JarFinder library.

I assume those aren't critical to reproing this?

Doug
0
Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

 

Author Comment

by:spectrumsofttech
Comment Utility
This is the JarFinder  class. It search for a jar file inside a folder and load the jar.
import java.io.File;
import java.net.URL;
import java.util.Enumeration;
import java.util.jar.JarEntry;
import java.util.jar.JarFile;

public class JarFinder {

	private String clzzName;

	public static JarFinder getInstance(String clzzName) {

		return new JarFinder(clzzName);

	}

	private JarFinder(String clzzName) {

		this.clzzName = clzzName;
	}

	public Class<?> jarFinder() {

		Class<?> c = null;
		try {

			boolean flag = false;

			File jarDir = new File(System.getProperty("user.home")
					+ "/.JarFolder");

			for (File file : jarDir.listFiles()) {

				if (file.isFile() && file.getName().endsWith(".jar")) {

					URL fileURL = file.toURI().toURL();
					String jarURL = "jar:" + fileURL + "!/";
					URL urls[] = { new URL(jarURL) };

					JarFile jarFile = new JarFile(file);

					Enumeration<JarEntry> jarEntries = jarFile.entries();

					while (jarEntries.hasMoreElements()) {

						JarEntry jarEntry = jarEntries.nextElement();

						String checkClass = clzzName.replaceAll("\\.", "/")
								+ ".class";
						if (jarEntry.getName().endsWith(".class")
								&& checkClass.equals(jarEntry.getName())) {

							c = JarClassLoader.getInstance(urls).loadClass(
									clzzName);
							JarClassLoader.getInstance(urls).close();
							flag = true;

						}

						if (flag) {

							jarFile.close();
							break;
						}

					}

				}

				if (flag) {
		
					break;
				}
			}
		} catch (Exception e) {

		}
		return c;

	}

	@Override
	protected void finalize() throws Throwable {
		super.finalize();
	}
}

Open in new window


This is the classloader I use to load the jar file.

import java.net.URL;
import java.net.URLClassLoader;

public class JarClassLoader extends URLClassLoader {

	ClassLoader currentThreadClassLoader = Thread.currentThread()
			.getContextClassLoader();

	public static JarClassLoader getInstance(URL[] urls) {

		return new JarClassLoader(urls);

	}

	private JarClassLoader(URL[] urls) {
		super(urls, JarClassLoader.class.getClassLoader());
		URLClassLoader ucl = new URLClassLoader(urls, currentThreadClassLoader);
		Thread.currentThread().setContextClassLoader(ucl);
		currentThreadClassLoader = Thread.currentThread()
				.getContextClassLoader();
	}

	@Override
	protected Class<?> findClass(String qualifiedClassName)
			throws ClassNotFoundException {


			synchronized (this) {
				Class<?> c = super.findClass(qualifiedClassName);

				return c;
		
		}
	}

	@Override
	public Class<?> loadClass(String name) throws ClassNotFoundException {

	
			synchronized (this) {

				Class<?> c = currentThreadClassLoader.loadClass(name);
				
				return c;
		}
	}

	@Override
	protected void finalize() throws Throwable {
		super.finalize();
	}
]

Open in new window


The jar being loaded is a Stand alone Spring Application.


Meanwhile, I have changed my code a bit. Now, I don't get any Out of memory error  due to Java heap.

But, now I get perm size Out of memory error.

I analyzed the dump file using Eclipse MAT. Under the leak suspect, It shows the following two:
 

The classloader/component "org.apache.catalina.loader.WebappClassLoader @ 0xe9159228" occupies 2,546,176 (11.87%) bytes. The memory is accumulated in classloader/component "org.apache.catalina.loader.WebappClassLoader @ 0xe9159228".

Keywords
org.apache.catalina.loader.WebappClassLoader @ 0xe9159228



4,838 instances of "java.lang.Class", loaded by "<system class loader>" occupy 4,741,584 (22.10%) bytes.

Biggest instances:

•class org.springframework.beans.CachedIntrospectionResults @ 0xe93e92a0 - 290,832 (1.36%) bytes.


What can be the problem for getting this error?
0
 
LVL 26

Accepted Solution

by:
dpearson earned 500 total points
Comment Utility
You get Perm Size errors when you are loading too many classes.  Classes (not instances of classes, the actual classes themselves) are loaded into the "PermGen" space - which means they last forever and don't get garbage collected.

This is usually no problem, but if you have your own class loader and it doesn't work quite right, you can end up loading a class over and over and eventually you run out of Perm Gen space.

Unfortunately debugging what your jar finder and class loader are doing, esp when you throw something like Spring into the mix isn't something I can really help with.  It's a long way from the original question about executors and threading, which I do know something about :)

Doug
0
 

Author Closing Comment

by:spectrumsofttech
Comment Utility
Thanks for your help and advice, Doug.
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Suggested Solutions

Developer tools in browsers have been around for a while, yet they are still heavily underused by developers. Developers still fix html or CSS then refresh page to see effect, or they put alert or debugger in JavaScript and then try again and again …
International Data Corporation (IDC) prognosticates that before the current the year gets over disbursing on IT framework products to be sent in cloud environs will be $37.1B.
Viewers will learn how to properly install Eclipse with the necessary JDK, and will take a look at an introductory Java program. Download Eclipse installation zip file: Extract files from zip file: Download and install JDK 8: Open Eclipse and …
Learn how to set-up custom confirmation messages to users who complete your Wufoo form. Include inputs from fields in your form, webpage redirects, and more with Wufoo’s confirmation options.

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now