java.lang.OutOfMemoryError: PermGen space

We encountered the dreaded java.lang.OutOfMemoryError: PermGen space error

What does it means - Permgen Space is the space where the class files are all loaded for the first time and there after they are loaded or referenced from there. This error means there is a hell lot of duplicate classes being loaded and not all are being unloaded. These duplicate class instances are cluttering the permgen space and resulting in this error. One expects only only class instance of a java file. This is basically a memory leak.

Temporary Solution -
'start tag' jvmarg value="-XX:+UseConcMarkSweepGC" 'end tag'
'start tag' jvmarg value="-XX:+CMSPermGenSweepingEnabled" 'end tag'
'start tag' jvmarg value="-XX:+CMSClassUnloadingEnabled" 'end tag'
- These allow perm gen sweeping and allows class unloading, I guess it works really well when there is no memory leak but low perm gen space allocated, so when there is a space deficiency it automatically unloads few classes and then reloads them when needs. Yes there will be a performance issue but am sure not many will notice it.
'start tag' jvmarg value="-XX:PermSize=256M" 'end tag'
'start tag' jvmarg value="-XX:MaxPermSize=512M" 'end tag'
- you can keep increasing this value between 128 - 512 or more but this will only delay the time after which the error is thrown.

Tools - Few tools that I used for analysing this issue
a) MAT - Eclipse Memory Analyser Tool (Also recommended - has some great features to zero in on the suspect / culprit)
b) IBM Heap Analyser
c) JConsole
d) Yourkit - trial version for 14 days (you better solve your issue within 14 days :)) [Highly recommended - it allows monitoring and a hell lot of features]
e) JMAP


Analysis & Solution -
There are two ways of going about this, you would never programmatically create duplicate classes and load them into the permgen, so the only few options are
First Way
1) Are you using different classloaders or custom classloaders to load few classes? If yes there are cases that same classes are loaded often because of different classloader instances. Check you own code.
Second Way
1) Analyse the various frameworks that you are using, check in particular if you in your code or these frameworks use dynamic proxying, which means creating proxy classes at run time for loosely coupling the classes - http://java.sun.com/j2se/1.4.2/docs/guide/reflection/proxy.html
PS: A tip - check if any framework uses CGLIB, in my case it was this.

Approach 1
1) I used jmap on the unix machine and got the heap reproduced in a file called heap.bin.
2) Then open the heap.bin using the free tool IBM Heap Analyser (ha36.jar Let me know if you need it) or MAT or Yourkit.
3) Now you can view different details of the heap, like duplicate class, which objects is being created in large number or objects based on size etc.
4) Now go to class instances and search for duplicate classes or sort the classes by name and then go through the list.
5) You should be able to see at some point few duplicate classes being loaded and they would have different extension
example- Class ABC you can see Class ABC, Class ABC$$EnchancerByCGLIB$$9a6119f_32$$FastClassByCGLIB$$123456, Class ABC$$EnchancerByCGLIB$$9a6119f_33$$FastClassByCGLIB$$123453, Class ABC$$EnchancerByCGLIB$$9a6119f_34$$FastClassByCGLIB$$123454, Class ABC$$EnchancerByCGLIB$$9a6119f_35$$FastClassByCGLIB$$123455 etc etc
6) Now you know this class is being created by CGLIB / dynamic proxying and because they are all different classes all together they will be surely kept in the perm gen adding to the clutter and the error.

Approach 2
1) Yourkit attaches to the live running application - the process is given in their site.
2) Monitor the live application and keep checking the permgen and number of classes loaded and unloaded - if the classes loaded keeps increasing even after a threshold we know we are in trouble (jconsole also can be used to see the classes loaded and unloaded)
3) In your kit keep checking the class instances and search for duplicate classes or sort the classes by name and then go through the list.
4) You should be able to see at some point few duplicate classes being loaded and they would have different extension
example- Class ABC you can see Class ABC, Class ABC$$EnchancerByCGLIB$$9a6119f_32$$FastClassByCGLIB$$123456, Class ABC$$EnchancerByCGLIB$$9a6119f_33$$FastClassByCGLIB$$123453, Class ABC$$EnchancerByCGLIB$$9a6119f_34$$FastClassByCGLIB$$123454, Class ABC$$EnchancerByCGLIB$$9a6119f_35$$FastClassByCGLIB$$123455 etc etc
5) Now you know this class is being created by CGLIB / dynamic proxying and because they are all different classes all together they will be surely kept in the perm gen adding to the clutter and the error.
6) This list will only keep increasing as time flies / more requests are being processed.


Solution -
1) Now check which version of the framework you are using and check any recorded bug against it - I was using cglib-nodep-2.1_3.jar and it seems to have a weak reference to all the classes that it creates causing this clutter. I then had to upgrade Spring to spring 2.5.6.SEC01, Hibernate and CGLIB to their production stable releases.
2) After this fix / upgrade I took regular imprints / monitored it live and found there were no more duplicate classes being created.
2.1) Another way is to take regular heap imprints and you would see that there is pattern say heap1 = 10 MB, heap2 = 50 MB, heap3 = 100 MB, heap4 = 80 MB, Heap5 ~= 80 MB.
In the case of the error prone application / memory leak plagued application the pattern would be heap1 = 10 MB, heap2 = 50 MB, heap3 = 100 MB, heap4 = 120 MB, Heap5 = 140 MB....


PS: Jconsole can be used to see the heap growth, perm gen growth, class loading, class loading etc at run time. All the graphs can have peaks and downs but must even out in due course, if they keep going north then you know for sure you are going to have trouble.

Few More
jmap -permstat 18254 > permstats_2.txt
jmap -histo:live 18254 > histo_live.txt
jmap -histo 18254 > histo.txt
-XX:-TraceClassLoading Trace loading of classes.
-XX:-TraceClassLoadingPreorder Trace all classes loaded in order referenced (not loaded). (Introduced in 1.4.2.)
-XX:-TraceClassResolution Trace constant pool resolutions. (Introduced in 1.4.2.)
-XX:-TraceClassUnloading Trace unloading of classes.

m.m