-
1. Re: Running Eclipse with byteman javaagent needs immense amount of memory
Hi Sven,
Sven Lange wrote:
. . .
Is byteman soooooo mem hungry??? My rules file has about 8 rules.
No, Byteman is soooooo *not* memory hungry. It works very hard to avoid retaining references which might use up memory, especially PermGen space. Normally, if you unload all your rules then Byteman will drop all references to any classes they were injected into.
This looks like it is a problem related to application redeployment. This is the most common cause of PermGen exhaustion -- especially inside Eclipse. The way your rules are written may well be causing references to application objects and, hence, classes in perm gen space to be retained across redeployments. If, for example, your rules use application instances as identifiers for Helper objects like counters, flags etc then these objects will be referenced from static fields (hashtables, ect) declared by Byteman's Helper class and that means their classes will contiue to be retained even after the rules whcih referred to them have been unloaded. Ideally a rule set should always delete such objects at some suitable point -- e.g. every rule set which uses createCounter or incrementCounter should, ideally, include a rule which calls deleteCounter. Of course, this is normally only a problem in environments like Eclipse where you redeploy classes -- PermGen space is rarely able to be reclaimed. Nor is this a problem unique to Byteman -- many other Java applications/tools fail with PermGen exhaustion
[aside: there are several options available to force dropping of such Helper data whenever all rules whcih might have created them are unloaded. None of them have been implemented because they cause issues with retainign backwards comaptibility for existing Byteman rule sets. if your problem turns out to be to do with unwanted retention of application instances thanks to helper calls then I wil be very interested in discussing options for enabling one of other of these forcing mechanisms based on your requirements.]
Another possibility is that you might have found a problem in the retransformer code in your JVM (I assume from your prompt that you are running on OSX). I have little experience of how well OSX implements the agent APIs but it has very likely been tested far less well than Linux. And, of course, you may just have found a nasty memory leak in Byteman itself. The fact that the problem is to do with PermGen space suggests this is to do with reloading classes rather than allocating instances so my expectation is that the problem is something to do with how Byteman is interacting with Eclipse or the JVM. If you could post your rules (or at least give an indication of what they are doing) and explain how you are running your application it would be much easier to identify what is going wrong and work out how to fix it.
-
2. Re: Running Eclipse with byteman javaagent needs immense amount of memory
Hi Andrew,
here is my rules script:
RULE trace main entry
CLASS XtendCompiler
METHOD getFavoriteVariableName
AT ENTRY
IF true
DO traceln("entering getFavoriteVariableName")
ENDRULE
RULE trace main exit
CLASS XtendCompiler
METHOD normalizeBlockExpression
AT ENTRY
IF true
DO traceln("entering normalizeBlockExpression")
ENDRULE
RULE trace main exit
CLASS XtendCompiler
METHOD doInternalToJavaStatement
AT ENTRY
IF true
DO traceln("entering doInternalToJavaStatement")
ENDRULE
RULE trace main exit
CLASS XtendCompiler
METHOD _toJavaStatement
AT ENTRY
IF true
DO traceln("entering _toJavaStatement")
ENDRULE
RULE trace main exit
CLASS XtendCompiler
METHOD internalToConvertedExpression
AT ENTRY
IF true
DO traceln("entering internalToConvertedExpression")
ENDRULE
RULE trace main exit
CLASS XtendCompiler
METHOD appendCatchClauseParameter
AT ENTRY
IF true
DO traceln("entering appendCatchClauseParameter")
ENDRULE
RULE trace main exit
CLASS XtendCompiler
METHOD appendVariableTypeAndName
AT ENTRY
IF true
DO traceln("entering appendVariableTypeAndName")
==== END
Eclipse is running normally until i open a Xtend (http://www.eclipse.org/xtend/) file with the Eclipse Xtend editor view. I can open Java or any other file exceot for Xtend files. As soon as I open a Xtend file I see the error messages on the console and eclipse freezes.
I am using Java 1.6.51 on a macbook with osx 10.4.3 / Eclipse 4.3 Kepler with Xtend 2.4.2
Best
Sven
-
3. Re: Running Eclipse with byteman javaagent needs immense amount of memory
1 of 1 people found this helpfulHi Sven,
Sven Lange wrote:
here is my rules script:
. . .RULE trace main exit
CLASS XtendCompiler
METHOD appendVariableTypeAndName
AT ENTRY
IF true
DO traceln("entering appendVariableTypeAndName")
==== END
Hmm, where it says ==== END does your rule actually say ENDRULE?
Sven Lange wrote:
Eclipse is running normally until i open a Xtend (http://www.eclipse.org/xtend/) file with the Eclipse Xtend editor view. I can open Java or any other file exceot for Xtend files. As soon as I open a Xtend file I see the error messages on the console and eclipse freezes.
I am using Java 1.6.51 on a macbook with osx 10.4.3 / Eclipse 4.3 Kepler with Xtend 2.4.2
Well, that is pretty weird. However, I think this is likely to be Eclipse fighting with Byteman in order to retain ownership of what Eclipse almost certainly considers to be one of its own classes.
Byteman uses the JVMTI Java agent capability of the JVM to modify the definition of the classes mentiond in Byteman rules. I don't know for sure what Eclipse does when running Java code but I do know that it will at some point use the JVMTI native library agent capabilities to perform debugging -- that's so because the debugger interface employed by Eclipse and any other program capable of debugging a JVM can only be based on these JVMTI native library APIs (this has been true since at least JDK6 -- I am not sure but possibly for JDK5 too). So, Eclipse will be using these JVMTI hooks, either directly or indirectly.
JVMTI native library agents can do far more than a Java agent like Byteman. Java agents normally only modify bytecode or, at first load only, Java class field, method and super hierarchy layout. Native agents can modify the behaviour of the JVM as well as redefine Java classes. This means it is hard to be sure exactly what Eclipse is doing without a thorough debug/audit of the Eclipse code itself. However, I suspect Eclipse will want to ensure that its own classes are either i) exempt from being transformed by Java agents or ii) redefined only in specific classloader contexts so that there is a vanilla version available for Eclipse itself to use. Given the symptoms you describe it looks to me like when Byteman changes the definition of class XtendCompiler that Eclipse is stepping in to provide another version and that this is ending up in some sort of recursive redefinition process.
Can you confirm that you are able to inect rules into other classes which are not related to Eclipse itself? (for example, try injecting code into a JDK class like Thread or String and check that you can still run your program).
Are you forking the code which employs Xtend in a separate JVM from the one running Eclipse? If not then could you investigate if the problem still manifests when you do so?
Thanks for raising this issue and for any further information you can provide. I will raise a JIRA for the problem so that it is documented for other Byteman users. If you can provide more feedback then I might be able to help resolve this problem but I doubt I will be able to do so as I have neither the requisite expertise in Eclipse to debug it properly nor the time to acquire said expertise. Let's hope that someone with more detailed knowledge of Eclipse will be able to idenitfy what is going wrong here and suggest a fix or workaround.
regards,
Andrew Dinn
-
4. Re: Running Eclipse with byteman javaagent needs immense amount of memory
Hi Andrew,
now i started Eclipse with this rule:
RULE trace main entry
CLASS Thread
METHOD run
AT ENTRY
IF true
DO traceln("entering run")
ENDRULE
When I now open a Java editor Eclipse dies immediately without any error message on the console.
Starting eclipse again it dies again with some errors printed to the console:
Svens-MacBook-Pro:Applications lange$ java -jar eclipse/plugins/org.eclipse.equinox.launcher_1.3.0.v20130327-1440.jar -XstartOnFirstThread -javaagent:${BYTEMAN_HOME}/lib/byteman.jar=script:~/Desktop/blah.btm -XX:MaxPermSize=1500m -Xms3000m -Xmx5000m
0 [Worker-1] WARN org.eclipse.xtext.ui.shared.internal.Activator - Disabling JDT use. : org/eclipse/jdt/ui/JavaUI
200 [Worker-1] WARN org.eclipse.xtext.ui.shared.internal.Activator - Disabling JDT use. : org/eclipse/jdt/ui/JavaUI
Job found still running after platform shutdown. Jobs should be canceled by the plugin that scheduled them during shutdown: org.eclipse.ui.internal.activities.MutableActivityManager$3$1
Job found still running after platform shutdown. Jobs should be canceled by the plugin that scheduled them during shutdown: org.eclipse.ui.internal.activities.MutableActivityManager$3$1
Job found still running after platform shutdown. Jobs should be canceled by the plugin that scheduled them during shutdown: org.eclipse.ui.internal.activities.MutableActivityManager$3$1
Job found still running after platform shutdown. Jobs should be canceled by the plugin that scheduled them during shutdown: org.eclipse.ui.internal.activities.MutableActivityManager$3$1
Job found still running after platform shutdown. Jobs should be canceled by the plugin that scheduled them during shutdown: org.eclipse.ui.internal.activities.MutableActivityManager$3$1
Job found still running after platform shutdown. Jobs should be canceled by the plugin that scheduled them during shutdown: org.eclipse.ui.internal.activities.MutableActivityManager$3$1
Job found still running after platform shutdown. Jobs should be canceled by the plugin that scheduled them during shutdown: org.eclipse.ui.internal.activities.MutableActivityManager$3$1
Job found still running after platform shutdown. Jobs should be canceled by the plugin that scheduled them during shutdown: org.eclipse.ui.internal.activities.MutableActivityManager$3$1
Job found still running after platform shutdown. Jobs should be canceled by the plugin that scheduled them during shutdown: org.eclipse.ui.internal.activities.MutableActivityManager$3$1
Job found still running after platform shutdown. Jobs should be canceled by the plugin that scheduled them during shutdown: org.eclipse.ui.internal.activities.MutableActivityManager$3$1
Job found still running after platform shutdown. Jobs should be canceled by the plugin that scheduled them during shutdown: org.eclipse.ui.internal.activities.MutableActivityManager$3$1
Job found still running after platform shutdown. Jobs should be canceled by the plugin that scheduled them during shutdown: org.eclipse.ui.internal.activities.MutableActivityManager$3$1
Job found still running after platform shutdown. Jobs should be canceled by the plugin that scheduled them during shutdown: org.eclipse.ui.internal.activities.MutableActivityManager$3$1
Job found still running after platform shutdown. Jobs should be canceled by the plugin that scheduled them during shutdown: org.eclipse.ui.internal.activities.MutableActivityManager$3$1
Job found still running after platform shutdown. Jobs should be canceled by the plugin that scheduled them during shutdown: org.eclipse.ui.internal.activities.MutableActivityManager$3$1
Job found still running after platform shutdown. Jobs should be canceled by the plugin that scheduled them during shutdown: org.eclipse.ui.internal.activities.MutableActivityManager$3$1
Job found still running after platform shutdown. Jobs should be canceled by the plugin that scheduled them during shutdown: org.eclipse.ui.internal.activities.MutableActivityManager$3$1
Job found still running after platform shutdown. Jobs should be canceled by the plugin that scheduled them during shutdown: org.eclipse.ui.internal.activities.MutableActivityManager$3$1
Job found still running after platform shutdown. Jobs should be canceled by the plugin that scheduled them during shutdown: org.eclipse.ui.internal.activities.MutableActivityManager$3$1
Job found still running after platform shutdown. Jobs should be canceled by the plugin that scheduled them during shutdown: org.eclipse.ui.internal.activities.MutableActivityManager$3$1
Job found still running after platform shutdown. Jobs should be canceled by the plugin that scheduled them during shutdown: org.eclipse.ui.internal.activities.MutableActivityManager$3$1
Job found still running after platform shutdown. Jobs should be canceled by the plugin that scheduled them during shutdown: org.eclipse.ui.internal.activities.MutableActivityManager$3$1
An error has occurred. See the log file
/Users/lange/Desktop/deleteme22/.metadata/.log.
the log is quite long......
i wont investigate any further. dont have time to dig deeper into this. thanks for your help!
-
5. Re: Running Eclipse with byteman javaagent needs immense amount of memory
Hi Sven,
Sven Lange wrote:
now i started Eclipse with this rule:
RULE trace main entry
CLASS Thread
METHOD run
AT ENTRY
IF true
DO traceln("entering run")
ENDRULE
When I now open a Java editor Eclipse dies immediately without any error message on the console.
Starting eclipse again it dies again with some errors printed to the console:
Svens-MacBook-Pro:Applications lange$ java -jar eclipse/plugins/org.eclipse.equinox.launcher_1.3.0.v20130327-1440.jar -XstartOnFirstThread -javaagent:${BYTEMAN_HOME}/lib/byteman.jar=script:~/Desktop/blah.btm -XX:MaxPermSize=1500m -Xms3000m -Xmx5000m
0 [Worker-1] WARN org.eclipse.xtext.ui.shared.internal.Activator - Disabling JDT use. : org/eclipse/jdt/ui/JavaUI
200 [Worker-1] WARN org.eclipse.xtext.ui.shared.internal.Activator - Disabling JDT use. : org/eclipse/jdt/ui/JavaUI
Job found still running after platform shutdown. Jobs should be canceled by the plugin that scheduled them during shutdown: org.eclipse.ui.internal.activities.MutableActivityManager$3$1
. . .Oh boy, look at all those worms wriggling inside that can!
Thanks for following this up. I admit I have no idea what is going on here -- I think this requires an Eclipse GURU :-(
Anyway, it's very good to know that there is a problem here.
regards,
Andrew Dinn