9 Replies Latest reply on Aug 4, 2015 1:46 PM by Scott Stark

    What are our middleware sun.misc.Unsafe uses?

    Scott Stark Master

      With the removal of the "private" sun.misc.Unsafe api on the horizon, we need to document the uses we have for the Unsafe apis and ensure there are public replacements in Java 9/10. The current view is that although sun.misc.Unsafe may not be visible by default in Java 9, there will be a flag that enables access to the class to avoid an immediate need to avoid disruption when Java 9 releases.

       

      The attached document is a summary of concerns that were raised by members of the Java Executive Committee(EC), and there was discussion around creating an EC working group to address this problem.

       

      I have pulled together the following comments from Jason, David and Andrew. Please post the specifics of APIs we are using and why so I can start tying together the use case with

       

      Jason Greene:

      *Everything* in Unsafe is useful. It is the only way to do the things it does, and to abuse the idiom, whats necessary for the goose (JDK API implementations) is necessary for the gander.

       

      Another use case is serialization. Without Unsafe or its Reflection internal API cousins, there is no way to implement a serialization framework with the same capabilities as Java serialization (e.g. the ability to serialize a class WITHOUT a public no-arg constructor).

       

      DML:

      One interesting note is that an Oracle employee was able to develop a

      version of the Atomic*FieldUpdater classes which had performance on par

      with Unsafe itself.  The trick is to generate a new *Updater subclass

      for each updater, which internally uses Unsafe.  Because most of the

      implementation methods in this case degenerate to straight calls to

      Unsafe with constant arguments (without all the complicated access

      checks that make these things slow), most of the additional cost

      disappears, other than having the virtual method dispatch.  I vaguely

      recall that this might even be mitigated in an existing or future

      hotspot version under certain circumstances: if you assign your updater

      to a static final field, then the exact type of that field's value

      should always be statically known to hotspot and thus it should be able

      to optimize and even inline method calls on that reference.  (If this

      isn't the case, then, well, it ought to be... )

       

      I do know that the performance results were quite good.  I would

      personally be reasonably satisfied with this solution in lieu of Unsafe

      (at least for CAS field accesses).  It doesn't do anything to address

      the other use cases though - like atomic memory access for example,

      which is a fairly important use case.



      Andrew Haley:

      I don't think that there are any real long-term reasons to keep

      Unsafe.  They are being addressed by things like VarHandles, my own

      work on speeding up heap ByteBuffers so that they're as fast as C

      arrays, and so on.


      TODO:

       

      ProjectUnsafe UsageReplacement
      jboss-modulesLow level primitives for synchronizationUsage will be dropped in 1.5 release when jdk7+ is required
      infinispanADDRESS_SIZE, ARRAY_OBJECT_INDEX_SCALETBD
      infinispancompareAndSwapInt/Long, objectFieldOffset, get/putObjectVolatile, arrayBaseOffset, ConcurrentHashMapV8 backport from JSR166TBD
      nettyConcurrentHashMapV8,CountedCompleter,ForkJoinPool,Striped64 JDK8 backports from JSR166, copyMemory, arrayBaseOffset, objectFieldOffset, get/put/{Byte,Short,Int,Long}TBD

      aesh

      sun.misc.Signal*No longer in use after 33.xx branch


      Imported Project Usage

       

      ProjectUsageReplacement
      Apache AvroSerializationTBD
      GuavaStriped64 backport from JSR166, arrayBaseOffset, getLong


        • 2. Re: What are our middleware sun.misc.Unsafe uses?
          Scott Stark Master

          Thanks Tomaz, I'm including the original jdeps report and Jason's original reply. I have dropped the xml related issues since that is simply bundled code.

           

          • for our JAXP providers so that we get consistent behavior on all JVM vendors and versions. The first two introduce the APIs that the report is complaining about. This is misreported as a JDK internal dep, because the JDK implementation of JAXP happens to be a fork of them, and so introduces those APIs as well.
          • org.relax.* - This is a similar issue. We bundle the sun JAXB impl of a fixed version in order to provide consistent results on all JVMs, the classes it is complaining about are actually introduced by a sub JAXB dep (rngom).
          • com.sun.misc.Unsafe - This class is critical to implementing high-performance concurrent data structures and algorithms (hence it’s usage by java.util.concurrent). Our general pattern is to attempt to use it if possible, and fallback to slower, less resilient, yet portable mechanisms. In practice all JVMs now include this class, since all providers prefer to reuse the java.util.concurrent code. If the day comes that java.util.concurrent no longer requires this (or an equivalent class), our primary need for it would be eliminated.
          • sun.reflect.ReflectionFactory - Serialization/Marshalling frameworks need to be able to instantiate classes which do not have public constructors (just like Java Serialization requires). Without this hook, or alternatively usage of unsafe, this is not possible.
          • sun.misc.Cleaner - This is useful in cases where you want to free the memory associated with a direct buffer immediately, preventing degenerate cases of massive native memory consumption, or failures caused by inaccurate limit checking. The main issue is that the garbage collector does not have proper insight into native memory space. and thus may defer freeing what it thinks is a single small reference which in reality is a large block of memory. PhantomReference isn’t a workable alternative. That said, in general we prefer to avoid non-static usage of direct buffers.
          • com.sun.org.apache.xerces.* - This looks like one of our third party deps, XOM, includes some code for Java 5 to workaround bugs and enable secure processing. This code will never actually be ran on WildFly since we require Java 7.
          • sun.misc.Signal/SignalHandler - This dependency has been removed in later versions and uses an alternative strategy.
          • sun.tools.jconsole.Utilities - We have a jconsole gui plugin that is using this. Aren’t all of these jconsole APIs considered unofficial?
          • GSS SPI/API usage - I’d have to research this further as to why jacorb needs these. However, we are looking into switching to a downstream fork of the oracle orb used in the JDK, so this one is likely to vanish as well.

            

          JDeps Report for wildfly-8.1.0.Final

            

                  We are using JDK 8's new          jdeps        analysis tool to identify usage of JDK-internal APIs. It has been advised not to rely on any JDK-internal API        (e.g. sun.* packages)        since 1996. The jdeps        tool along with a        wiki page        were created to help you migrate to use the JDK supported APIs.  

                    

                  Your team should plan to eliminate any dependency to JDK-internal APIs.  Migrating to        use the JDK supported APIs now will give your team adequate time for regression        testing before the release of JDK 9.        For JDK-internal APIs that your application has to depend on and there is no        supported API providing equivalent functionality, we would appreciate an        explanation about why and how these particular APIs, if any, are used in your        application. jdeps is a static analysis tool and therefore  use of JDK-internal        APIs via reflection or dynamically generated bytecode are not shown in this report        while such dependency should also be replaced.  

            

                  Development teams can use        jdeps        directly to identify any use of JDK-internal APIs and update your code or components        appropriately.          Alternately if many development groups need something changed in the        spec, we encourage participation in either        OpenJDK or the        Java Community Process.  

                    

                  We have analyzed jar files within wildfly-8.1.0.Final and found 26        jar files depend on JDK-Internal-API. Our        observations are  

              
          • APIs that have known replacements: 7  
          • APIs that your team should plan to migrate away and provide us            with explanations if necessary: 87  

            

           

                    

          APIs that have                        known replacements:

            

                                                                                          

          IDReplace Usage ofWithInside
          7sun.misc.CleanerUse java.lang.ref.PhantomReference @since 1.2
          • bin/client/jboss-client.jar
          • modules/system/layers/base/io/netty/main/netty-all-4.0.15.Final.jar

            

          APIs that your team should plan to migrate away and provide us            with explanations if necessary:

            

                                                                                                                                                                                                                                                            

          IDInternal APIs requiring clarificationAre used by
          75sun.misc.Signal
          • bin/client/jboss-cli-client.jar
          • modules/system/layers/base/org/jboss/aesh/main/aesh-0.33.11.jar
          Explanation if necessary...
          76sun.misc.SignalHandler
          • bin/client/jboss-cli-client.jar
          • modules/system/layers/base/org/jboss/aesh/main/aesh-0.33.11.jar
          Explanation if necessary...
          77sun.misc.Unsafe
          • bin/client/jboss-client.jar
          • jboss-modules.jar
          • modules/system/layers/base/com/google/guava/main/guava-16.0.1.jar
          • modules/system/layers/base/io/netty/main/netty-all-4.0.15.Final.jar
          • modules/system/layers/base/io/undertow/core/main/undertow-core-1.0.15.Final.jar
          • modules/system/layers/base/javax/rmi/api/main/jboss-rmi-api_1.0_spec-1.0.4.Final.jar
          • modules/system/layers/base/org/apache/avro/main/avro-1.7.6.jar
          • modules/system/layers/base/org/infinispan/commons/main/infinispan-commons-6.0.2.Final.jar
          • modules/system/layers/base/org/wildfly/mod_cluster/undertow/main/wildfly-mod_cluster-undertow-8.1.0.Final.jar
          Explanation if necessary...
          78sun.nio.cs.Surrogate
          • modules/system/layers/base/com/sun/codemodel/main/codemodel-2.6.jar
          Explanation if necessary...
          79sun.nio.cs.Surrogate$Parser
          • modules/system/layers/base/com/sun/codemodel/main/codemodel-2.6.jar
          Explanation if necessary...
          80sun.reflect.Reflection
          • bin/client/jboss-cli-client.jar
          • modules/system/layers/base/org/wildfly/security/manager/main/wildfly-security-manager-1.0.0.Final.jar
          Explanation if necessary...
          81sun.reflect.ReflectionFactory
          • bin/client/jboss-cli-client.jar
          • bin/client/jboss-client.jar
          • modules/system/layers/base/io/undertow/servlet/main/undertow-servlet-1.0.15.Final.jar
          • modules/system/layers/base/javax/rmi/api/main/jboss-rmi-api_1.0_spec-1.0.4.Final.jar
          • modules/system/layers/base/org/jboss/marshalling/main/jboss-marshalling-1.4.6.Final.jar
          Explanation if necessary...
          82sun.reflect.ReflectionFactory$GetReflectionFactoryAction
          • bin/client/jboss-cli-client.jar
          • bin/client/jboss-client.jar
          • modules/system/layers/base/javax/rmi/api/main/jboss-rmi-api_1.0_spec-1.0.4.Final.jar
          • modules/system/layers/base/org/jboss/marshalling/main/jboss-marshalling-1.4.6.Final.jar
          Explanation if necessary...
          83sun.security.jgss.spi.GSSContextSpi
          • modules/system/layers/base/org/jacorb/main/jacorb-2.3.2-jbossorg-5.jar
          Explanation if necessary...
          84sun.security.jgss.spi.GSSCredentialSpi
          • modules/system/layers/base/org/jacorb/main/jacorb-2.3.2-jbossorg-5.jar
          Explanation if necessary...
          85sun.security.jgss.spi.GSSNameSpi
          • modules/system/layers/base/org/jacorb/main/jacorb-2.3.2-jbossorg-5.jar
          Explanation if necessary...
          86sun.security.jgss.spi.MechanismFactory
          • modules/system/layers/base/org/jacorb/main/jacorb-2.3.2-jbossorg-5.jar
          Explanation if necessary...
          87sun.tools.jconsole.Utilities
          • bin/client/jboss-cli-client.jar
          • modules/system/layers/base/org/jboss/as/cli/main/wildfly-cli-8.1.0.Final.jar
          Explanation if necessary...

                            

          Identify External Replacements

            

          In addition to the above, your application might be using JDK-internal APIs that do not have            direct Oracle-provided replacements. These APIs are subject to            significant change or removal in JDK 9. Proper fixing may involve            identifying a separate third-party library that performs this            functionality.

            

                                                                                              

          IDInternal API (grouped by package)Used ByIdentify External Replacement
          • 3. Re: What are our middleware sun.misc.Unsafe uses?
            William Burns Expert

            There was an equivalent discussion on infinispan-dev a while back as well: [infinispan-dev] Analysis of infinispan-6.0.2 dependency on JDK-Internal APIs & [infinispan-dev] Analysis of infinispan-6.0.2 dependency on JDK-Internal APIs

             

            Our usage of Unsafe as Dan has mentioned is due to use a modified variation of the ConcurrentHashMap to allow us to use a different type of equivalence (required for arrays).

             

            Since this prior discussion we have added usage of Unsafe for the purpose of estimating the in memory size of our data container.  This uses the static ARRAY_BYTE_BASE_OFFSET, ARRAY_BYTE_INDEX_SCALE, ADDRESS_SIZE & ARRAY_OBJECT_INDEX_SCALE variables to estimate sizing on heap (since these take into consideration 32 or 64 bit jvm and compressed pointers).

            • 4. Re: What are our middleware sun.misc.Unsafe uses?
              Scott Stark Master

              William, it looks the usage is only when running in a VM prior to Java 8. Is that correct?

              • 5. Re: What are our middleware sun.misc.Unsafe uses?
                Jozef Hartinger Master

                Weld optionally uses Unsafe#allocateInstance(Class) for instantiating client proxies for classes that do not have a no-arg constructor. Note that Unsafe is only used if a special non-default "relaxed construction" configuration option is set. By default, CDI requires that proxied beans have no-arg constructors.

                • 6. Re: What are our middleware sun.misc.Unsafe uses?
                  William Burns Expert

                  Scott Stark wrote:

                   

                  William, it looks the usage is only when running in a VM prior to Java 8. Is that correct?

                  When Infinispan was using Java 7 we "backported" the ConcurrentHashMap from Java 8.  Now that we support Java 8 we still have to use a custom ConcurrentHashMap to support our custom equality, so unfortunately it is still in use.

                  • 7. Re: What are our middleware sun.misc.Unsafe uses?
                    Michel Graciano Newbie

                    Just trying to help a little.

                    There is a recent updated JEP 193: Variable Handles which states about the exposure of some Unsafe operations. Maybe it is interesting that some of you take a look at this, if already haven't.

                     

                    Best regards

                    • 8. Re: What are our middleware sun.misc.Unsafe uses?
                      Scott Stark Master

                      This email to the openjdk dev list introduces JEP 260: Encapsulate Most Internal APIs, which is about:


                      Description

                      Based upon analyses of various large collections of code, including Maven Central, and also feedback received since the release of JDK 8 and its dependency analysis tool (jdeps), we can divide the JDK's internal APIs into two broad categories:

                      • Those which do not appear to be used by code outside of the JDK, or are used by outside code merely for convenience, i.e., for functionality that is available in supported APIs or can easily be provided by libraries (e.g.,sun.misc.BASE64Decoder); and
                      • Those which provide critical functionality that would be difficult, if not impossible, to implement outside of the JDK itself (e.g., sun.misc.Unsafe).

                      In JDK 9 we propose to:

                      • Encapsulate all non-critical internal APIs by default: The modules that define them will not export their packages for outside use. (Access to such APIs will be available, as a last resort, via a command-line flag at both compile time and run time, unless those APIs are revised or removed for other reasons.)
                      • Encapsulate critical internal APIs for which supported replacements exist in JDK 8, in the same manner and with the same last-resort workaround. (A supported replacement is one that is either part of the Java SE 8 standard (i.e., in a java.* or javax.* package) or else JDK-specific and annotated with @jdk.Exported (typically in a com.sun.* or jdk.* package).)
                      • Not encapsulate critical internal APIs for which supported replacements do not exist in JDK 8 and, further, deprecate those which have supported replacements in JDK 9 with the intent to encapsulate them, or possibly even remove them, in JDK 10.

                      The critical internal APIs proposed to remain accessible in JDK 9 are:

                      • sun.misc.Cleaner
                      • sun.misc.{Signal,SignalHandler}
                      • sun.misc.Unsafe (The functionality of many of the methods in this class is now available via variable handles (JEP 193).)
                      • sun.reflect.Reflection::getCallerClass (The functionality of this method may be provided in a standard form via JEP 259.)
                      • sun.reflect.ReflectionFactory

                       

                      ----- Forwarded Message -----

                      From: "mark reinhold" <mark.reinhold@oracle.com>

                      To: jigsaw-dev@openjdk.java.net

                      Sent: Tuesday, August 4, 2015 7:48:39 AM

                      Subject: Encapsulating internal APIs in JDK 9 (sun.misc.Unsafe, etc.)

                       

                       

                      As part of the overall modularization effort [1] we're going to

                      encapsulate most of the JDK's internal APIs within the modules that

                      define and use them so that, by default, they are not accessible to

                      code outside the JDK.

                       

                       

                      This change will improve the integrity of the platform, since many of

                      these internal APIs define privileged, security-sensitive operations.

                      In the long run it will also reduce the costs borne by the maintainers

                      of the JDK itself and by the maintainers of libraries and applications

                      that, knowingly or not, make use of these non-standard, unstable, and

                      unsupported internal APIs.

                       

                       

                      It's well-known that some popular libraries make use of a few of these

                      internal APIs, such as sun.misc.Unsafe, to invoke methods that would be

                      difficult, if not impossible, to implement outside of the JDK.  To ensure

                      the broad testing and adoption of the release we propose to treat these

                      critical APIs as follows:

                       

                       

                        - If it has a supported replacement in JDK 8 then we will encapsulate

                          it in JDK 9;

                       

                       

                        - If it does not have a supported replacement in JDK 8 then we will not

                          encapsulate it in JDK 9, so that it remains accessible to outside

                          code; and, further,

                       

                       

                        - If it has a supported replacement in JDK 9 then we will deprecate it

                          in JDK 9 and encapsulate it, or possibly even remove it, in JDK 10.

                       

                       

                      The critical internal APIs proposed to remain accessible in JDK 9 are

                      listed in JEP 260 [2].  Suggested additions to the list, justified by

                      real-world use cases and estimates of developer and end-user impact,

                      are welcome.

                       

                       

                      - Mark

                       

                       

                       

                       

                      [1] http://openjdk.java.net/jeps/200

                      [2] http://openjdk.java.net/jeps/260

                      • 9. Re: What are our middleware sun.misc.Unsafe uses?
                        Scott Stark Master

                        Thanks. That JEP and some others are referenced in the JEP 260 that was just introduced to formally address the encapsulation problem.