Native Memory Tracking in 7u40

Since we don’t have any nice NMT (Native Memory Tracking) MBean in HotSpot (yet), and therefore not in the JMC console, I thought I’d show how it can be done using command line arguments and JCMD. Please note that you’ll get a 5-10% performance hit if you enable this.

Step 1 – Enabling NMT

This is done by using the following command line:

-XX:NativeMemoryTracking=[off|summary|detail]

Where the different options are:

off NMT is turned off. This is the default.
summary Only collect memory usage aggregated by subsystem.
detail Collect memory usage by individual call sites.

 

Again, note that enabling this will cause you a 5-10% overhead.

 

Step 2 – Using JCMD to Access the Data

To use JCMD to dump the data collected thus far, and to optionally compare it to the last baseline, use the following command:

jcmd <pid> VM.native_memory [summary | detail | baseline | summary.diff | detail.diff | shutdown] [scale= KB | MB | GB]

summary Print a summary aggregated by category.
detail
  • Print memory usage aggregated by category
  • Print virtual memory map
  • Print memory usage aggregated by call site
baseline Create a new memory usage snapshot to diff against.
summary.diff Print a new summary report against the last baseline.
detail.diff Print a new detail report against the last baseline.
shutdown Shutdown NMT.

 

Here is a table with the different memory categories (this may change):

Category

Description

Java Heap

The heap. This is where your objects live.

Class

Class meta data.

Code

Generated code.

GC

Data use by the GC, such as card table.

Compiler

Memory used by the compiler when generating code.

Symbol

Symbols.

Memory Tracking

Memory used by the NMT itself.

Pooled Free Chunks

Memory used by chunks in the arena chunk pool.

Shared space for classes

Memory mapped to class data sharing archive.

Thread

Memory used by threads, including thread data structure, resource area and handle area, etc.

Thread stack

Thread stack. It is marked as committed memory, but it might not be completely committed by the OS.

Internal

Memory which does not fit the above categories, such as the memory used by the command line parser, JVMTI, properties, etc.

Unknown

When memory category can not be determined.

  • Arena: when arena is used as stack or value object
  • Virtual Memory: when type information has not yet arrived

 

Additional Options

Use the following NMT options to print a report on VM exit:

-XX:+PrintNMTStatistics -XX:+UnlockDiagnosticVMOptions

NMT will shut itself down when it detects low resource conditions, such as when running low on native memory. Sometimes that may be the situation when you would really want the data! This is the flag to disable auto shutdown:

-XX:-AutoShutdownNMT

Note that detail level information is not supported on ARM.

Final Thoughts

The information in this blog post mostly comes from a wiki document authored by Zhengyu Gu, who also did the HotSpot implementation. Also, shout out to Fredrik Öhrström who came up with the NMT idea, and who did the original JRockit implementation of it!

8 Responses to "Native Memory Tracking in 7u40"

  1. PhilC says:

    Hi, does this also help track memory from JNI invocations?

  2. Marcus says:

    Not in this first phase. This first version only tracks HotSpot internal memory usage. I believe there is a second phase of the NMT project where third party native code allocation will be tracked. Not sure when it is targeted for implementation though.

  3. Dmitry says:

    Is it supported in Mac OS X? I’m getting the following error running jcmd :

    com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded
    at sun.tools.attach.BsdVirtualMachine.(BsdVirtualMachine.java:90)
    at sun.tools.attach.BsdAttachProvider.attachVirtualMachine(BsdAttachProvider.java:63)
    at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:213)
    at sun.tools.jcmd.JCmd.executeCommandForPid(JCmd.java:140)
    at sun.tools.jcmd.JCmd.main(JCmd.java:129)

  4. Marcus says:

    Yep, I’m writing this on my MacBook Pro. Works fine here. Make sure you’ve eliminated the usual attach suspects, such as not being able to write to the hsperfdata folder.

    Kind regards,
    Marcus

  5. Hassan says:

    Any further update on JNI memory tracking? Is this already added in latest release? Appreciate your help.

    > Not in this first phase. This first version only tracks HotSpot internal memory usage. I believe
    > there is a second phase of the NMT project where third party native code allocation will be
    > tracked. Not sure when it is targeted for implementation though.

    >> Hi, does this also help track memory from JNI invocations?

  6. Marcus says:

    Hi Hassan,

    I believe there it at least a performance upgrade planned for the existing memory tracking in an upcoming version of the JDK (not out yet). Not sure if that also includes any new functionality.

    Kind regards,
    Marcus

  7. Dmitry says:

    Are there any options for native memory leaks diagnostics? I’ve tried to detect unclosed java.util.zip.Deflater leak using NMT, but it doesn’t seem to show the leak anythere.

  8. It is interesting to note that Unsafe.allocateMemory() is tallied under “Internal” and this applies to ByteBuffer.directAllocate() as well as other uders of Unsafe.allocateMemeory. It wont be visible in the call site details.

Leave a Reply

Your email address will not be published.