We’re interrupting the ordinary programming for a series of Raspberry PI related articles. Yep, I finally bought one and started having some fun.

I know I am sooo late to the party. My excuse is that having really young kids translates into a constant time shortage. That said, I bought myself a Raspberry Pi just in time for some vacation. One of the first things I did was to get an LCD shield for the Raspberry PI. I settled on this particular kit:
http://www.adafruit.com/products/1115

It’s a really nice little LCD kit which also features a few buttons. The shield uses a port extender (MCP23017) to get another 16 GPIO ports over I²C. This is quite nice, since you probably want to use your GPIO ports for other things. Also, you can hook up several I²C devices on the same bus, so using the I²C pins for the LCD shield doesn’t mean that you’ve used them up.

The LCD shield can be run in various different modes. My little example library uses 16×2 characters by default.

Building the Kit

The instructions for building the kit are excellent, and there are just a few things I’d like to note:

1. Get a stacking header
   I fortunately came to the conclusion I would need a stacking header before soldering everything together, but unfortunately after I had ordered the LCD shield. I am oh so happy I decided to wait for the header to arrive before finally putting the kit together.
2. If you want to attach a standard flat 26 pin cable, trim out some space in the LCD PCB for the little orientation peg on the cable header 
   This I did not do – I ended up trimming down the cable header with my Dremel instead, since it’s too hard to access that area after soldering on the LCD.
3. (Optional) Unused GPIO pins for free!
   E.g. GPA5 appears to not be in use. Feel free to use it for your own purposes. The library initializes it as yet another input pin by default, but that is easily changed.
4. (Optional) Use interrupts for the buttons
   This will cost you a GPIO pin (and you cannot use one of the unused ones on the LCD shield’s MCP23017), but will allow you to poll the button pins a lot less often without risking missing a button press. You may want to do some additional soldering on pin 20 of the MCP23017 – INTA is not hooked up by default. This also requires you to hack my Java code a little bit to initialize GPINTEN, and to read from INTF and INTCAP,

Testing the Kit

Once you’ve built the kit, follow the instructions here to try out the LCD kit using the demo python code. Once you have your LCD shield up and running you can move on to getting everything to run from Java.

Using the LCD Shield from Java

Once everything is working from python, you are ready to run things from Java. I mostly did a straight port of the Python library.

1. First download and install Pi4J
   This will be required for most future posts I might put on the blog.
2. Next download lcd.jar
   The source is available with lcd_src.zip.

Now you can run the included demos by just invoking the jar like this:
sudo java –Xbootclasspath/a:/opt/pi4j/lib/pi4j-core.jar –jar lcd.jar

When running the demos, use the up/down buttons to select a demo, then press select to run a demo. When the demo is done, use up/down to select a new demo.

API Examples

Here is how the API can be used to show some text:

LCD lcd = new LCD();
lcd.setText("Hello World!\n2nd Hello World!");

Here is an example with some buttons:

final LCD lcd = new LCD();
lcd.setText("LCD Test!\nPress a button...");
ButtonPressedObserver observer = new ButtonPressedObserver(lcd);
observer.addButtonListener(new ButtonListener() {
    @Override public void onButtonPressed(Button button) {
        lcd.clear();
        lcd.setText(button.toString());
    }
});

For more examples, check out the demos in the se.hirt.pi.adafruitlcd.demo package. I currently have most of the implementation in my python port LCD class. Anything that isn’t purely reading/setting registers, I’ve handled outside of that class (such as the polling of buttons). If I ever find some time I might clean up the API and move away from the current python port.

Tips and Tricks

• If you want to teach the LCD to do additional tricks, the LCD controller used is the HD44780
  Note that it might be valuable to read the spec if for no other reason than to understand the limits (e.g. quite small DDRAM buffer).
• MCP23017 is an awesome little thing
  Ever wanted more GPIO ports? Using the MCP23017 is one way to get plenty more. Just remember to provide a unique address by configuring A0-A2 (see page 8 of the spec). Connecting them all to ground will yield 0x20 (which is what the LCD shield uses). Hooking them all up to 5v will yield 0x27.

• Be careful with how much current you source/sink, both using the MCP23017 (25mA) and the RPi (16mA GPIO, 50mA 3.3v rail, 5V rail pretty much depends on your power supply). If you need to source/sink more than the IO pins can handle, you can use a Darlington Array, for example ULN2803.

This blog post is the third in a series of posts on using unsupported functionality in the Oracle JDK and/or Java Mission Control.

I will use the JMC flight recorder parser in this example, and since the JMC parser is unsupported I’ll just go ahead and use my usual disclaimer:

The following blog entry will describe UNSUPPORTED functionality. This means that relying on the described APIs or functionality may BREAK your code/plugin with any given update of the JDK and/or Mission Control.

I strongly recommend against using these APIs in production code. There will be supported APIs for reading reading recordings eventually and they may be quite different to what I show here. Also note that older versions of the parsers are not guaranteed to be forward compatible with newer versions of the binary format. In short, this is NOT supported and WILL CHANGE!

You may want to first read the introduction to the JFR parsers and about the relational key before continuing.

Ready? Here we go…

A Flight Recording

First off we need a recording to parse. If you have your own favourite recording, you’re all set. If you don’t you can have mine. The recording I linked to is a recording of WLS running the MedRec example app with some load. The benefit of using that recording is that it, aside from the standard events, also contains events provided by the Java API for creating third party events (by the WebLogic Diagnostics Framework). Quite useful events, all bound together using the relational key in various ways. Don’t forget to unzip the recording before use.

The recording is pretty new and really requires a JMC 5.3.0 with the WLS plug-in for the best result. JMC 5.3.0 has not been released yet, but don’t worry; this blog entry will not even require you to start JMC.

That said, since all blog posts require a pretty picture, here is one of the servlet invocations in the recording:

Looking at Metadata

Let’s first write a little program that lists the available event types and their metadata. Use the knowledge obtained in the JFR parser blog entry to obtain a copy of the JMC Flight Recorder Parser.

Next, use the example below to iterate through the metadata and print out the event type name, all the attributes and the relational key for the the individual attributes:

import java.io.File;
import java.util.Collection;

import com.jrockit.mc.flightrecorder.FlightRecording;
import com.jrockit.mc.flightrecorder.FlightRecordingLoader;
import com.jrockit.mc.flightrecorder.spi.IEventType;
import com.jrockit.mc.flightrecorder.spi.IField;

public class FlightRecorderMetaData {
    public static void main(String[] args) throws ClassNotFoundException {
        FlightRecording recording = FlightRecordingLoader.loadFile(new File("C:\\demo\\wldf.jfr"));

        for (IEventType type : recording.getEventTypes()) {
            System.out.println(type.getName());
            printAttributes(type.getFields());
        }
    }

    private static void printAttributes(Collection<IField> fields) {
        for (IField field : fields) {
            System.out.println(String.format("   %s (relkey: %s)", field.getName(), field.getRelationalKey()));
        }
    }
}

Here is a slightly more convoluted example, printing everything out in a nice tree, also showing how many events were found per event type plus some additional stats. It takes a flight recording as its only argument.

Using the Relational Key

Looking at the metadata for the WLDF recording, it can be seen that a lot of the event attributes provided with the WLDF specific events are actually using the relational key for various different things. There is a relational key for servlet URIs, there is another one for the ECID (Enterprise Context ID) and so on. Let’s say we wanted to find all the events with a relational key for ECID, and then group them on ECID (effectively grouping the events per transaction). That could look something like this:

import java.io.File;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;

import com.jrockit.mc.flightrecorder.FlightRecording;
import com.jrockit.mc.flightrecorder.FlightRecordingLoader;
import com.jrockit.mc.flightrecorder.spi.IEvent;
import com.jrockit.mc.flightrecorder.spi.IEventType;
import com.jrockit.mc.flightrecorder.spi.IField;
import com.jrockit.mc.flightrecorder.spi.IView;

public class FlightRecorderECID {
    private static final String KEY_ECID = "http://www.oracle.com/fmw/ECID";

    public static void main(String[] args) throws ClassNotFoundException {
        FlightRecording recording = FlightRecordingLoader.loadFile(new File("C:\\demo\\wldf.jfr"));    

        Collection<IEventType> ecidTypes = new LinkedList<>();
        for (IEventType type : recording.getEventTypes()) {
            if (type.getPath().startsWith("wls/")) { // just optimization, can remove
                if (hasECID(type)) {
                    ecidTypes.add(type);                   
                }
            }
        }
       
        IView ecidView = recording.createView();
        ecidView.setEventTypes(ecidTypes);
        Map<String, List<IEvent>> eventMap = buildEventMap(ecidView);
       
        System.out.println("ECIDS:");
        for (Entry<String, List<IEvent>> ecid : eventMap.entrySet()) {
            System.out.println(String.format("%s [%d events]", ecid.getKey(), ecid.getValue().size()));
        }
    }

    private static Map<String, List<IEvent>> buildEventMap(IView ecidView) {
        Map<String, List<IEvent>> map = new HashMap<>();
        for (IEvent event : ecidView) {
            String ecid = getEcid(event);
            if (ecid != null) {
                addToMap(map, ecid, event);
            }
        }
        return map;
    }

    private static void addToMap(Map<String, List<IEvent>> map, String ecid, IEvent event) {
        List<IEvent> eventList = map.get(ecid);
        if (eventList == null) {
            eventList = new ArrayList<>();
            map.put(ecid, eventList);
        }
        eventList.add(event);
    }

    private static String getEcid(IEvent event) {
        IField ecidField = getEcidField(event.getEventType());
        if (ecidField == null) {
            return null;
        }
        return String.valueOf(ecidField.getValue(event));
    }

    private static boolean hasECID(IEventType type) {
        return getEcidField(type) != null;
    }

    private static IField getEcidField(IEventType type) {
        for (IField field : type.getFields()) {
            if (field.isRelational() && KEY_ECID.equals(field.getRelationalKey())) {
                return field;
            }
        }
        return null;
    }
}

The output should look something like this:

8ec006a7-30e9-4fac-be7f-716f42d3cbc8-00000ed0 [1 events]
8ec006a7-30e9-4fac-be7f-716f42d3cbc8-00000237 [45 events]
8ec006a7-30e9-4fac-be7f-716f42d3cbc8-00000236 [2 events]
8ec006a7-30e9-4fac-be7f-716f42d3cbc8-00000d58 [2 events]
8ec006a7-30e9-4fac-be7f-716f42d3cbc8-0000022a [2 events]
8ec006a7-30e9-4fac-be7f-716f42d3cbc8-0000022d [2 events]
8ec006a7-30e9-4fac-be7f-716f42d3cbc8-0000022c [12 events]

Running the same example but using the relational key for URI (http://www.oracle.com/wls/Servlet/servletName) will look something like this:

/console/index.jsp [10 events]
/_async/AsyncResponseService [195 events]
/medrec/registerPatient.action [90 events]
/physician-web/physician/addPrescription.action [448 events]
/console/images/button_bg_mo.png [7 events]
/physician-web/physician/viewPatients.action [432 events]
/medrec/admin/home.action [48 events]
/console/jsp/changemgmt/ChangeManager.jsp [9 events]
/physician-web/physician/createRecord.action [720 events]
/medrec-jaxws-services/PatientFacadeService [275 events]
/medrec-jaxrpc-services/JaxRpcRecordCreationFacadeBroker [235 events]

Conclusion

This blog provided some examples on how to read out metadata from a flight recording and a simple example showing the use of the relational key.

This blog post explains the relational key – a little piece of event attribute metadata that is used to associate attributes in different event types with each other. This is a quick break in my series of posts about unsupported functionality, the reason for the break being that you will need to be armed with this knowledge for my next instalment in that series.

The Relational Key

The relational key is a little piece of event type attribute metadata on the form of an URI. It is normally used as a marker to hint to JMC that it often makes sense to find all events (of different event types) with the same attribute value for attributes with the same relational key. For example, the Java Flight Recorder has quite a lot of different GC events. Almost all of them are related to a certain GC ID. It often makes sense to find all events related to the same GC ID. The picture below lists some common GC events, and their attribute metadata.

The relational key can be seen in action in how Mission Control presents the alternatives for the Operative Set. If you go to the GC view, pick a garbage collection and open the context menu, you will find that there is an option to add all the events with the same GC ID to the operative set.

Moving over to the events tab, and selecting to only show the operative set, a range of different events, all associated with that particular garbage collection, can be seen.

Another very useful example is the ECID attribute in the events produced by the WebLogic Diagnostics Framework (see the “Putting it All Together” section of my post about the Operative Set).

Conclusion

The relational key is a handy little piece of attribute metadata that is used to give hints to the JMC user interface that different event types are related to each other. It is used by Mission Control to, for example, control what attributes are listed in the Operative Set context menu.

This blog post is the second in a series of posts on using unsupported functionality in the Oracle JDK and/or Java Mission Control.

Just as there is no supported way to add custom events to the Java Flight Recorder (yet), there is even less in terms of programmatic support to read recordings. There are however two different unsupported parsers available. Both are POJO APIs, and except for supportability / future API breakage issues, there is in practice nothing stopping you from using either one to read your recordings.

Disclaimer:

The following blog entry will describe UNSUPPORTED functionality. This means that relying on the described APIs or functionality may BREAK your code/plugin with any given update of the JDK and/or Mission Control.

I strongly recommend against using these APIs in production code. There will be supported APIs for reading reading recordings eventually and they may be quite different. Also note that older versions of the parsers are not guaranteed to be forward compatible with newer versions of the binary format. In short, this is NOT supported and WILL CHANGE!

That said, it can sometimes be quite useful to have programmatic access to flight recorder data. One could, for example, do automatic filtering and/or reporting on recordings, and then use JMC to do detailed analysis on the ones that are flagged as interesting.

The Reference Parser

The reference parser shipped with the Oracle JDK can be likened to a SAX XML parser. It allows you to iterate through the recording, uses little memory, is somewhat limited, and provides a very simple Java model. If your needs are simple, this may be the perfect parser to use. It is available by default in JDK 7u4 and later. The parser API is located in the JDK_HOME/jre/lib/jfr.jar file.

Here is an example snippet that iterates through a recording and prints out the total event count:

I will not spend too much time on this parser.

Strengths:

• Uses little memory
• Included in the Hotspot JDK (>=7u4)

Weaknesses:

• Slow
• Simple model/API

The JMC Parser

If the reference parser is the SAX XML parser equivalent, then the JMC parser is the DOM equivalent. The jar for the parser is easily nicked from the JMC distribution, either from the update site or from the JDK installation. In the JDK, you’ll find the jar containing the JMC flight recorder parser at JDK_HOME/lib/missioncontrol/plugins/com.jrockit.mc.flightrecorder_<version>.jar.

You will also need to add the JDK_HOME/lib/missioncontrol/plugins/com.jrockit.mc.common_<version>.jar to your path.

Here is the equivalent event counting example using the JMC parser:

Strengths:

• Fast
• The jar can be used with any JDK (>=6)
• Rich API
• Good for complex analysis scenarios

Weaknesses:

• Builds a Java model of the recording – uses more memory

Summary

This blog post lists two different UNSUPPORTED parsers that can be used to read java flight recordings programmatically from Java. A very simple example showing how to iterate through the events in a recording was shown for both of the parsers.

This blog post is the first in a series of posts on using unsupported functionality in the Oracle JDK and/or Java Mission Control.

Since the 7u40 version of the Oracle JDK, the Java Flight Recorder provides a wealth of information about the operating system, the JVM and the Java application running in the JVM. To make the wealth of information easier to digest, it can often be quite useful to provide some contextual information to the events available from the Flight Recorder. For instance, WebLogic Diagnostics Framework integrates with the Flight Recorder to add information about requests, and there are trial projects that add information about JUnit tests run, JavaFX pulse logger information and more. Having that contextual information makes it much easier to get a good starting point for where you want to study the recording in more detail, for instance homing in on the longest servlet requests taking the longest time to complete.

This blog will show how to use the Flight Recorder Java API to record your own data into the Java Flight Recorder.

Disclaimer:

The following blog entry will describe UNSUPPORTED functionality. This means that relying on the described APIs or functionality may BREAK your code/plugin with any given update of the JDK and/or Mission Control.

I strongly recommend against using these APIs in production code. There will be supported APIs for recording custom events into the Java Flight Recorder eventually and they will likely be quite different. Also note that code using these APIs will require an Oracle JDK >= 7u4, both at compile and run-time. And finally, note that the entire API for recording your own events entered the Oracle JDK deprecated. This is NOT supported, and WILL CHANGE!

That said, I still think it is valuable to share this information since:

• Sometimes having this ability can mean the difference between solving a problem in a reasonable time and not.
• This is an API for recording information. If proper care is taken, you can insulate yourself from the effects of not having the API available at runtime. This will never be the kind of API that is critical to the execution of your application. That said, the API and having access to the flight recorder infrastructure can be pretty addictive if you’re interested in production time profiling and diagnostics.
• There are plenty of opportunities where you can use this API to a great advantage without even touching your application code. For example, having an event type for unit tests or maybe coding your own BCI agent to insert the code required to emit events on the fly.
• Getting some early feedback on these APIs would be quite helpful.

Different Kind of Events

Before we start recording events, we must discuss the different kind of event types available. There are currently four different ones:

1. Instant events
   These are events with only one time – the time the event occurred.
2. Requestable events
   These are events with a user configurable period.
3. Duration events
   These are events that have a start and end time (a duration).
4. Timed events
   These are duration events with a user configurable threshold. If an event has a duration shorter than the defined threshold, it will not be recorded.

To record events we need to define two things:

1. A producer
   The producer is basically some metadata and a namespace for your events. In the Java API, you will register your event types with the producer. Once the producer is no longer referenced, your associated event types will be cleared. Do NOT lose the reference to your producer.
2. One or more event types
   In event types you define the metadata used to describe the events as well as the attributes (data) available.

Defining the Producer

To define a producer, simply instantiate a Producer instance:

Producer myProducer = new Producer("Demo Producer", "A demo event producer.", http://www.example.com/demo/);

The producer is simply defined using a name – the name is what will be shown to the user in the JMC GUI, a description and an identifier. The identifier is on an URI format.

Next we must register the producer:

myProducer.register();

After making sure to keep a reference to the producer, we can start defining the event types.

Defining an Event Type

Event types are defined using a Java class, with annotations used to declare metadata. Step one is to subclass the class appropriate for the kind of event you want, for example a timed event. Next you need to decide what attributes (data) should be available in the events. The following example declares an event type named My Event, with a single attribute named Message:

@EventDefinition(path = "demo/myevent", name = "My Event", description = "An event triggered by doStuff.", stacktrace = true, thread = true)
private class MyEvent extends TimedEvent {
    @ValueDefinition(name = "Message", description = "The logged important stuff.")
    private String text;

Registering the Event Type

Next you have to register the event type with the producer. The registration method returns something called an event token. Keeping a reference to the event token and using it as an argument to the construction of an event instance is much more efficient than using the default event constructor. Here is how you may go about registering your event:

EventToken myToken = myProducer.addEvent(MyEvent.class);

Emitting Events

Now we’re ready to start emitting our event. This is very easily done:

1. Construct an instance of the event (or reuse a previous instance, as discussed later).
2. Call the start method to take the start timestamp.
3. Set the attributes you wish to set.
4. Call the end method to take the end timestamp.
5. Set some more attributes if necessary.
6. Call the commit method to commit the event to the recording engine.

So this is how it would look in code:

public void doStuff() {
    MyEvent event = new MyEvent(myToken);
    event.begin();
    String importantResultInStuff = "";
    // Generate the string, then set it...
    event.setText(importantResultInStuff);
    event.end();
    event.commit();
}


Now, you may come to the conclusion that this will cause an object allocation for each event. If you would rather not allocate a new object for each event, you can reuse one of them, like this:

private MyEvent event = new MyEvent(myToken);
public void doStuffReuse() {
    event.reset();
    event.begin();
    String importantResultInStuff = "";
    // Generate the string, then set it...
    event.setText(importantResultInStuff);
    event.end();
    event.commit();
}

(The only difference is that you need to remember to reset the event between each invocation.)

Conclusion

There is a Java API for adding events to the Java Flight Recorder available in the Oracle JDK since 7u4. It is NOT SUPPORTED, and it will change. That said, it can be quite useful for providing contextual information to the rest of the data provided.

I am very interested in feedback on this API. Such feedback will be fed back into a future, officially supported, API.

The Operative Set is an easily overlooked but quite powerful feature in the Mission Control Flight Recorder user interface. It is a selection of events that you can reach and utilize from almost every tab in the Flight Recorder user interface.

This blog will explain how the Operative Set can be used to quickly drill down into a set of events having some very specific properties. The Operative Set may take a bit of time and experimenting to get used to. If you want to try out this feature whilst reading this blog, I’ve made the actual recording used in the examples available here.

Note: If JMC warns you the recording is too large, see this blog post.

Altering the Operative Set

Before we start using the Operative Set, I’ll just note that the tab groups in JMC really are of two different types:

1. Tab groups specific to one or more specific event types.
   These tab groups are pre-configured to show off specific events. The Event Types view will not affect what is shown in the tab groups.
2. The general Events tab group.
   The tabs in this tab group will show events of the event types selected in the Event Types view.

The event type specific tabs are usually used to add and remove events of interest to/from the operative set, and the general events can be used to both study and modify the set of events in the operative set. The general Events tabs are also the only way to analyse events of types for which there is no specific user interface. Of course, it is easy to build a new user interface for events where there is no specific user interface yet, but that is for another blog post.

To alter the Operative Set, simply select the events that you want to add/remove and use the context menu. For example, to add the the events for the most contended lock, simply go to the contention tab, right click on the row representing what you’re interested in, go to the Operative Set menu, then choose Add Selection.

Voila – the events have now been added to the Operative Set. You can now move over to the general Events tabs to play with them.

Using the Tabs in the Events Tab Group

The first thing to check when moving over to the Events tabs, is to make sure that the events of interest aren’t filtered out. This is done by checking the settings of the Event Type view, which is normally to the left of the Flight Recorder editor.

In our case, we have just added Java Monitor Blocked event to the Operative Set, so there isn’t anything else we need to do. When in doubt, enable all event types.

So let’s see what the Events tabs can be used for.

Overview

This tab is a good place to sanity check what you actually have in your operative set. Click the checkboxes for the Operative Set, and you will see what producers the events originate from, as well as the distribution per event type. This page will be quite boring with our current selection of events.

Log

The Log is simply a table of the available events. When you select one of the events, all information (attribute values) in the event will be shown to you in the lower half of the tab. Again you can opt to only see what is in the Operative Set. Note that in the 5.2.0 version of Mission Control, this is one of the places where you’ll get to see the actual line number in the stack traces. This means that if you’re not happy with the line number-less traces in the aggregated stack traces, you can always select to add the events you are interested to the operative set, and then look at the events in the log.

Graph

The graph view will show you the events in a per thread graph. This is very useful to get a visualization of what is happening in terms of distribution over both threads and time. In our simple example, we can make out what threads are involved in blocking on the log4j logger, as well as the time periods for which we have contention.

Threads

The threads simply list the events aggregated per thread. This can be useful for setting the operative set to only the events occurring in a certain thread.

Stack Trace

In this tab the aggregated stack traces for the selected events can be viewed:

It can be noted that for the events in our Operative Set (the blocking events), we seem to spending the most time waiting to do tracing rather than debug or info level logging.

Histogram

This tab is not for creating graphical histograms, but rather for grouping events on a specified attribute. In our case we can for instance select to group our events on monitor address, then select to set the operative set to the busiest lock:

Then we can, for example, move back to check what threads are involved in locking on the particular lock instance.

The Relational Key

Now, there is something called a relational key in the event metadata for an attribute. The relational key is used to tell that an attribute contains a value that can bind together events of different event types. For instance all the different GC events that are associated with a certain GC ID are connected through a relational key:

The Operative Set context menu has a submenu called Add Related Events. That menu will typically try to be helpful by listing attributes that have a relational key. So for instance, to find all GC events that are related to the longest garbage collection taking place in the recording, go to the Memory tab group, select the Garbage Collections tab and sort the GC table on time. Next select the longest recording and choose Operative Set | Add Related Events | With GC ID = (the id of the longest GC in the supplied recording has ID 76).

Looking at the log, you can, for example, look at all the related GC events in the log tab:

Putting it All Together

So let’s look at what is happening during an invocation of the viewPatients servlet at the different levels of the recording. First go to the servlet tab in the WebLogic tab group, next right click in the table and select Operative Set | Clear. Most WLS events are related to a certain request using something called the ECID (Enterprise Context ID). Let’s next add the events associated with the first found ECID associated with an invocation of viewPatients:

Now we can check what was occurring on the WLS level through the request by simply looking at the events in the log. First enable all event types in the Event Type view. Next move over to the Log tab:

Lastly, let’s check what was happening with the other available events occurring in the same thread during the same time as these high level WLS events. First select all the events in the log (in the Operative Set) (ctrl-A). Next right click and select add concurrent:

You will now see related events such as allocations and exceptions popping up:

The other tabs in the Events tab group can of course be used with the new operative set as usual. For example, to get a graphic overview of all the events involved in the request, go to the Graph tab:

Summary

The Operative Set is a feature that allows power users of Mission Control to look at events from different angles and quickly home in on related events having certain properties. It is a power user feature – it may take a little bit of getting used to, but once you’re used to it you’ll be scary effective.

I just realized that I could not open my favourite example recordings using the standard settings included in JMC. It seems sometimes the heuristics for the default max heap size on the 32-bit HotSpot JVM results in a relatively modest value. So, if you are seeing this a lot…

…the next step would be to try to get JMC a bit more memory. The easiest way to do this is to simply edit the jmc.ini file, which is located next to the jmc launcher in the bin folder of the JDK. It will typically look something like this:

-startup
../lib/missioncontrol/plugins/org.eclipse.equinox.launcher_1.3.0.v20120522-1813.jar
--launcher.library
../lib/missioncontrol/plugins/org.eclipse.equinox.launcher.win32.win32.x86_1.1.200.v20120913-144807
-vm
./javaw.exe
-vmargs
-XX:+UseG1GC
-XX:+UnlockCommercialFeatures
-XX:+FlightRecorder
-Djava.net.preferIPv4Stack=true


What you would typically want next is to set the max heap and initial heap to the same, and to as much as you can possibly afford. On my tiny laptop, maybe something like a gig or so:

-startup
../lib/missioncontrol/plugins/org.eclipse.equinox.launcher_1.3.0.v20120522-1813.jar
--launcher.library
../lib/missioncontrol/plugins/org.eclipse.equinox.launcher.win32.win32.x86_1.1.200.v20120913-144807
-vm
./javaw.exe
-vmargs
-XX:+UseG1GC
-Xmx1024m
-Xms1024m
-XX:+UnlockCommercialFeatures
-XX:+FlightRecorder
-Djava.net.preferIPv4Stack=true

Since we don’t have any nice NMT (Native Memory Tracking) MBean in HotSpot (yet), and therefore not in the JMC console, I thought I’d show how it can be done using command line arguments and JCMD. Please note that you’ll get a 5-10% performance hit if you enable this.

Step 1 – Enabling NMT

This is done by using the following command line:

-XX:NativeMemoryTracking=[off|summary|detail]

Where the different options are:

Again, note that enabling this will cause you a 5-10% overhead.

Step 2 – Using JCMD to Access the Data

To use JCMD to dump the data collected thus far, and to optionally compare it to the last baseline, use the following command:

jcmd <pid> VM.native_memory [summary | detail | baseline | summary.diff | detail.diff | shutdown] [scale= KB | MB | GB]

Here is a table with the different memory categories (this may change):

Additional Options

Use the following NMT options to print a report on VM exit:

-XX:+PrintNMTStatistics -XX:+UnlockDiagnosticVMOptions

NMT will shut itself down when it detects low resource conditions, such as when running low on native memory. Sometimes that may be the situation when you would really want the data! This is the flag to disable auto shutdown:

-XX:-AutoShutdownNMT

Note that detail level information is not supported on ARM.

Final Thoughts

The information in this blog post mostly comes from a wiki document authored by Zhengyu Gu, who also did the HotSpot implementation. Also, shout out to Fredrik Öhrström who came up with the NMT idea, and who did the original JRockit implementation of it!

I just got a question on how to get the Java Discovery protocol up and running, so I though I’d better do a quick blog on it.

First of all you need a 7u40 or later of the HotSpot JDK. Next you need to know how to start the external management agent, since the JDP server follows the life cycle of the external JMX management agent. Since it is getting late, I hereby promise to do a really long blog on how to properly set it up in a secure manner later, and simply show how to ABSOLUTELY NOT do it for now. The following example will open up an agent with absolutely no security and wide access into your JVM with both the RMI Registry and RMI Server at port 7091:

-Dcom.sun.management.jmxremote.port=7091 -Dcom.sun.management.jmxremote.rmi.port=7091 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

Now, the next step is to enable JDP. This is simply done by adding the flag:

-Dcom.sun.management.jmxremote.autodiscovery=true

Finally, a nice structured display name can be provided to JDP by setting the name parameter like this:

-Dcom.sun.management.jdp.name=MyCluster/MyJVM

You should see something similar to this:

Note that there are other parameters that can be used with JDP, such as changing the multicast group, or changing the TTL for wider reach. Please see the command line documentation for further information.

Using JCMD

JDP can also be started when using JCMD to fire up the external management agent. Here is an example:

jcmd 5596 ManagementAgent.start jmxremote.ssl=false jmxremote.port=7091 jmxremote.rmi.port=7091 jmxremote.authenticate=false jmxremote.autodiscovery=true

Where 5596 is the PID of the process for which you want to start up the management agent. Again, do not start up your management agents with all the security turned off. This is just an example.

Unfortunately the name parameter is not available when starting JDP through JCMD. This is a known bug, and it will be fixed in a later version of the JVM.

With the Hotspot JDK 7u40 there is a nifty new tool called Java Mission Control. Users of the nifty old tool JRockit Mission Control will recognize a lot of the features. This particular blog entry focuses on how to do allocation profiling with the Java Flight Recorder.

The JVM is a wonderful little piece of virtualization technology that gives you the illusion of an infinite heap. However that comes at a cost. Sometimes the JVM will need to find out exactly what on your heap is in use, and throw the rest away – a so called garbage collection. This is because the physical memory available to the JVM indeed is limited, and memory will have to be reclaimed and reused when it is no longer needed. In timing sensitive applications, such as trading systems and telco applications, such pauses can be quite costly. There are various tuning that can be done for the GC to make it less likely to occur. But I digress. One way to garbage collect less, is of course to allocate less.

Sometimes you may want to find out where the allocation pressure is caused by your application. There are various reasons as to why you may have reached this conclusion. The most common one is probably that the JVM is having to garbage collect more often and/or longer than you think is reasonable.

The JFR Allocation Event

In the Java Flight Recorder implementation in HotSpot 7u40 there are two allocation events that can assist in finding out where the allocations are taking place in your application: the Allocation in new TLAB and the Allocation outside TLAB events.

Just like for most events provided with the HotSpot JDK, there is of course a custom user interface for analyzing this in Mission Control, but before going there I thought we’d take a moment to discuss the actual events.

First you need to make sure they are recorded. If you bring up the default (continuous) template, you will notice that allocation profiling is off by default. Either turn it on, or use the profiling template. The reason it is off by default is that it may produce quite a lot of events, and it is a bit hard to estimate the overhead since it will vary a lot with the particular allocation behaviour of your application.

The Log tab in the Events tab group is an excellent place to look at individual events. Let’s check out what these actually contain:

They contain the allocation size of whatever was allocated, the stack trace for what caused the allocation, the class of what was allocated, and time information. In the case of the inside TLAB allocation events, they also contain the size of the TLAB.

Note that we, in the case of the (inside) TLAB allocation events, are not emitting an event for each and every location – that would be way too expensive. We are instead creating an event for the first allocation in a new TLAB. This means that we get a sampling of sorts of the thread local allocations taking place.

Also, note that in JRockit we used to have TLA events that perfectly corresponded to the allocations done in the nursery, and large object allocation events that corresponded to allocation done directly in old space. This is a little bit more complicated in HotSpot. The outside TLAB allocation events can both correspond to allocations done due to an allocation causing a young collection, and the subsequent allocation of the object in the Eden as well as the direct allocation of a humongous object directly in old space. In other words, don’t worry too much if you see the allocation of a few small objects among the outside of TLAB events. Normally these would be very few though, so in practice the distinction is probably not that important.

Using the Allocations Tab

The allocations tab actually consists of three sub tabs:

1. General
2. Allocation in new TLAB
3. Allocation outside TLAB

The General tab provides an overview and some statistics of the available events, such as total memory allocated for the two different event types. Depending on what your overarching goal is (e.g. tuning TLAB size, reducing overall allocation) you may use this information differently. In the example below it would seem prudent to focus on the inside TLAB allocations, as they contribute way more to the total allocation pressure.

The Allocation in new TLAB tab provides three more types of visualization specific to the in new TLAB events: Allocation by Class, Allocation by Thread and Allocation Profile. The Allocation Profile is simply the aggregated stack trace tree for all of the events. In the example below it can be seen that Integer object allocations correspond to almost all the allocations in the system. Selecting the Integer class in the histogram table shows the aggregated stack traces for all the Integer allocations, and in this example all of them originate from the same place, as seen below (click in pictures to show them in full size).

The Allocation outside TLAB tab works exactly the same way as the Allocation in new TLAB tab, only this time it is, of course, for the Allocation outside TLAB events:

Limitations

Since the Allocation in new TLAB events only represents a sample of the total thread local allocations, having only one event will say very little about the actual distribution. The more events, the more accurate the picture. Also, if the allocation behaviour vary wildly during the recording, it may be hard to establish a representative picture of the thread local allocations.

Thanks to Nikita Salnikov for requesting this blog entry!

Further reading and useful links

Related Blogs:
Java Mission Control Finally Released
Creating Flight Recordings
My JavaOne 2013 Sessions
Low Overhead Method Profiling with Mission Control

The Mission Control home page:
http://oracle.com/missioncontrol

Mission Control Base update site for Eclipse:
http://download.oracle.com/technology/products/missioncontrol/updatesites/base/5.2.0/eclipse/

Mission Control Experimental update site (Mission Control plug-ins):
http://download.oracle.com/technology/products/missioncontrol/updatesites/experimental/5.2.0/eclipse/

The Mission Control Facebook Community Page (not kidding):
http://www.facebook.com/pages/Java-Mission-Control/275169442493206

Mission Control on Twitter:
@javamissionctrl

Me on Twitter:
@hirt