Using the (Very Unsupported) Java Flight Recorder API

Since version 7u4 of the Hotspot JDK, there is a Java API for adding flight recording events to the Java Flight Recorder (JFR). Using this API to add your own event producers is quite unsupported, but very useful. It can, for instance, be used to add application specific events, such as a transaction event, to provide a context for all the other events being recorded by the flight recorder.

Now, in the interest of full disclosure, the API entered the Hotspot code base as @deprecated, since:

  1. It is in the com.oracle.jrockit namespace.
  2. The API will eventually be superceded by an improved “2.0” version.

So, the API is deprecated and unsupported. It is subject to change without notice. Don’t come complaining if it changes radically between releases! That being said, since the API is in use by WLS and others, it will likely remain available in its current form for the foreseeable future.

This article will show how user defined events can be put into the Flight Recorder. More specifically, how to:

  1. Add a new event producer,
  2. add a new event type, and,
  3. record events of the newly created event type.

In later articles I will show different ways to dump data from the flight recorder, different ways to get programmatic access to the events, as well as how to use the Mission Control client to analyse your events and how to build custom user interfaces in Mission Control for them.

Step 1 – Adding a Producer

To record events you need to add metadata about your event producer. As with all metadata, it is important to spend some time to ensure that the metadata is correct and useful. In my example we will create an event to track the trading of Smurfberries among the Smurfs on the fictional Smurfberry Exchange market. (Yes, I do know that Smurfs are fictional too, but please don’t tell my daughter.)

Here is a picture of papa smurf carrying smurfberries, for reference:
smx64

To create your producer you need to do something along the following:

private static final String PRODUCER_URI = "http://www.smx.com/smx/";
private static final Producer PRODUCER;
 
static {
    PRODUCER = createProducer();
}
 
private static Producer createProducer() {
    try {
        return new Producer("Smurfberry Exchange producer",
                "A demo event producer for the fictional Smurfberry Exchange.", PRODUCER_URI);
    } catch (Exception e) {
        // Add proper exception handling.
        e.printStackTrace();
    }
    return null;
}

Step 2 – Adding Event Type(s)

Next we declare the event types to be produced by our producer. In our case we want an event to be generated for each Smurfberry transaction.
We need to:

  1. Create our new event class.
  2. Register our new event type with the producer.

There are several different types of events that we can create.

  • Duration events
    These are events that last over a period of time, or, in other words, events that have a start and end time. The Garbage Collection event is an example of a duration event.
  • Timed events
    These are events for which a threshold can be configured in the recording engine. If the event duration is less than the threshold, the event will not be recorded. Java Sleep and Java Wait events are examples of timed events.
  • Instant events
    These events only have the time when the event took place. The Exception and Event Settings Changed are examples of instant events.
  • Requestable events
    These events can be configured to be polled periodically by the recording engine. The event implements a callback. An example of a requestable event is the CPU Load Sample event.

Conceptually Duration and Timed events are very similar, and from now on I will interchangebly refer to them collectively as duration events. Duration events are among the most useful ones, since they allow us to time things going on in our application. Since we want to time the transactions, we will use a duration event in the example. And since we want the end user creating a recording of our system to be able to decide how slow transactions to record, we will be using a TimedEvent.

Creating the event is easily done by subclassing the TimedEvent class, and annotating the class with the metadata for the event:

import com.oracle.jrockit.jfr.ContentType;
import com.oracle.jrockit.jfr.EventDefinition;
import com.oracle.jrockit.jfr.EventToken;
import com.oracle.jrockit.jfr.TimedEvent;
import com.oracle.jrockit.jfr.ValueDefinition;
 
@EventDefinition(path="smx/transaction", name = "SMX Transaction", description="An Smurfberry Exchange transaction.", stacktrace=true, thread=true)
public class TransactionEvent extends TimedEvent {
    @ValueDefinition(name="Price", description="The price at which the transaction was made.", contentType=ContentType.None)
    private float price;
 
    @ValueDefinition(name="Quantity", description="The quantity of smurfberries transferred.", contentType=ContentType.None)
    private int quantity;
 
    public TransactionEvent(EventToken eventToken) {
        super(eventToken);
    }
 
    public float getPrice() {
        return price;
    }
     
    public void setPrice(float price) {
        this.price = price;
    }
 
    public int getQuantity() {
        return quantity;
    }
 
    public void setQuantity(int quantity) {
        this.quantity = quantity;
    }
}

A few notes on the code above:

  • The path is where the event will be “mounted” in the overall EventType tree. Choose carefully.
  • The metadata will be used by all users to understand your event. Take care to explain your event properly.
  • The metadata about stacktrace and thread lets flightrecorder know that we want to capture the stacktrace from where the event is originating and thread information.
  • In this case, no appropriate content types are available, but if you are exposing things like time in nanos, bytes, Class names or similar, use the correct content type, and you will get a lot for free in the user interfaces for analysing your events.
  • If you have a field that will be shared among several events, and where the value can be used to join events from the different types, the field should also have a relational key. A relation key is a unique URI signifying a relationship between different events based on the values of specific fields, and work much like keys in a relational database. While these relations have no significance in the running application, they tell a renderer of flight recorder data that certain events should be associated and grouped together, typically to form an event chain. Here is an example:
    @ValueDefinition(name="transactionId", description="The transaction id for a fictive transaction.",
    	contentType=ContentType.None, relationKey="http://example.com/myserver/transactionId")
    private String transactionId;

Next we need to register the event type, and our producer, propely. Here is the extended code from Example Snippet 1:

private final static String PRODUCER_URI = "http://www.smx.com/smx/";
    private static final Producer PRODUCER;
    private static final EventToken TRANSACTION_EVENT_TOKEN;
     
    static {
        PRODUCER = createProducer();
        TRANSACTION_EVENT_TOKEN = createToken();
        PRODUCER.register();
    }
 
    /**
     * Creates our producer.
     */
    private static Producer createProducer() {
        try {
            return new Producer("Smurfberry Exchange producer",
                    "A demo event producer for the fictional Smurfberry Exchange.", PRODUCER_URI);
        } catch (Exception e) {
            // Add proper exception handling.
            e.printStackTrace();
        }
        return null;
    }
 
    /**
     * Creates our event token.
     */
    private static EventToken createToken() {
        try {
            return PRODUCER.addEvent(TransactionEvent.class);
        } catch (Exception e) {
            // Add proper exception handling.
            e.printStackTrace();
        }
        return null;
    }

Step 3 – Recording Events

Now we can start recording the smurfy transactions! It is quite simple – for a duration event, create the event, set the properties, end the event and then commit it to the flight recorder:

public void doTransaction(...) {
        TransactionEvent event = new TransactionEvent(SmurfBerryExchange.TRANSACTION_EVENT_TOKEN);
        event.begin();
        doWork();
        float price = getPrice();
        int quantity = getQuantity();
        event.setPrice(price);
        event.setQuantity(quantity);
        event.end();
        event.commit();
    }

That’s all there is to it. If you know you’re thread safe, you can dispens with the event object creation and simply reuse the event like this:

private TransactionEvent event = new TransactionEvent(SmurfBerryExchange.TRANSACTION_EVENT_TOKEN);
    public void doTransaction(...) {
        event.reset();
        event.begin();
        doWork();
        float price = getPrice();
        int quantity = getQuantity();
        event.setPrice(price);
        event.setQuantity(quantity);
        event.end();
        event.commit();
    }

Troubleshooting

If you can’t get it to work, you may want to check the following:

  1. In JRockit the flight recorder was always enabled. Not so in Hotspot. Don’t forget to enable the flight recorder: -XX:+UnlockCommercialFeatures -XX:+FlightRecorder
  2. You need a 7u4 or later to use this with a Hotspot JDK.
  3. Did you forget to keep a reference to the Producer?
  4. Once you start recording your new events, you need to enable them. Easiest way is to setup a template in Mission Control (5.0 and later) and use that template for your recordings. If you are using pre-7u12 JVMs the server and client side templates are different, and you will need another type of template for jcmd to work. In 7u12 (JMC 5.2) and later they are the same.

For more information on using the Flight Recorder, see the following book.

Using the Mission Control DTrace Plug-in

There is a downloadable plug-in for Mission Control which adds DTrace support to Mission Control. It can be used to record running Java processes with highly detailed information from the underlying platform. It can also be highly customized and configured to record your own probes. It does so with a twist; for the recorded data to be the most useful, it needs to be in a self describing format that can easily be consumed by a client. Therefore Mission Control adds a few capabilities to DScript and, if installed into Eclipse from the eclipse update site, adds editing functionality of dscript files with the new syntax.

Together with the capability to quickly build your own user interface for the new probes, from within Mission Control itself, the DTrace plug-in provides quite powerful functionality for the platforms where DTrace exist.

Quick Overview

The DTrace plug-in lets you go from editing your self describing DTrace events…

…via DTrace execution on the server… …to the DTrace data visualized in Mission Control!
de_editor_small2 d_script_small2 cpu_tab_small2

Installing Mission Control & the DTrace Plug-in

If you are an Oracle employee, you should be using the latest builds of Java Mission Control 5.2. Contact me if you need information on how to get builds. If you are not, there is an early version of the DTrace plug-in available in JRockit Mission Control 4.1.

No matter which version you use, you have the choice to either use the stand alone version of Mission Control, or the Eclipse version. For DTrace they both have their own distinct advantages: the stand alone version provides easy access to the built in GUI editor, and the Eclipse version has nice editor support for the special DScript Event syntax.

For the stand alone version:

  1. Download JRockit: http://www.oracle.com/technetwork/middleware/jrockit/downloads/index.html
    Note that the client were not built for Solaris in the good old JRockit days, so be prepared to install the client on one of the other platforms. You can also try installing the plug-ins into an Eclipse. If you are an Oracle Employee, and you’re using Java Mission Control 5.2, there are builds for most platforms.
  2. Install the DTrace plug-in by selecting Help->Install Plug-ins, and then selecting the plugins under DTrace Recorder.

For the Eclipse version:

  1. Install Mission Control from the following update site: http://download.oracle.com/technology/products/missioncontrol/updatesites/base/4.1.0/eclipse/
  2. Install the DTrace Recorder plug-ins from the following update site: http://download.oracle.com/technology/products/missioncontrol/updatesites/experimental/4.1.0/eclipse/

Using the Plug-in As Is

After installing Mission Control and the DTrace plug-in, open the Help in Mission Control, or in Eclipse. Navigate to the Using the DTrace Recorder chapter. It will tell you all that is needed for basic use of the DTrace Recorder plug-in. Really, the documentation is quite good, and I would recommend taking a break here and go read it before proceeding. Ler

Adding Your Own Probes

This is most easily done if you have installed Mission Control and the DTrace plug-in into Eclipse. After installing and running a DTrace recording, a sample .de script file will be available to you, as explained in the documentation. A .de file is a dscript file embellished with our own syntax for creating self describing events. A .de file, together with the user input provided in the recording wizard, will be compiled into an ordinary dscript file.

The anatomy of the new .de script files are:

  • The Parameters
  • The Templates
  • Event Declarations
  • Probes

Parameters

The parameters are user defined parameters that will be presented to the user when a recording is started. Some are treated special by Mission Control and will be filled out by the framework. In the parameters section the meta data for the parameters are specified. Check out how the objectAllocationCutoff parameter is used in the Template for actual parameter use. Here is an example of a parameter metadata declaration:

@name = "Allocation Block Size" 
@type = "memory"
parameter objectAllocationCutoff;

The name annotation is the user friendly name that will be presented to a user of the parameter in the recording wizard. The type decides how input and rendering will be handled. The parameter keyword declares the parameter.

Templates

The template section declares the different templates the user can choose from. It declares which parameters will be available, and which probesets the user will be able to turn on and off. The following snippets show the standard profiling template provided with the plug-in:

@name ="Hotspot Profiling JDK6/JDK7";
template hotspotProfiling
{
    parameter processIdentifier (application = "java*");
    parameter objectAllocationCutoff(min = 1, value = 4096, max = 200000000);
 
    probeset Allocation (enabled = true);
    probeset ClassLoading (enabled = true);
    probeset GC (enabled = true);
    probeset IO (enabled = true);
    probeset JVMLocks (enabled = true);
    probeset JVMCompilation (enabled = true);
    probeset JVMOperations (enabled = true);
    probeset JavaLocks (enabled = true);
    probeset Paging (enabled = true);
    probeset Scheduling (enabled = true);
    probeset SystemCalls (enabled = true);
};

The template shows that the user will be able to choose processes with java in its application name. It also defines the range for the object allocation cutoff and a set of probesets that the user can select from, all of which be enabled by default.

A probe set is just what it sounds like – a set of probes that will be handled by the template as a group. A probeset is simply a listing of all the probes it includes:

@name ="Scheduling";
@description = "Records thread related information.";
probeset Scheduling
{
    probe cpu_on;
    probe cpu_off;
    probe cpu_exit;
    probe changed_priority;
    probe thread_start;
    probe thread_stop;
    probe thread_park_begin;
    probe thread_park_end;
    probe thread_unpark;
    probe thread_sleep_begin;
    probe thread_sleep_end;
    probe thread_yield;
    probe thread_migration;
};

Event Declarations

To be able to make sense of the events we emit, we need to set up some metadata to associate with them. The metadata includes the path to the event type, the user friendly name, the preferred color to render the events in, and the data fields used. This is an example on how to declare an event:

@name = "Running Thread";
@color = rgb(50, 255, 50);
@path = "Operating_System/Thread/Running";
event ThreadRunning
{
    @name ="Thread Name";
    string threadName;
 
    @name = "Thread";
    @description = "The thread the event occured in.";
    @type = "thread";
    int thread;
};

Note that, to help Mission Control make the most out of your data, it is quite useful to specify the content type with the @type annotation. There are often a lot of metadata that can be shared between different events. To help with that, there is a mixin keyword to specify metadata that can be shared. You can then use the extends keywork to import that data into your declaration, like this:

mixin Threadable
{
    @name = "Thread";
    @description = "The thread the event occured in.";
    @type = "thread";
    int thread;
};
...
@name = "Java Blocked";
@description = "Blocked on a Java monitor";
@color = rgb(255, 109, 100);
@path = "Java_Application/Monitor_Blocked";
event JavaMonitorBlocked extends Threadable, ClassLockable
{
};

Note, that when editing these files in Eclipse, you have editor support with full syntax highlighting. You can also use ctrl (command on Mac) clicks to jump around in the source, like this:

jump

Probes

The final section is the probes. These are quite like dscript probes, with the important difference that the emit keyword is used for emitting data. Here is an example:

probe read_io
syscall::*read:entry
/ pid == $processIdentifier /
{
    emit IORead(thread = tid, size = arg2);
}

It is also possible to pair events together to form duration events, using the begin and end keywords:

probe syscall_entry
syscall:::entry
/ pid == $processIdentifier /
{
    begin SystemCall(thread = tid, function = probefunc);
}
 
probe syscall
syscall:::
/ pid == $processIdentifier /
{
    end SystemCall(thread = tid, function = probefunc);
}

Note that the pairing is done by matching up the arguments.

Adding Your Own GUI

Mission Control has a built in GUI designer that allows you to redesign the GUI from within itself. Since the designer is somewhat tricky to use, it is not yet supported. However, if you can live with the quirks, it is quite useful. You can, for instance, easily build a custom user interface for your own events.

First you need to get into open the design view.

For JRockit Mission Control 4.1, this is most easily done from within the stand alone (RCP) version of the tool. This is how you enable it:

  1. Start JAVA_HOME/bin/jmc
  2. Press ctrl-1. The Commands view should open.
  3. In the Commands View, type enabledesignview, and press the play icon.

    commands
  4. The Designer View should open.

In late builds of Java Mission Control 5.2, this is simply done by selecting the Window->Show View->Designer menu. The designer view will control all the editing of your tabs, as well as control what tabgroups and tabs are available. It is also from the design view that you export your custom tabs as plug-ins. Now, the first quirk is:

You Must Prime Your Design View!

You prime it by loading a recording with the events that you want to build GUIs for. When you have loaded a recording, however, you cannot change the tabgroups or what tabs are available. So, if you want to add or remove tab groups or tabs, you normally directly close your newly opened recording. Yes. Quirk.

Designer view after opening recording:

designer1

Designer view after closing recording:

designer2

Now, to add a new tab group, you simply right click on the top node, and select New->New Group from the context menu. Add icons accordingly. If you want to change the relative position of your group, you can add a Placement Path. The Placement Path is a string that will be lexically compared with other strings to determine the position of the group. A placement path starting with ~ (tilde) is typically used to put a tab group towards the end.

To add a tab, simply right click on the tab group node, and select New->Tab.

To edit your tab, load the recording containing the events for which you want to build a GUI, then press the stop button in the upper right corner of the Designer View. This will stop the GUI for that tab, and allow it to be edited. Assign components as required. For example, to add an event histogram:

  1. Right click on the empty area, and select Assign->Table->Event Histogram:addhistogram
  2. Fill out an identifier (anything that helps you remember the component later)
  3. Go to the Columns tab and press the Browse button, then browse button again, and select your event:browse_event
  4. Next click Group By and hit the Browse button. Select the attribute in your event that you want to group by:group_by
  5. Next add the columns you want in the table, by pressing add and selecting the attributes. For histograms, having the group by attribute first usually makes sense. You can easily make an event count field, by selecting any attribute, and then selecting Aggregation Function and Content Type to Count:

    count
  6. Press play in the upper right corner to see your working user interface:

    donegui

Next you may want to store your new GUI as a plug-in, so that it can be shared with other users. This is easily done like this:

  1. Close your recording.
  2. Right click on the top node, and select Export UI to Plug-In.
  3. Next select the tab to export. Note that you usually just want to export your own additions:select_tab_to_export
  4. Next select plug-in properties, such as the name and version. Note that higher versions of your plug-in will override older versions:

    plug-in-properties
  5. Finally select where it should be saved. Once exported, you can place the resulting jar file in the plug-in folder of another Mission Control to enjoy your user interface there.

That’s it! Have fun, and please let me know if you find this to be useful functionality that you would like the Mission Control team to continue to improve upon!

Big thanks to Klara Ward (@klaraward), the DTrace plug-in maintainer, for the help on this blog-post!

Finally Got Fiber Installed!

Yay! I finally got a fiber installed. This is really good news, since it means that my uplink is gazillion times faster than it was before, which in turn means it is possible to read my somewhat graphics intense blog without throwing a rage fit. Yay again!

Here is a picture of the speeds I measured yesterday:
image

Notice how you can actually look at this blog without patiently waiting for a few minutes! Blinkar