JMC 6 Automated Analysis Headless

This post will be a little bit wider than the title implies; it will be about using the JMC core JFR APIs in general. That said, one of the things we will be using it for, towards the end, will be to run the JMC 6 automated analysis headless.

This article will also cheat by talking about JMC 6.1, so that anyone reading this article will have relevant information for JMC 6.1 as well. Winking smile 

The JAR-files Needed

To run these examples you will need to have the following Jar-files from the JDK_HOME/lib/missioncontrol/plugins folder on your classpath:

Prefix

Version needed

Comment

com.oracle.jmc.common

>= 6.0.0

Classes common to JMC, such as stacktrace definitions, the content type system (quantities/ units of measurement) etc.
com.oracle.jmc.flightrecorder

>= 6.0.0

The Java Flight Recorder parser, classes for extracting information from a recording.
com.oracle.jmc.flightrecorder.rules

>= 6.0.0

The core definitions and classes for automated analysis of recordings.
com.oracle.jmc.flightrecorder.rules.jdk

>= 6.0.0

Contains the rules for the JDK (such as rules for synchronization trouble, long lasting VM operations and much more).
com.oracle.jmc.flightrecorder.ext.wls.parser

>= 6.0.0

(Optional, only relevant for users wanting WebLogic Server extensions)
com.oracle.jmc.flightrecorder.ext.wls.rules

>= 6.0.0

(Optional, only relevant for users wanting WebLogic Server extensions)

Loading a Recording

Loading a recording is done through a call to the JfrLoaderToolkit#loadEvents(). It takes a file argument, and returns an IItemCollection. The IItemCollection can be thought of as a collection of events.

The IItemCollection supports operations like filter(), and getAggregate(). Used correctly, you should rarely need to rely on external iteration.

Here is an example which loads a recording, and calculates the standard deviation for the java monitor enter events in a recording:

import java.io.File;

import com.oracle.jmc.common.IDisplayable;
import com.oracle.jmc.common.item.Aggregators;
import com.oracle.jmc.common.item.IItemCollection;
import com.oracle.jmc.common.item.ItemFilters;
import com.oracle.jmc.common.unit.IQuantity;
import com.oracle.jmc.flightrecorder.JfrAttributes;
import com.oracle.jmc.flightrecorder.JfrLoaderToolkit;
import com.oracle.jmc.flightrecorder.jdk.JdkTypeIDs;

/**
 * Finds out the standard deviation for the java monitor enter events.
 */
public class LoadRecording {
	public static void main(String[] args) throws Exception {
		DemoToolkit.verifyFirstFileArgument(RunRulesOnFile.class, args);
		
		IItemCollection events = JfrLoaderToolkit.loadEvents(new File(args[0]));
		IQuantity aggregate = events.apply(ItemFilters.type(JdkTypeIDs.MONITOR_ENTER))
				.getAggregate(Aggregators.stddev(JfrAttributes.DURATION));
		
		System.out.println("The standard deviation for the Java monitor enter events was "
				+ aggregate.displayUsing(IDisplayable.AUTO));
	}
}

Note the stream-like syntax. The JMC libraries work well together with streams, but are compiled on JDK 7, and can run on a JDK 7 compliant runtime. Also note that some class names start with Jfr whilst others class names start with Jdk. The difference being that some concepts, like DURATION, are intrinsic to JFR, whilst other are defined in terms of the Java JDK classes or runtime, like MONITOR_ENTER.

The JMC core libraries provide common statistical aggregators, and accessors for common attributes. Should you feel something is missing, it is easy to add to the built-in operations.

Also note the IQuantity returned by the aggregator. The built-in system for handling quantities and units of measurement makes it easy to calculate with, and display, quantities in different units of measurement.

Performing an Automated Analysis Headless

It is also easy to run the JMC rules headless using these core libraries. Simply evaluate the rules against the IItemCollection. Here is a simple example iterating through the rules and evaluating them one by one:

import java.io.File;
import java.util.concurrent.RunnableFuture;

import com.oracle.example.jmc6jfr.rules.util.DemoToolkit;
import com.oracle.jmc.common.item.IItemCollection;
import com.oracle.jmc.common.util.IPreferenceValueProvider;
import com.oracle.jmc.flightrecorder.JfrLoaderToolkit;
import com.oracle.jmc.flightrecorder.rules.IRule;
import com.oracle.jmc.flightrecorder.rules.Result;
import com.oracle.jmc.flightrecorder.rules.RuleRegistry;

public class RunRulesOnFileSimple {
	public static void main(String[] args) throws Exception {
		File recording = DemoToolkit.verifyRecordingArgument(RunRulesOnFileSimple.class, args);
		IItemCollection events = JfrLoaderToolkit.loadEvents(recording);
		
		for (IRule rule : RuleRegistry.getRules()) {
			RunnableFuture<Result> future = rule.evaluate(events, IPreferenceValueProvider.DEFAULT_VALUES);
			future.run();
			Result result = future.get();
			if (result.getScore() > 50) {
				System.out.println(String.format("[Score: %3.0f] Rule ID: %s, Rule name: %s, Short description: %s",
						result.getScore(), result.getRule().getId(), result.getRule().getName(),
						result.getShortDescription()));
			}
		}
	}
}

That said, if you are not constrained to have to run on JDK 7, you can always run the rules in parallel, for example by employing parallel streams:

import java.io.File;
import java.util.List;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Executor;
import java.util.concurrent.Executors;
import java.util.concurrent.RunnableFuture;
import java.util.stream.Collectors;
import java.util.stream.Stream;

import com.oracle.jmc.common.item.IItemCollection;
import com.oracle.jmc.common.util.IPreferenceValueProvider;
import com.oracle.jmc.flightrecorder.JfrLoaderToolkit;
import com.oracle.jmc.flightrecorder.rules.IRule;
import com.oracle.jmc.flightrecorder.rules.Result;
import com.oracle.jmc.flightrecorder.rules.RuleRegistry;

/**
 * Runs the rules on the events in the specified file in parallel, then prints
 * them in order of descending score.
 */
public class RunRulesOnFile {
	private final static Executor EXECUTOR = Executors
			.newFixedThreadPool(Runtime.getRuntime().availableProcessors() - 1);
	private static int limit;

	public static void main(String[] args) throws Exception {
		if (args.length == 0) {
			System.out.println(
					"Usage: RunRulesOnFile <recording file> [<limit>]\n\tThe recording file must be a flight recording from JDK 7 or above. The limit, if set, will only report rules triggered with a score higher or equal than the limit.");
			System.exit(2);
		}
		IItemCollection events = JfrLoaderToolkit.loadEvents(new File(args[0]));
		if (args.length > 1) {
			limit = Integer.parseInt(args[1]);
		}
		Stream<RunnableFuture<Result>> resultFutures = RuleRegistry.getRules().stream()
				.map((IRule r) -> evaluate(r, events));
		List<Result> results = resultFutures.parallel().map((RunnableFuture<Result> runnable) -> get(runnable))
				.collect(Collectors.toList());
		results.sort((Result r1, Result r2) -> Double.compare(r2.getScore(), r1.getScore()));
		results.stream().forEach(RunRulesOnFile::printResult);
	}

	public static RunnableFuture<Result> evaluate(IRule rule, IItemCollection events) {
		RunnableFuture<Result> evaluation = rule.evaluate(events, IPreferenceValueProvider.DEFAULT_VALUES);
		EXECUTOR.execute(evaluation);
		return evaluation;
	}

	public static Result get(RunnableFuture<Result> resultFuture) {
		try {
			return resultFuture.get();
		} catch (InterruptedException | ExecutionException e) {
			e.printStackTrace();
		}
		return null;
	}

	private static void printResult(Result result) {
		if (result.getScore() >= limit) {
			System.out.printf("(%.0f) [%s]: %s\nDetails:\n%s\n============<End of Result>============\n",
					result.getScore(), result.getRule().getId(), result.getShortDescription(),
					result.getLongDescription() == null ? "<no description>" : result.getLongDescription());
		}
	}
}

If you do not need that kind of control, there is a class available for performing automated analysis included with the JMC core library. To run an automatic analysis, simply run the class:

com.oracle.jmc.flightrecorder.rules.report.JfrRulesReport <files> –format <format> –min <severity>

…where files is one or more recordings, and <format> is and <min> is minimum severity to report [ok | info | warning]. In JMC 6.1.0 there is one additional class for generating the file, which produces an HTML in a format and style very similar to the looks of the JMC 6.0.0 Automated Analysis page:

com.oracle.jmc.flightrecorder.rules.report.html.JfrHtmlRulesReport <file> [<outputfile>]

…where <file> is the (single) recording to analyze, and where the optional <outputfile> is where the resulting HTML should be written. If no outputfile is specified, the result of the analysis will be written on stdout.

Adding Custom Rules

It’s easy to add you own heuristics. Do you find yourself with thousands of recordings, and would like to add a Bayesiean network to do some machine learning? The rules are all pure Java, so you can pretty much do whatever you want in a rule.

The easiest way to get started writing your own custom rules is to get yourself an Eclipse Oxygen or later, and install JMC and the experimental PDE plug-in into your Eclipse.

  1. Install Eclipse Oxygen or later.
  2. Got to the Mission Control homepage, and find the Eclipse Update site.
  3. Follow the instructions to install the Mission Control plug-ins.
    image
  4. Got to the Mission Control homepage, and find the Experimental Update site.
  5. Install the PDE plug-in.
    image

Here is a cheat sheet for the eclipse update sites for the JMC 6.0.0 release:

http://download.oracle.com/technology/products/missioncontrol/updatesites/base/6.0.0/eclipse/

http://download.oracle.com/technology/products/missioncontrol/updatesites/supported/6.0.0/eclipse/

http://download.oracle.com/technology/products/missioncontrol/updatesites/experimental/6.0.0/eclipse/

(JMC 6.1.0 will be released according to the same URL pattern.)

To start building your rule, press ctrl-n (or click the File | New | Other… menu) to bring up the New wizard.

image

Select Plug-in Project and hit Next. Name your rule project something exciting.

pic

Unclick that this plug-in will make contributions to the UI and hit Next.

pic2

Next select the Simple JFR Rule Wizard and click Finish (or Next, if you really wish to do some further customizations).

pic3

You will now have a new project in your workspace, containing an example rule. If you have compilation errors, you need to set JMC to be your target platform (see the next section). You can either just export your rule, put it on the class path with the other core libraries, or you can try out your rule by running JMC from within Eclipse with your new rule.

Starting JMC from within Eclipse with Workspace Plug-ins

Running JMC from within Eclipse with any plug-ins you are currently developing is a simple matter of setting the plug-in development target platform to your JMC installation, and launching that platform with your workspace plug-ins.

First set JMC to be your target platform.

  1. Go to Preferences in the main menu (Window | Preferences on Windows).
  2. Find Target Platform by typing Tar in the filter box:
    image
  3. Click Add… and type JMC 6 as name.
    image
  4. Press Add… and select Installation. Hit Next.
    image
  5. Browse to the JMC 6 installation directory (JDK_9_HOME/lib/missioncontrol), and hit Ok.
    SNAGHTML1538a63d
  6. Hit Finish, and Finish. You should now see your new platform.
    image
  7. Select the new JMC 6 platform, and hit Apply and Close to activate it.
    image

Everything should now compile cleanly. Next step is to run JMC with your new rule.

  1. Context click on your project and select Run as Eclipse Application
    SNAGHTML153e5cb4
  2. This should normally be it. If you for some reason run into trouble, go to Run Configurations, and make sure that your launcher is using the com.oracle.jmc.rcp.application.product.
    SNAGHTML1540e14b

You should now be able to see your rule in action. If you have not changed the rule code, try setting the environment variable in the launcher to various values between 0 and 100 and see what happens when you run the rule in JMC:

image

image

Exporting a Plug-in

You can export your rule by context-clicking on the project and selecting Export. In the Export wizard, select Deployable plug-ins and Fragments.

image 

Click Next, select a folder to export the plug-in to and Finish. The resulting jar can either be put on the class path to be included in the headless analysis, or put in the dropins folder of any Java Mission Control installation where you would want the rule to be available.

Summary

This blog described how to:

  • Get started using the JMC core libraries to read Java Flight Recordings (JDK 7, 8 and 9)
  • Get started doing headless analysis of Java Flight Recordings (JDK 7, 8 and 9)
  • Get started creating custom rules for analyzing Java Flight Recordings

Hope it helps someone!

Java Flight Recorder in JDK 9

As you probably already know, JFR is on the road to be open sourced. The plans are being drawn, and if everything goes well, this could happen as early as in JDK 10 (JDK 18.3). No promises, and please do not give anyone a hard time if this doesn’t work out. Know that we tried. Anyways, what we did recently do was to release JDK 9, and there are a LOT of exciting stuff happening for JFR already in JDK 9. Smile

Let’s dive into it…

Supported APIs

Yep. The JFR APIs in the JDK are supported. This means that unlike all my other posts on JFR related APIs, this post does not contain any disclaimer for using these APIs. They are supported! By Oracle! Woho!

On the sad side, this also means that the APIs have changed package, and signatures. On the plus side, they have been immensely enhanced for readability, performance and ease of use.

I will group the API as follows:

  • API for Custom Events
  • API for Controlling the Flight Recorder
  • API for Consuming Recordings

Now, there is an advanced API for consuming recordings included with JMC. That API sports internal iteration, aggregation in various forms, JDK 7, 8 and 9 JFR file compatibility, JDK 7 and later execution compatibility, POJO (no weird dependencies), rules evaluation, and more. This API will also be open sourced, and I will discuss it in a later post.

Creating Custom Events

The API for creating and recording your own events is very simple. All you need to do is to subclass jdk.jfr.Event.and annotate the fields you wish to record with jdk.jfr.Label.

import jdk.jfr.Event;
import jdk.jfr.Label;

public class Hello {
	@Label("Hello World")
	static class HelloWorldEvent extends Event {
		@Label("Message")
		String message;
	}
	public static void main(String... args) {
		HelloWorldEvent event = new HelloWorldEvent();
		event.message = "Hello World event message!";
		event.commit();
	}
}

It is good form to provide a description for your event class and for non-trivial attributes. Here is an example of an event from a JUnit plug-in for JUnit 5 (that I have not released for JDK 9 yet):

/**
 * Event for a test run and completed normally (without exception).
 * 
 * @author Marcus Hirt
 */
@Label("Test Run")
@Description("JUnit test executed and completed normally without exception")
@Category("JUnit")
public class TestEvent extends Event {
    @Label("Display Name")
    @Description("The JUnit display name for the test")
    private String displayName;

    public void setDisplayName(String displayName) {
        this.displayName = displayName;
    }

    public String getDisplayName() {
        return displayName;
    }
}

Other important annotations are, for example, the annotation for providing information about the content. These annotations are put on fields to explain to tools how to interpret the data. For example, Java Mission Control would be able to properly render timestamps and durations properly, if the appropriate @Timestamp and @Timespan annotations were used. I have provided a quick overview of some of the most commonly used ones below:

Annotation Comment
@Address The value is a memory address
@BooleanFlag The value is a boolean flag. Strictly speaking perhaps not the most useful one.
@Frequency The value is a frequency, in Hz
@MemoryAddress The value is a memory address.
@MemoryAmount The value is an amount of memory, in bytes.
@NetworkAddress The value is a network address. Value is a long below 2^32 or an unsigned integer and clients will likely display it in the usual octet format (e.g. 127.0.0.1).
@Percentage The event is a fraction, typically between 0 and 1, and is best represented as a percentage
@Timespan The event is a duration. Defaults to nanoseconds, but can be changed by setting value to one of the static fields defined in Timespan: MICROSECONDS, MILLISECONDS, NANOSECONDS, SECONDS, TICKS. It is always nice to explicitly set value (even for NANOSECONDS), so that it can be readily seen what was intended.
@Timestamp The event is a timestamp. Defaults to MILLISECONDS_TO_EPOCH, but can be changed by setting value to one of the static fields defined in Timestamp: MILLISECONDS_SINCE_EPOCH, TICKS. It is always nice to explicitly set value (even for MILLISECONDS_SINCE_EPOCH), so that it can be readily seen what was intended.

There are other annotations worth knowing about such as these:

Annotation Comment
@Enabled Determines if the event should be enabled by default. Remember the bad old days when you added an event, and then did not see it recorded, since the template did not include it? No more. Also, events are enabled by default, so be careful to explicitly set this to false if the event is expensive. Enable is true by default.
@Experimental The element is experimental, and may change in future versions without prior notice.
@Label The human readable label for the element. Use headline-style capitalization, capitalize first and last words, and all nouns, pronouns, verbs and adverbs. Do not include punctuation.
@Period The default period for a periodic event.
@Registered Events are automatically registered by default when they are first used. This annotation can override this behaviour. If you want to make sure that an event which is not likely to be encountered is registered, perhaps to make the metadata available to tools, you can use FlightRecorder.register(class) to register the event programatically.
@Threshold Defines the default threshold for the event. When an even have a duration lower than the threshold, it will not be recorded. The user can override this threshold in the template settings.
@Unsigned The field value should be treated as an unsigned value.


This is by no means an exhaustive list of all the annotations, and as soon as I find out where the javadocs will end up, I will add a link to them.

Note that the programming model is simpler, compared to previous releases. There is no Producer with which you need to register the events. The life cycle of an event simply follows the life cycle of the event class. Also, new event types are enabled by default. As a matter of fact, all the defaults for an event type can be configured using annotations. Also, compared to previous releases, you do not need to explicitly reuse event objects and keep event tokens around; care has been taken to ensure that events on the hot path should be easy to scalarize.

API for Controlling the FlightRecorder

In JDK 9 there are actually two, related, APIs for controlling the FlightRecorder. Just like before, there will be a JMX API. This API provides similar capabilities as the JDK 8 JMX API, but the API has been cleaned up and moved from the com.oracle.jrockit domain to the jdk.management.flightrecorder domain.

image

I will not spend much time on the JMX API. A good example on how to use the different versions of it will be provided in the JMC source, once it is open sourced.

Another addition is an easy to use local API for controlling the Flight Recorder. Here is an example which starts a recording, records the calculation of the first 50 Fibonacci numbers and writes the recording to disk:

import java.io.IOException;
import java.nio.file.Path;
import java.nio.file.Paths;

import com.oracle.example.jdk9jfr.fib.Fibonacci;
import com.oracle.example.jdk9jfr.fib.FibonacciEvent;

import jdk.jfr.Recording;

/**
 * Example program both controlling the recorder and parsing the resulting data.
 * 
 * The program will calculate the 50 first Fibonacci numbers, then print them
 * out.
 */
public class RecordOnly {
	public static void main(String[] args) throws IOException {
		if (args.length == 0) {
			System.out.println("Need to specify a location for the recording!");
			System.exit(2);
		}
		
		Path path = Paths.get(args[0]);

		try (Recording recording = new Recording()) {
			recording.setName("Fibonacci Recording");
			recording.start();
			recording.enable(FibonacciEvent.class);
			for (int n = 0; n &lt; 50; n++) {
				System.out.println("Calculating fib " + n);
				FibonacciEvent event = new FibonacciEvent();
				event.number = n;
				event.begin();
				event.value = Fibonacci.fibonacciIterative(n);
				event.commit();
			}
			recording.stop();
			recording.dump(path);
		}
	}
}

Consuming Recordings

There is a very easy to use API for consuming the recordings included in the JDK as well. It is only supported to use it to read recordings produced from the same JDK as the API reading it, and it is only supported to run the parser in the same JDK with which it is provided. So, in short, JFR recordings produced by JDK 9, and the parser running on JDK 9.

As I mentioned in the beginning, there is another parser included in JMC, with supported reading JDK 7, 8 and 9 recordings, and which supports internal iteration, aggregation in various forms, JDK 7 and later execution compatibility, and rules evaluation. It is still POJO, so it can be included into anything. Which parser to use depends on what you are trying to accomplish.

Here is an addition to the previous example which also reads the produced recording using the Parser built into the JDK, and which then writes out information from the captured events on stdout:

import java.io.IOException;
import java.nio.file.Path;
import java.nio.file.Paths;

import com.oracle.example.jdk9jfr.fib.Fibonacci;
import com.oracle.example.jdk9jfr.fib.FibonacciEvent;

import jdk.jfr.Recording;
import jdk.jfr.consumer.RecordedEvent;
import jdk.jfr.consumer.RecordingFile;

/**
 * Example program both controlling the recorder and parsing the resulting data.
 * 
 * The program will calculate the 50 first Fibonacci numbers, then print them
 * out.
 */
public class RecordAndConsume {
	public static void main(String[] args) throws IOException {
		if (args.length == 0) {
			System.out.println("Need to specify a location for the recording!");
			System.exit(2);
		}
		
		Path path = Paths.get(args[0]);

		try (Recording recording = new Recording()) {
			recording.setName("Fibonacci Recording");
			recording.start();
			recording.enable(FibonacciEvent.class);
			for (int n = 0; n &lt; 50; n++) {
				System.out.println("Calculating fib " + n);
				FibonacciEvent event = new FibonacciEvent();
				event.number = n;
				event.begin();
				event.value = Fibonacci.fibonacciIterative(n);
				event.commit();
			}
			recording.stop();
			recording.dump(path);
			for (RecordedEvent event : RecordingFile.readAllEvents(path)) {
				int number = event.getValue("number");
				long value = event.getValue("value");
				System.out.printf("fibonacci(%d) = %d (time: %dns)\n", number, value, event.getDuration().getNano());
			}
		}
	}
}

Other Improvements to JFR in JDK 9

Here is a list of some other improvements to JFR in JDK9:

  • Modularized APIs (if you do not need the JMX API, don’t include it).
  • Extensible APIs (can provide custom settings for event types, can provide custom metadata using annotations)
  • Performance improvements (more on this later)
  • Can emit data even in bad situations (Out of Memory for example)
  • New events
    • More detailed safe point information
    • More detailed code cache information
    • New PLAB (promotion) events
    • New compiler events for detailed inlining information
    • New H1 specific information for better visualization of region states
    • Module events (loaded modules, full transitive closure)
    • NativeLibrary (load, a periodic event, by default emitted each chunk)
    • …and more. Just make a test recording and include them all. Winking smile

Performance Enhancements

Here is a graphic (provided by my excellent friend Erik Gahlin at the Oracle JPG Serviceability Team), which shows some of the areas where JFR has improved in JDK 9:

image

Let’s take them in order:

  1. The time spent in safepoints due to chunk rotation to disk has been reduced from a bit over 7ms to almost nothing.
  2. If you have code with a disabled Java event, the overhead of having the event related code in your method will be zero (if that code is on the hot path). After optimization, any trace of the event related code will be optimized away.
  3. The overhead of writing an event (without capturing stack traces), has gone from 8ns to 6ns.
  4. The file size (and buffer memory usage) has roughly been cut by half, due to the introduction of compressed integers.
  5. The time required to capture a stack trace has been almost cut in half.
  6. Even the startup has been slightly improved.

So, not only have the APIs become easier to use, and more comprehensive. JFR has also become significantly faster. That is a bit of a feat, considering that it wasn’t very slow to begin with.

Summary

The new Java release, JDK 9, provides plenty of improvements to JFR:

  1. Supported API
  2. Simplified programming model
  3. Higher performance

This blog provided a quick introduction to JFR in JDK 9, with some simple examples.

Here is a link to the full source for the examples: source_jdk9_jfr_examples.zip

Stop Wasting Your Heap

The Java Virtual Machine is a wonderful little piece of software. It provides the illusion of an infinite heap – as long as you stop referring to instances you no longer need, you can keep going forever. Allocate as many little temporary object as you like; you do not have to worry about running out of memory. The JVM will automatically reclaim the memory when needed.

That said, the amount of memory that you reference at any given time, sometimes referred to as the live set, is limited by whatever memory is available to the Java heap. If you get an OutOfMemoryError, you either have a memory leak, or your application is trying to keep more memory live than can fit on the heap.

For solving the memory leak case, I will soon be demoing something pretty exciting at JavaOne 2017 that we’ve been working on (stay tuned). But what if the application is simply keeping alive too much memory? How do you go about reducing the amount of live memory required to run your application?

One way is to limit your application in various ways. Only allow N number of simultaneous users, crank down various caches you may have used in your application to speed up certain operations etc. Another is to try to look at various heap usage patterns that are known to waste heap memory. You usually end up doing a little bit of everything.

There is a little known experimental plug-in for Java Mission Control called JOverflow which can be helpful for estimating how much memory is wasted due to common heap usage anti-patterns.

Installing JOverflow

JOverflow can easily be installed from within JMC like this:

  1. Start JMC
  2. Go to Help | Install New Software
    SNAGHTML13a75ed2
  3. Check Heap Analysis/JOverflow
    image
  4. Click Next, Next, and finally accept the license and Finish

If you are installing into Eclipse, simply use the Experimental update site (can be found from the homepage, http://oracle.com/missioncontrol)

The Example

The example I will be using is a heap dump from an earlier version of Java Mission Control, JMC 4.1.0. In that version we sacrificed quite a lot of heap memory, for a dubious performance optimization.

The heap dump can be downloaded from here: jmc41dump.hprof

Having the example open in JOverflow will allow you to experiment with the UI, and will likely help facilitate the understanding of the rest of this blog post.

Understanding the JOverflow User Inteface

Start by opening a heap dump. If JOverflow is properly installed in JMC (or Eclipse, if running JMC in Eclipse), you should now be presented with the JOverflow user interface.

This is what it looks like on my computer, with the example heap dump (click pictures for full size):

image

There are four quadrants in the JOverflow user interface. Also notice the little reset button in the upper right corner (clip_image002). It will reset all the selections in the user interface.
 

Object Selection

The top left quadrant, Object Selection, will show you what heap usage anti-patterns the analysis has found. The first column in the Object Selection table show the kind of objects found, implicitly telling you what was analyzed for. The second how much memory they use in total. The third column, Overhead, shows how much of the memory was wasted, in percent of the total heap used.

SNAGHTML13c3d562

Referrer Tree-table

The top right quadrant contains the Referrer tree-table. This tree-table will show the aggregated reference chains for whatever is selected. Note that the way to reset the selections in the Referrer tree-table is to right click in the table. This is since you can make multiple consecutive selections to arrive at the reference chain you are interested in.

SNAGHTML13c68918

Note that the calculated overhead is all zero. This is since we have not selected the result of any analysis in the Object Selection table yet.

Class Histogram

The lower left shows a class histogram for whatever is selected, allowing you to filter on class.

SNAGHTML13cb55ea

Note that selections can be done directly in the pie chart. If you want to reset your selection, click the button representing the selection you have made.

SNAGHTML13cd5757

Again, note that the Overhead column is still zero. This is since we have not selected the result of any analysis in the Object Selection table. That said, by selecting something directly in the class histogram, without having something selected in the Object Selection table, we can see what analysis is relevant for the selected class.

Ancestor referrer

The final table, in the lower right, will show the objects grouped by the closest ancestor referrer.

SNAGHTML13d3cf97

It provides a pie chart to show the memory distribution, and filter box, making it easy to home in on instances of classes belonging to specific packages. Now let’s take a look at the actual example.

Reducing Memory Usage – Example

From the Object Selection table, it seems we are wasting 17% of our used heap with 710 sparse arrays. Selecting Sparse Arrays in the Object Selection table shows us a lot immediately:

image

Almost all of the sparse arrays are Object arrays (see class histogram) accounting for 16% of the heap. The Referrer table-tree and the Ancestor referrer table both show that the field CircularArray.m_array keep holding on 8 of these, and that these 8 instances are responsible for almost all of overhead. So, whilst there are indeed 710 sparse arrays in our system, just 8 are responsible for almost all the heap waste.

We want to know where these circular arrays are being used, so we click the top entry in the Referrer tree-table (remember to use right-click if you want to reset the selections you make here).

SNAGHTML13e08c36

So, it would seem that it is our JMX console attribute subscription storage that is wasting all that memory. This was implemented differently in a later release…

Let’s look at the second offender, the Duplicate Strings. First we reset the user interface (clip_image002). Then select Duplicate Strings in the Object Selection table. We immediately recognize that this will be a tougher job, and the law of diminishing returns probably would perhaps make us stop here. That said, it would seem like Eclipse interning some preference related keys, and JMC interning some JMC related Strings would help.

For a more fun example with duplicate strings see the following file: eclipse.hprof, where various duplicate path segment strings account for a lot of the memory waste:

image

Note: IIRC I tried this on a more recent version of Eclipse, and it was no longer an issue.

Summary

  • JOverflow is a plug-in for Java Mission Control
  • It can be used to identify potentially bad heap usage patterns
  • Used correctly it can reduce the amount of live memory required to run your application

Flight Recorder and Mission Control Sessions at JavaOne 2017

So, I did a quick search for Java Mission Control and Java Flight Recorder related sessions, and these are the ones I found:

Session Title

Session ID

Date

Start Time

End Time

Room

Distributed JVM Monitoring and Profiling Made Simple CON6836 2017-10-03
(Tuesday)
8:30 9:15 Moscone West – Room 2022
The Art of Performance Tuning CON4027 2017-10-03
(Tuesday)
9:30 10:15 Moscone West – Room 2022
Distributed Application Diagnostics Made Simple in Next-Gen Oracle Cloud CON6529
(OOW)
2017-10-03
(Tuesday)
12:45 13:30 Moscone West – Room 3924
From Concept to Robotic Overlord with Robo4J CON3021 2017-10-03
(Tuesday)
15:00 15:45 Moscone West – Room 2009
G1GC Concepts and Performance Tuning CON4577 2017-10-04
(Wednesday)
12:45 13:30 Moscone West – Room 2016
Java Flight Recorder in JDK 9/Java Mission Control 6 HOL3018 2017-10-04
(Wednesday)
13:30 15:30 Hilton San Francisco Union Square (Lobby Level) – Golden Gate 4/5
Deep Diagnostics: APM and Oracle Java Flight Recorder in Oracle Cloud CON6916 2017-10-04
(Wednesday)
14:45 15:30 Moscone West – Room 2005

Let me know if I missed any session!

My JavaOne Sessions 2017

Though I feel truly honored to be one of the Featured Speakers at the JavaOne 2017 home page, I wish someone could have warned me in advance. That picture…😎😱

featured_speaker

Anyways, clicking on the link from from my little speaker square, the one saying “See all sessions from this speaker”, currently shows me as having one talk. A Hands-on-Lab. This is not correct.

Here are the sessions (at JavaOne and OOW) where I will be speaking this year:

Session Title

Session ID

Date

Start Time

End Time

Room

Create a Robot and Bring it to Life KID7398 2017-09-30
(Saturday)
13:00 15:00 Hilton San Francisco Union Square – Continental Ballroom 2/3
Distributed JVM Monitoring and Profiling Made Simple CON6836 2017-10-03
(Tuesday)
8:30 9:15 Moscone West – Room 2022
Distributed Application Diagnostics Made Simple in Next-Gen Oracle Cloud CON6529
(OOW)
2017-10-03
(Tuesday)
12:45 13:30 Moscone West – Room 3924
From Concept to Robotic Overlord with Robo4J CON3021 2017-10-03
(Tuesday)
15:00 15:45 Moscone West – Room 2009
Java Flight Recorder in JDK 9/Java Mission Control 6 HOL3018 2017-10-04
(Wednesday)
13:30 15:30 Hilton San Francisco Union Square (Lobby Level) – Golden Gate 4/5

Looking forward to seeing you at JavaOne/Oracle Open World! Smile

Magnetometer Calibration in Robo4J

This article will explain how to calibrate your magnetometer in Robo4J. This will be required if you want to get as precise results as possible from your magnetometer.

Earth’s magnetic field can be distorted in various ways. If there is no distortion, rotating your magnetometer 360 degrees around the  Z-axis would result in a perfect circle (in the XY plane) centered around the origo. If there is distortion, the circle could be offset or even turned into a tilted ellipse.

There are two different type of distortions commonly discussed:

    1. Hard iron
    2. Soft iron

Hard iron effects are caused by a constant additive field, for example by a magnet. As long as the magnet stays put, relative to the magnetometer, the effect will be constant. This is what we call “bias” in Robo4J.

Soft iron effects are caused by materials that can influence the magnetic field, but which do not necessarily emit a magnetic field. For example iron and nickel. These effects are more complicated and typically turn that perfect circle (from the example above) into an ellipse.

Now, since we are dealing with a 3D magnetometer here, we are really talking about trying to turn an ellipsoid at an offset to origo, into a sphere centered around origo.

So, this is how to calibrate a magnetometer in Robo4J:

    1. Gather data from your magnetometer.
    2. Calculate the bias vector and transform matrix.
    3. Use the calculated bias vector and transformation matrix when instantiating your magnetometer.

Getting Data

If you have an Adafruit 10 DOF breakout board, or an LSM303 device, you are in luck. This is the first magnetometer we support in Robo4J, but we will add more devices as we go. This blog will talk about how to do the calibration given this device.

    1. First, clone robo4j from github
    2. Next build and install robo4j:
      ./gradlew install

    3. Next source the script with the environment variables:
      source /scripts/rpi/environment.sh

    4. Then run the test program, with CSV as output format, and redirect the output to a file:
      java -cp $ROBO4J_PATH com.robo4j.hw.rpi.i2c.magnetometer.MagnetometerLSM303Test 20 1 CSV > /home/pi/magnetometer.csv

It is important that you try to hit as many points as possible, so rotate that thing like there is no tomorrow. I built a simple calibration tool that can help a little bit (but only with one axis). I have been thinking about building a massive one, using three slip rings and ball bearings, but since it would have to be large enough to accept a full sized robot to be truly useful, I know building one would put an unnecessary strain on my marriage. Smile with tongue out

Calculate the Bias Vector and Transform Matrix

This sounds hard, but is rather easy, as we’ve already added a very spiffy application to Robo4J for both visualizing your magnetometer data, and for calculating the bias vector and transform matrix. It is located in the robo4j-tools repo. Simply build and run it with your file as argument.

This is what it will look like when you start it:

raw

If you have some spurious weird values, you may want to filter out the ones that stand out the most. Simply flip the Filter panel open, and click Filter Points. By default the values within 1.0 standard deviation (range from calculated center) will be kept. You can play with the Stddev value to get the result you want.

You can also show what the result would look like if corrected with the Bias Vector and Transformation Matrix, by clicking the “Corrected” checkbox in the Data Selection pane:

corrected

The visualization keep rotating so that the points will be shown alternatively from the XY and the ZY axis.

Using the Data When Instantiating the Magnetometer

In robo4j-hw-rpi the magnetometer can be instantiated with a Tuple3f, and a Matrix3f. Simply instantiate the tuple with the values from the bias vector, and the matrix with the values from the Transform Matrix:

Tuple3f bias = new Tuple3f(-44.689f, -2.0665f, -15.240f);
Matrix3f transform = new Matrix3f(1.887f, 5.987f, -5.709f, 5.987f, 1.528f, -2.960f, -5.709f, -2.960f, 9.761f);
MagnetometerLSM303Device magnetometer = new MagnetometerLSM303Device(I2CBus.BUS_1, 0x1e, Mode.CONTINUOUS_CONVERSION, Rate.RATE_7_5, false, bias, transform);

Summary

This blog explained how to use the Robo4J MagViz tool to calibrate a magnetometer.

Radio Silence & Robo4J

So, first of all I want to apologize for the radio silence on this blog. There are two major reasons for this radio silence:

  1. My twins
  2. Me doing blogs over at Robo4J

So, I thought I’d just do a recap of my Robo4J blogs, in the order I published them (which is probably the order someone new to Robo4J may want to read them). Smile

Here is the list:

My JavaOne 2016 Sessions

This year I’ll be doing two sessions and one HoL:

Title: Java Mission Control and JFR in JDK 9: A Sneak Peek [CON1509]

Abstract:
In JDK 9, the JFR APIs will become supported, so you can now rely on the JFR APIs for both controlling Oracle Java Flight Recorder and introducing your own custom JFR data into the recordings. Also, with JDK 9, a new major, very different version of the Oracle Java Mission Control feature of Oracle Java SE Advanced will be released.

This session takes a sneak peek into what the new APIs for controlling the Oracle Java Flight Recorder feature will look like and provides migration guidelines from the old APIs. It also goes through some of the highlights of the completely redesigned Oracle Java Mission Control 6.0.0, such as the automatic analysis of flight recordings. A quick intro to the new bytecode instrumentation agent used internally by Oracle Java Mission Control 6 is also provided.

 

Title: Using Oracle Java Flight Recorder in an Autonomous Robotic Vehicle [CON1511]

Abstract:
This session shows how the speaker used the Oracle Java Mission Control and Oracle Java Flight Recorder features of Oracle Java SE Advanced to record large quantities of data from the sensors in a little hobby project: a small autonomous robotic vehicle running Oracle Java SE Embedded.

The session focuses on how to use Oracle Java Flight Recorder to great advantage where resources are scarce and where overhead can cause significant problems. It also discusses how to build custom integration for Oracle Java Flight Recorder, using APIs already available in Oracle’s HotSpot JDK, and how the speaker went about designing, 3-D-printing, and building the actual hardware and software.

 

Title: Java Mission Control 5.5 [HOL1510] (with David Buck)

Abstract:
This session shows how the Oracle Java Mission Control and Oracle Java Flight Recorder features of Oracle Java SE Advanced can be used to solve various commonly encountered production-time profiling and diagnostics problems. It also shows how various Oracle Java Mission Control plug-ins can be installed and put to good use to further extend the functionality of Oracle Java Mission Control.

Among other things, the session looks at
• Reducing memory pressure
• Maximizing throughput
• Reducing heap usage (heap waste analysis)

 

Looking forward to seeing you at JavaOne 2016! 🙂

/M

Using the JMC Designer View

So several people have asked me about this very unsupported feature recently. First it came up in a Google groups discussion. Then it came up again when I was helping one of the Oracle Cloud teams, and used the feature myself to modify some of the tabs to show more relevant data. And now that a colleague asked if I had a blog on how to use the JMC designer, I felt that I had to write one. So, against my better judgement, I’ll go ahead and describe, in some detail, how to use this feature in JMC 5.5.

Before we start, I would just like put in the following disclaimers:

  • Plug-ins created in JMC 5.5 will not work in JMC 6.
  • There will not be a Design view in JMC 6.
  • This is not supported functionality.

Here is the mandatory standard disclaimer:

The following blog entry will describe UNSUPPORTED functionality. This means that relying on the described APIs or functionality may BREAK your code/plugin with any given update of the JDK and/or Mission Control.

As mentioned, this disclaimer is very much applicable this time, as it is absolutely certain that this will no longer work in JMC 6.

There. I think I’ve said it plenty enough to dare continuing now.

The Designer View

Maybe you have, at times, wished that there were no dials in the Overview tab (mentioned in a Google groups discussion). Or that a certain table contained the standard deviation or max value of the aggregated events for a certain attribute (happens all the time).

Well, all but the last tab group in JMC are really customized reports that show off some specific part of the recording, usually focusing on a few different event types. And most people usually want these reports to look slightly different.

Enter the Designer View. This view is really how the rest of the JMC UI was designed. All the different tabs in JMC were created by the JMC team using the Designer View. All the tabs, except for the ones in the Events tab group, are really custom reports that highlight some aspect of the recording to help solve problems.

Starting the Designer

Starting the Designer is very easy. First open a recording. Any recording will do, but depending on what you want to do, it is good if it contains events from the event type you want to design for. Open the Designer View by hitting the Window | Show View | Designer (Unsupported) menu. Did I mention that this is unsupported?

Let’s start by removing the dials in the Overview tab. Don’t worry about messing something up – you can always reset the user interface in the preferences Window | Preferences, Java Mission Control / Flight Recorder -> Reset User Interface. That said, that extreme option will reset all your modifications to factory defaults. You can also undo changes in the design mode, which is more local and usually enough.

Here we go. Once the Designer View opens, it will show you a tree of the available tabs. In our case we want to edit the Overview. Select the General / Overview Tab, and click the “stop” button to open the Overview Tab in design mode.

designer_init

This will show a layout view with boxes representing the different components. Bring up the context menu for one of the dials and select the Delete menu.

delete

Repeat this until you have deleted all the Dials, as well as the container for the dials.

deleted

Press the play button in the Designer View to take a look at what your changes look like when live.

deleted_live

No more dials! Let’s take another example.

Adding Table Columns

Let’s say that you want to add some statistics for Servlet invocation events. You have the WLS tabs, but you find the Servlet tab sadly lacking in the critical-pieces-of-information department.

Note: If you do not have a WLS recording, you can download and follow this tutorial to get one. To install the plug-in, simply go to Help | Install New Software… and select the WLS tab pack.

install_wls

When looking at the fabled Servlet tab, we decide that it would be very nice to see the longest lasting servlet invocations, as well as the standard deviations. First step is to open the Designer View on the Servlet tab. Right click on the Servlet Invocations by URI table (the bottom area), and select properties. You should see something like this:

edit_servlets

Select the Columns tab. What do you know; max duration is already there, but it is not visible by default.

If you find that a table is missing a column with important information, always first check to see if the information is hidden. Go to the table, use the context menu to check what is available under Visible Columns. If there are many columns, use Configure… to see them all.

visiblecolumns 

Click max duration and check the Visible check box.

max_duration_visible

Next we will add a column for the standard deviation. Click the Add… button and select the duration attribute. Click OK in the Add Event Attribute dialog when done. Next edit the Name and Description to something meaningful, then select Standard Deviation as you aggregate function.

stddev

Then hit play to look at your new and now visible column. If you sort on Max Duration, it may look something along the lines of this:

maxdurations

So, what if you want to add your own tab group? Well, let’s use the Smurfberry Exchange example from the tutorial.

Adding Tab Groups and Designing Tabs from Scratch

It is a well known fact that the Smurfs are trading smurfberries on their Smurfberry Exchange (SMX). They have performance problems though, so they have asked us for help. As part of solving their performance issue, we’ve recorded the actual transactions taking place.

For more information on the actual events recorded, see http://hirt.se/blog/?p=277. To solve their problem, look at the jmx.jfr in the tutorial and try to figure out what is going on. The jmx_fixed.jfr contains a recording when the problem is fixed.

This blog entry is however not about helping the smurfs solve their performance problem, but rather about adding a nifty UI so that we can see graphs over the exchange price development over time.

First close any open recording. Structural changes will not be seen in the UI until you have reloaded your recording(s). Next create a group by context clicking the root in the Designer View, and selecting New | New Group.

new_group

Edit your group to your satisfaction. Note that the placement path is a lexical comparison of strings. Usually the following format is used: /#1.0. I’ll just add it last.

smurfgroup

Next, we want to create a tab in the tab group we just created. Context click the tab group and select New | New tab to add the tab. Fill out the New Tab wizard in a similar fashion.

 smxpricetab

That is enough to get the structure in place. Now open the recording containing the events for wish you want to design. In this case the smx.jfr file. Our new tab will, not very surprisingly, be quite empty.

emptysmxtab

Let’s click the stop button and get to work. We want one of those nifty automatic range navigators on the top, and a big chart showing the price over time. First we fix the layout. Context click on the empty area, and select Assign | Container | Rows to split the area into two.

containers

The range navigators are all usually using 100 pixels, so select the upper container and set min and max to 100 pixels.

minmax

Next context click the upper container and assign it an Autoconfiguring Range Navigator (Assign | Other | Autoconfiguring Range Navigator). The Autoconfiguring Range Navigator shows where all the events that are represented on the tab are located in time, and allows the user to shift the time for all the components of the page, i.e. zoom into various parts of the recording.

assignrangenav

Next we add the price chart. Select the lower container and use the context menu to select Assign | Graphics | Chart. You will now see the properties dialog for the newly added chart. Fill out the base properties.

The only thing of note here is the role selection. It can be used to define a relationship between a master component and a slave component. The slave will only show whatever is selected in the master. We noted that this were never done in more than three steps, so we ended up building the editor around that simple use case. A component can be a Master, Slave or a Slave Child. This is one of the reasons we never started supporting the Designer View. It was built only to be used by JMC developers to quickly build JMC itself. It can sometimes be a bit quirky.

Note that we are not slaving this chart to selections in any master component, so let the role remain Independent.

Next we need to configure the Left Y-axis, and the data series associated with that axis. We will only have one axis. Price is numeric, so the Content Type should be Number.

lefty 

Finally we add the price data series. Click the Data Series tab and the Add… button to add the price attribute. Note that you can type “SMX” in the filter box at the top, to quickly find the right event type to select attributes from.

addprice

Next we configure the rendering of the price series. Select to show the legend (mostly useful for when there are multiple series, and you want the user to be able to select which ones to render by clicking the legenes). Select Line and Fill. Don’t forget to select Line (x,y) as style, and three different colors for Line Color, Top Fill Color and Bottom Fill Color, so that you get a nice gradient. 😉

configurepriceseries

Press the play button in the Designer View to see what it all looks like, working range navigator and all.

smxfinished

The Smurf economy seems to be doing fairly well. Prices are indeed increasing over time. Or perhaps there is a shortage of Smurfberries. Hard to tell. When looking at the recording where the performance problem has been fixed, the prices seem to falling towards the end. Could be a symptom of the smurfs effectively getting into high frequency trading after fixing the performance bottleneck. 😉

smxfixed

 

Saving and Sharing Designs

Now that you’ve built some amazing custom visualization, perhaps even for your own custom events, you may want to share it with others. Or even use it yourself. It is very nice to have it stored if you mistakenly press the Reset User Interface button, or if you accidentally FUBAR the user interface making it absolutely necessary to press that button. Or if you want install your changes into JMCs in multiple versions of the JDK.

To export your design as a plug-in, simply right click on the root node in the Designer View and select Export UI to Plug-in.

export_as_plugin

In the export wizard that opens, select the tabs to export. Note that you can select any tabs, even default tabs that you have overridden.

exporttabs

Next you get to select the ID and version of your plug-in. Higher versions will override lower versions.

versionselect

Click OK to select where to store it. The resulting plug-in can be shared. To use it, simply add the plug-in to the JDK_HOME\lib\missioncontrol\dropins folder.

Summary

In this blog you learnt how to wreak absolute havoc on the JMC user interface. It’s not supported, so don’t come crying if/when something breaks. 😉

I’ve added the SMX plug-in here.

I will so very much regret posting this.