Recording JMX data into JFR

I recently saw a question on Stack Overflow regarding recording JMX data into JFR. I proposed that one could use the JFR Java API to record MBean attributes, but I realized I never did a blog post on dynamic events. So this blog post will be about dynamic JFR event generation, and how to write a configurable agent which you can use to periodically record declared JMX attributes into the Flight Recorder.

If you couldn’t care less about the implementation details, you can just download the agent jar and the example configuration file at the end of this article. 😉

And here is the usual disclaimer:

The following blog entry will describe UNSUPPORTED functionality. This means that relying on the described APIs or functionality may BREAK your code/plugin with any given update of the JDK and/or Mission Control.

 

Dynamic JFR Events

The JFR event API can be very useful to record custom events into the Java Flight Recorder. But what do you do when you do not know beforehand what fields the event needs to contain? Or if you wish to programmatically generate event types based on some input?

Dynamic JFR events to the rescue. You will, as usual, need a Producer:

Producer p = new Producer("Dynamic Event Example", "This example will generate a dynamic event over and over", "http://www.hirt.se/example/dynamic/");

Next you need to create your DynamicEventToken. This token is what you will later use to create your events.

token = p.createDynamicInstantEvent("Dynamic Event Name", "Dynamic event description", "dynamic/path", true, true,
    new DynamicValue("rndstr", "Random String", "Contains a random string", ContentType.None, String.class),
    new DynamicValue("fraction", "Random Fraction", "Contains a random fraction between 0 and 1", ContentType.Percentage, Float.TYPE),
    new DynamicValue("timestamp", "Timestamp", "Contains a timestamp", ContentType.Timestamp, Long.TYPE));

Don’t forget to register the producer! Also note that there is one DynamicValue per attribute to record. The ContentType is used to provide the data consumer (usually JMC) with a hints for helpful visualization of the data. Check out the jmxprobes.xml file for more information on the ContentTypes.

Here is the full source for an example recording dynamic events of the same event type, but different payloads, over and over again:

package se.hirt.jmx2jfr;

import java.util.Random;

import com.oracle.jrockit.jfr.ContentType;
import com.oracle.jrockit.jfr.DynamicEventToken;
import com.oracle.jrockit.jfr.DynamicValue;
import com.oracle.jrockit.jfr.InstantEvent;
import com.oracle.jrockit.jfr.Producer;

@SuppressWarnings("deprecation")
public class DynamicExample implements Runnable {
	private static final Random RND = new Random();
	private final DynamicEventToken token;

	public DynamicExample() throws Exception {
		Producer p = new Producer("Dynamic Event Example", "This example will generate a dynamic event over and over", "http://www.hirt.se/example/dynamic/");
		token = p.createDynamicInstantEvent("Dynamic Event Name", "Dynamic event description", "dynamic/path", /* record thread */ true, /* record stack trace */true,
				new DynamicValue("rndstr", "Random String", "Contains a random string", ContentType.None, String.class), 
				new DynamicValue("fraction", "Random Fraction", "Contains a random fraction between 0 and 1", ContentType.Percentage, Float.TYPE),
				new DynamicValue("timestamp", "Timestamp", "Contains a timestamp", ContentType.Timestamp, Long.TYPE));
		p.register();
	}

	public void run() {
		while (true) {
			InstantEvent e = token.newInstantEvent();
			e.setValue("rndstr", randomString());
			e.setValue("fraction", RND.nextDouble());
			e.setValue("timestamp", System.currentTimeMillis());
			e.commit();
			sleep(1000);
		}
	}
	
	public final static void main(String[] args) throws Exception {		
		Thread t = new Thread(new DynamicExample(), "Event generation thread");
		t.setDaemon(true);
		t.start();
		System.out.println("Press enter to exit!");
		System.in.read();
	}
	
	private static String randomString() {
		StringBuilder builder = new StringBuilder();
		for (int i = 0; i < RND.nextInt(10) + 1; i++) {
			builder.append(Character.toString((char) (RND.nextInt(26) + 64)));
		}
		return builder.toString();
	}
	
	private void sleep(long millis) {
		try {
			Thread.sleep(millis);
		} catch (InterruptedException e) {
		}
	}
}

To run the example, especially if on a JDK prior to 8u40, remember to start with:

-XX:+UnlockCommercialFeatures –XX:+FlightRecorder

To make a recording, start JMC and hook up to the running process. Don’t forget to enable the events!

enable_dyn_events

If you do a 10 second recording, you should have recorded about 10 events. Notice that properly setting the content type allows JMC to present the information in a better way. See, for example, the timestamp field:

generatedevents

 

 

Using the Agent

The agent allows you to configure and record JMX data into JFR without having to change any code. There is a small test snippet that you can run in the agent jar to try it out (download the agent and the settings file from the end of this article):

java -XX:+UnlockCommercialFeatures -XX:+FlightRecorder -javaagent:jmx2jfr.jar=jmxprobes.xml -cp jmx2jfr.jar se.hirt.jmx2jfr.test.HelloWorldTest

Next hook up to the HelloWorldTest process from JMC and create a 10s recording. Don’t forget to enable the MBean events:

enablembeanevents

When the recording finishes, go to the Events | Log tab. Select to only see the MBean events in the Event Types view, and select an event. You will see that MBean data has been recorded from the platform MBean server.

mbeanslog

Use the configurations file to configure what MBeans and attributes to record, and what content types to use for the different attributes. There will be one Event Type per MBean, filled with whatever attributes you have selected:

<jmx2jfr>
        <!-- Time to wait before retrying getting an MBean attribute in ms -->
	<retrytime>60000</retrytime>
	
	<!-- Time to wait until starting in ms --> 
	<delay>10000</delay>
	
	<!-- How often to get the attributes and emit events for them, in ms -->
	<period>1000</period>
	
	<!-- clean: Use name or type keys for event name. Leads to cleaner event 
	     type names, but can result in name collisions if you have MBeans with 
	     the same name in the same domain.
	     canonical: ugly event type names, but guaranteed not to collide -->
	<namingmode>clean</namingmode>
	
	<!-- objectname:    MBean name - look it up in JMC
	     attributename: Attribute name - look it up in JMC
	     contenttype:   [None | Bytes | Timestamp | Millis | Nanos | Ticks | Address] -->
	<attributes>
		<attribute>
			<objectname>java.lang:type=OperatingSystem</objectname>
			<attributename>FreeSwapSpaceSize</attributename>
			<contenttype>Bytes</contenttype>
		</attribute>
		<attribute>
			<objectname>java.lang:type=OperatingSystem</objectname>
			<attributename>TotalSwapSpaceSize</attributename>
			<contenttype>Bytes</contenttype>
		</attribute>
...

This should be fairly self explanatory. Edit the file to get the MBean attributes that you are interested in.

Summary

TL;DR – agent to easily record data from the platform MBean server into the Flight Recorder. Just edit the xml settings file, add the agent to the command line and point it to your edited settings file.

Downloads:

Splitting Flight Recorder Files

Flight recorder is most commonly used to dump the last few minutes of data. Either by doing a profiling recording for a minute or so, or by having a continuous recording which is later dumped when some anomaly was discovered using a monitoring tool, for example the JMC JMX console.

Sometimes though, people bring me enormous recordings. Now, enormous recordings are not handled well in JMC. Most of the time, JMC will discover that you are trying to open something that is too large, and prompt you for what subset of the recording you want to open. 

rl5di

Sometimes it is easier to simply split the recording into smaller recordings instead.

Here is a small tool that can be used for splitting flight recordings:

split.jar

Usage:

java -jar split.jar filename [targetSizePerFile in MiB (default 50)]

Note that the jfr file will be split along chunk boundaries using a best effort algorithm. The individual files will be as close to the specified target size as the chunk boundaries allow.

Let me know if you run into trouble! 🙂

The BMP-180 Pressure Sensor

For detecting small altitude changes, reading the barometric altitude, or for just detecting changes in pressure, you may want to use a pressure sensor. The Bosch BMP-180 is a very reasonably priced one, which is accessed over I2C. There are several nice breakout boards available from Adafruit.featuring the BMP-180. There is, for example, one featuring only the BMP-180, and another one featuring gyros and accelerometers as well.

BMP

No matter which, you would access it in a similar fashion.

As usual, before you start, you need to install PI4J and set up the Rasberry for i2c communication.

Next decide in which mode you want the device to run. Check out the OperatingMode for details. The higher resolution, the longer time it takes to read values, and the higher the current draw.

If you are using the standard address and i2c bus, next create a BMPDevice:

BMPDevice bmp = new BMPDevice(OperatingMode.STANDARD);

Next, reading the pressure, altitude and temperature is done like this:

System.out.println(String.format(“Temperature: %.1fC, Pressure: %dhPa, Altitude: %.1fm”,
        bmp.readTemperature(),
        bmp.readPressure() / 100,
        bmp.readAltitude()));

To run the standard example, which will print the above every two seconds:

sudo java -classpath .:classes:/opt/pi4j/lib/*:./bmp.jar se.hirt.pi.adafruit.bmp.test.BMPTest

Summary

The Bosh BMP-180 is an easy way to measure the barometric altitude and/or changes in pressure.

Java Mission Control 5.5.0 Released!

The latest version of Java Mission Control was released a few moments ago, together with Oracle Java SE 8u40. It’s a minor release; most of the development is taking place in the upcoming major version of JMC, but there are nevertheless some interesting features and fixes in this release. I have selected a few of the highlights below.

Dynamic Enablement of Java Flight Recorder

If you forgot to enable flight recorder on the command line (in 8u40 and later), all is no longer lost – JFR can be enabled dynamically, after the fact that the JVM process has been started! If you try to connect to a JVM which has not enabled JFR, you will be presented with a dialog like this:

dynamic

Clicking yes will dynamically enable the Flight Recorder and allow you to start recordings.

JMC Now Using Eclipse RCP 4.4

Up until now we have based the JMC RCP application on Eclipse RCP 3.8.2. The reason was due to a performance problem when running with Eclipse RCP 4.x. For JMC 5.5.0 we worked around this problem, and are now using Eclipse RCP 4.4. As a result, JMC will also feel much snappier when running inside of Eclipse, as a set of Eclipse plug-ins.

Upgrading to Eclipse 4.4 affects many different parts of the stand-alone JMC application. For a starter, plenty of bugs have been fixed in the platform over the past years. Fixes that JMC can now take advantage of. Also, various enhancements done to the platform are now available, such as themes:

darkjmc

It is supported to run JMC in the very latest version of Eclipse, and the speed should be comparable to running in Eclipse 3.8.2.

JMC Plug-ins Are Now Signed

JMC can be used as a set of plug-ins in Eclipse. This has the added benefit that you can jump to your source code from anywhere we display a class, or a method frame or similar. There is also an experimental update site from where you can install extra JMC content, for example the WebLogic Server plug-in, or the D-Trace plug-in. Previously all the JMC plug-ins were unsigned. Installing them into Eclipse required you to accept to install unsigned content into your Eclipse:

unsigned

This is no longer the case.

 

JMC Friendlier to Users of Dynamic Languages

Looking at recordings from applications running in implementations of dynamic languages making heavy use of Lambda Forms will be much friendlier. Say, for example, a recording of a Java Script application running on Nashorn. Just like the JVM by default hides @hidden annotated Lambda Form methods, so will JMC. If you still want to see them in all their glory, the setting can be toggled in the preferences.

Here is a picture of the same recording opened twice – to the left opened with @hidden annotated methods hidden, and to the right with @hidden annotated methods visible:

hiddenlambdaforms

And here is what a stack trace can look like with them visible:

hiddenshowing

Suffice it to say, you will usually want to leave them hidden… 😉

 

Bugfixes

  • Performance workarounds for certain CTabFolder related Eclipse bugs, which allows us to run faster in Eclipse 4.x
  • Numerous bugs fixed from upgrading to a newer version of the platform
  • The JMX Console no longer assumes it is connected to the platform MBean server – any MBean server should be okay, and functionality degraded gracefully
  • JMC no longer assumes that projects are physically located in the workspace when running in Eclipse
  • When jumping to source from a stack trace in the JFR UI, you will end up on the correct line number (previously you always ended up in the method declaration, even when aggregating per line number)
  • You can now use the –vmargs flag to append JVM arguments when launching JMC. Previously –vmargs would replace all JVM arguments

 

Summary

  • Yay, JMC 5.5.0!
  • Yay, Java SE 8u40!

The Adafruit Ultimate GPS

This blog will provide a tiny library for communicating with the Adafruit Ultimate GPS board from Java SE Embedded. I only implemented UART communication, but it is trivial to adapt the code for use over the USB serial, should you want to. The library provides position information as well as speed, as GGA and VTG was the only info I need (right now). As usual, first install PI4J before attempting to use the library.

GPS

Here is how to use the thing:

GPS gps = new GPS();
gps.addListener(new GPSListener() {
    @Override
    public void onEvent(PositionEvent event) {
        System.out.println(event);
    }

    @Override
    public void onEvent(VelocityEvent event) {
        System.out.println(event);
    }
});

 

That’s it. The PositionEvent will provide you with the 2D location, the altitude about mean sea level, the altitude above the ellipsoid, an estimate of the max error, number of satellites used for the fix etc. The VelocityEvent will provide the heading and ground speed. Hope this helps!

 

Summary

Almost all projects are improved by a GPS. 🙂 This tiny library makes it easy to access the Adafruit GPS from Java SE Embedded. Here are the links:

Reading JMX Metrics

I found this question on Stack Overflow about how to connect to a locally running process and read JMX metrics. I thought I’d explain a little bit more in detail here on the blog. The steps involved are:

    1. Try to get the JMX Service URL for the local JMX agent.
    2. If you couldn’t, try to start the local JMX agent.
    3. Open an MBeanServerConnection to the local JMX agent.
    4. Do whatever MBean-ish stuff you were planning on doing, for example reading attributes.

This is pretty much as simple as it sounds. Let’s do them in order.

Getting the Service URL

When the local management agent is started, it publishes the service URL (which also contains the serialized stub for communication – take a look at the URL, it’s really quite funny) in a well known location. This can be retrieved by using the sun.management.ConnectorAddressLink class like this:

private static JMXServiceURL getLocalStubServiceURLFromPID(int pid)
        throws IOException {
    String address = ConnectorAddressLink.importFrom(pid);
    if (address != null) {
        return new JMXServiceURL(address);
    }
    return null;
}

If you didn’t get any service url, then the local management agent was not started.

Starting the Local Management Agent

Starting the local management agent can be done in various ways. In late versions of the JDK this is quite easily done. I am only going to show you the simple way today. In older you might need to resort to the attach command to load java agents (VirtualMachine vm = VirtualMachine.attach(pid).loadAgent(<path to management-agent.jar>,”com.sun.management.jmxremote”)).

To start the local agent, we simply execute the diagnostic command for starting it:

executeCommandForPID(vm, pid, “ManagementAgent.start_local”);

private static void executeCommandForPID(VirtualMachine vm, String pid,
        String cmd) throws IOException {
    HotSpotVirtualMachine hsvm = (HotSpotVirtualMachine) vm;
    hsvm.executeJCmd(cmd);
}

 

Opening the MBeanServer Connection

This, as well as doing the MBeany things you want to do once you have the connection, is done just as you would normally do it:

JMXConnector jmxc = JMXConnectorFactory.connect(url, null);
MBeanServerConnection connection = jmxc.getMBeanServerConnection();

Getting attributes:

connection.getAttribute(“java.lang:type=Memory”, “HeapMemoryUsage”);

 

Summary

    • Hooking up to the local management agent is quite easy.
    • There are valuable metrics that can be collected.
    • Here is a full example getting the JVM CPU load as well as used heap.

Controlling the Adafruit PWM Driver from Java

This article will show how to control the Adafruit 16CH 12 bit PWM driver from a Raspberry PI using Java.

  1. First step is to configure your Raspberry PI for I2C.
  2. Next step is to set up PI4J.
  3. Next is to download the tiny little library for accessing the PWM driver I just made (and/or the source).

IMG_2385

This is the raspberry hooked up to various things, one of them being the PWM driver. The easiest way to test the library is to hook up a servo to channel 0 and run the jar. The test program also accepts a servo on channel 1, and motors (via H-bridges) on channel 2 and 3. Do not, under any circumstance, run the test program with servos on channel 2 and 3, unless you edit it first.

Here is how to run the test program:

sudo java -classpath .:classes:/opt/pi4j/lib/*:./pwm.jar se.hirt.pi.adafruit.pwm.test.PWMTest

You should see something like this:

And this is how to use it:

First construct a PWMDevice:

PWMDevice device = new PWMDevice();

If you’re using the default address (0x40) on I2C bus 1, you can safely use the default constructor. Next select the PWM frequency to use. 50 Hz is, for example, good for RC servo control:

device.setPWMFreqency(50);

Next select the channel for which you want to alter the PWM signal, for example channel 0:

PWMChannel servo0 = device.getChannel(0);

Setting the PWM signal is done with the setPWM(int on, int off) channel method. We have 12 bit fidelity with which to choose when to go from low to high, and vice versa. That gives us a valid number from 0 to 4095, with 0 meaning in the beginning of the pulse, and 4095 meaning the end. So, for most servos, the internetz say that a 1ms pulse means minimum servo travel, and a 2ms pulse means maximum servo travel. 1.5 ms is centered. 50Hz means every pulse is about 20ms. 1/20 of 4096 means that minimum servo travel should be about 205. Max travel should be around 410. Of course, your mileage may vary, and servos can usually travel a bit further than this. Anyways, this would be setting min:

servo0.setPWM(0, 205);

See the test program in the source jar for more examples. Have fun!

Conclusion and Tips

  • The Adafruit 16CH 12 bit PWM driver is a wonderful little breakout board for controlling servos and motors (via an H-bridge).
  • You can use 6V for driving the servos (V+) – just make sure that the servos can handle it.
  • Servos that are specified to handle 6V are usually faster/stronger when running on 6V.
  • I am using a Polulu voltage regulator for driving the Raspberry PI off of a 3S LiPo-battery. I am using a separate BEC for servos/motors.

My Favourite JMC Quotes

So, recently I came across some lovely JMC related quotes and articles. The first one is regarding the allocation profiling in JFR and how it does not affect scalarization (stack local allocation of objects):

http://psy-lob-saw.blogspot.se/2014/12/the-escape-of-arraylistiterator.html?utm_content=bufferc532a&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer

“Java Mission Control is my current favourite profiler for the Oracle JVM.”

This tweet was heart warming too:

image

Not to mention that there were recently JMC presentations from Houston to Sri Lanka.

This, and maybe Christmas lighting induced general nostalgia, made me think of some other neat quotes I’ve seen regarding JMC.

Here are some of my favourite ones:

“Mission Control is what you wanted instead of profiler, but were afraid to ask.”

– Oleksandr Otenko, Oracle Performance Engineer

“JMC, because in my experience other tools cause way more harm than good.”

– David Buck, Oracle SE

“JMC is my main tool for getting insight into the rhythm of a JVM and the running applications. JMCs low overhead enables usage of an identical always-active setup in development and production environments. JMCs approach to recordings is very useful for creation of performance baselines to support both testing and profiling. I have used recordings extensively to document non-functional test results during development to validate SLA requirements and ensure good JVM citizenship. I have used recordings to resolve critical production issues caused by latency, memory-leaks or threading.”

– Allan Thrane Andersen, Trygg

“It’s an invaluable piece of work.”

– Maurizio Cimadamore, Oracle (Java Lang Tools)

“I managed to do in 1 day what I’ve tried to do in 2+ weeks using <tool> and <other tool>.”

– Maurizio Cimadamore, Oracle (Java Lang Tools)

“For the record: Java Mission Control is the best profiler ever, I use it daily, and so should you.”

– Marcus Lagergren, Oracle (Consulting Member of Technical Staff)

“If I could get only one present for Christmas, it would be Oracle Java Mission Control.”

– Morten G. Hermansen (tweet)

“I am ACS engineer since 2008, delivering local Middleware support to several customers. Since I started to work with Java/JRockit Mission Control, it became a key tool for my work, helping me to troubleshooting, identifying root causes and bottlenecks, and also for doing proactive follow up services to customers. Without it, I would be blind.”

– Iratxe Etxebarria, Oracle (ACS)

“JMC and JFR is really convenient tool. These helps with not only trouble shooting but also tuning. It means that JMC/JFR can reduce business impact of customers caused by less performance and application trouble.”

– Chihiro Ito, Oracle (Japan, Consulting group)

“JFR is an absolutely necessary function as much as AWR or DTrace. These are not nice-to-have but must-have.”

– Shingo Yamanari, Oracle (Japan, Sales Consulting)

“JFR is really impressive for partners in Japan. Members who have trouble shooting tasks totally understand the necessity of JFR.”

– Tomofumi Nijo, Oracle (Japan, Sales Consulting)

“JRockit Flight Recorder (JFR) provides detailed information of application runtime such as memory consumption, thread usage, etc. This kind of information will support trouble shooting by work for reproducing. Plus, JFR has less impact to a system because it runs with very low load. JFR provides deep insight of application behavior. It allows efficient system management, rapid trouble shooting and impact analysis.”

– Tadaaki Yoshida, Manager, NTT DOCOMO, INC.

“I love it, what a great tool! It is perfectly intuitive for me personally, maybe because I was a Precise consultant for years, during my time away from Sun/Oracle; JFR is like Precise, only done right :-)”

– Dave Fisk, Principal Software Engineer, Oracle

One might argue that the one from my friend Lagergren is a bit biased, but he really does use JMC all the time.

If you think I’ve left out an amazing quote, or if you’d like to share one, please drop me an e-mail! 🙂

JMC Tutorial

I know. JavaOne has already come and gone, and I promised, after my JavaOne submission snafu, to publish a Tutorial here by the end of JavaOne. Well, life happened, and here we are, a few weeks after JavaOne.

That said, here it is finally, the first draft of the JMC tutorial:

http://hirt.se/downloads/oracle/jmc_tutorial.zip

The tutorial instructions are in the JMC_Tutorial.pdf included in the root of the zip.

In this tutorial you can:

  • Install JMC (either the stand alone version, the Eclipse version, or both)
  • Do Java Flight Recordings
  • Analyse Flight Recordings
  • Watch a laser kill monsters
  • Use experimental plug-ins such as the Java FX and WLS plug-ins for Flight Recorder, as well as JOverflow for doing memory dump analysis.
  • Get to play with jcmd

The Tutorial has hours worth of material, so do the exercises that you find the most interesting first. Also, you may want to print out the tutorial and keep it beside your computer to minimize the switching back and forth between JMC, Eclipse and the pdf.

Hope you find this helpful!

(Please don’t hesitate to send suggestions or any bugs/typos found my way!)

 

Other Resources

Here are some other blog posts that will make for fine companions to the Tutorial:

Improving the Fidelity of the JFR Method Profiler

One nice property of the JFR method profiler is that it does not require for the threads to be at safe points for stacks to be sampled. However, since the common case is that stacks will only be walked at safe points, HotSpot normally does not provide metadata for non-safe point parts of the code, which means that such samples will not be properly resolved to the correct line number and BCI. That is, unless you specify:

-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints

With the DebugNonSafepoints, the compiler will generate the necessary metadata for the parts of the code not at safe points as well.

Big thanks to Doug Simon and Gilles Duboscq for noting this!