Deep Distributed Tracing with OpenTracing and the JDK Flight Recorder

Recently I had a talk at Code One about using OpenTracing together with the JDK Flight Recorder to do deep tracing. Since the session wasn’t recorded, I though I’d do a blog about it instead. Here we go…

Distributed tracing has been of interest for a very long time. Multiple companies have sprung up around the idea over the years, and most APM (Application Performance Management) solutions are built around the idea. Google released a paper around their large scale distributed systems tracing infrastructure in 2010 – Dapper, and there are now several open source alternatives for distributed tracing available inspired by the paper, such as Jaeger and Zipkin.

In Java land, pretty much all of the APMs are doing pretty much the same thing: they use BCI (byte code instrumentation) for getting the data, and then they present that data to the end-user in various ways, oftentimes using some kind of analysis to recognize common problems and suggesting solutions to the end users of the APM. The real differentiation is knowing what data to get, and what to do with the data once captured.

Since there was no standard, one problem was for vendors to inject helpful, vendor specific, information into the distributed traces. The vendor of a software component may have a quite good idea about what information would be helpful to solve problems. Some vendors support APM specific APIs for contributing the data, but more often than not the instrumentation is done using BCI by scores of developers working for the various APM companies. The same is true for maintainers of open source components – either skip the problem entirely and let the APM vendors come up with good instrumentations points (if your component is popular enough), or pick a popular APM and integrate with it. That is, until OpenTracing came along…

Introduction to OpenTracing

OpenTracing is an open source, vendor neutral, distributed tracing API. In other words, library developers can interact with one API to support multiple APM/Tracer vendors. Also, customers can add contextual information to distributed traces without worrying about vendor lock-in. Contributors to OpenTracing include LightStep, Jaeger, Skywalking and Datadog, and the specification is available on GitHub:

https://github.com/opentracing

The core API concepts in OpenTracing are (from the slides of my talk, DEV5435):


Trace


– A distributed operation, potentially spanning multiple processes


– Implicitly defined by the individual Spans in the trace (more soon)


– Can be thought of as a directed acyclic graph (DAG) of Spans


– The span in the root of the DAG is called the root Span


– The edges between the Spans are called References


Span


– Has an operation name


– Has a start timestamp


– Has a finish timestamp


– Has a SpanContext


• Has Baggage Items (key/value pairs which cross process boundaries)


• Implementation specific state used to identify the span across process boundaries)


– Zero or more key/value Span Tags


– Zero or more Span Logs (key/value + timestamp)


Reference


– Defines a direct casual relationship between two spans


– ChildOf


• Parent depends on the child in some way


• Note that it is legal for a finish timestamp of a child to be after that of any parent


– FollowFrom


• Parent does not depend on the result of the child in any way


• Note that it is legal for a FollowsFrom child to be started after the end of any ancestor

Also worth noting is that a Scope is a thread local activation of a span.

The Example

As an example, we’ll be using a simple application consisting of three microservices. It is part of the back-end of a fictional robot store. Robots can be ordered at the Orders service, and they will be produced in a Factory. There is also a Customers service keeping track of the customers. Finally there is a load generator that can be used to exercise the services.

image 

The code is available under https://github.com/thegreystone/problematic-microservices. (Yes, as the name indicates, the services come pre-packaged with built-in problems. :))

The services, as well as the load generator, have built-in tracing support, so for a full systems run with the load generator, you would get a trace (a DAG of spans), looking something along the lines of:

image

Or, in Jaeger, where you have time on the X axis:

tracing

In this case I have scrolled down a bit to focus on the factory. As can be seen, there is great variability in the time it takes to create a chassis and/or paint a robot. We have multiple production lanes, and we’d expect times across the factory lanes to be more even, not to mention much faster. So what gives?

Well, we can expand the operation to see if there was some additional information:

tracing_details

Now, sometimes the tags may include crucial pieces information that may help you solve the problem without needing any additional information. In this particular case, though, knowing that we were building a pink BB-8 isn’t really doing the trick.

What would be the next step? All too often the next step would be to look at the code around the instrumentation point, trying to figure out what was going on at the time simply from analyzing the code. Sometimes that may be quite hard. The problem may be in third party code not expected to behave badly. There may even be some other piece of code not directly in the code path causing the problems, perhaps an agent misbehaving and causing long lasting safe points in the JVM.

So, we’re screwed then? Nah. What if you had a magic tool that could record what was going on in the JVM and the application at the time of the incident? Something providing not only method profiling information, but a deeper view, including information about vm operations, memory allocation profiling, events for the usual application caused thread halts and much, much more. Something that could be always on, with very low overhead. And let’s say you ran with a tracer that added some contextual information, such as information that could be used to identify traces, spans and thread local span activations in the recorded data, and which allowed you to use your favourite tracer too? Then things would get interesting indeed…

Running with the JFR Tracer

For Code One I wrote a little delegating JFR tracer, which allows you to record contextual information into the flight recorder. It was meant as an example on how to do deep distributed tracing. Deep enough to solve entire classes of problems that are hard to solve without more detailed knowledge.

The tracer works with Oracle JDK 7+ and OpenJDK 11+ (it is a multi-release jar, a.k.a. mrjar), and the source is available on GitHub here:

https://github.com/thegreystone/jfr-tracer

The bundle is available from Maven Central, and here is the dependency you need to add:

<dependency>
<groupId>se.hirt.jmc</groupId>
<artifactId>jfr-tracer</artifactId>
<version>0.0.3</version>
</dependency>

Next you need to initiate your tracer and pass it to the constructor of the DelegatingJfrTracer, like so:

GlobalTracer.register(new DelegatingJfrTracer(yourFavTracer));

That’s it. When the tracer is running you will get contextual information recorded into the flight recorder.

Looking at the Recording

Dumping the flight recorder for the factory, and looking at the dump in the Threads view, might look something like this:

image

We can see that we have these long lasting monitor enter (Java Blocking) events, and looking at the stack traces directly by selecting individual events, or at the Lock Instances page, it is fairly obvious where the contention is:

image

We can, of course, create a custom OpenTracing view to make it easier to directly finding and homing in on long lasting traces (I’ll create a repo for a ready made one with some more flair at some point). Simply go the the Event Browser, and right click on the Open Tracing folder. Select “Create a new page using the selected event types”. You will now have a new page in the Outline. You can right click on the title on the page to rename it and switch icon.

Next select an arbitrary event, and right click on it. Select Group-By->Trace Id. In the new Group By table that appeared, select Visible Columns to enable (at least) the attribute showing the longest duration (the total duration of (wall clock) time the trace spent in the process that the recording came from). Next sort on the Longest Duration column.

In this case I’ve ran a few more (press enter in the single step load generator a few times, or let it just continuously add load):

image

You can, of course add additional tables with groupings that can be useful, for example, per thread. To quickly home in the entire user interface on a trace id of interest, just select a trace and choose “Store and Set as Focused Selection”:

SNAGHTML38aee12

Now you can go back to, for example, the Threads view, and click the Time Range: Set button in the upper right corner. Voila, you are in exactly the right place. You may also want to view concurrently occurring events in the same threads (see check boxes on top), and enable additional thread lanes:

SNAGHTML38f17fd

Summary

  • Distributed tracing is great, especially in today’s world of (very µ and plenty) µ-services.
  • For the Java platform, injecting trace/span-identifying information as contextual information into the JDK Flight Recorder is dynamite.
  • A simple example on how to do this automagically is available on my GitHub as a delegating Tracer, in an mrjar, supporting Oracle JDK 7+ and OpenJDK 11+:
        https://github.com/thegreystone/jfr-tracer
  • The slides for my Code One presentations can be found here:
        https://oracle.rainfocus.com/widget/oracle/oow18/catalogcodeone18?search=hirt
    (The relevant session for this blog is DEV5435.)
  • The JDK Flight Recorder r0xx0rz.
  • JDK Mission Control r0xx0rz.

Note that since the article was written, I have donated the tracer to OpenTracing.
See https://github.com/opentracing-contrib/java-jfr-tracer.

Dudes and Dudettes, Things Just Got Better!

Oh my god. The amount of FUD concerning the JDK licensing for JDK 11 is just amazing.

So, unless I’ve missed something, Oracle does the following:

  1. Contributes pretty much all of the closed source technologies (or what was originally to become closed source) of the Oracle JDK to OpenJDK, for example giving the community:
    • JDK Flight Recorder
    • JDK Mission Control
    • ZGC
    • …and probably more stuff I can’t think of right now
  2. Ensures the Oracle JDK and the OpenJDK builds are virtually indistinguishable, except for licensing
  3. Moves to, from what I’ve been told, a very competitively priced subscription model (as opposed to the rather, IMHO, highly priced Java SE Advanced licenses)
  4. Starts providing a free OpenJDK build (which includes all these donated technologies)
  5. Provides uncountable man hours of maintaining and innovating the Java platform
  6. Ensures that the community knows where to find the free bits by linking to them, and slaps on a bright yellow warning sign, so that everyone can see that the licensing has changed:

    image

And how does the community react, you wonder? Yep, that’s right. “Oracle is the Devil”, “This is a bait and switch operation” etc. Ad nauseum.

So, this is my personal take on open source: if I like a certain open source technology, and it helps me in my work, I support it. Either by contributing, or by paying (gasp) money for it. Especially if I would like the technology to thrive in the future. Technologies that are not supported, tend to die and be forgotten. I have personally, for a very long time, paid a yearly contribution of 35$ to Eclipse. And that is even though my team, and countless of other teams at Oracle, have contributed to various Eclipse projects over the years. And, no, Eclipse does not provide me support for it.

Summary

Oracle gives away countless of highly regarded technologies and starts releasing free OpenJDK builds. Parts of the Java community throws a fit.

My Sessions at Code One 2018

If anyone would like to catch up with me at Code One, here are some specific times where my location is known in advance. 😉

Session Title

ID

Date Start Time End Time

Room

Contributing to the Mission Control
OpenJDK Project

[DEV4506]

Monday,
Oct 22

10:30

11:15

Moscone West
Room 2004
Robotics on Java Simplified

[DEV6089]

Monday,
Oct 22

14:30

15:15

Moscone West
Room 2024

Production-Time Profiling
and Diagnostics on the JVM

[DEV4507]

Wednesday,
Oct 24

10:30

11:15

Moscone West
Room 2004
OpenJDK Mission Control:
The Hands-on-Lab

[HOL4508]

Wednesday,
Oct 24

12:30

14:30

Moscone West
Room 2001A

Diagnose Your Microservices:
OpenTracing/Oracle Application
Performance Monitoring Cloud

[DEV5435]

Wednesday,
Oct 24

16:00

16:45

Moscone West
Room 2011

Getting Started with the
(Open Source) JDK Mission Control

[DEV4509]

Thursday,
Oct 25

11:00

11:45

Moscone West
Room 2014

Note that the last few years, the HoL has been full – it may be a good idea to register for it early. Especially now that JMC/JFR is being open sourced (JDK 11, JMC 7).

Looking forward to seeing you at Code One!

Here is a link to the sessions in the content catalogue.

JMC Open Sourced!

This is going to be a short blog post, because it is past bedtime and I’m frankly pretty beat. Just wanted to say thank you to everyone who helped open source Java Mission Control in the relatively short period of time it was done in.

So a huge thank you to everyone on the Stockholm JMC team: Klara Ward, Ola Westin, Henrik Dafgård, Erik Greijus and Per Kroon. And to Dalibor Topic, who had the thankless job to go through all the source and built artifacts checking on, among other things, licenses and copyrights. Also thank you to Donald Smith for handling the internal license approval process, and Mark Reinhold for advice. Also, a big thank you to Guru, who is prepping to start building Oracle binary builds of the OpenJDK project. And to Iris who published the repos and set up the project on OpenJDK. And to everyone who I’ve forgotten to mention, since my brain is no longer really functioning at 2 a.m. You guys rock!

The repositories are available here:

      http://hg.openjdk.java.net/jmc

  • The jmc repo contains the source, which is buildable using Maven.
    Everything you need to know about building JMC is in the README.txt
    Please read it first, thoroughly, before asking questions.
  • The jmc-graphics repo contains artwork and graphic files, like splash screens and icons.

We also have a slack at https://jdkmissioncontrol.slack.com/.

If you have any questions, don’t hesitate to ask!

Kind regards,
Marcus

JMC & JFR in JDK 9 Labs

So JavaOne 2017 has now come and gone. This year it was a pretty great one! The Java keynote was IMHO the best one in years, with plenty of great announcements and interesting content. The sessions I managed to attend were also very high quality. Not to mention the dinners and the parties. Winking smile

One of the sessions I did this year was a hands-on-lab on Java Mission Control 6 and Java Flight Recorder in JDK 9.

IMG_6670

Since it was full, and since people who could not attend have asked for it, I promised to put the HoL on my blog. Well, here it is!

There are three files:

  1. The lab instructions.
  2. The full lab content for Windows x64 (easiest, but largest download, 856MiB).
  3. The project files only (all other platforms, 39MiB).

Hope this helps!

Please let me know if you find any errors in the labs, or if you get stuck!

/M

Java Flight Recorder in JDK 9

As you probably already know, JFR is on the road to be open sourced. The plans are being drawn, and if everything goes well, this could happen as early as in JDK 10 (JDK 18.3). No promises, and please do not give anyone a hard time if this doesn’t work out. Know that we tried. Anyways, what we did recently do was to release JDK 9, and there are a LOT of exciting stuff happening for JFR already in JDK 9. Smile

Let’s dive into it…

Supported APIs

Yep. The JFR APIs in the JDK are supported. This means that unlike all my other posts on JFR related APIs, this post does not contain any disclaimer for using these APIs. They are supported! By Oracle! Woho!

On the sad side, this also means that the APIs have changed package, and signatures. On the plus side, they have been immensely enhanced for readability, performance and ease of use.

I will group the API as follows:

  • API for Custom Events
  • API for Controlling the Flight Recorder
  • API for Consuming Recordings

Now, there is an advanced API for consuming recordings included with JMC. That API sports internal iteration, aggregation in various forms, JDK 7, 8 and 9 JFR file compatibility, JDK 7 and later execution compatibility, POJO (no weird dependencies), rules evaluation, and more. This API will also be open sourced, and I will discuss it in a later post.

Creating Custom Events

The API for creating and recording your own events is very simple. All you need to do is to subclass jdk.jfr.Event.and annotate the fields you wish to record with jdk.jfr.Label.

import jdk.jfr.Event;
import jdk.jfr.Label;

public class Hello {
	@Label("Hello World")
	static class HelloWorldEvent extends Event {
		@Label("Message")
		String message;
	}
	public static void main(String... args) {
		HelloWorldEvent event = new HelloWorldEvent();
		event.message = "Hello World event message!";
		event.commit();
	}
}

It is good form to provide a description for your event class and for non-trivial attributes. Here is an example of an event from a JUnit plug-in for JUnit 5 (that I have not released for JDK 9 yet):

/**
 * Event for a test run and completed normally (without exception).
 * 
 * @author Marcus Hirt
 */
@Label("Test Run")
@Description("JUnit test executed and completed normally without exception")
@Category("JUnit")
public class TestEvent extends Event {
    @Label("Display Name")
    @Description("The JUnit display name for the test")
    private String displayName;

    public void setDisplayName(String displayName) {
        this.displayName = displayName;
    }

    public String getDisplayName() {
        return displayName;
    }
}

Other important annotations are, for example, the annotation for providing information about the content. These annotations are put on fields to explain to tools how to interpret the data. For example, Java Mission Control would be able to properly render timestamps and durations properly, if the appropriate @Timestamp and @Timespan annotations were used. I have provided a quick overview of some of the most commonly used ones below:

Annotation Comment
@Address The value is a memory address
@BooleanFlag The value is a boolean flag. Strictly speaking perhaps not the most useful one.
@Frequency The value is a frequency, in Hz
@MemoryAddress The value is a memory address.
@MemoryAmount The value is an amount of memory, in bytes.
@NetworkAddress The value is a network address. Value is a long below 2^32 or an unsigned integer and clients will likely display it in the usual octet format (e.g. 127.0.0.1).
@Percentage The event is a fraction, typically between 0 and 1, and is best represented as a percentage
@Timespan The event is a duration. Defaults to nanoseconds, but can be changed by setting value to one of the static fields defined in Timespan: MICROSECONDS, MILLISECONDS, NANOSECONDS, SECONDS, TICKS. It is always nice to explicitly set value (even for NANOSECONDS), so that it can be readily seen what was intended.
@Timestamp The event is a timestamp. Defaults to MILLISECONDS_TO_EPOCH, but can be changed by setting value to one of the static fields defined in Timestamp: MILLISECONDS_SINCE_EPOCH, TICKS. It is always nice to explicitly set value (even for MILLISECONDS_SINCE_EPOCH), so that it can be readily seen what was intended.

There are other annotations worth knowing about such as these:

Annotation Comment
@Enabled Determines if the event should be enabled by default. Remember the bad old days when you added an event, and then did not see it recorded, since the template did not include it? No more. Also, events are enabled by default, so be careful to explicitly set this to false if the event is expensive. Enable is true by default.
@Experimental The element is experimental, and may change in future versions without prior notice.
@Label The human readable label for the element. Use headline-style capitalization, capitalize first and last words, and all nouns, pronouns, verbs and adverbs. Do not include punctuation.
@Period The default period for a periodic event.
@Registered Events are automatically registered by default when they are first used. This annotation can override this behaviour. If you want to make sure that an event which is not likely to be encountered is registered, perhaps to make the metadata available to tools, you can use FlightRecorder.register(class) to register the event programatically.
@Threshold Defines the default threshold for the event. When an even have a duration lower than the threshold, it will not be recorded. The user can override this threshold in the template settings.
@Unsigned The field value should be treated as an unsigned value.


This is by no means an exhaustive list of all the annotations, and as soon as I find out where the javadocs will end up, I will add a link to them.

Note that the programming model is simpler, compared to previous releases. There is no Producer with which you need to register the events. The life cycle of an event simply follows the life cycle of the event class. Also, new event types are enabled by default. As a matter of fact, all the defaults for an event type can be configured using annotations. Also, compared to previous releases, you do not need to explicitly reuse event objects and keep event tokens around; care has been taken to ensure that events on the hot path should be easy to scalarize.

API for Controlling the FlightRecorder

In JDK 9 there are actually two, related, APIs for controlling the FlightRecorder. Just like before, there will be a JMX API. This API provides similar capabilities as the JDK 8 JMX API, but the API has been cleaned up and moved from the com.oracle.jrockit domain to the jdk.management.flightrecorder domain.

image

I will not spend much time on the JMX API. A good example on how to use the different versions of it will be provided in the JMC source, once it is open sourced.

Another addition is an easy to use local API for controlling the Flight Recorder. Here is an example which starts a recording, records the calculation of the first 50 Fibonacci numbers and writes the recording to disk:

import java.io.IOException;
import java.nio.file.Path;
import java.nio.file.Paths;

import com.oracle.example.jdk9jfr.fib.Fibonacci;
import com.oracle.example.jdk9jfr.fib.FibonacciEvent;

import jdk.jfr.Recording;

/**
 * Example program both controlling the recorder and parsing the resulting data.
 * 
 * The program will calculate the 50 first Fibonacci numbers, then print them
 * out.
 */
public class RecordOnly {
	public static void main(String[] args) throws IOException {
		if (args.length == 0) {
			System.out.println("Need to specify a location for the recording!");
			System.exit(2);
		}
		
		Path path = Paths.get(args[0]);

		try (Recording recording = new Recording()) {
			recording.setName("Fibonacci Recording");
			recording.start();
			recording.enable(FibonacciEvent.class);
			for (int n = 0; n &lt; 50; n++) {
				System.out.println("Calculating fib " + n);
				FibonacciEvent event = new FibonacciEvent();
				event.number = n;
				event.begin();
				event.value = Fibonacci.fibonacciIterative(n);
				event.commit();
			}
			recording.stop();
			recording.dump(path);
		}
	}
}

Consuming Recordings

There is a very easy to use API for consuming the recordings included in the JDK as well. It is only supported to use it to read recordings produced from the same JDK as the API reading it, and it is only supported to run the parser in the same JDK with which it is provided. So, in short, JFR recordings produced by JDK 9, and the parser running on JDK 9.

As I mentioned in the beginning, there is another parser included in JMC, with supported reading JDK 7, 8 and 9 recordings, and which supports internal iteration, aggregation in various forms, JDK 7 and later execution compatibility, and rules evaluation. It is still POJO, so it can be included into anything. Which parser to use depends on what you are trying to accomplish.

Here is an addition to the previous example which also reads the produced recording using the Parser built into the JDK, and which then writes out information from the captured events on stdout:

import java.io.IOException;
import java.nio.file.Path;
import java.nio.file.Paths;

import com.oracle.example.jdk9jfr.fib.Fibonacci;
import com.oracle.example.jdk9jfr.fib.FibonacciEvent;

import jdk.jfr.Recording;
import jdk.jfr.consumer.RecordedEvent;
import jdk.jfr.consumer.RecordingFile;

/**
 * Example program both controlling the recorder and parsing the resulting data.
 * 
 * The program will calculate the 50 first Fibonacci numbers, then print them
 * out.
 */
public class RecordAndConsume {
	public static void main(String[] args) throws IOException {
		if (args.length == 0) {
			System.out.println("Need to specify a location for the recording!");
			System.exit(2);
		}
		
		Path path = Paths.get(args[0]);

		try (Recording recording = new Recording()) {
			recording.setName("Fibonacci Recording");
			recording.start();
			recording.enable(FibonacciEvent.class);
			for (int n = 0; n &lt; 50; n++) {
				System.out.println("Calculating fib " + n);
				FibonacciEvent event = new FibonacciEvent();
				event.number = n;
				event.begin();
				event.value = Fibonacci.fibonacciIterative(n);
				event.commit();
			}
			recording.stop();
			recording.dump(path);
			for (RecordedEvent event : RecordingFile.readAllEvents(path)) {
				int number = event.getValue("number");
				long value = event.getValue("value");
				System.out.printf("fibonacci(%d) = %d (time: %dns)\n", number, value, event.getDuration().getNano());
			}
		}
	}
}

Other Improvements to JFR in JDK 9

Here is a list of some other improvements to JFR in JDK9:

  • Modularized APIs (if you do not need the JMX API, don’t include it).
  • Extensible APIs (can provide custom settings for event types, can provide custom metadata using annotations)
  • Performance improvements (more on this later)
  • Can emit data even in bad situations (Out of Memory for example)
  • New events
    • More detailed safe point information
    • More detailed code cache information
    • New PLAB (promotion) events
    • New compiler events for detailed inlining information
    • New H1 specific information for better visualization of region states
    • Module events (loaded modules, full transitive closure)
    • NativeLibrary (load, a periodic event, by default emitted each chunk)
    • …and more. Just make a test recording and include them all. Winking smile

Performance Enhancements

Here is a graphic (provided by my excellent friend Erik Gahlin at the Oracle JPG Serviceability Team), which shows some of the areas where JFR has improved in JDK 9:

image

Let’s take them in order:

  1. The time spent in safepoints due to chunk rotation to disk has been reduced from a bit over 7ms to almost nothing.
  2. If you have code with a disabled Java event, the overhead of having the event related code in your method will be zero (if that code is on the hot path). After optimization, any trace of the event related code will be optimized away.
  3. The overhead of writing an event (without capturing stack traces), has gone from 8ns to 6ns.
  4. The file size (and buffer memory usage) has roughly been cut by half, due to the introduction of compressed integers.
  5. The time required to capture a stack trace has been almost cut in half.
  6. Even the startup has been slightly improved.

So, not only have the APIs become easier to use, and more comprehensive. JFR has also become significantly faster. That is a bit of a feat, considering that it wasn’t very slow to begin with.

Summary

The new Java release, JDK 9, provides plenty of improvements to JFR:

  1. Supported API
  2. Simplified programming model
  3. Higher performance

This blog provided a quick introduction to JFR in JDK 9, with some simple examples.

Here is a link to the full source for the examples: source_jdk9_jfr_examples.zip

My JavaOne Sessions 2017

Though I feel truly honored to be one of the Featured Speakers at the JavaOne 2017 home page, I wish someone could have warned me in advance. That picture…😎😱

featured_speaker

Anyways, clicking on the link from from my little speaker square, the one saying “See all sessions from this speaker”, currently shows me as having one talk. A Hands-on-Lab. This is not correct.

Here are the sessions (at JavaOne and OOW) where I will be speaking this year:

Session Title

Session ID

Date

Start Time

End Time

Room

Create a Robot and Bring it to Life KID7398 2017-09-30
(Saturday)
13:00 15:00 Hilton San Francisco Union Square – Continental Ballroom 2/3
Distributed JVM Monitoring and Profiling Made Simple CON6836 2017-10-03
(Tuesday)
8:30 9:15 Moscone West – Room 2022
Distributed Application Diagnostics Made Simple in Next-Gen Oracle Cloud CON6529
(OOW)
2017-10-03
(Tuesday)
12:45 13:30 Moscone West – Room 3924
From Concept to Robotic Overlord with Robo4J CON3021 2017-10-03
(Tuesday)
15:00 15:45 Moscone West – Room 2009
Java Flight Recorder in JDK 9/Java Mission Control 6 HOL3018 2017-10-04
(Wednesday)
13:30 15:30 Hilton San Francisco Union Square (Lobby Level) – Golden Gate 4/5

Looking forward to seeing you at JavaOne/Oracle Open World! Smile