The Operative Set is an easily overlooked but quite powerful feature in the Mission Control Flight Recorder user interface. It is a selection of events that you can reach and utilize from almost every tab in the Flight Recorder user interface.
This blog will explain how the Operative Set can be used to quickly drill down into a set of events having some very specific properties. The Operative Set may take a bit of time and experimenting to get used to. If you want to try out this feature whilst reading this blog, I’ve made the actual recording used in the examples available here.
Note: If JMC warns you the recording is too large, see this blog post.
Altering the Operative Set
Before we start using the Operative Set, I’ll just note that the tab groups in JMC really are of two different types:
- Tab groups specific to one or more specific event types.
These tab groups are pre-configured to show off specific events. The Event Types view will not affect what is shown in the tab groups. - The general Events tab group.
The tabs in this tab group will show events of the event types selected in the Event Types view.
The event type specific tabs are usually used to add and remove events of interest to/from the operative set, and the general events can be used to both study and modify the set of events in the operative set. The general Events tabs are also the only way to analyse events of types for which there is no specific user interface. Of course, it is easy to build a new user interface for events where there is no specific user interface yet, but that is for another blog post.
To alter the Operative Set, simply select the events that you want to add/remove and use the context menu. For example, to add the the events for the most contended lock, simply go to the contention tab, right click on the row representing what you’re interested in, go to the Operative Set menu, then choose Add Selection.
Voila – the events have now been added to the Operative Set. You can now move over to the general Events tabs to play with them.
Using the Tabs in the Events Tab Group
The first thing to check when moving over to the Events tabs, is to make sure that the events of interest aren’t filtered out. This is done by checking the settings of the Event Type view, which is normally to the left of the Flight Recorder editor.
In our case, we have just added Java Monitor Blocked event to the Operative Set, so there isn’t anything else we need to do. When in doubt, enable all event types.
So let’s see what the Events tabs can be used for.
Overview
This tab is a good place to sanity check what you actually have in your operative set. Click the checkboxes for the Operative Set, and you will see what producers the events originate from, as well as the distribution per event type. This page will be quite boring with our current selection of events.
Log
The Log is simply a table of the available events. When you select one of the events, all information (attribute values) in the event will be shown to you in the lower half of the tab. Again you can opt to only see what is in the Operative Set. Note that in the 5.2.0 version of Mission Control, this is one of the places where you’ll get to see the actual line number in the stack traces. This means that if you’re not happy with the line number-less traces in the aggregated stack traces, you can always select to add the events you are interested to the operative set, and then look at the events in the log.
Graph
The graph view will show you the events in a per thread graph. This is very useful to get a visualization of what is happening in terms of distribution over both threads and time. In our simple example, we can make out what threads are involved in blocking on the log4j logger, as well as the time periods for which we have contention.
Threads
The threads simply list the events aggregated per thread. This can be useful for setting the operative set to only the events occurring in a certain thread.
Stack Trace
In this tab the aggregated stack traces for the selected events can be viewed:
It can be noted that for the events in our Operative Set (the blocking events), we seem to spending the most time waiting to do tracing rather than debug or info level logging.
Histogram
This tab is not for creating graphical histograms, but rather for grouping events on a specified attribute. In our case we can for instance select to group our events on monitor address, then select to set the operative set to the busiest lock:
Then we can, for example, move back to check what threads are involved in locking on the particular lock instance.
The Relational Key
Now, there is something called a relational key in the event metadata for an attribute. The relational key is used to tell that an attribute contains a value that can bind together events of different event types. For instance all the different GC events that are associated with a certain GC ID are connected through a relational key:
The Operative Set context menu has a submenu called Add Related Events. That menu will typically try to be helpful by listing attributes that have a relational key. So for instance, to find all GC events that are related to the longest garbage collection taking place in the recording, go to the Memory tab group, select the Garbage Collections tab and sort the GC table on time. Next select the longest recording and choose Operative Set | Add Related Events | With GC ID = (the id of the longest GC in the supplied recording has ID 76).
Looking at the log, you can, for example, look at all the related GC events in the log tab:
Putting it All Together
So let’s look at what is happening during an invocation of the viewPatients servlet at the different levels of the recording. First go to the servlet tab in the WebLogic tab group, next right click in the table and select Operative Set | Clear. Most WLS events are related to a certain request using something called the ECID (Enterprise Context ID). Let’s next add the events associated with the first found ECID associated with an invocation of viewPatients:
Now we can check what was occurring on the WLS level through the request by simply looking at the events in the log. First enable all event types in the Event Type view. Next move over to the Log tab:
Lastly, let’s check what was happening with the other available events occurring in the same thread during the same time as these high level WLS events. First select all the events in the log (in the Operative Set) (ctrl-A). Next right click and select add concurrent:
You will now see related events such as allocations and exceptions popping up:
The other tabs in the Events tab group can of course be used with the new operative set as usual. For example, to get a graphic overview of all the events involved in the request, go to the Graph tab:
Summary
The Operative Set is a feature that allows power users of Mission Control to look at events from different angles and quickly home in on related events having certain properties. It is a power user feature – it may take a little bit of getting used to, but once you’re used to it you’ll be scary effective. 