{"id":920,"date":"2017-09-22T18:00:25","date_gmt":"2017-09-22T17:00:25","guid":{"rendered":"http:\/\/hirt.se\/blog\/?p=920"},"modified":"2017-09-21T20:26:50","modified_gmt":"2017-09-21T19:26:50","slug":"jmc-6-automated-analysis-headless","status":"publish","type":"post","link":"https:\/\/hirt.se\/blog\/?p=920","title":{"rendered":"JMC 6 Automated Analysis Headless"},"content":{"rendered":"

This post will be a little bit wider than the title implies; it will be about using the JMC core JFR APIs in general. That said, one of the things we will be using it for, towards the end, will be to run the JMC 6 automated analysis headless.<\/p>\n

This article will also cheat by talking about JMC 6.1, so that anyone reading this article will have relevant information for JMC 6.1 as well. \"Winking <\/p>\n

The JAR-files Needed<\/h2>\n

To run these examples you will need to have the following Jar-files from the JDK_HOME\/lib\/missioncontrol\/plugins folder on your classpath:<\/p>\n\n\n\n\n\n\n\n\n\n
\n

Prefix<\/p>\n<\/th>\n

\n

Version needed<\/p>\n<\/th>\n

\n

Comment<\/p>\n<\/th>\n<\/tr>\n

com.oracle.jmc.common<\/td>\n\n

>= 6.0.0<\/p>\n<\/td>\n

Classes common to JMC, such as stacktrace definitions, the content type system (quantities\/ units of measurement) etc. <\/td>\n<\/tr>\n
com.oracle.jmc.flightrecorder<\/td>\n\n

>= 6.0.0<\/p>\n<\/td>\n

The Java Flight Recorder parser, classes for extracting information from a recording.<\/td>\n<\/tr>\n
com.oracle.jmc.flightrecorder.rules<\/td>\n\n

>= 6.0.0<\/p>\n<\/td>\n

The core definitions and classes for automated analysis of recordings.<\/td>\n<\/tr>\n
com.oracle.jmc.flightrecorder.rules.jdk<\/td>\n\n

>= 6.0.0<\/p>\n<\/td>\n

Contains the rules for the JDK (such as rules for synchronization trouble, long lasting VM operations and much more).<\/td>\n<\/tr>\n
com.oracle.jmc.flightrecorder.ext.wls.parser<\/td>\n\n

>= 6.0.0<\/p>\n<\/td>\n

(Optional<\/font>, only relevant for users wanting WebLogic Server extensions)<\/td>\n<\/tr>\n
com.oracle.jmc.flightrecorder.ext.wls.rules<\/td>\n\n

>= 6.0.0<\/p>\n<\/td>\n

(Optional<\/font>, only relevant for users wanting WebLogic Server extensions)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Loading a Recording<\/h2>\n

\nLoading a recording is done through a call to the JfrLoaderToolkit#loadEvents(). It takes a file argument, and returns an IItemCollection. The IItemCollection can be thought of as a collection of events.<\/p>\n

The IItemCollection<\/font> supports operations like filter(), and getAggregate(). Used correctly, you should rarely need to rely on external iteration.<\/p>\n

Here is an example which loads a recording, and calculates the standard deviation for the java monitor enter events in a recording:<\/p>\n

\r\nimport java.io.File;\r\n\r\nimport com.oracle.jmc.common.IDisplayable;\r\nimport com.oracle.jmc.common.item.Aggregators;\r\nimport com.oracle.jmc.common.item.IItemCollection;\r\nimport com.oracle.jmc.common.item.ItemFilters;\r\nimport com.oracle.jmc.common.unit.IQuantity;\r\nimport com.oracle.jmc.flightrecorder.JfrAttributes;\r\nimport com.oracle.jmc.flightrecorder.JfrLoaderToolkit;\r\nimport com.oracle.jmc.flightrecorder.jdk.JdkTypeIDs;\r\n\r\n\/**\r\n * Finds out the standard deviation for the java monitor enter events.\r\n *\/\r\npublic class LoadRecording {\r\n\tpublic static void main(String[] args) throws Exception {\r\n\t\tDemoToolkit.verifyFirstFileArgument(RunRulesOnFile.class, args);\r\n\t\t\r\n\t\tIItemCollection events = JfrLoaderToolkit.loadEvents(new File(args[0]));\r\n\t\tIQuantity aggregate = events.apply(ItemFilters.type(JdkTypeIDs.MONITOR_ENTER))\r\n\t\t\t\t.getAggregate(Aggregators.stddev(JfrAttributes.DURATION));\r\n\t\t\r\n\t\tSystem.out.println("The standard deviation for the Java monitor enter events was "\r\n\t\t\t\t+ aggregate.displayUsing(IDisplayable.AUTO));\r\n\t}\r\n}\r\n<\/pre>\n<\/p>\n
\nNote the stream-like syntax. The JMC libraries work well together with streams, but are compiled on JDK 7, and can run on a JDK 7 compliant runtime. Also note that some class names start with Jfr<\/font> whilst others class names start with Jdk<\/font>. The difference being that some concepts, like DURATION, are intrinsic to JFR, whilst other are defined in terms of the Java JDK classes or runtime, like MONITOR_ENTER.<\/p>\n
The JMC core libraries provide common statistical aggregators, and accessors for common attributes. Should you feel something is missing, it is easy to add to the built-in operations. <\/p>\n
Also note the IQuantity returned by the aggregator. The built-in system for handling quantities and units of measurement makes it easy to calculate with, and display, quantities in different units of measurement.<\/p>\n
Performing an Automated Analysis Headless<\/h2>\n
It is also easy to run the JMC rules headless using these core libraries. Simply evaluate the rules against the IItemCollection<\/font>. Here is a simple example iterating through the rules and evaluating them one by one:<\/p>\n
\r\nimport java.io.File;\r\nimport java.util.concurrent.RunnableFuture;\r\n\r\nimport com.oracle.example.jmc6jfr.rules.util.DemoToolkit;\r\nimport com.oracle.jmc.common.item.IItemCollection;\r\nimport com.oracle.jmc.common.util.IPreferenceValueProvider;\r\nimport com.oracle.jmc.flightrecorder.JfrLoaderToolkit;\r\nimport com.oracle.jmc.flightrecorder.rules.IRule;\r\nimport com.oracle.jmc.flightrecorder.rules.Result;\r\nimport com.oracle.jmc.flightrecorder.rules.RuleRegistry;\r\n\r\npublic class RunRulesOnFileSimple {\r\n\tpublic static void main(String[] args) throws Exception {\r\n\t\tFile recording = DemoToolkit.verifyRecordingArgument(RunRulesOnFileSimple.class, args);\r\n\t\tIItemCollection events = JfrLoaderToolkit.loadEvents(recording);\r\n\t\t\r\n\t\tfor (IRule rule : RuleRegistry.getRules()) {\r\n\t\t\tRunnableFuture<Result> future = rule.evaluate(events, IPreferenceValueProvider.DEFAULT_VALUES);\r\n\t\t\tfuture.run();\r\n\t\t\tResult result = future.get();\r\n\t\t\tif (result.getScore() > 50) {\r\n\t\t\t\tSystem.out.println(String.format("[Score: %3.0f] Rule ID: %s, Rule name: %s, Short description: %s",\r\n\t\t\t\t\t\tresult.getScore(), result.getRule().getId(), result.getRule().getName(),\r\n\t\t\t\t\t\tresult.getShortDescription()));\r\n\t\t\t}\r\n\t\t}\r\n\t}\r\n}\r\n<\/pre>\n<\/p>\n
That said, if you are not constrained to have to run on JDK 7, you can always run the rules in parallel, for example by employing parallel streams:<\/p>\n
\r\nimport java.io.File;\r\nimport java.util.List;\r\nimport java.util.concurrent.ExecutionException;\r\nimport java.util.concurrent.Executor;\r\nimport java.util.concurrent.Executors;\r\nimport java.util.concurrent.RunnableFuture;\r\nimport java.util.stream.Collectors;\r\nimport java.util.stream.Stream;\r\n\r\nimport com.oracle.jmc.common.item.IItemCollection;\r\nimport com.oracle.jmc.common.util.IPreferenceValueProvider;\r\nimport com.oracle.jmc.flightrecorder.JfrLoaderToolkit;\r\nimport com.oracle.jmc.flightrecorder.rules.IRule;\r\nimport com.oracle.jmc.flightrecorder.rules.Result;\r\nimport com.oracle.jmc.flightrecorder.rules.RuleRegistry;\r\n\r\n\/**\r\n * Runs the rules on the events in the specified file in parallel, then prints\r\n * them in order of descending score.\r\n *\/\r\npublic class RunRulesOnFile {\r\n\tprivate final static Executor EXECUTOR = Executors\r\n\t\t\t.newFixedThreadPool(Runtime.getRuntime().availableProcessors() - 1);\r\n\tprivate static int limit;\r\n\r\n\tpublic static void main(String[] args) throws Exception {\r\n\t\tif (args.length == 0) {\r\n\t\t\tSystem.out.println(\r\n\t\t\t\t\t"Usage: RunRulesOnFile <recording file> [<limit>]\\n\\tThe recording file must be a flight recording from JDK 7 or above. The limit, if set, will only report rules triggered with a score higher or equal than the limit.");\r\n\t\t\tSystem.exit(2);\r\n\t\t}\r\n\t\tIItemCollection events = JfrLoaderToolkit.loadEvents(new File(args[0]));\r\n\t\tif (args.length > 1) {\r\n\t\t\tlimit = Integer.parseInt(args[1]);\r\n\t\t}\r\n\t\tStream<RunnableFuture<Result>> resultFutures = RuleRegistry.getRules().stream()\r\n\t\t\t\t.map((IRule r) -> evaluate(r, events));\r\n\t\tList<Result> results = resultFutures.parallel().map((RunnableFuture<Result> runnable) -> get(runnable))\r\n\t\t\t\t.collect(Collectors.toList());\r\n\t\tresults.sort((Result r1, Result r2) -> Double.compare(r2.getScore(), r1.getScore()));\r\n\t\tresults.stream().forEach(RunRulesOnFile::printResult);\r\n\t}\r\n\r\n\tpublic static RunnableFuture<Result> evaluate(IRule rule, IItemCollection events) {\r\n\t\tRunnableFuture<Result> evaluation = rule.evaluate(events, IPreferenceValueProvider.DEFAULT_VALUES);\r\n\t\tEXECUTOR.execute(evaluation);\r\n\t\treturn evaluation;\r\n\t}\r\n\r\n\tpublic static Result get(RunnableFuture<Result> resultFuture) {\r\n\t\ttry {\r\n\t\t\treturn resultFuture.get();\r\n\t\t} catch (InterruptedException | ExecutionException e) {\r\n\t\t\te.printStackTrace();\r\n\t\t}\r\n\t\treturn null;\r\n\t}\r\n\r\n\tprivate static void printResult(Result result) {\r\n\t\tif (result.getScore() >= limit) {\r\n\t\t\tSystem.out.printf("(%.0f) [%s]: %s\\nDetails:\\n%s\\n============<End of Result>============\\n",\r\n\t\t\t\t\tresult.getScore(), result.getRule().getId(), result.getShortDescription(),\r\n\t\t\t\t\tresult.getLongDescription() == null ? "<no description>" : result.getLongDescription());\r\n\t\t}\r\n\t}\r\n}\r\n<\/pre>\n<\/p>\n
If you do not need that kind of control, there is a class available for performing automated analysis included with the JMC core library. To run an automatic analysis, simply run the class:<\/p>\n
\n
com.oracle.jmc.flightrecorder.rules.report.JfrRulesReport <files> \u2013format <format> \u2013min <severity><\/p>\n<\/blockquote>\n
\u2026where files is one or more recordings, and <format> is  and <min> is minimum severity to report [ok | info | warning]. In JMC 6.1.0<\/em><\/strong> there is one additional class for generating the file, which produces an HTML in a format and style very similar to the looks of the JMC 6.0.0 Automated Analysis page:<\/p>\n
\n
com.oracle.jmc.flightrecorder.rules.report.html.JfrHtmlRulesReport <file> [<outputfile>]<\/p>\n<\/blockquote>\n
…where <file> is the (single) recording to analyze, and where the optional <outputfile> is where the resulting HTML should be written. If no outputfile is specified, the result of the analysis will be written on stdout.<\/p>\n
Adding Custom Rules<\/h2>\n
It’s easy to add you own heuristics. Do you find yourself with thousands of recordings, and would like to add a Bayesiean network to do some machine learning? The rules are all pure Java, so you can pretty much do whatever you want in a rule.<\/p>\n
The easiest way to get started writing your own custom rules is to get yourself an Eclipse Oxygen or later, and install JMC and the experimental PDE plug-in into your Eclipse. <\/p>\n
    \n
  1. Install Eclipse Oxygen or later.<\/li>\n
  2. Got to the Mission Control homepage<\/a>, and find the Eclipse Update site.<\/li>\n
  3. Follow the instructions to install the Mission Control plug-ins.
    \"image\"<\/a><\/li>\n
  4. Got to the Mission Control homepage<\/a>, and find the Experimental Update site.<\/li>\n
  5. Install the PDE plug-in.
    \"image\"<\/a><\/li>\n<\/ol>\n
    Here is a cheat sheet for the eclipse update sites for the JMC 6.0.0 release: <\/p>\n
    \n
    http:\/\/download.oracle.com\/technology\/products\/missioncontrol\/updatesites\/base\/6.0.0\/eclipse\/<\/a><\/p>\n
    http:\/\/download.oracle.com\/technology\/products\/missioncontrol\/updatesites\/supported\/6.0.0\/eclipse\/<\/a><\/p>\n
    http:\/\/download.oracle.com\/technology\/products\/missioncontrol\/updatesites\/experimental\/6.0.0\/eclipse\/<\/a><\/p>\n<\/blockquote>\n
    (JMC 6.1.0 will be released according to the same URL pattern.)<\/p>\n
    To start building your rule, press ctrl-n (or click the File | New | Other\u2026<\/font> menu) to bring up the New wizard. <\/p>\n
    \"image\"<\/a><\/p>\n
    Select Plug-in Project<\/font> and hit Next<\/font>. Name your rule project something exciting.<\/p>\n
    \"pic\"<\/a><\/p>\n
    Unclick that this plug-in will make contributions to the UI and hit Next.<\/p>\n
    \"pic2\"<\/a><\/p>\n
    Next select the Simple JFR Rule Wizard<\/font> and click Finish<\/font> (or Next<\/font>, if you really wish to do some further customizations).<\/p>\n
    \"pic3\"<\/a><\/p>\n
    You will now have a new project in your workspace, containing an example rule. If you have compilation errors, you need to set JMC to be your target platform (see the next section). You can either just export your rule, put it on the class path with the other core libraries, or you can try out your rule by running JMC from within Eclipse with your new rule. <\/p>\n
    <\/h2>\n
    Starting JMC from within Eclipse with Workspace Plug-ins<\/h3>\n
    Running JMC from within Eclipse with any plug-ins you are currently developing is a simple matter of setting the plug-in development target platform to your JMC installation, and launching that platform with your workspace plug-ins.<\/p>\n
    First set JMC to be your target platform.<\/p>\n
      \n
    1. Go to Preferences<\/font> in the main menu (Window | Preferences<\/font> on Windows).<\/li>\n
    2. Find Target Platform<\/font> by typing Tar in the filter box:
      \"image\"<\/a><\/li>\n
    3. Click Add\u2026 and type JMC 6 as name.
      \"image\"<\/a><\/li>\n
    4. Press Add\u2026 and select Installation<\/font>. Hit Next. 
      \"image\"<\/a><\/li>\n
    5. Browse<\/font> to the JMC 6 installation directory (JDK_9_HOME\/lib\/missioncontrol), and hit Ok<\/font>.
      \"SNAGHTML1538a63d\"<\/a><\/li>\n
    6. Hit Finish<\/font>, and Finish<\/font>. You should now see your new platform.
      \"image\"<\/a><\/li>\n
    7. Select the new JMC 6 platform, and hit Apply and Close to activate it.
      \"image\"<\/a><\/li>\n<\/ol>\n
      Everything should now compile cleanly. Next step is to run JMC with your new rule.<\/p>\n
        \n
      1. Context click on your project and select Run as Eclipse Application
        \"SNAGHTML153e5cb4\"<\/a><\/li>\n
      2. This should normally be it. If you for some reason run into trouble, go to Run Configurations<\/font>, and make sure that your launcher is using the com.oracle.jmc.rcp.application.product<\/font>.
        \"SNAGHTML1540e14b\"<\/a><\/li>\n<\/ol>\n
        You should now be able to see your rule in action. If you have not changed the rule code, try setting the environment variable in the launcher to various values between 0 and 100 and see what happens when you run the rule in JMC:<\/p>\n
        \"image\"<\/a><\/p>\n
        \"image\"<\/a><\/p>\n
        <\/p>\n
        Exporting a Plug-in<\/h3>\n
        You can export your rule by context-clicking on the project and selecting Export. In the Export wizard, select Deployable plug-ins and Fragments.<\/p>\n
        \"image\"<\/a> <\/p>\n
        Click Next<\/font>, select a folder to export the plug-in to and Finish<\/font>. The resulting jar can either be put on the class path to be included in the headless analysis, or put in the dropins folder of any Java Mission Control installation where you would want the rule to be available.<\/p>\n
        <\/p>\n
        Summary<\/h1>\n
        This blog described how to:<\/p>\n
          \n
        • Get started using the JMC core libraries to read Java Flight Recordings (JDK 7, 8 and 9)<\/li>\n
        • Get started doing headless analysis of Java Flight Recordings (JDK 7, 8 and 9)<\/li>\n
        • Get started creating custom rules for analyzing Java Flight Recordings<\/li>\n<\/ul>\n
          Hope it helps someone!<\/p>\n","protected":false},"excerpt":{"rendered":"
          This post will be a little bit wider than the title implies; it will be about using the JMC core JFR APIs in general. That said, one of the things we will be using it for, towards the end, will be to run the JMC 6 automated analysis headless. This article will also cheat by […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[17,16,13,15],"class_list":["post-920","post","type-post","status-publish","format-standard","hentry","category-java-mission-control-fair-dj","tag-java-flight-recorder","tag-java-mission-control","tag-jfr","tag-jmc"],"_links":{"self":[{"href":"https:\/\/hirt.se\/blog\/index.php?rest_route=\/wp\/v2\/posts\/920","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/hirt.se\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hirt.se\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hirt.se\/blog\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/hirt.se\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=920"}],"version-history":[{"count":10,"href":"https:\/\/hirt.se\/blog\/index.php?rest_route=\/wp\/v2\/posts\/920\/revisions"}],"predecessor-version":[{"id":930,"href":"https:\/\/hirt.se\/blog\/index.php?rest_route=\/wp\/v2\/posts\/920\/revisions\/930"}],"wp:attachment":[{"href":"https:\/\/hirt.se\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=920"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hirt.se\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=920"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hirt.se\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=920"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}