{"id":1166,"date":"2019-11-09T11:00:00","date_gmt":"2019-11-09T10:00:00","guid":{"rendered":"http:\/\/hirt.se\/blog\/?p=1166"},"modified":"2019-11-25T23:09:04","modified_gmt":"2019-11-25T22:09:04","slug":"compressing-jfr-2-0-recordings","status":"publish","type":"post","link":"https:\/\/hirt.se\/blog\/?p=1166","title":{"rendered":"Compressing Flight Recordings"},"content":{"rendered":"<p>Flight recordings are nifty binary recordings of what is going on in the runtime and the application running on it. A flight recording contains a wide variety of information, such as various kinds of profiling information, threat stall information and a whole host of other information. All adhering to a common event model and with the ability to dynamically add new event types.<\/p>\n<p>In the versions of JFR since JDK 9, some care was taken to reduce the memory footprint by <a href=\"https:\/\/en.wikipedia.org\/wiki\/LEB128\">LEB 128<\/a> encoding integers, noting that many things, like constant pool indices, usually occupy relatively low numbers. The memory footprint was cut in about half, compared to previous versions of JFR.<\/p>\n<p>Now, sometimes you may want to compress the JFR data even further. The question then is \u2013 how much can you save if you compress the recordings further, and what algorithms would be best suited for doing the compression? What if you want the compression activity to use as little CPU as possible?<\/p>\n<p>My friend and colleague at Datadog, <a href=\"https:\/\/twitter.com\/yardus\">Jaroslav Bachorik<\/a>, set out to answer that question for some typical recording shapes that we see at Datadog, using a set of compression algorithms from <a href=\"https:\/\/commons.apache.org\/proper\/commons-compress\/\">Apache Commons Compress<\/a> (bzip2, LZMA, LZ4), the built in <a href=\"https:\/\/docs.oracle.com\/javase\/8\/docs\/api\/index.html?java\/util\/zip\/package-summary.html\">GZip<\/a>, a dedicated <a href=\"https:\/\/github.com\/lz4\/lz4-java\">LZ4 library<\/a>, <a href=\"https:\/\/github.com\/mstevens83\/xz-java\">XZ<\/a>, and <a href=\"https:\/\/xerial.org\/snappy-java\/\">Snappy<\/a>.<\/p>\n<p>Below is a table of his findings for \u201csmall\u201d (~1.5 MiB) and \u201clarge\u201d (~5 MiB) recordings from one of our services. The benchmark was run on a MacBook Pro 2019. Now, you\u2019d have to test on your own recordings to truly know, but I suspect that these results will hold up pretty well with other kinds of loads as well.<\/p>\n<table border=\"1\" cellspacing=\"0\" cellpadding=\"2\">\n<tbody>\n<tr bgcolor=\"#505050\">\n<th valign=\"top\"><span style=\"color: #ffffff\">Algorithm<\/span><\/th>\n<th valign=\"top\"><span style=\"color: #ffffff\">Recording Size<\/span><\/th>\n<th valign=\"top\"><span style=\"color: #ffffff\">Throughput<\/span><\/th>\n<th valign=\"top\"><span style=\"color: #ffffff\">Compression Ratio<\/span><\/th>\n<th valign=\"top\"><span style=\"color: #ffffff\">Utility<\/span><\/th>\n<\/tr>\n<tr>\n<td valign=\"top\">Gzip<\/td>\n<td valign=\"top\">small<\/td>\n<td valign=\"top\">24.299<\/td>\n<td valign=\"top\">3.28<\/td>\n<td valign=\"top\">79.647<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\">Gzip<\/td>\n<td valign=\"top\">large<\/td>\n<td valign=\"top\">5.762<\/td>\n<td valign=\"top\">3.54<\/td>\n<td valign=\"top\">20.436<\/td>\n<\/tr>\n<tr bgcolor=\"#dddddd\">\n<td valign=\"top\">BZip2<\/td>\n<td valign=\"top\">small<\/td>\n<td valign=\"top\">6.616<\/td>\n<td valign=\"top\">3.51<\/td>\n<td valign=\"top\">23.198<\/td>\n<\/tr>\n<tr bgcolor=\"#dddddd\">\n<td valign=\"top\">BZip2<\/td>\n<td valign=\"top\">large<\/td>\n<td valign=\"top\">1.518<\/td>\n<td valign=\"top\">3.84<\/td>\n<td valign=\"top\">5.826<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\">LZ4<\/td>\n<td valign=\"top\">small<\/td>\n<td valign=\"top\">133.115<\/td>\n<td valign=\"top\">2.40<\/td>\n<td valign=\"top\"><span style=\"background-color: #ffff00\">319.394<\/span><\/td>\n<\/tr>\n<tr>\n<td valign=\"top\">LZ4<\/td>\n<td valign=\"top\">large<\/td>\n<td valign=\"top\"><span style=\"background-color: #ffff00\">38.901<\/span><\/td>\n<td valign=\"top\">2.57<\/td>\n<td valign=\"top\"><span style=\"background-color: #ffff00\">100.009<\/span><\/td>\n<\/tr>\n<tr bgcolor=\"#dddddd\">\n<td valign=\"top\">LZ4 (Apache)<\/td>\n<td valign=\"top\">small<\/td>\n<td valign=\"top\">0.055<\/td>\n<td valign=\"top\">2.74<\/td>\n<td valign=\"top\">0.152<\/td>\n<\/tr>\n<tr bgcolor=\"#dddddd\">\n<td valign=\"top\">LZ4 (Apache)<\/td>\n<td valign=\"top\">large<\/td>\n<td valign=\"top\">0.013<\/td>\n<td valign=\"top\">3.00<\/td>\n<td valign=\"top\">0.039<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\">LZMA<\/td>\n<td valign=\"top\">small<\/td>\n<td valign=\"top\">1.828<\/td>\n<td valign=\"top\"><span style=\"background-color: #ffff00\">4.31<\/span><\/td>\n<td valign=\"top\">7.882<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\">LZMA<\/td>\n<td valign=\"top\">large<\/td>\n<td valign=\"top\">0.351<\/td>\n<td valign=\"top\"><span style=\"background-color: #ffff00\">4.37<\/span><\/td>\n<td valign=\"top\">1.533<\/td>\n<\/tr>\n<tr bgcolor=\"#dddddd\">\n<td valign=\"top\">Snappy<\/td>\n<td valign=\"top\">small<\/td>\n<td valign=\"top\"><span style=\"background-color: #ffff00\">134.598<\/span><\/td>\n<td valign=\"top\">2.27<\/td>\n<td valign=\"top\">305.494<\/td>\n<\/tr>\n<tr bgcolor=\"#dddddd\">\n<td valign=\"top\">Snappy<\/td>\n<td valign=\"top\">large<\/td>\n<td valign=\"top\">35.986<\/td>\n<td valign=\"top\">2.49<\/td>\n<td valign=\"top\">89.692<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\">XZ<\/td>\n<td valign=\"top\">small<\/td>\n<td valign=\"top\">1.847<\/td>\n<td valign=\"top\"><span style=\"background-color: #ffff00\">4.31<\/span><\/td>\n<td valign=\"top\">7.964<\/td>\n<\/tr>\n<tr>\n<td valign=\"top\">XZ<\/td>\n<td valign=\"top\">large<\/td>\n<td valign=\"top\">0.349<\/td>\n<td valign=\"top\"><span style=\"background-color: #ffff00\">4.37<\/span><\/td>\n<td valign=\"top\">1.523<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Throughput is <span style=\"font-family: Courier New\">recordings\/s<\/span>. Utility is <span style=\"font-family: Courier New\">throughput * compression ratio<\/span>, and meant to capture the combination of compression strength and performance. Note that the numbers are not normalized \u2013 only compare numbers in the same size category.<\/p>\n<h2>Summary \/ TL;DR<\/h2>\n<ul>\n<li>The built-in GZip is doing a fairly good\/balanced job of compressing flight recordings<\/li>\n<li>You can get the best utility out of LZ4, closely followed by Snappy, but you sacrifice some compression<\/li>\n<li>If you\u2019re prepared to pay for it, LZMA and XZ give a good compression ratio<\/li>\n<li>All credz to <a href=\"https:\/\/twitter.com\/yardus\">Jaroslav<\/a> for his <a href=\"https:\/\/openjdk.java.net\/projects\/code-tools\/jmh\/\">JMH-benchmark<\/a> and all the data<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Flight recordings are nifty binary recordings of what is going on in the runtime and the application running on it. A flight recording contains a wide variety of information, such as various kinds of profiling information, threat stall information and a whole host of other information. All adhering to a common event model and with [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[12,17,23,13],"class_list":["post-1166","post","type-post","status-publish","format-standard","hentry","category-java-mission-control-fair-dj","tag-java","tag-java-flight-recorder","tag-jdk-flight-recorder","tag-jfr"],"_links":{"self":[{"href":"https:\/\/hirt.se\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1166","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/hirt.se\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hirt.se\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hirt.se\/blog\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/hirt.se\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1166"}],"version-history":[{"count":12,"href":"https:\/\/hirt.se\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1166\/revisions"}],"predecessor-version":[{"id":1181,"href":"https:\/\/hirt.se\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1166\/revisions\/1181"}],"wp:attachment":[{"href":"https:\/\/hirt.se\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1166"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hirt.se\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1166"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hirt.se\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1166"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}