| <?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en-US"><generator uri="https://jekyllrb.com/" version="3.9.2">Jekyll</generator><link href="http://nemo.apache.org//feed.xml" rel="self" type="application/atom+xml" /><link href="http://nemo.apache.org//" rel="alternate" type="text/html" hreflang="en-US" /><updated>2022-09-10T00:27:36+09:00</updated><id>http://nemo.apache.org//feed.xml</id><title type="html">Nemo</title><subtitle>A Data Processing System for Flexible Employment With Different Deployment Characteristics. |
| </subtitle><entry><title type="html">Nemo Release 0.4</title><link href="http://nemo.apache.org//blog/2022/09/10/release-note-0.4/" rel="alternate" type="text/html" title="Nemo Release 0.4" /><published>2022-09-10T00:00:00+09:00</published><updated>2022-09-10T00:00:00+09:00</updated><id>http://nemo.apache.org//blog/2022/09/10/release-note-0.4</id><content type="html" xml:base="http://nemo.apache.org//blog/2022/09/10/release-note-0.4/"><p>Release Notes - Apache Nemo - Version 0.4</p> |
| |
| <h3 id="heres-a-brief-summary-about-the-release">Here’s a brief summary about the release:</h3> |
| |
| <ul> |
| <li>This release fixes a few high-priority bugs in 0.3 and has a variety |
| of smaller fixes.</li> |
| <li>The release also includes several major functionalities regarding |
| stream processing.</li> |
| <li>Installation script for nemo prerequisites for easier usage by new users</li> |
| <li>Cleanup of unused stream metrics</li> |
| <li>Addition of critical stream metrics (e.g. latency, throughput)</li> |
| <li>Addition of examples of stream applications</li> |
| <li>Scripts for network profiling within the Nemo cluster</li> |
| <li>Bump library versions to react to security risks and keep libraries |
| up-to-date with new features</li> |
| <li>Fix the heisenbug regarding communication through stream pipe channels</li> |
| <li>Upgrading the CI with new platforms</li> |
| </ul> |
| |
| <h2> Sub-task |
| </h2> |
| <ul> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-398">NEMO-398</a>] - ExecutorRepresenter interface and LambdaExecutorRepresenter |
| </li> |
| </ul> |
| |
| <h2> Bug |
| </h2> |
| <ul> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-299">NEMO-299</a>] - WindowedWordCountITCase Hangs (Heisenbug) |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-460">NEMO-460</a>] - Setting coders in CombinePerKey transformation |
| </li> |
| </ul> |
| |
| <h2> New Feature |
| </h2> |
| <ul> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-481">NEMO-481</a>] - Add Stream examples |
| </li> |
| </ul> |
| |
| <h2> Improvement |
| </h2> |
| <ul> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-394">NEMO-394</a>] - Exchange data via shared memory when two tasks are in the same Executor in streaming |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-429">NEMO-429</a>] - Dec 2nd, 2019 Code Session |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-434">NEMO-434</a>] - Logical DAG modification for Dynamic sampling of task metrics during the execution of a stage |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-436">NEMO-436</a>] - Dynamic re-configuration based on the Sampled Metric Data |
| </li> |
| </ul> |
| |
| <h2> Task |
| </h2> |
| <ul> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-422">NEMO-422</a>] - SonarCloud issues |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-440">NEMO-440</a>] - Migrate to Java11 and use Java 11 Features |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-482">NEMO-482</a>] - Keep library versions updated |
| </li> |
| </ul> |
| |
| <h2> Umbrella |
| </h2> |
| <ul> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-483">NEMO-483</a>] - Record Metrics associated with stream processing |
| </li> |
| </ul></content><author><name>Won Wook SONG</name></author><summary type="html">Release Notes - Apache Nemo - Version 0.4</summary></entry><entry><title type="html">Nemo Release 0.3</title><link href="http://nemo.apache.org//blog/2022/06/07/release-note-0.3/" rel="alternate" type="text/html" title="Nemo Release 0.3" /><published>2022-06-07T00:00:00+09:00</published><updated>2022-06-07T00:00:00+09:00</updated><id>http://nemo.apache.org//blog/2022/06/07/release-note-0.3</id><content type="html" xml:base="http://nemo.apache.org//blog/2022/06/07/release-note-0.3/"><p>Release Notes - Apache Nemo - Version 0.3</p> |
| |
| <h3 id="heres-a-brief-summary-about-the-release">Here’s a brief summary about the release:</h3> |
| |
| <ul> |
| <li>This release fixes a few high-priority bugs in 0.2 and has a variety of smaller fixes.</li> |
| <li>The release also includes several major functionalities regarding stream processing and dynamic optimizations, such as dynamic task sizing, as well as implementation of the simulation scheduler.</li> |
| </ul> |
| |
| <h2> Sub-task |
| </h2> |
| <ul> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-351">NEMO-351</a>] - Empowering Nemo with fast I/O using Apache Crail |
| </li> |
| </ul> |
| |
| <h2> Bug |
| </h2> |
| <ul> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-377">NEMO-377</a>] - Fix watermark emission when there are no outputs in GBKWindowTransform |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-411">NEMO-411</a>] - Bug in ScheduleGroupPass, OutputTag, DuplicateEdgeGroup |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-447">NEMO-447</a>] - Fix beam pom.xml to resolve build failure |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-457">NEMO-457</a>] - Improve test coverage score on sonar cloud test |
| </li> |
| </ul> |
| |
| <h2> New Feature |
| </h2> |
| <ul> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-438">NEMO-438</a>] - Create a Simulator for Simulating an Execution of an Execution Plan |
| </li> |
| </ul> |
| |
| <h2> Improvement |
| </h2> |
| <ul> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-324">NEMO-324</a>] - Distinguish Beam&#39;s run and waitUntilFinish methods. |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-392">NEMO-392</a>] - Support combine in streaming |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-433">NEMO-433</a>] - Improvement of Task Metrics and Collecting Them For Sampling |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-439">NEMO-439</a>] - Upgrade current working version to 0.3-SNAPSHOT |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-459">NEMO-459</a>] - Enable Automatic Analysis for sonar cloud |
| </li> |
| </ul></content><author><name>Won Wook SONG</name></author><summary type="html">Release Notes - Apache Nemo - Version 0.3</summary></entry><entry><title type="html">Nemo Release 0.2</title><link href="http://nemo.apache.org//blog/2020/03/09/release-note-0.2/" rel="alternate" type="text/html" title="Nemo Release 0.2" /><published>2020-03-09T00:00:00+09:00</published><updated>2020-03-09T00:00:00+09:00</updated><id>http://nemo.apache.org//blog/2020/03/09/release-note-0.2</id><content type="html" xml:base="http://nemo.apache.org//blog/2020/03/09/release-note-0.2/"><p>Release Notes - Apache Nemo - Version 0.2</p> |
| |
| <h2> Sub-task |
| </h2> |
| <ul> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-350">NEMO-350</a>] - Implement Off-heap SerializedMemoryStore |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-385">NEMO-385</a>] - Support Lambda Pass with lambda policy and lambda resource property |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-387">NEMO-387</a>] - Support Lambda scheduler |
| </li> |
| </ul> |
| |
| <h2> Bug |
| </h2> |
| <ul> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-80">NEMO-80</a>] - SLF4J: Failed to load class “org.slf4j.impl.StaticLoggerBinder” |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-304">NEMO-304</a>] - Fail-fast for mis-configuration in user application |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-321">NEMO-321</a>] - Fix the data skew pass metric mismatch |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-327">NEMO-327</a>] - Fix skew handling for multi shuffle edge receiver |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-357">NEMO-357</a>] - Fix broken link on README |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-368">NEMO-368</a>] - NEMO-353 breaks the application from running in YARN environments |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-402">NEMO-402</a>] - Broken guava version conflicts cause ERROR: Trying to remove a RunningJob that is unknown |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-413">NEMO-413</a>] - Fix index checking for byte access of MemoryChunk using UNSAFE |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-414">NEMO-414</a>] - Command-line specified runtime data plane configurations not applied |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-416">NEMO-416</a>] - Guava vendor version conflict when deserializing Task object |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-420">NEMO-420</a>] - OffHeapMemory configuration only supports a single type of executor |
| </li> |
| </ul> |
| |
| <h2> New Feature |
| </h2> |
| <ul> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-335">NEMO-335</a>] - Using a database for recording metric data |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-336">NEMO-336</a>] - Cost prediction using the metric data |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-359">NEMO-359</a>] - implementation of getEstimatedSizeBytes in SourceVertex |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-360">NEMO-360</a>] - Implementing an &#39;XGBoostPolicy&#39; |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-388">NEMO-388</a>] - Off-heap memory management (reuse ByteBuffer) |
| </li> |
| </ul> |
| |
| <h2> Improvement |
| </h2> |
| <ul> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-11">NEMO-11</a>] - Generalize Equality of Int Predicates for Loops |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-162">NEMO-162</a>] - Add insertVertex() API in optimization pass |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-182">NEMO-182</a>] - Consider reshaping in run-time optimization |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-253">NEMO-253</a>] - Refactor getInternal(Main/Additional)OutputMap in TaskExecutor |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-275">NEMO-275</a>] - Eager Garbage Collection for GroupByKey |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-319">NEMO-319</a>] - Fix path to beam resources in examples in README |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-320">NEMO-320</a>] - Make WebUI scale to big workloads |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-323">NEMO-323</a>] - Upgrade current working version to 0.2-SNAPSHOT |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-328">NEMO-328</a>] - Refactor IRDAG |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-332">NEMO-332</a>] - Refactor RunTimePass |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-337">NEMO-337</a>] - IRDAG Unit Tests |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-338">NEMO-338</a>] - SkewSamplingPass |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-340">NEMO-340</a>] - SonarCloud for PRs |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-353">NEMO-353</a>] - Launch NEXMark applications |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-356">NEMO-356</a>] - Visualize the name of beam transform in DAG |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-358">NEMO-358</a>] - Recycling vertex ids while cloning a vertex |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-369">NEMO-369</a>] - DirectByteArrayOutputStream usage refactoring |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-390">NEMO-390</a>] - Address SonarCloud issues for the IR package |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-391">NEMO-391</a>] - Set GrpcMessageEnvironment as a default implementation |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-395">NEMO-395</a>] - Address SonarCloud issues for the scheduler package |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-404">NEMO-404</a>] - Provide user argument to use lambda executor representer |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-412">NEMO-412</a>] - Address Sonar Cloud issue for MemoryChunk |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-418">NEMO-418</a>] - BlockFetchFailureProperty |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-424">NEMO-424</a>] - Fix Sonarcloud bugs regarding Optional |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-428">NEMO-428</a>] - Ignore .factorypath for rat check and version control |
| </li> |
| </ul> |
| |
| <h2> Task |
| </h2> |
| <ul> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-25">NEMO-25</a>] - Improve WebUI to use RESTful APIs by Nemo |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-153">NEMO-153</a>] - IR-based dynamic optimization for WordCount application |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-154">NEMO-154</a>] - Handle skewness information in SchedulingConstraint |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-193">NEMO-193</a>] - Revised version of IR-based dynamic optimization |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-322">NEMO-322</a>] - Committer&#39;s guide |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-361">NEMO-361</a>] - Consistency on indentations |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-362">NEMO-362</a>] - Upgrade of checkstyle version |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-364">NEMO-364</a>] - Upgrade Beam |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-375">NEMO-375</a>] - Add option to turn off metric collection to DB |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-379">NEMO-379</a>] - Change javadoc goal to a proper one |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-383">NEMO-383</a>] - Implement DirectByteBufferOutputStream for Off-heap SerializedMemoryStore |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-384">NEMO-384</a>] - Implement DirectByteBufferInputStream for Off-heap SerializedMemoryStore |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-397">NEMO-397</a>] - Separation of JVM heap region and off-heap memory region |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-399">NEMO-399</a>] - Include the official WordCount example on the Beam website |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-400">NEMO-400</a>] - Javadoc compile error |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-421">NEMO-421</a>] - Release v0.2 |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-437">NEMO-437</a>] - Support Java version 11 |
| </li> |
| </ul> |
| |
| <h2> Umbrella |
| </h2> |
| <ul> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-60">NEMO-60</a>] - IR-based dynamic optimization |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-156">NEMO-156</a>] - Support Beam Nemo Runner |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-157">NEMO-157</a>] - Support Nemo Streaming |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-158">NEMO-158</a>] - Support Spark SQL Example |
| </li> |
| <li>[<a href="https://issues.apache.org/jira/browse/NEMO-159">NEMO-159</a>] - Nemo Web UI |
| </li> |
| </ul> |
| |
| <h3 id="list-of-contributors">List of Contributors</h3> |
| |
| <p>Nemo 0.2 was the work of many contributors:</p> |
| |
| <ul> |
| <li>Arun Lakshman R</li> |
| <li>Byung-Gon Chun</li> |
| <li>Davor Bonaci</li> |
| <li>Eunji Jeong</li> |
| <li>Geon Woo Kim</li> |
| <li>Gyewon Lee</li> |
| <li>Haeyoon Cho</li> |
| <li>Jae Hyeon Park</li> |
| <li>JangHo Seo</li> |
| <li>Jangho Seo</li> |
| <li>Jean-Baptiste Onofré</li> |
| <li>Jeongyoon Eo</li> |
| <li>John Yang</li> |
| <li>Joo Yeon Kim</li> |
| <li>Kenn Knowles</li> |
| <li>Markus Weimer</li> |
| <li>Minhyeok Kweun</li> |
| <li>Sanha Lee</li> |
| <li>Seonghyun Park</li> |
| <li>Soojeong Kim</li> |
| <li>Taegeon Um</li> |
| <li>Won Wook SONG</li> |
| <li>Wooyeon Lee</li> |
| <li>Yunseong Lee</li> |
| </ul></content><author><name>Wooyeon Lee</name></author><summary type="html">Release Notes - Apache Nemo - Version 0.2 Sub-task [NEMO-350] - Implement Off-heap SerializedMemoryStore [NEMO-385] - Support Lambda Pass with lambda policy and lambda resource property [NEMO-387] - Support Lambda scheduler Bug [NEMO-80] - SLF4J: Failed to load class “org.slf4j.impl.StaticLoggerBinder” [NEMO-304] - Fail-fast for mis-configuration in user application [NEMO-321] - Fix the data skew pass metric mismatch [NEMO-327] - Fix skew handling for multi shuffle edge receiver [NEMO-357] - Fix broken link on README [NEMO-368] - NEMO-353 breaks the application from running in YARN environments [NEMO-402] - Broken guava version conflicts cause ERROR: Trying to remove a RunningJob that is unknown [NEMO-413] - Fix index checking for byte access of MemoryChunk using UNSAFE [NEMO-414] - Command-line specified runtime data plane configurations not applied [NEMO-416] - Guava vendor version conflict when deserializing Task object [NEMO-420] - OffHeapMemory configuration only supports a single type of executor New Feature [NEMO-335] - Using a database for recording metric data [NEMO-336] - Cost prediction using the metric data [NEMO-359] - implementation of getEstimatedSizeBytes in SourceVertex [NEMO-360] - Implementing an &#39;XGBoostPolicy&#39; [NEMO-388] - Off-heap memory management (reuse ByteBuffer) Improvement [NEMO-11] - Generalize Equality of Int Predicates for Loops [NEMO-162] - Add insertVertex() API in optimization pass [NEMO-182] - Consider reshaping in run-time optimization [NEMO-253] - Refactor getInternal(Main/Additional)OutputMap in TaskExecutor [NEMO-275] - Eager Garbage Collection for GroupByKey [NEMO-319] - Fix path to beam resources in examples in README [NEMO-320] - Make WebUI scale to big workloads [NEMO-323] - Upgrade current working version to 0.2-SNAPSHOT [NEMO-328] - Refactor IRDAG [NEMO-332] - Refactor RunTimePass [NEMO-337] - IRDAG Unit Tests [NEMO-338] - SkewSamplingPass [NEMO-340] - SonarCloud for PRs [NEMO-353] - Launch NEXMark applications [NEMO-356] - Visualize the name of beam transform in DAG [NEMO-358] - Recycling vertex ids while cloning a vertex [NEMO-369] - DirectByteArrayOutputStream usage refactoring [NEMO-390] - Address SonarCloud issues for the IR package [NEMO-391] - Set GrpcMessageEnvironment as a default implementation [NEMO-395] - Address SonarCloud issues for the scheduler package [NEMO-404] - Provide user argument to use lambda executor representer [NEMO-412] - Address Sonar Cloud issue for MemoryChunk [NEMO-418] - BlockFetchFailureProperty [NEMO-424] - Fix Sonarcloud bugs regarding Optional [NEMO-428] - Ignore .factorypath for rat check and version control Task [NEMO-25] - Improve WebUI to use RESTful APIs by Nemo [NEMO-153] - IR-based dynamic optimization for WordCount application [NEMO-154] - Handle skewness information in SchedulingConstraint [NEMO-193] - Revised version of IR-based dynamic optimization [NEMO-322] - Committer&#39;s guide [NEMO-361] - Consistency on indentations [NEMO-362] - Upgrade of checkstyle version [NEMO-364] - Upgrade Beam [NEMO-375] - Add option to turn off metric collection to DB [NEMO-379] - Change javadoc goal to a proper one [NEMO-383] - Implement DirectByteBufferOutputStream for Off-heap SerializedMemoryStore [NEMO-384] - Implement DirectByteBufferInputStream for Off-heap SerializedMemoryStore [NEMO-397] - Separation of JVM heap region and off-heap memory region [NEMO-399] - Include the official WordCount example on the Beam website [NEMO-400] - Javadoc compile error [NEMO-421] - Release v0.2 [NEMO-437] - Support Java version 11 Umbrella [NEMO-60] - IR-based dynamic optimization [NEMO-156] - Support Beam Nemo Runner [NEMO-157] - Support Nemo Streaming [NEMO-158] - Support Spark SQL Example [NEMO-159] - Nemo Web UI</summary></entry><entry><title type="html">Nemo Release 0.1</title><link href="http://nemo.apache.org//blog/2019/03/02/release-note-0.1/" rel="alternate" type="text/html" title="Nemo Release 0.1" /><published>2019-03-02T00:00:00+09:00</published><updated>2019-03-02T00:00:00+09:00</updated><id>http://nemo.apache.org//blog/2019/03/02/release-note-0.1</id><content type="html" xml:base="http://nemo.apache.org//blog/2019/03/02/release-note-0.1/"><p>Nemo 0.1 is an initial release that brings several features and performance enhancements. The most visible features are the frontend support for Beam and Spark; IR DAG with execution properties, optimization passes and policies; support for loops; dynamic optimization; multiple-job submission; and various features of the execution runtime.</p> |
| |
| <p>You can view the JIRA-generated release note here: <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12321823&amp;version=12344546">link</a>.</p> |
| |
| <h2 id="new-features--improvements">New features / Improvements</h2> |
| |
| <h3 id="frontend">Frontend</h3> |
| |
| <ul> |
| <li>Beam</li> |
| <li>Official Support for Apache Beam (details available on <a href="https://beam.apache.org/documentation/runners/nemo/">this link</a>)</li> |
| <li>Spark</li> |
| </ul> |
| |
| <h3 id="core">Core</h3> |
| |
| <ul> |
| <li>Intermediate representation (IR)</li> |
| <li>Execution properties</li> |
| <li>Passes</li> |
| <li>Policies</li> |
| <li>Loop support</li> |
| <li>Dynamic optimization</li> |
| <li>Multi-job submission</li> |
| </ul> |
| |
| <h3 id="execution-runtime">Execution Runtime</h3> |
| |
| <ul> |
| <li>Data stores</li> |
| <li>Metadata management</li> |
| <li>Inter-executor data transfer (memory/disk/gluserFS for batch, pipe for streaming)</li> |
| <li>Data encoding &amp; decoding</li> |
| <li>Partitioners</li> |
| <li>Block manager</li> |
| <li>Data location based scheduling</li> |
| <li>Scheduler (based on different execution properties including locality, site, anti affinity, …)</li> |
| <li>RDD caching</li> |
| </ul> |
| |
| <h3 id="other-components">Other Components</h3> |
| <ul> |
| <li>Web UI</li> |
| </ul> |
| |
| <h3 id="known-issues">Known Issues</h3> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/NEMO-302?jql=project%20%3D%20NEMO%20AND%20issuetype%20%3D%20Bug%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20key%20DESC">A number of bugs &amp; unresolved JIRA issues</a></li> |
| </ul> |
| |
| <h3 id="miscellaneous-fixes">Miscellaneous Fixes</h3> |
| <ul> |
| <li>Website</li> |
| <li>CI integration</li> |
| <li>Documentations</li> |
| </ul> |
| |
| <h2 id="maven-artifacts">Maven Artifacts</h2> |
| <p>Nemo is available in Maven Central, making it easier to link into your programs without having to build as a JAR. Use the following Maven identifiers to add it to a project:</p> |
| |
| <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>&lt;!-- https://mvnrepository.com/artifact/org.apache.nemo/nemo-project --&gt; |
| &lt;dependency&gt; |
| &lt;groupId&gt;org.apache.nemo&lt;/groupId&gt; |
| &lt;artifactId&gt;nemo-project&lt;/artifactId&gt; |
| &lt;version&gt;0.1&lt;/version&gt; |
| &lt;type&gt;pom&lt;/type&gt; |
| &lt;/dependency&gt; |
| </code></pre></div></div> |
| |
| <h3 id="list-of-contributors">List of Contributors</h3> |
| |
| <p>Nemo 0.1 was the work of many contributors:</p> |
| |
| <ul> |
| <li>Arun Lakshman R</li> |
| <li>Byung-Gon Chun</li> |
| <li>Davor Bonaci</li> |
| <li>Eunji Jeong</li> |
| <li>Geon Woo Kim</li> |
| <li>Gyewon Lee</li> |
| <li>Jae Hyeon Park</li> |
| <li>JangHo Seo</li> |
| <li>Jangho Seo</li> |
| <li>Jean-Baptiste Onofré</li> |
| <li>Jeongyoon Eo</li> |
| <li>John Yang</li> |
| <li>Joo Yeon Kim</li> |
| <li>Kenn Knowles</li> |
| <li>Markus Weimer</li> |
| <li>Minhyeok Kweun</li> |
| <li>Sanha Lee</li> |
| <li>Seonghyun Park</li> |
| <li>Soojeong Kim</li> |
| <li>Taegeon Um</li> |
| <li>Won Wook SONG</li> |
| <li>Wooyeon Lee</li> |
| <li>Yunseong Lee</li> |
| </ul></content><author><name>Won Wook SONG</name></author><summary type="html">Nemo 0.1 is an initial release that brings several features and performance enhancements. The most visible features are the frontend support for Beam and Spark; IR DAG with execution properties, optimization passes and policies; support for loops; dynamic optimization; multiple-job submission; and various features of the execution runtime.</summary></entry><entry><title type="html">Harnessing transient resources using Nemo</title><link href="http://nemo.apache.org//blog/2018/03/23/pado-on-nemo/" rel="alternate" type="text/html" title="Harnessing transient resources using Nemo" /><published>2018-03-23T00:00:00+09:00</published><updated>2018-03-23T00:00:00+09:00</updated><id>http://nemo.apache.org//blog/2018/03/23/pado-on-nemo</id><content type="html" xml:base="http://nemo.apache.org//blog/2018/03/23/pado-on-nemo/"><p>To increase datacenter utilization, data processing jobs are increasingly being deployed on transient resources temporarily borrowed from latency-critical jobs. However, these transient resources must be evicted whenever latency-critical jobs require them again. Resource evictions often lead to cascading recomputations that substantially degrade job performance.</p> |
| |
| <p>Pado[1] is an optimization technique that reduces the number of recomputations triggered by eviction of transient resources. Specifically, Pado uses a placement algorithm for selectively retaining intermediate results on reserved resources that are not evicted.</p> |
| |
| <p>Nemo provides an optimization policy interface that makes it easy for users to employ techniques like Pado to improve application performance. To demonstrate the flexibility of Nemo, we have developed and evaluated PadoPolicy. We summarize preliminary evaluation results as follows.</p> |
| |
| <h3 id="experimentation-setup">Experimentation setup</h3> |
| |
| <ul> |
| <li>Systems: Spark 2.2.0, Nemo with PadoPolicy</li> |
| <li>Resources: |
| <ul> |
| <li>m4.2xlarge AWS EC2 instances (8 vCPU, 32GB memory)</li> |
| <li>35 transient nodes, and 5 reserved nodes |
| <ul> |
| <li>5-minute mean poisson eviction rate for each transient node</li> |
| <li>An evicted transient node is immediately replaced with a new transient node</li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li>Dataset: 10GB Yahoo! music ratings dataset[2]</li> |
| <li>Application: A machine learning recommendation algorithm - Alternating least squares (ALS) |
| <ul> |
| <li>Spark MLlib ALS implementation for Spark, and our ALS implementation written in Beam for Nemo</li> |
| </ul> |
| </li> |
| </ul> |
| |
| <h3 id="job-completion-time-jct">Job completion time (JCT)</h3> |
| |
| <p>Spark was not able to complete the job even after running for <strong>120 minutes</strong>, because it was stuck repeatedly recomputing intermediate results that are lost due to eviction. In contrast, Nemo completed in <strong>18 minutes</strong> by selectively retaining intermediate results on reserved nodes using the Pado technique. This <strong>6.7X speedup</strong> validates the flexibility of Nemo.</p> |
| |
| <p>[1] Youngseok Yang, Geon-Woo Kim, Won Wook Song, Yunseong Lee, Andrew Chung, Zhengping Qian, Brian Cho, and Byung-Gon Chun. 2017. Pado: A Data Processing Engine for Harnessing Transient Resources in Datacenters. In Proceedings of the Twelfth European Conference on Computer Systems (EuroSys ‘17).</p> |
| |
| <p>[2] Yahoo! Music User Ratings of Songs with Artist, Album, and Genre Meta Information, v. 1.0. https://webscope. sandbox.yahoo.com/catalog.php?datatype=r.</p></content><author><name>John Yang</name></author><summary type="html">To increase datacenter utilization, data processing jobs are increasingly being deployed on transient resources temporarily borrowed from latency-critical jobs. However, these transient resources must be evicted whenever latency-critical jobs require them again. Resource evictions often lead to cascading recomputations that substantially degrade job performance.</summary></entry><entry><title type="html">Optimizing shuffle performance using Nemo</title><link href="http://nemo.apache.org//blog/2018/03/23/shuffle-on-nemo/" rel="alternate" type="text/html" title="Optimizing shuffle performance using Nemo" /><published>2018-03-23T00:00:00+09:00</published><updated>2018-03-23T00:00:00+09:00</updated><id>http://nemo.apache.org//blog/2018/03/23/shuffle-on-nemo</id><content type="html" xml:base="http://nemo.apache.org//blog/2018/03/23/shuffle-on-nemo/"><p>Data shuffle is a key operation that underlies almost all large-scale data processing jobs. A shuffle operation typically involves writing intermediate data to disk, and reading the data back later when the successive computations are scheduled.</p> |
| |
| <p>Sailfish[1] is an optimization technique that reduces disk overheads associated with a shuffle operation. Specifically, Sailfish minimizes the number of disk seeks involved in reading intermediate data back from disk. Jobs that handle large volumes of data can especially benefit from the Sailfish technique.</p> |
| |
| <p>Nemo provides an optimization policy interface that makes it easy for users to employ techniques like Sailfish to improve application performance. To demonstrate the flexibility of Nemo, we have developed and evaluated SailfishPolicy. We summarize preliminary evaluation results as follows.</p> |
| |
| <h3 id="experimentation-setup">Experimentation setup</h3> |
| |
| <ul> |
| <li>Systems: Spark[2] 2.3.0 (a state-of-the-art system), and Nemo with SailfishPolicy</li> |
| <li>Resources: 20 h1.4xlarge (16 vCPU, 64GB memory, 2 HDDs) AWS instances |
| <ul> |
| <li>One of the disk is used by a HDFS cluster, and the other is used as a scratch disk by Nemo and Spark for maintaining intermediate data</li> |
| </ul> |
| </li> |
| <li>Dataset: 2TB Wikipedia pageview statistics[3] stored in the HDFS cluster</li> |
| <li>Application: A MapReduce application that reads input data from HDFS, computes the sum of pageview counts per Wikipedia project, and writes the results to HDFS |
| <ul> |
| <li>Spark’s app is written in Spark DSL, and Nemo’s app is written in Beam</li> |
| </ul> |
| </li> |
| </ul> |
| |
| <h3 id="job-completion-time-jct">Job completion time (JCT)</h3> |
| |
| <p><img src="https://user-images.githubusercontent.com/6691311/37783061-d7c62970-2e37-11e8-89d5-9ef3da8fd846.png" alt="Figure 1" /></p> |
| <center>Figure 1</center> |
| |
| <p>As shown in Figure 1, Nemo outperforms Spark by 2.26X primarily because Nemo’s reduce stage completes faster than Spark’s.</p> |
| |
| <h3 id="mean-disk-throughput-mbs">Mean disk throughput (MB/s)</h3> |
| |
| <p><img src="https://user-images.githubusercontent.com/6691311/37783098-f17b55d4-2e37-11e8-9cf3-bf082562c1e6.png" alt="Figure 2" /></p> |
| <center>Figure 2</center> |
| |
| <p>To understand the performance difference, we’ve measured the mean throughput of the scratch disks that Nemo and Spark use for handling intermediate data. As depicted in Figure 2, Nemo’s reduce stage enjoys much higher disk read throughput with a smaller number of disk seeks. This explains why Nemo’s reduce stage was able to complete more quickly, and validates the effectiveness of SailfishPolicy.</p> |
| |
| <p>[1] Sriram Rao, Raghu Ramakrishnan, Adam Silberstein, Mike Ovsiannikov, and Damian Reeves. 2012. Sailfish: a framework for large scale data processing. In Proceedings of the Third ACM Symposium on Cloud Computing (SoCC ‘12).</p> |
| |
| <p>[2] Apache Spark. https://spark.apache.org/.</p> |
| |
| <p>[3] Wikipedia pageview statistics. https://dumps.wikimedia.org/other/pagecounts-raw/.</p></content><author><name>John Yang</name></author><summary type="html">Data shuffle is a key operation that underlies almost all large-scale data processing jobs. A shuffle operation typically involves writing intermediate data to disk, and reading the data back later when the successive computations are scheduled.</summary></entry><entry><title type="html">Nemo blog published!</title><link href="http://nemo.apache.org//blog/2018/03/20/nemo-blog-published/" rel="alternate" type="text/html" title="Nemo blog published!" /><published>2018-03-20T00:00:00+09:00</published><updated>2018-03-20T00:00:00+09:00</updated><id>http://nemo.apache.org//blog/2018/03/20/nemo-blog-published</id><content type="html" xml:base="http://nemo.apache.org//blog/2018/03/20/nemo-blog-published/"><p>Our blog is published and is online! We’ll be posting exciting news related to our project on our blog.</p> |
| |
| <p>Your contribution is welcome!</p></content><author><name>Won Wook SONG</name></author><summary type="html">Our blog is published and is online! We’ll be posting exciting news related to our project on our blog.</summary></entry></feed> |
