commit | 680f7be4d9db7b330ec474c248b48e89fb114779 | [log] [tgz] |
---|---|---|
author | dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> | Tue Dec 17 15:43:30 2019 +0900 |
committer | Won Wook SONG <wonook@apache.org> | Tue Dec 17 15:43:30 2019 +0900 |
tree | 5c5d5e8673ac39c40aeaff274d43bb4eaae643c5 | |
parent | c8a11993ff954dfa0f815f1033051db3d428a419 [diff] |
Bump netty-all from 4.1.16.Final to 4.1.42.Final (#248) Bumps [netty-all](https://github.com/netty/netty) from 4.1.16.Final to 4.1.42.Final. <details> <summary>Commits</summary> - [`bd907c3`](https://github.com/netty/netty/commit/bd907c3b3acf3d746ad1143270224aa0f784cd1b) [maven-release-plugin] prepare release netty-4.1.42.Final - [`2791f0f`](https://github.com/netty/netty/commit/2791f0fefac82663ca09dc8aa5dda024152c92f8) Avoid use of global AtomicLong for ScheduledFutureTask ids ([#9599](https://github-redirect.dependabot.com/netty/netty/issues/9599)) - [`86ff76a`](https://github.com/netty/netty/commit/86ff76a4f77662fe1bf61deeec5f3e39e16d4020) Fix incorrect comment ([#9598](https://github-redirect.dependabot.com/netty/netty/issues/9598)) - [`5e69a13`](https://github.com/netty/netty/commit/5e69a13c21bcf89bf26dbf1421c5ec8188561a17) Cleanup JNI code to always correctly free memory when loading fails and also ... - [`eb3c4bd`](https://github.com/netty/netty/commit/eb3c4bd926e697da3ea16079a5f4211adb375ea5) ChunkedNioFile can use absolute FileChannel::read to read chunks ([#9592](https://github-redirect.dependabot.com/netty/netty/issues/9592)) - [`76592db`](https://github.com/netty/netty/commit/76592db0bde9d4908191c2d7e54af3466ffd4673) Close eventfd shutdown/wakeup race by closely tracking epoll edges ([#9586](https://github-redirect.dependabot.com/netty/netty/issues/9586)) - [`0a2d85f`](https://github.com/netty/netty/commit/0a2d85f1d32749eb278b27502ba7de20de958485) Fix GraalVM native image build error ([#9593](https://github-redirect.dependabot.com/netty/netty/issues/9593)) - [`dc4de7f`](https://github.com/netty/netty/commit/dc4de7fbb4e17c606c92c542d5015042ada48e38) We need to use NewGloblRef when caching jclass instances ([#9595](https://github-redirect.dependabot.com/netty/netty/issues/9595)) - [`4499384`](https://github.com/netty/netty/commit/4499384135cdaca5ccdda85b0c9610d4ecd6c3cc) Update to netty-tcnative 2.0.26.Final ([#9589](https://github-redirect.dependabot.com/netty/netty/issues/9589)) - [`8648171`](https://github.com/netty/netty/commit/8648171abc27497d87a8b47e509e8adabd622f5b) Fix *SslEngineTest to not throw ClassCastException and pass in all cases ([#9588](https://github-redirect.dependabot.com/netty/netty/issues/9588)) - Additional commits viewable in [compare view](https://github.com/netty/netty/compare/netty-4.1.16.Final...netty-4.1.42.Final) </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=io.netty:netty-all&package-manager=maven&previous-version=4.1.16.Final&new-version=4.1.42.Final)](https://help.github.com/articles/configuring-automated-security-fixes) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot ignore this [patch|minor|major] version` will close this PR and stop Dependabot creating any more for this minor/major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/incubator-nemo/network/alerts). </details>
A Data Processing System for Flexible Employment With Different Deployment Characteristics.
Details about Nemo and its development can be found in:
Please refer to the Contribution guideline to contribute to our project.
export HADOOP_HOME=/path/to/hadoop-2.7.2 export YARN_HOME=$HADOOP_HOME export PATH=$PATH:$HADOOP_HOME/bin
On Ubuntu 14.04 LTS and its point releases:
$ sudo apt-get install protobuf-compiler
On Ubuntu 16.04 LTS and its point releases:
$ sudo add-apt-repository ppa:snuspl/protobuf-250 $ sudo apt update $ sudo apt install protobuf-compiler=2.5.0-9xenial1
On macOS:
$ brew tap homebrew/versions $ brew install protobuf@2.5
Or build from source:
$ ./configure
$ make
$ make check
$ sudo make install
To check for a successful installation of version 2.5.0, run $ protoc --version
$ mvn clean install -T 2C
$ mvn clean install -DskipITs -T 2C
Apache Nemo is an official runner of Apache Beam, and it can be executed from Beam, using NemoRunner, as well as directly from the Nemo project. The details of using NemoRunner from Beam is shown on the NemoRunner page of the Apache Beam website. Below describes how Beam applications can be run directly on Nemo.
-job_id
: ID of the Beam job-user_main
: Canonical name of the Beam application-user_args
: Arguments that the Beam application accepts-optimization_policy
: Canonical name of the optimization policy to apply to a job DAG in Nemo Compiler-deploy_mode
: yarn
is supported(default value is local
)## WordCount example from the Beam website (Count words from a document) $ ./bin/run_beam.sh \ -job_id beam_wordcount \ -optimization_policy org.apache.nemo.compiler.optimizer.policy.DefaultPolicy \ -user_main org.apache.nemo.examples.beam.BeamWordCount \ -user_args "--runner=NemoRunner --inputFile=`pwd`/examples/resources/inputs/test_input_wordcount --output=`pwd`/outputs/wordcount" $ less `pwd`/outputs/wordcount* ## MapReduce WordCount example (Count words from the Wikipedia dataset) $ ./bin/run_beam.sh \ -job_id mr_default \ -executor_json `pwd`/examples/resources/executors/beam_test_executor_resources.json \ -optimization_policy org.apache.nemo.compiler.optimizer.policy.DefaultPolicy \ -user_main org.apache.nemo.examples.beam.WordCount \ -user_args "`pwd`/examples/resources/inputs/test_input_wordcount `pwd`/outputs/wordcount" $ less `pwd`/outputs/wordcount* ## YARN cluster example $ ./bin/run_beam.sh \ -deploy_mode yarn \ -job_id mr_transient \ -executor_json `pwd`/examples/resources/executors/beam_test_executor_resources.json \ -user_main org.apache.nemo.examples.beam.WordCount \ -optimization_policy org.apache.nemo.compiler.optimizer.policy.TransientResourcePolicy \ -user_args "hdfs://v-m:9000/test_input_wordcount hdfs://v-m:9000/test_output_wordcount" ## NEXMark streaming Q0 (query0) example $ ./bin/run_nexmark.sh \ -job_id nexmark-Q0 \ -executor_json `pwd`/examples/resources/executors/beam_test_executor_resources.json \ -user_main org.apache.beam.sdk.nexmark.Main \ -optimization_policy org.apache.nemo.compiler.optimizer.policy.StreamingPolicy \ -scheduler_impl_class_name org.apache.nemo.runtime.master.scheduler.StreamingScheduler \ -user_args "--runner=NemoRunner --streaming=true --query=0 --numEventGenerators=1"
-executor_json
command line option can be used to provide a path to the JSON file that describes resource configuration for executors. Its default value is config/default.json
, which initializes one of each Transient
, Reserved
, and Compute
executor, each of which has one core and 1024MB memory.
num
(optional): Number of containers. Default value is 1type
: Three container types are supported:Transient
: Containers that store eviction-prone resources. When batch jobs use idle resources in Transient
containers, they can be arbitrarily evicted when latency-critical jobs attempt to use the resources.Reserved
: Containers that store eviction-free resources. Reserved
containers are used to reliably store intermediate data which have high eviction cost.Compute
: Containers that are mainly used for computation.memory_mb
: Memory size in MBcapacity
: Number of Task
s that can be run in an executor. Set this value to be the same as the number of CPU cores of the container.[ { "num": 12, "type": "Transient", "memory_mb": 1024, "capacity": 4 }, { "type": "Reserved", "memory_mb": 1024, "capacity": 2 } ]
This example configuration specifies
Nemo Compiler and Engine can store JSON representation of intermediate DAGs.
-dag_dir
command line option is used to specify the directory where the JSON files are stored. The default directory is ./dag
. Using our online visualizer, you can easily visualize a DAG. Just drop the JSON file of the DAG as an input to it.$ ./bin/run_beam.sh \ -job_id als \ -executor_json `pwd`/examples/resources/executors/beam_test_executor_resources.json \ -user_main org.apache.nemo.examples.beam.AlternatingLeastSquare \ -optimization_policy org.apache.nemo.compiler.optimizer.policy.TransientResourcePolicy \ -dag_dir "./dag/als" \ -user_args "`pwd`/examples/resources/inputs/test_input_als 10 3"
-db_enabled
: Whether or not to turn on the DB (true
or false
).-db_address
: Address of the DB. (ex. PostgreSQL DB starts with jdbc:postgresql://...
)-db_id
: ID of the DB from the given address.-db_password
: Credentials for the DB from the given address.