| <!DOCTYPE html> |
| <!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> |
| <!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| |
| |
| <link rel="shortcut icon" href="../../img/favicon.ico"> |
| <title>Configuration - Apache Gearpump(incubating)</title> |
| <link href='https://fonts.googleapis.com/css?family=Lato:400,700|Roboto+Slab:400,700|Inconsolata:400,700' rel='stylesheet' type='text/css'> |
| |
| <link rel="stylesheet" href="../../css/theme.css" type="text/css" /> |
| <link rel="stylesheet" href="../../css/theme_extra.css" type="text/css" /> |
| <link rel="stylesheet" href="../../css/highlight.css"> |
| |
| <script> |
| // Current page data |
| var mkdocs_page_name = "Configuration"; |
| var mkdocs_page_input_path = "deployment/deployment-configuration.md"; |
| var mkdocs_page_url = "/deployment/deployment-configuration/index.html"; |
| </script> |
| |
| <script src="../../js/jquery-2.1.1.min.js"></script> |
| <script src="../../js/modernizr-2.8.3.min.js"></script> |
| <script type="text/javascript" src="../../js/highlight.pack.js"></script> |
| |
| </head> |
| |
| <body class="wy-body-for-nav" role="document"> |
| |
| <div class="wy-grid-for-nav"> |
| |
| |
| <nav data-toggle="wy-nav-shift" class="wy-nav-side stickynav"> |
| <div class="wy-side-nav-search"> |
| <a href="../../index.html" class="icon icon-home"> Apache Gearpump(incubating)</a> |
| <div role="search"> |
| <form id ="rtd-search-form" class="wy-form" action="../../search.html" method="get"> |
| <input type="text" name="q" placeholder="Search docs" /> |
| </form> |
| </div> |
| </div> |
| |
| <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> |
| <ul class="current"> |
| |
| |
| <li class="toctree-l1"> |
| |
| <a class="" href="../../index.html">Overview</a> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">Introduction</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../../introduction/submit-your-1st-application/index.html">Submit Your 1st Application</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../introduction/commandline/index.html">Client Command Line</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../introduction/basic-concepts/index.html">Basic Concepts</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../introduction/features/index.html">Technical Highlights</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../introduction/message-delivery/index.html">Reliable Message Delivery</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../introduction/performance-report/index.html">Performance</a> |
| </li> |
| </ul> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">Deployment</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../deployment-local/index.html">Local Mode</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../deployment-standalone/index.html">Standalone Mode</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../deployment-yarn/index.html">YARN Mode</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../deployment-docker/index.html">Docker Mode</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../deployment-ui-authentication/index.html">UI Authentication</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../deployment-ha/index.html">High Availability</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../deployment-msg-delivery/index.html">Reliable Message Delivery</a> |
| </li> |
| <li class=" current"> |
| |
| <a class="current" href="index.html">Configuration</a> |
| <ul class="subnav"> |
| |
| <li class="toctree-l3"><a href="#master-and-worker-configuration">Master and Worker configuration</a></li> |
| |
| |
| <li class="toctree-l3"><a href="#configuration-for-user-submitted-application-job">Configuration for user submitted application job</a></li> |
| |
| |
| <li class="toctree-l3"><a href="#logging">Logging</a></li> |
| |
| <ul> |
| |
| <li><a class="toctree-l4" href="#to-change-the-log-level-for-master-and-worker-daemon">To change the log level for master and worker daemon</a></li> |
| |
| <li><a class="toctree-l4" href="#to-change-the-log-level-for-application-job">To change the log level for application job</a></li> |
| |
| </ul> |
| |
| |
| <li class="toctree-l3"><a href="#gearpump-default-configuration">Gearpump Default Configuration</a></li> |
| |
| |
| </ul> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../deployment-resource-isolation/index.html">Resource Isolation</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../deployment-security/index.html">YARN Security Guide</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../get-gearpump-distribution/index.html">How to Get Your Gearpump Distribution</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../hardware-requirement/index.html">Hardware Requirement</a> |
| </li> |
| </ul> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">Programming Guide</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../../dev/dev-write-1st-app/index.html">Write Your 1st App</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../dev/dev-custom-serializer/index.html">Customized Message Passing</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../dev/dev-connectors/index.html">Gearpump Connectors</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../dev/dev-storm/index.html">Storm Compatibility</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../dev/dev-ide-setup/index.html">IDE Setup</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../dev/dev-non-streaming-example/index.html">Non Streaming Examples</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../dev/dev-rest-api/index.html">REST API</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../api/scala/index.html">Scala API</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../api/java/index.html">Java API</a> |
| </li> |
| </ul> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">Internals</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../../internals/gearpump-internals/index.html">Gearpump Internals</a> |
| </li> |
| </ul> |
| </li> |
| |
| </ul> |
| </div> |
| |
| </nav> |
| |
| <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> |
| |
| |
| <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> |
| <i data-toggle="wy-nav-top" class="fa fa-bars"></i> |
| <a href="../../index.html">Apache Gearpump(incubating)</a> |
| </nav> |
| |
| |
| <div class="wy-nav-content"> |
| <div class="rst-content"> |
| <div role="navigation" aria-label="breadcrumbs navigation"> |
| <ul class="wy-breadcrumbs"> |
| <li><a href="../../index.html">Docs</a> »</li> |
| |
| |
| |
| <li>Deployment »</li> |
| |
| |
| |
| <li>Configuration</li> |
| <li class="wy-breadcrumbs-aside"> |
| |
| <a href="https://github.com/apache/incubator-gearpump/edit/master/docs/contents/deployment/deployment-configuration.md" |
| class="icon icon-github"> Edit on GitHub</a> |
| |
| </li> |
| </ul> |
| <hr/> |
| </div> |
| <div role="main"> |
| <div class="section"> |
| |
| <h2 id="master-and-worker-configuration">Master and Worker configuration</h2> |
| <p>Master and Worker daemons will only read configuration from <code>conf/gear.conf</code>.</p> |
| <p>Master reads configuration from section master and gearpump:</p> |
| <pre class="codehilite"><code class="language-bash">master { |
| } |
| gearpump{ |
| }</code></pre> |
| |
| |
| <p>Worker reads configuration from section worker and gearpump:</p> |
| <pre class="codehilite"><code class="language-bash">worker { |
| } |
| gearpump{ |
| }</code></pre> |
| |
| |
| <h2 id="configuration-for-user-submitted-application-job">Configuration for user submitted application job</h2> |
| <p>For user application job, it will read configuration file <code>gear.conf</code> and <code>application.conf</code> from classpath, while <code>application.conf</code> has higher priority. |
| The default classpath contains:</p> |
| <ol> |
| <li><code>conf/</code></li> |
| <li>current working directory.</li> |
| </ol> |
| <p>For example, you can put a <code>application.conf</code> on your working directory, and then it will be effective when you submit a new job application.</p> |
| <h2 id="logging">Logging</h2> |
| <p>To change the log level, you need to change both <code>gear.conf</code>, and <code>log4j.properties</code>.</p> |
| <h3 id="to-change-the-log-level-for-master-and-worker-daemon">To change the log level for master and worker daemon</h3> |
| <p>Please change <code>log4j.rootLevel</code> in <code>log4j.properties</code>, <code>gearpump-master.akka.loglevel</code> and <code>gearpump-worker.akka.loglevel</code> in <code>gear.conf</code>.</p> |
| <h3 id="to-change-the-log-level-for-application-job">To change the log level for application job</h3> |
| <p>Please change <code>log4j.rootLevel</code> in <code>log4j.properties</code>, and <code>akka.loglevel</code> in <code>gear.conf</code> or <code>application.conf</code>.</p> |
| <h2 id="gearpump-default-configuration">Gearpump Default Configuration</h2> |
| <p>This is the default configuration for <code>gear.conf</code>.</p> |
| <table> |
| <thead> |
| <tr> |
| <th>config item</th> |
| <th>default value</th> |
| <th>description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td>gearpump.hostname</td> |
| <td>"127.0.0.1"</td> |
| <td>hostname of current machine. If you are using local mode, then set this to 127.0.0.1. If you are using cluster mode, make sure this hostname can be accessed by other machines.</td> |
| </tr> |
| <tr> |
| <td>gearpump.cluster.masters</td> |
| <td>["127.0.0.1:3000"]</td> |
| <td>Config to set the master nodes of the cluster. If there are multiple master in the list, then the master nodes runs in HA mode. For example, you may start three master, on node1: <code>bin/master -ip node1 -port 3000</code>, on node2: <code>bin/master -ip node2 -port 3000</code>, on node3: <code>bin/master -ip node3 -port 3000</code>, then you need to set <code>gearpump.cluster.masters = ["node1:3000","node2:3000","node3:3000"]</code></td> |
| </tr> |
| <tr> |
| <td>gearpump.task-dispatcher</td> |
| <td>"gearpump.shared-thread-pool-dispatcher"</td> |
| <td>default dispatcher for task actor</td> |
| </tr> |
| <tr> |
| <td>gearpump.metrics.enabled</td> |
| <td>true</td> |
| <td>flag to enable the metrics system</td> |
| </tr> |
| <tr> |
| <td>gearpump.metrics.sample-rate</td> |
| <td>1</td> |
| <td>We will take one sample every <code>gearpump.metrics.sample-rate</code> data points. Note it may have impact that the statistics on UI portal is not accurate. Change it to 1 if you want accurate metrics in UI</td> |
| </tr> |
| <tr> |
| <td>gearpump.metrics.report-interval-ms</td> |
| <td>15000</td> |
| <td>we will report once every 15 seconds</td> |
| </tr> |
| <tr> |
| <td>gearpump.metrics.reporter</td> |
| <td>"akka"</td> |
| <td>available value: "graphite", "akka", "logfile" which write the metrics data to different places.</td> |
| </tr> |
| <tr> |
| <td>gearpump.retainHistoryData.hours</td> |
| <td>72</td> |
| <td>max hours of history data to retain, Note: Due to implementation limitation(we store all history in memory), please don't set this to too big which may exhaust memory.</td> |
| </tr> |
| <tr> |
| <td>gearpump.retainHistoryData.intervalMs</td> |
| <td>3600000</td> |
| <td>time interval between two data points for history data (unit: ms). Usually this is set to a big value so that we only store coarse-grain data</td> |
| </tr> |
| <tr> |
| <td>gearpump.retainRecentData.seconds</td> |
| <td>300</td> |
| <td>max seconds of recent data to retain. This is for the fine-grain data</td> |
| </tr> |
| <tr> |
| <td>gearpump.retainRecentData.intervalMs</td> |
| <td>15000</td> |
| <td>time interval between two data points for recent data (unit: ms)</td> |
| </tr> |
| <tr> |
| <td>gearpump.log.daemon.dir</td> |
| <td>"logs"</td> |
| <td>The log directory for daemon processes(relative to current working directory)</td> |
| </tr> |
| <tr> |
| <td>gearpump.log.application.dir</td> |
| <td>"logs"</td> |
| <td>The log directory for applications(relative to current working directory)</td> |
| </tr> |
| <tr> |
| <td>gearpump.serializers</td> |
| <td>a map</td> |
| <td>custom serializer for streaming application, e.g. <code>"scala.Array" = ""</code></td> |
| </tr> |
| <tr> |
| <td>gearpump.worker.slots</td> |
| <td>1000</td> |
| <td>How many slots each worker contains</td> |
| </tr> |
| <tr> |
| <td>gearpump.appmaster.vmargs</td> |
| <td>"-server -Xss1M -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC -XX:NewRatio=3 -Djava.rmi.server.hostname=localhost"</td> |
| <td>JVM arguments for AppMaster</td> |
| </tr> |
| <tr> |
| <td>gearpump.appmaster.extraClasspath</td> |
| <td>""</td> |
| <td>JVM default class path for AppMaster</td> |
| </tr> |
| <tr> |
| <td>gearpump.executor.vmargs</td> |
| <td>"-server -Xss1M -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC -XX:NewRatio=3 -Djava.rmi.server.hostname=localhost"</td> |
| <td>JVM arguments for executor</td> |
| </tr> |
| <tr> |
| <td>gearpump.executor.extraClasspath</td> |
| <td>""</td> |
| <td>JVM default class path for executor</td> |
| </tr> |
| <tr> |
| <td>gearpump.jarstore.rootpath</td> |
| <td>"jarstore/"</td> |
| <td>Define where the submitted jar file will be stored. This path follows the hadoop path schema. For HDFS, use <code>hdfs://host:port/path/</code>, and HDFS HA, <code>hdfs://namespace/path/</code>; if you want to store on master nodes, then use local directory. <code>jarstore.rootpath = "jarstore/"</code> will point to relative directory where master is started. <code>jarstore.rootpath = "/jarstore/"</code> will point to absolute directory on master server</td> |
| </tr> |
| <tr> |
| <td>gearpump.scheduling.scheduler-class</td> |
| <td>"org.apache.gearpump.cluster.scheduler.PriorityScheduler"</td> |
| <td>Class to schedule the applications.</td> |
| </tr> |
| <tr> |
| <td>gearpump.services.host</td> |
| <td>"127.0.0.1"</td> |
| <td>dashboard UI host address</td> |
| </tr> |
| <tr> |
| <td>gearpump.services.port</td> |
| <td>8090</td> |
| <td>dashboard UI host port</td> |
| </tr> |
| <tr> |
| <td>gearpump.netty.buffer-size</td> |
| <td>5242880</td> |
| <td>netty connection buffer size</td> |
| </tr> |
| <tr> |
| <td>gearpump.netty.max-retries</td> |
| <td>30</td> |
| <td>maximum number of retries for a netty client to connect to remote host</td> |
| </tr> |
| <tr> |
| <td>gearpump.netty.base-sleep-ms</td> |
| <td>100</td> |
| <td>base sleep time for a netty client to retry a connection. Actual sleep time is a multiple of this value</td> |
| </tr> |
| <tr> |
| <td>gearpump.netty.max-sleep-ms</td> |
| <td>1000</td> |
| <td>maximum sleep time for a netty client to retry a connection</td> |
| </tr> |
| <tr> |
| <td>gearpump.netty.message-batch-size</td> |
| <td>262144</td> |
| <td>netty max batch size</td> |
| </tr> |
| <tr> |
| <td>gearpump.netty.flush-check-interval</td> |
| <td>10</td> |
| <td>max flush interval for the netty layer, in milliseconds</td> |
| </tr> |
| <tr> |
| <td>gearpump.netty.dispatcher</td> |
| <td>"gearpump.shared-thread-pool-dispatcher"</td> |
| <td>default dispatcher for netty client and server</td> |
| </tr> |
| <tr> |
| <td>gearpump.shared-thread-pool-dispatcher</td> |
| <td>default Dispatcher with "fork-join-executor"</td> |
| <td>default shared thread pool dispatcher</td> |
| </tr> |
| <tr> |
| <td>gearpump.single-thread-dispatcher</td> |
| <td>PinnedDispatcher</td> |
| <td>default single thread dispatcher</td> |
| </tr> |
| <tr> |
| <td>gearpump.serialization-framework</td> |
| <td>"org.apache.gearpump.serializer.FastKryoSerializationFramework"</td> |
| <td>Gearpump has built-in serialization framework using Kryo. Users are allowed to use a different serialization framework, like Protobuf. See <code>org.apache.gearpump.serializer.FastKryoSerializationFramework</code> to find how a custom serialization framework can be defined</td> |
| </tr> |
| <tr> |
| <td>worker.executor-share-same-jvm-as-worker</td> |
| <td>false</td> |
| <td>whether the executor actor is started in the same jvm(process) from which running the worker actor, the intention of this setting is for the convenience of single machine debugging, however, the app jar need to be added to the worker's classpath when you set it true and have a 'real' worker in the cluster</td> |
| </tr> |
| </tbody> |
| </table> |
| |
| </div> |
| </div> |
| <footer> |
| |
| <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> |
| |
| <a href="../deployment-resource-isolation/index.html" class="btn btn-neutral float-right" title="Resource Isolation">Next <span class="icon icon-circle-arrow-right"></span></a> |
| |
| |
| <a href="../deployment-msg-delivery/index.html" class="btn btn-neutral" title="Reliable Message Delivery"><span class="icon icon-circle-arrow-left"></span> Previous</a> |
| |
| </div> |
| |
| |
| <hr/> |
| |
| <div role="contentinfo"> |
| <!-- Copyright etc --> |
| |
| </div> |
| |
| Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. |
| </footer> |
| |
| </div> |
| </div> |
| |
| </section> |
| |
| </div> |
| |
| <div class="rst-versions" role="note" style="cursor: pointer"> |
| <span class="rst-current-version" data-toggle="rst-current-version"> |
| |
| <a href="https://github.com/apache/incubator-gearpump" class="fa fa-github" style="float: left; color: #fcfcfc"> GitHub</a> |
| |
| |
| <span><a href="../deployment-msg-delivery/index.html" style="color: #fcfcfc;">« Previous</a></span> |
| |
| |
| <span style="margin-left: 15px"><a href="../deployment-resource-isolation/index.html" style="color: #fcfcfc">Next »</a></span> |
| |
| </span> |
| </div> |
| <script src="../../js/theme.js"></script> |
| |
| </body> |
| </html> |
