Added nofollow to all external links
diff --git a/contributor/index.html b/contributor/index.html
index 7bb2141..6ce9275 100644
--- a/contributor/index.html
+++ b/contributor/index.html
@@ -67,8 +67,8 @@
<ul>
<li><a href="https://issues.apache.org/jira/browse/GOBBLIN" target="_blank">JIRA</a> for tracking issues</li>
- <li><a href="https://github.com/apache/incubator-gobblin/pulls" target="_blank">GitHub</a> for code reviews (via pull requests)</li>
- <li><a href="https://travis-ci.org/apache/incubator-gobblin" target="_blank">TravisCI</a> for automated builds</li>
+ <li><a href="https://github.com/apache/incubator-gobblin/pulls" target="_blank" rel="nofollow">GitHub</a> for code reviews (via pull requests)</li>
+ <li><a href="https://travis-ci.org/apache/incubator-gobblin" target="_blank" rel="nofollow">TravisCI</a> for automated builds</li>
</ul>
<h2 id="development-docs">Development docs<a class="header-link" href="#development-docs"><i class="fa fa-link"></i></a></h2>
diff --git a/docs/404.html b/docs/404.html
index 1802b94..308f566 100644
--- a/docs/404.html
+++ b/docs/404.html
@@ -505,7 +505,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/Getting-Started/index.html b/docs/Getting-Started/index.html
index 4c2b052..761b163 100644
--- a/docs/Getting-Started/index.html
+++ b/docs/Getting-Started/index.html
@@ -530,7 +530,7 @@
<li>Getting Started</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/Getting-Started.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/Getting-Started.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -620,14 +620,14 @@
<p>Here we show how to run a Gobblin daemon. A Gobblin daemon tracks a directory and finds job configuration files in it (jobs with extensions <code>*.pull</code>). Job files can be either run once or scheduled jobs. Gobblin will automatically execute this jobs as they are received following the schedule.</p>
<p>For this example, we will once again run the Wikipedia example. The records will be stored as Avro files.</p>
<h2 id="preliminary">Preliminary</h2>
-<p>Each Gobblin job minimally involves several constructs, e.g. <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/Source.java">Source</a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/extractor/Extractor.java">Extractor</a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/writer/DataWriter.java">DataWriter</a> and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/publisher/DataPublisher.java">DataPublisher</a>. As the names suggest, Source defines the source to pull data from, Extractor implements the logic to extract data records, DataWriter defines the way the extracted records are output, and DataPublisher publishes the data to the final output location. A job may optionally have one or more Converters, which transform the extracted records, as well as one or more PolicyCheckers that check the quality of the extracted records and determine whether they conform to certain policies.</p>
-<p>Some of the classes relevant to this example include <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/java/org/apache/gobblin/example/wikipedia/WikipediaSource.java">WikipediaSource</a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/java/org/apache/gobblin/example/wikipedia/WikipediaExtractor.java">WikipediaExtractor</a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/java/org/apache/gobblin/example/wikipedia/WikipediaConverter.java">WikipediaConverter</a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/writer/AvroHdfsDataWriter.java">AvroHdfsDataWriter</a> and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/publisher/BaseDataPublisher.java">BaseDataPublisher</a>.</p>
-<p>To run Gobblin in standalone daemon mode we need a Gobblin configuration file (such as uses <a href="https://github.com/apache/incubator-gobblin/blob/master/conf/standalone/application.conf">application.conf</a>). And for each job we wish to run, we also need a job configuration file (such as <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull">wikipedia.pull</a>). The Gobblin configuration file, which is passed to Gobblin as a command line argument, should contain a property <code>jobconf.dir</code> which specifies where the job configuration files are located. By default, <code>jobconf.dir</code> points to environment variable <code>GOBBLIN_JOB_CONFIG_DIR</code>. Each file in <code>jobconf.dir</code> with extension <code>.job</code> or <code>.pull</code> is considered a job configuration file, and Gobblin will launch a job for each such file. For more information on Gobblin deployment in standalone mode, refer to the <a href="user-guide/Gobblin-Deployment#Standalone-Deployment">Standalone Deployment</a> page.</p>
+<p>Each Gobblin job minimally involves several constructs, e.g. <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/Source.java" rel="nofollow">Source</a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/extractor/Extractor.java" rel="nofollow">Extractor</a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/writer/DataWriter.java" rel="nofollow">DataWriter</a> and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/publisher/DataPublisher.java" rel="nofollow">DataPublisher</a>. As the names suggest, Source defines the source to pull data from, Extractor implements the logic to extract data records, DataWriter defines the way the extracted records are output, and DataPublisher publishes the data to the final output location. A job may optionally have one or more Converters, which transform the extracted records, as well as one or more PolicyCheckers that check the quality of the extracted records and determine whether they conform to certain policies.</p>
+<p>Some of the classes relevant to this example include <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/java/org/apache/gobblin/example/wikipedia/WikipediaSource.java" rel="nofollow">WikipediaSource</a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/java/org/apache/gobblin/example/wikipedia/WikipediaExtractor.java" rel="nofollow">WikipediaExtractor</a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/java/org/apache/gobblin/example/wikipedia/WikipediaConverter.java" rel="nofollow">WikipediaConverter</a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/writer/AvroHdfsDataWriter.java" rel="nofollow">AvroHdfsDataWriter</a> and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/publisher/BaseDataPublisher.java" rel="nofollow">BaseDataPublisher</a>.</p>
+<p>To run Gobblin in standalone daemon mode we need a Gobblin configuration file (such as uses <a href="https://github.com/apache/incubator-gobblin/blob/master/conf/standalone/application.conf" rel="nofollow">application.conf</a>). And for each job we wish to run, we also need a job configuration file (such as <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull" rel="nofollow">wikipedia.pull</a>). The Gobblin configuration file, which is passed to Gobblin as a command line argument, should contain a property <code>jobconf.dir</code> which specifies where the job configuration files are located. By default, <code>jobconf.dir</code> points to environment variable <code>GOBBLIN_JOB_CONFIG_DIR</code>. Each file in <code>jobconf.dir</code> with extension <code>.job</code> or <code>.pull</code> is considered a job configuration file, and Gobblin will launch a job for each such file. For more information on Gobblin deployment in standalone mode, refer to the <a href="user-guide/Gobblin-Deployment#Standalone-Deployment">Standalone Deployment</a> page.</p>
<p>A list of commonly used configuration properties can be found here: <a href="user-guide/Configuration-Properties-Glossary">Configuration Properties Glossary</a>.</p>
<h2 id="steps_1">Steps</h2>
<ul>
<li>
-<p>Create a folder to store the job configuration file. Put <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull">wikipedia.pull</a> in this folder, and set environment variable <code>GOBBLIN_JOB_CONFIG_DIR</code> to point to this folder. Also, make sure that the environment variable <code>JAVA_HOME</code> is set correctly.</p>
+<p>Create a folder to store the job configuration file. Put <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull" rel="nofollow">wikipedia.pull</a> in this folder, and set environment variable <code>GOBBLIN_JOB_CONFIG_DIR</code> to point to this folder. Also, make sure that the environment variable <code>JAVA_HOME</code> is set correctly.</p>
</li>
<li>
<p>Create a folder as Gobblin's working directory. Gobblin will write job output as well as other information there, such as locks and state-store (for more information, see the <a href="user-guide/Gobblin-Deployment#Standalone-Deployment">Standalone Deployment</a> page). Set environment variable <code>GOBBLIN_WORK_DIR</code> to point to that folder.</p>
@@ -682,11 +682,11 @@
</code></pre>
<p><code>output.json</code> will contain all retrieved records in JSON format.</p>
-<p>Note that since this job configuration file we used (<a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull">wikipedia.pull</a>) doesn't specify a job schedule, the job will run immediately and will run only once. To schedule a job to run at a certain time and/or repeatedly, set the <code>job.schedule</code> property with a cron-based syntax. For example, <code>job.schedule=0 0/2 * * * ?</code> will run the job every two minutes. See <a href="http://www.quartz-scheduler.org/documentation/quartz-2.1.x/tutorials/crontrigger.html">this link</a> (Quartz CronTrigger) for more details.</p>
+<p>Note that since this job configuration file we used (<a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull" rel="nofollow">wikipedia.pull</a>) doesn't specify a job schedule, the job will run immediately and will run only once. To schedule a job to run at a certain time and/or repeatedly, set the <code>job.schedule</code> property with a cron-based syntax. For example, <code>job.schedule=0 0/2 * * * ?</code> will run the job every two minutes. See <a href="http://www.quartz-scheduler.org/documentation/quartz-2.1.x/tutorials/crontrigger.html" rel="nofollow">this link</a> (Quartz CronTrigger) for more details.</p>
<h1 id="other-example-jobs">Other Example Jobs</h1>
-<p>Besides the Wikipedia example, we have another example job <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/simplejson.pull">SimpleJson</a>, which extracts records from JSON files and store them in Avro files.</p>
-<p>To create your own jobs, simply implement the relevant interfaces such as <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/Source.java">Source</a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/extractor/Extractor.java">Extractor</a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/converter/Converter.java">Converter</a> and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/writer/DataWriter.java">DataWriter</a>. In the job configuration file, set properties such as <code>source.class</code> and <code>converter.class</code> to point to these classes.</p>
-<p>On a side note: while users are free to directly implement the Extractor interface (e.g., WikipediaExtractor), Gobblin also provides several extractor implementations based on commonly used protocols, e.g., <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaExtractor.java">KafkaExtractor</a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/restapi/RestApiExtractor.java">RestApiExtractor</a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-sql/src/main/java/org/apache/gobblin/source/jdbc/JdbcExtractor.java">JdbcExtractor</a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/sftp/SftpExtractor.java">SftpExtractor</a>, etc. Users are encouraged to extend these classes to take advantage of existing implementations.</p>
+<p>Besides the Wikipedia example, we have another example job <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/simplejson.pull" rel="nofollow">SimpleJson</a>, which extracts records from JSON files and store them in Avro files.</p>
+<p>To create your own jobs, simply implement the relevant interfaces such as <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/Source.java" rel="nofollow">Source</a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/extractor/Extractor.java" rel="nofollow">Extractor</a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/converter/Converter.java" rel="nofollow">Converter</a> and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/writer/DataWriter.java" rel="nofollow">DataWriter</a>. In the job configuration file, set properties such as <code>source.class</code> and <code>converter.class</code> to point to these classes.</p>
+<p>On a side note: while users are free to directly implement the Extractor interface (e.g., WikipediaExtractor), Gobblin also provides several extractor implementations based on commonly used protocols, e.g., <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaExtractor.java" rel="nofollow">KafkaExtractor</a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/restapi/RestApiExtractor.java" rel="nofollow">RestApiExtractor</a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-sql/src/main/java/org/apache/gobblin/source/jdbc/JdbcExtractor.java" rel="nofollow">JdbcExtractor</a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/sftp/SftpExtractor.java" rel="nofollow">SftpExtractor</a>, etc. Users are encouraged to extend these classes to take advantage of existing implementations.</p>
</div>
</div>
@@ -709,7 +709,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/Gobblin-Architecture/index.html b/docs/Gobblin-Architecture/index.html
index 3f95451..8a19a72 100644
--- a/docs/Gobblin-Architecture/index.html
+++ b/docs/Gobblin-Architecture/index.html
@@ -532,7 +532,7 @@
<li>Architecture</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/Gobblin-Architecture.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/Gobblin-Architecture.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -587,7 +587,7 @@
<ol>
<li>
-<p>A Gobblin job starts with an optional phase of acquiring a job lock. The purpose of doing this is to prevent the next scheduled run of the same job from starting until the current run finishes. This phase is optional because some job schedulers such as <a href="http://azkaban.github.io/">Azkaban</a> is already doing this. </p>
+<p>A Gobblin job starts with an optional phase of acquiring a job lock. The purpose of doing this is to prevent the next scheduled run of the same job from starting until the current run finishes. This phase is optional because some job schedulers such as <a href="http://azkaban.github.io/" rel="nofollow">Azkaban</a> is already doing this. </p>
</li>
<li>
<p>The next thing the job does is to create an instance of the <code>Source</code> class specified in the job configuration. A <code>Source</code> is responsible for partitioning the data ingestion work into a set of <code>WorkUnit</code>s, each of which represents a logic unit of work for extracting a portion of the data from a data source. A <code>Source</code> is also responsible for creating a <code>Extractor</code> for each <code>WorkUnit</code>. A <code>Extractor</code>, as the name suggests, actually talks to the data source and extracts data from it. The reason for this design is that Gobblin's <code>Source</code> is modeled after Hadoop's <code>InputFormat</code>, which is responsible for partitioning the input into <code>Split</code>s as well as creating a <code>RecordReader</code> for each <code>Split</code>. </p>
@@ -669,7 +669,7 @@
<h2 id="handling-of-failures">Handling of Failures</h2>
<p>As a fault tolerance data ingestion framework, Gobblin employs multiple level of defenses against job and task failures. For job failures, Gobblin keeps track of the number of times a job fails consecutively and optionally sends out an alert email if the number exceeds a defined threshold so the owner of the job can jump in and investigate the failures. For task failures, Gobblin retries failed tasks in a job run up to a configurable maximum number of times. In addition to that, Gobblin also provides an option to enable retries of <code>WorkUnit</code>s corresponding to failed tasks across job runs. The idea is that if a task fails after all retries fail, the <code>WorkUnit</code> based on which the task gets created will be automatically included in the next run of the job if this type of retries is enabled. This type of retries is very useful in handling intermittent failures such as those due to temporary data source outrage.</p>
<h2 id="job-scheduling">Job Scheduling</h2>
-<p>Like mentioned above, a Gobblin job typically runs periodically on some schedule. Gobblin can be integrated with job schedulers such as <a href="http://azkaban.github.io/">Azkaban</a>,<a href="http://oozie.apache.org/">Oozie</a>, or Crontab. Out-of-the-box, Gobblin also ships with a built-in job scheduler backed by a <a href="http://quartz-scheduler.org/">Quartz</a> scheduler, which is used as the default job scheduler in the standalone deployment and it supports cron-based triggers using the configuration property <code>job.schedule</code> for defining the cron schedule. An important feature of Gobblin is that it decouples the job scheduler and the jobs scheduled by the scheduler such that different jobs may run in different deployment settings. This is achieved using the abstraction <code>JobLauncher</code> that has different implementations for different deployment settings. For example, a job scheduler may have 5 jobs scheduled: 2 of them run locally on the same host as the scheduler using the <code>LocalJobLauncher</code>, whereas the rest 3 run on a Hadoop cluster somewhere using the <code>MRJobLauncher</code>. Which <code>JobLauncher</code> to use can be simply configured using the property <code>launcher.type</code>. Please refer to this <a href="http://quartz-scheduler.org/documentation/quartz-2.2.x/tutorials/tutorial-lesson-06">tutorial</a> for more information on how to use and configure a cron-based trigger.</p>
+<p>Like mentioned above, a Gobblin job typically runs periodically on some schedule. Gobblin can be integrated with job schedulers such as <a href="http://azkaban.github.io/">Azkaban</a>,<a href="http://oozie.apache.org/">Oozie</a>, or Crontab. Out-of-the-box, Gobblin also ships with a built-in job scheduler backed by a <a href="http://quartz-scheduler.org/" rel="nofollow">Quartz</a> scheduler, which is used as the default job scheduler in the standalone deployment and it supports cron-based triggers using the configuration property <code>job.schedule</code> for defining the cron schedule. An important feature of Gobblin is that it decouples the job scheduler and the jobs scheduled by the scheduler such that different jobs may run in different deployment settings. This is achieved using the abstraction <code>JobLauncher</code> that has different implementations for different deployment settings. For example, a job scheduler may have 5 jobs scheduled: 2 of them run locally on the same host as the scheduler using the <code>LocalJobLauncher</code>, whereas the rest 3 run on a Hadoop cluster somewhere using the <code>MRJobLauncher</code>. Which <code>JobLauncher</code> to use can be simply configured using the property <code>launcher.type</code>. Please refer to this <a href="http://quartz-scheduler.org/documentation/quartz-2.2.x/tutorials/tutorial-lesson-06" rel="nofollow">tutorial</a> for more information on how to use and configure a cron-based trigger.</p>
</div>
</div>
@@ -692,7 +692,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/Powered-By/index.html b/docs/Powered-By/index.html
index 626c328..7bdaa96 100644
--- a/docs/Powered-By/index.html
+++ b/docs/Powered-By/index.html
@@ -492,7 +492,7 @@
<li>Companies Powered By Gobblin</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/Powered-By.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/Powered-By.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -508,21 +508,21 @@
<p>
A few companies known to be powered by Gobblin:
<ul>
- <li><a href="http://www.linkedin.com" target="_blank"><b>LinkedIn</b></li>
- <li><a href="http://www.intel.com" target="_blank"><b>Intel</b></li>
- <li><a href="http://www.paypal.com" target="_blank"><b>Paypal</b></li>
- <li><a href="http://www.microsoft.com" target="_blank"><b>Microsoft</b></li>
- <li><a href="http://www.ibm.com" target="_blank"><b>IBM</b></li>
- <li><a href="http://www.home.cern/" target="_blank"><b>CERN</b></li>
- <li><a href="http://www.apple.com/" target="_blank"><b>Apple</b></li>
- <li><a href="http://www.stunlockstudios.com" target="_blank"><b>Stunlock Studios</b></li>
- <li><a href="http://www.swisscom.ch" target="_blank"><b>Swisscom</b></li>
- <li><a href="http://www.prezi.com" target="_blank"><b>Prezi</b></li>
- <li><a href="http://www.cleverleaf.co.uk" target="_blank"><b>Cleverdata</b></li>
- <li><a href="http://www.applift.com" target="_blank"><b>AppLift</b></li>
- <li><a href="http://www.nerdwallet.com" target="_blank"><b>Nerdwallet</b></li>
- <li><a href="http://www.sandia.gov/" target="_blank"><b>Sandia National Laboratories</b></li>
- <li><a href="http://www.bpuholdings.com/" target="_blank"><b>BPU Holdings</b></li>
+ <li><a href="http://www.linkedin.com" target="_blank" rel="nofollow"><b>LinkedIn</b></li>
+ <li><a href="http://www.intel.com" target="_blank" rel="nofollow"><b>Intel</b></li>
+ <li><a href="http://www.paypal.com" target="_blank" rel="nofollow"><b>Paypal</b></li>
+ <li><a href="http://www.microsoft.com" target="_blank" rel="nofollow"><b>Microsoft</b></li>
+ <li><a href="http://www.ibm.com" target="_blank" rel="nofollow"><b>IBM</b></li>
+ <li><a href="http://www.home.cern/" target="_blank" rel="nofollow"><b>CERN</b></li>
+ <li><a href="http://www.apple.com/" target="_blank" rel="nofollow"><b>Apple</b></li>
+ <li><a href="http://www.stunlockstudios.com" target="_blank" rel="nofollow"><b>Stunlock Studios</b></li>
+ <li><a href="http://www.swisscom.ch" target="_blank" rel="nofollow"><b>Swisscom</b></li>
+ <li><a href="http://www.prezi.com" target="_blank" rel="nofollow"><b>Prezi</b></li>
+ <li><a href="http://www.cleverleaf.co.uk" target="_blank" rel="nofollow"><b>Cleverdata</b></li>
+ <li><a href="http://www.applift.com" target="_blank" rel="nofollow"><b>AppLift</b></li>
+ <li><a href="http://www.nerdwallet.com" target="_blank" rel="nofollow"><b>Nerdwallet</b></li>
+ <li><a href="http://www.sandia.gov/" target="_blank" rel="nofollow"><b>Sandia National Laboratories</b></li>
+ <li><a href="http://www.bpuholdings.com/" target="_blank" rel="nofollow"><b>BPU Holdings</b></li>
</ul>
</p>
@@ -545,7 +545,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/adaptors/Gobblin-Distcp/index.html b/docs/adaptors/Gobblin-Distcp/index.html
index 2000198..1b58df2 100644
--- a/docs/adaptors/Gobblin-Distcp/index.html
+++ b/docs/adaptors/Gobblin-Distcp/index.html
@@ -537,7 +537,7 @@
<li>Gobblin Distcp</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/adaptors/Gobblin-Distcp.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/adaptors/Gobblin-Distcp.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -893,7 +893,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/adaptors/Hive-Avro-To-ORC-Converter/index.html b/docs/adaptors/Hive-Avro-To-ORC-Converter/index.html
index 22e9980..32be77d 100644
--- a/docs/adaptors/Hive-Avro-To-ORC-Converter/index.html
+++ b/docs/adaptors/Hive-Avro-To-ORC-Converter/index.html
@@ -526,7 +526,7 @@
<li>Hive Avro-To-Orc Converter</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/adaptors/Hive-Avro-To-ORC-Converter.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/adaptors/Hive-Avro-To-ORC-Converter.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -582,12 +582,12 @@
</ul>
<h1 id="job-constructs">Job Constructs</h1>
<h2 id="source-and-extractor">Source and Extractor</h2>
-<p>Gobblin provides <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/source/HiveSource.java"><code>HiveSource</code></a> which is a generic source that connects to the hive metastore and creates <code>WorkUnits</code> for any Hive <code>Partitions</code> and <code>Tables</code> whitelisted. The <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/extractor/HiveConvertExtractor.java"><code>HiveConvertExtractor</code></a> is a Gobblin <code>Extractor</code> to extracts work for Avro to ORC conversion.</p>
-<p>The <code>HiveSource</code> uses the <code>HiveDatasetFinder</code> to find all hive tables and partitions that satisfy a whitelist. For each table/partition it creates a workunit is the <code>updateTime</code> is greater than the <code>lowWatermark</code>. By default a <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/watermarker/PartitionLevelWatermarker.java"><code>PartitionLevelWatermarker</code></a> is used. This watermarker tracks watermarks for every partition of the table. Gobblin also provides a <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/watermarker/TableLevelWatermarker.java"><code>TableLevelWatermarker</code></a> that keeps one watermark per table.</p>
+<p>Gobblin provides <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/source/HiveSource.java" rel="nofollow"><code>HiveSource</code></a> which is a generic source that connects to the hive metastore and creates <code>WorkUnits</code> for any Hive <code>Partitions</code> and <code>Tables</code> whitelisted. The <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/extractor/HiveConvertExtractor.java" rel="nofollow"><code>HiveConvertExtractor</code></a> is a Gobblin <code>Extractor</code> to extracts work for Avro to ORC conversion.</p>
+<p>The <code>HiveSource</code> uses the <code>HiveDatasetFinder</code> to find all hive tables and partitions that satisfy a whitelist. For each table/partition it creates a workunit is the <code>updateTime</code> is greater than the <code>lowWatermark</code>. By default a <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/watermarker/PartitionLevelWatermarker.java" rel="nofollow"><code>PartitionLevelWatermarker</code></a> is used. This watermarker tracks watermarks for every partition of the table. Gobblin also provides a <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/watermarker/TableLevelWatermarker.java" rel="nofollow"><code>TableLevelWatermarker</code></a> that keeps one watermark per table.</p>
<p>The <code>HiveConvertExtractor</code> builds <code>QueryBasedHiveConversionEntity</code>s. The extractor makes necessary calls to the Hive Metastore to get table/partition metadata. The metadata is then wrapped into a <code>QueryBasedHiveConversionEntity</code>.</p>
<h2 id="converter">Converter</h2>
<p>The converter builds the Hive DDLs/DMLs required to perform the Avro to ORC conversion. Gobblin supports conversion of Avro to both flattened ORC and nested ORC.
-The abstract converter <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/converter/AbstractAvroToOrcConverter.java"><code>AbstractAvroToOrcConverter</code></a> builds DDLs/DMLs for any destination ORC format. Concrete subclass <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/converter/HiveAvroToFlattenedOrcConverter.java"><code>HiveAvroToFlattenedOrcConverter</code></a> provides the configurations required for Avro to flattened ORC conversion and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/converter/HiveAvroToNestedOrcConverter.java"><code>HiveAvroToNestedOrcConverter</code></a> provides the configurations required for Avro to nested ORC conversion. In the job configurations, both converters can be chained to perform flattend and nested ORC conversion in the same job. Each converter can also be used independent of other.</p>
+The abstract converter <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/converter/AbstractAvroToOrcConverter.java" rel="nofollow"><code>AbstractAvroToOrcConverter</code></a> builds DDLs/DMLs for any destination ORC format. Concrete subclass <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/converter/HiveAvroToFlattenedOrcConverter.java" rel="nofollow"><code>HiveAvroToFlattenedOrcConverter</code></a> provides the configurations required for Avro to flattened ORC conversion and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/converter/HiveAvroToNestedOrcConverter.java" rel="nofollow"><code>HiveAvroToNestedOrcConverter</code></a> provides the configurations required for Avro to nested ORC conversion. In the job configurations, both converters can be chained to perform flattend and nested ORC conversion in the same job. Each converter can also be used independent of other.</p>
<p>The converter builds the following different DDLs/DMLs</p>
<ul>
<li>Create staging table DDL - ORC data is written to a staging table first. The publisher then publishes them to the final ORC table. These DDLs are to create the staging table. A staging table looks like <code><orc_db_name>.<orc_table_name>_staging_<timestamp></code></li>
@@ -600,9 +600,9 @@
<li>Drop staging table DDL - Cleanup the staging table after data is published from staging to final tables</li>
</ul>
<h2 id="writer">Writer</h2>
-<p>The writer in this context executes the Hive DDLs/DMLs generated by the converter. <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/writer/HiveQueryExecutionWriter.java"><code>HiveQueryExecutionWriter</code></a> uses Hive JDBC connector to execute the DDLs. The DDLs write ORC data into staging tables. After the writer has completed <code>HiveQueryExecutionWriter#write()</code>, ORC data will be available in the staging tables.</p>
+<p>The writer in this context executes the Hive DDLs/DMLs generated by the converter. <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/writer/HiveQueryExecutionWriter.java" rel="nofollow"><code>HiveQueryExecutionWriter</code></a> uses Hive JDBC connector to execute the DDLs. The DDLs write ORC data into staging tables. After the writer has completed <code>HiveQueryExecutionWriter#write()</code>, ORC data will be available in the staging tables.</p>
<h2 id="publisher">Publisher</h2>
-<p>The publisher <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/publisher/HiveConvertPublisher.java"><code>HiveConvertPublisher</code></a> executes hive DDLs to publish staging ORC tables to final ORC tables. The publisher also cleans up staging tables.
+<p>The publisher <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/publisher/HiveConvertPublisher.java" rel="nofollow"><code>HiveConvertPublisher</code></a> executes hive DDLs to publish staging ORC tables to final ORC tables. The publisher also cleans up staging tables.
By default publishing happens per dataset (dataset = table in this context). If a dataset fails, other datasets will still be published but the job will fail. The commit policy is configurable.</p>
<h1 id="job-config-properties">Job Config Properties</h1>
<p>These are some of the job config properties used by <code>HiveAvroToOrcSource</code> and <code>HiveConvertExtractor</code>.</p>
@@ -866,7 +866,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/case-studies/Hive-Distcp/index.html b/docs/case-studies/Hive-Distcp/index.html
index 133676e..d31c85c 100644
--- a/docs/case-studies/Hive-Distcp/index.html
+++ b/docs/case-studies/Hive-Distcp/index.html
@@ -525,7 +525,7 @@
<li>Hive Distcp</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/case-studies/Hive-Distcp.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/case-studies/Hive-Distcp.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -553,10 +553,10 @@
</ul>
</div>
<h1 id="introduction">Introduction</h1>
-<p>Gobblin hive distcp is built on top of <a href="http://gobblin.readthedocs.io/en/latest/adaptors/Gobblin-Distcp/">Gobblin distcp</a>. It uses Hive metastore to find datasets to copy, then performs regular file listings to find the actual files to copy. After finishing the copy, the Hive registrations in the source are replicated on the target.</p>
+<p>Gobblin hive distcp is built on top of <a href="http://gobblin.readthedocs.io/en/latest/adaptors/Gobblin-Distcp/" rel="nofollow">Gobblin distcp</a>. It uses Hive metastore to find datasets to copy, then performs regular file listings to find the actual files to copy. After finishing the copy, the Hive registrations in the source are replicated on the target.</p>
<p>This document will show an sample job config of running Gobblin hive distcp, and explain how it works.</p>
<h1 id="configure-hive-distcp-job">Configure Hive Distcp Job</h1>
-<p>Below is the sample job config of running Gobblin hive distcp. Gobblin job constructs and data flow are the same as <a href="http://gobblin.readthedocs.io/en/latest/adaptors/Gobblin-Distcp/">Gobblin distcp</a>. The only difference is the <code>gobblin.data.profile.class</code> and hive related properties.</p>
+<p>Below is the sample job config of running Gobblin hive distcp. Gobblin job constructs and data flow are the same as <a href="http://gobblin.readthedocs.io/en/latest/adaptors/Gobblin-Distcp/" rel="nofollow">Gobblin distcp</a>. The only difference is the <code>gobblin.data.profile.class</code> and hive related properties.</p>
<pre><code>job.name=SampleHiveDistcp
job.group=HiveDistcp
job.description=Sample job config for hive distcp
@@ -663,7 +663,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/case-studies/Kafka-HDFS-Ingestion/index.html b/docs/case-studies/Kafka-HDFS-Ingestion/index.html
index 7e1f6b5..bb9051b 100644
--- a/docs/case-studies/Kafka-HDFS-Ingestion/index.html
+++ b/docs/case-studies/Kafka-HDFS-Ingestion/index.html
@@ -579,7 +579,7 @@
<li>Kafka-HDFS Ingestion</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/case-studies/Kafka-HDFS-Ingestion.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/case-studies/Kafka-HDFS-Ingestion.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -733,12 +733,12 @@
<p>After the job finishes, the job output file will be in <code>/gobblintest/job-output/test</code> in HDFS, and the metrics will be in <code>/gobblin-kafka/metrics</code>.</p>
<h1 id="job-constructs">Job Constructs</h1>
<h2 id="source-and-extractor">Source and Extractor</h2>
-<p>Gobblin provides two abstract classes, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaSource.java"><code>KafkaSource</code></a> and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaExtractor.java"><code>KafkaExtractor</code></a>. <code>KafkaSource</code> creates a workunit for each Kafka topic partition to be pulled, then merges and groups the workunits based on the desired number of workunits specified by property <code>mr.job.max.mappers</code> (this property is used in both standalone and MR mode). More details about how workunits are merged and grouped is available <a href="#merging-and-grouping-workunits-in-kafkasource">here</a>. <code>KafkaExtractor</code> extracts the partitions assigned to a workunit, based on the specified low watermark and high watermark.</p>
+<p>Gobblin provides two abstract classes, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaSource.java" rel="nofollow"><code>KafkaSource</code></a> and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaExtractor.java" rel="nofollow"><code>KafkaExtractor</code></a>. <code>KafkaSource</code> creates a workunit for each Kafka topic partition to be pulled, then merges and groups the workunits based on the desired number of workunits specified by property <code>mr.job.max.mappers</code> (this property is used in both standalone and MR mode). More details about how workunits are merged and grouped is available <a href="#merging-and-grouping-workunits-in-kafkasource">here</a>. <code>KafkaExtractor</code> extracts the partitions assigned to a workunit, based on the specified low watermark and high watermark.</p>
<p>To use them in a Kafka-HDFS ingestion job, one should subclass <code>KafkaExtractor</code> and implement method <code>decodeRecord(MessageAndOffset)</code>, which takes a <code>MessageAndOffset</code> object pulled from the Kafka broker and decodes it into a desired object. One should also subclass <code>KafkaSource</code> and implement <code>getExtractor(WorkUnitState)</code> which should return an instance of the Extractor class.</p>
-<p>As examples, take a look at <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaSimpleSource.java"><code>KafkaSimpleSource</code></a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaSimpleExtractor.java"><code>KafkaSimpleExtractor</code></a>, and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaExtractor.java"><code>KafkaAvroExtractor</code></a>.</p>
+<p>As examples, take a look at <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaSimpleSource.java" rel="nofollow"><code>KafkaSimpleSource</code></a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaSimpleExtractor.java" rel="nofollow"><code>KafkaSimpleExtractor</code></a>, and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaExtractor.java" rel="nofollow"><code>KafkaAvroExtractor</code></a>.</p>
<p><code>KafkaSimpleExtractor</code> simply returns the payload of the <code>MessageAndOffset</code> object as a byte array. A job that uses <code>KafkaSimpleExtractor</code> may use a <code>Converter</code> to convert the byte array to whatever format desired. For example, if the desired output format is JSON, one may implement an <code>ByteArrayToJsonConverter</code> to convert the byte array to JSON. Alternatively one may implement a <code>KafkaJsonExtractor</code>, which extends <code>KafkaExtractor</code> and convert the <code>MessageAndOffset</code> object into a JSON object in the <code>decodeRecord</code> method. Both approaches should work equally well. <code>KafkaAvroExtractor</code> decodes the payload of the <code>MessageAndOffset</code> object into an Avro <a href="http://avro.apache.org/docs/current/api/java/index.html?org/apache/avro/generic/GenericRecord.html"><code>GenericRecord</code></a> object.</p>
<h2 id="writer-and-publisher">Writer and Publisher</h2>
-<p>Any desired writer and publisher can be used, e.g., one may use the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/writer/AvroHdfsDataWriter.java"><code>AvroHdfsDataWriter</code></a> and the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/publisher/BaseDataPublisher.java"><code>BaseDataPublisher</code></a>, similar as the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull">Wikipedia example job</a>. If plain text output file is desired, one may use <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/writer/SimpleDataWriter.java"><code>SimpleDataWriter</code></a>.</p>
+<p>Any desired writer and publisher can be used, e.g., one may use the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/writer/AvroHdfsDataWriter.java" rel="nofollow"><code>AvroHdfsDataWriter</code></a> and the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/publisher/BaseDataPublisher.java" rel="nofollow"><code>BaseDataPublisher</code></a>, similar as the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull" rel="nofollow">Wikipedia example job</a>. If plain text output file is desired, one may use <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/writer/SimpleDataWriter.java" rel="nofollow"><code>SimpleDataWriter</code></a>.</p>
<h1 id="job-config-properties">Job Config Properties</h1>
<p>These are some of the job config properties used by <code>KafkaSource</code> and <code>KafkaExtractor</code>.</p>
<table>
@@ -788,7 +788,7 @@
<h1 id="metrics-and-events">Metrics and Events</h1>
<h2 id="task-level-metrics">Task Level Metrics</h2>
-<p>Task level metrics can be created in <code>Extractor</code>, <code>Converter</code> and <code>Writer</code> by extending <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/instrumented/extractor/InstrumentedExtractor.java"><code>InstrumentedExtractor</code></a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/instrumented/converter/InstrumentedConverter.java"><code>InstrumentedConverter</code></a> and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/instrumented/writer/InstrumentedDataWriter.java"><code>InstrumentedDataWriter</code></a>.</p>
+<p>Task level metrics can be created in <code>Extractor</code>, <code>Converter</code> and <code>Writer</code> by extending <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/instrumented/extractor/InstrumentedExtractor.java" rel="nofollow"><code>InstrumentedExtractor</code></a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/instrumented/converter/InstrumentedConverter.java" rel="nofollow"><code>InstrumentedConverter</code></a> and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/instrumented/writer/InstrumentedDataWriter.java" rel="nofollow"><code>InstrumentedDataWriter</code></a>.</p>
<p>For example, <code>KafkaExtractor</code> extends <code>InstrumentedExtractor</code>. So you can do the following in subclasses of <code>KafkaExtractor</code>:</p>
<pre><code>Counter decodingErrorCounter = this.getMetricContext().counter("num.of.decoding.errors");
decodingErrorCounter.inc();
@@ -796,9 +796,9 @@
<p>Besides Counter, Meter and Histogram are also supported.</p>
<h2 id="task-level-events">Task Level Events</h2>
-<p>Task level events can be submitted by creating an <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/event/EventSubmitter.java"><code>EventSubmitter</code></a> instance and using <code>EventSubmitter.submit()</code> or <code>EventSubmitter.getTimingEvent()</code>.</p>
+<p>Task level events can be submitted by creating an <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/event/EventSubmitter.java" rel="nofollow"><code>EventSubmitter</code></a> instance and using <code>EventSubmitter.submit()</code> or <code>EventSubmitter.getTimingEvent()</code>.</p>
<h2 id="job-level-metrics">Job Level Metrics</h2>
-<p>To create job level metrics, one may extend <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/AbstractJobLauncher.java"><code>AbstractJobLauncher</code></a> and create metrics there. For example:</p>
+<p>To create job level metrics, one may extend <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/AbstractJobLauncher.java" rel="nofollow"><code>AbstractJobLauncher</code></a> and create metrics there. For example:</p>
<pre><code>Optional<JobMetrics> jobMetrics = this.jobContext.getJobMetricsOptional();
if (!jobMetrics.isPresent()) {
LOG.warn("job metrics is absent");
@@ -814,7 +814,7 @@
<p>For more details about metrics, events and reporting them, please see Gobblin Metrics section.</p>
<h1 id="grouping-workunits">Grouping Workunits</h1>
<p>For each topic partition that should be ingested, <code>KafkaSource</code> first retrieves the last offset pulled by the previous run, which should be the first offset of the current run. It also retrieves the earliest and latest offsets currently available from the Kafka cluster and verifies that the first offset is between the earliest and the latest offsets. The latest offset is the last offset to be pulled by the current workunit. Since new records may be constantly published to Kafka and old records are deleted based on retention policies, the earliest and latest offsets of a partition may change constantly.</p>
-<p>For each partition, after the first and last offsets are determined, a workunit is created. If the number of Kafka partitions exceeds the desired number of workunits specified by property <code>mr.job.max.mappers</code>, <code>KafkaSource</code> will merge and group them into <code>n</code> <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/workunit/MultiWorkUnit.java"><code>MultiWorkUnit</code></a>s where <code>n=mr.job.max.mappers</code>. This is done using <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/workunit/packer/KafkaWorkUnitPacker.java"><code>KafkaWorkUnitPacker</code></a>, which has two implementations: <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/workunit/packer/KafkaSingleLevelWorkUnitPacker.java"><code>KafkaSingleLevelWorkUnitPacker</code></a> and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/workunit/packer/KafkaBiLevelWorkUnitPacker.java"><code>KafkaBiLevelWorkUnitPacker</code></a>. The packer packs workunits based on the estimated size of each workunit, which is obtained from <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/workunit/packer/KafkaWorkUnitSizeEstimator.java"><code>KafkaWorkUnitSizeEstimator</code></a>, which also has two implementations, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/workunit/packer/KafkaAvgRecordSizeBasedWorkUnitSizeEstimator.java"><code>KafkaAvgRecordSizeBasedWorkUnitSizeEstimator</code></a> and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/workunit/packer/KafkaAvgRecordTimeBasedWorkUnitSizeEstimator.java"><code>KafkaAvgRecordTimeBasedWorkUnitSizeEstimator</code></a>.</p>
+<p>For each partition, after the first and last offsets are determined, a workunit is created. If the number of Kafka partitions exceeds the desired number of workunits specified by property <code>mr.job.max.mappers</code>, <code>KafkaSource</code> will merge and group them into <code>n</code> <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/workunit/MultiWorkUnit.java" rel="nofollow"><code>MultiWorkUnit</code></a>s where <code>n=mr.job.max.mappers</code>. This is done using <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/workunit/packer/KafkaWorkUnitPacker.java" rel="nofollow"><code>KafkaWorkUnitPacker</code></a>, which has two implementations: <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/workunit/packer/KafkaSingleLevelWorkUnitPacker.java" rel="nofollow"><code>KafkaSingleLevelWorkUnitPacker</code></a> and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/workunit/packer/KafkaBiLevelWorkUnitPacker.java" rel="nofollow"><code>KafkaBiLevelWorkUnitPacker</code></a>. The packer packs workunits based on the estimated size of each workunit, which is obtained from <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/workunit/packer/KafkaWorkUnitSizeEstimator.java" rel="nofollow"><code>KafkaWorkUnitSizeEstimator</code></a>, which also has two implementations, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/workunit/packer/KafkaAvgRecordSizeBasedWorkUnitSizeEstimator.java" rel="nofollow"><code>KafkaAvgRecordSizeBasedWorkUnitSizeEstimator</code></a> and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/workunit/packer/KafkaAvgRecordTimeBasedWorkUnitSizeEstimator.java" rel="nofollow"><code>KafkaAvgRecordTimeBasedWorkUnitSizeEstimator</code></a>.</p>
<h2 id="single-level-packing">Single-Level Packing</h2>
<p>The single-level packer uses a worst-fit-decreasing approach for assigning workunits to mappers: each workunit goes to the mapper that currently has the lightest load. This approach balances the mappers well. However, multiple partitions of the same topic are usually assigned to different mappers. This may cause two issues: (1) many small output files: if multiple partitions of a topic are assigned to different mappers, they cannot share output files. (2) task overhead: when multiple partitions of a topic are assigned to different mappers, a task is created for each partition, which may lead to a large number of tasks and large overhead.</p>
<h2 id="bi-level-packing">Bi-Level Packing</h2>
@@ -857,29 +857,29 @@
<p>The type of <code>Deserializer</code> to be used in <code>KafkaDeserializerExtractor</code> can be specified by the property <code>kafka.deserializer.type</code>. This property can either be set to any of the pre-defined <code>Deserializer</code>s such as <code>CONFLUENT_AVRO</code>, <code>CONFLUENT_JSON</code>, <code>GSON</code>, <code>BYTE_ARRAY</code>, and <code>STRING</code> (see the section on <a href="#confluent-integration">Confluent Integration</a> and <a href="#kafkagsondeserializer">KafkaGsonDeserializer</a> for more details). The value of this property can point to the full-qualified path of a <code>Deserializer</code> implementation. If the value is set a class name, then a <code>kafka.schema.registry.class</code> must also be provided so that the <code>Extractor</code> knows how to retrieve the schema for the topic.</p>
<h2 id="gobblin-deserializer-implementations">Gobblin <code>Deserializer</code> Implementations</h2>
<h3 id="kafkagsondeserializer">KafkaGsonDeserializer</h3>
-<p>The <code>KafkaGsonDeserializer</code> is an implementation of the <code>Deserializer</code> class that converts <code>byte[]</code> to <a href="https://google.github.io/gson/apidocs/com/google/gson/JsonObject.html">JSONObject</a>s. It uses <a href="https://github.com/google/gson">GSON</a> to do this.</p>
+<p>The <code>KafkaGsonDeserializer</code> is an implementation of the <code>Deserializer</code> class that converts <code>byte[]</code> to <a href="https://google.github.io/gson/apidocs/com/google/gson/JsonObject.html" rel="nofollow">JSONObject</a>s. It uses <a href="https://github.com/google/gson" rel="nofollow">GSON</a> to do this.</p>
<p>This class is useful for converting Kafka data to JSON Objects.</p>
<p>Using this class simply requires setting <code>kafka.deserializer.type</code> to <code>GSON</code>.</p>
<h2 id="comparison-with-kafkasimplesource">Comparison with <code>KafkaSimpleSource</code></h2>
<p>Gobblin's <code>KafkaSimpleSource</code> and <code>KafkaSimpleExtractor</code> are very useful when data just needs to be read from Kafka and written to a text file. However, it does not provide good support for writing to more complex data file formats such as <a href="https://avro.apache.org/">Avro</a> or <a href="https://orc.apache.org/">ORC</a>. It also doesn't provide good support for record level manipulations such as Gobblin <code>Converter</code>s and it lacks good support for use with Gobblin's <code>WriterPartitioner</code>. The reason is that <code>KafkaSimpleExtractor</code> simply returns a <code>byte[]</code>, which is just a black-box of data. It is much easier to maniuplate the record if it is converted to a Java Object. This is where Gobblin's <code>KafkaDeserializerExtractor</code> becomes useful.</p>
<h1 id="confluent-integration">Confluent Integration</h1>
-<p><a href="http://www.confluent.io/">Confluent</a> provides a standardized distribution of <a href="http://kafka.apache.org/">Apache Kafka</a>, along with other useful tools for working with Kafka. One useful tool that Confluent provides is a generic <a href="http://www.confluent.io/blog/schema-registry-kafka-stream-processing-yes-virginia-you-really-need-one">Schema Registry</a>.</p>
-<p>Gobblin has integration with <a href="https://github.com/confluentinc/schema-registry">Confluent's Schema Registry Library</a> which provides a service to register and get <a href="https://avro.apache.org/docs/1.8.0/spec.html">Avro Schemas</a> and provides a generic <a href="https://github.com/confluentinc/schema-registry/blob/master/avro-serializer/src/main/java/io/confluent/kafka/serializers/KafkaAvroDeserializer.java">Avro Deserializer</a> and <a href="https://github.com/confluentinc/schema-registry/blob/master/json-serializer/src/main/java/io/confluent/kafka/serializers/KafkaJsonDeserializer.java">JSON Deserializer</a>.</p>
+<p><a href="http://www.confluent.io/">Confluent</a> provides a standardized distribution of <a href="http://kafka.apache.org/">Apache Kafka</a>, along with other useful tools for working with Kafka. One useful tool that Confluent provides is a generic <a href="http://www.confluent.io/blog/schema-registry-kafka-stream-processing-yes-virginia-you-really-need-one" rel="nofollow">Schema Registry</a>.</p>
+<p>Gobblin has integration with <a href="https://github.com/confluentinc/schema-registry">Confluent's Schema Registry Library</a> which provides a service to register and get <a href="https://avro.apache.org/docs/1.8.0/spec.html">Avro Schemas</a> and provides a generic <a href="https://github.com/confluentinc/schema-registry/blob/master/avro-serializer/src/main/java/io/confluent/kafka/serializers/KafkaAvroDeserializer.java" rel="nofollow">Avro Deserializer</a> and <a href="https://github.com/confluentinc/schema-registry/blob/master/json-serializer/src/main/java/io/confluent/kafka/serializers/KafkaJsonDeserializer.java" rel="nofollow">JSON Deserializer</a>.</p>
<h2 id="confluent-schema-registry">Confluent Schema Registry</h2>
-<p>Gobblin integrates with Confluent's <a href="https://github.com/confluentinc/schema-registry/blob/master/client/src/main/java/io/confluent/kafka/schemaregistry/client/SchemaRegistryClient.java">SchemaRegistryClient</a> class in order to register and get Avro Schema's from the Confluent <a href="https://github.com/confluentinc/schema-registry/blob/master/core/src/main/java/io/confluent/kafka/schemaregistry/storage/SchemaRegistry.java">SchemaRegistry</a>. This is implemented in the <code>ConfluentKafkaSchemaRegistry</code> class, which extends Gobblin's <code>KafkaSchemaRegistry</code> class. The <code>ConfluentKafkaSchemaRegistry</code> can be used by setting <code>kafka.schema.registry.class</code> to <code>gobblin.source.extractor.extract.kafka.ConfluentKafkaSchemaRegistry</code>.</p>
+<p>Gobblin integrates with Confluent's <a href="https://github.com/confluentinc/schema-registry/blob/master/client/src/main/java/io/confluent/kafka/schemaregistry/client/SchemaRegistryClient.java" rel="nofollow">SchemaRegistryClient</a> class in order to register and get Avro Schema's from the Confluent <a href="https://github.com/confluentinc/schema-registry/blob/master/core/src/main/java/io/confluent/kafka/schemaregistry/storage/SchemaRegistry.java" rel="nofollow">SchemaRegistry</a>. This is implemented in the <code>ConfluentKafkaSchemaRegistry</code> class, which extends Gobblin's <code>KafkaSchemaRegistry</code> class. The <code>ConfluentKafkaSchemaRegistry</code> can be used by setting <code>kafka.schema.registry.class</code> to <code>gobblin.source.extractor.extract.kafka.ConfluentKafkaSchemaRegistry</code>.</p>
<h2 id="confluent-deserializers">Confluent Deserializers</h2>
<p>Confluent's Schema Registry Library also provides a few useful <code>Deserializer</code> implementations:</p>
<ul>
-<li><a href="https://github.com/confluentinc/schema-registry/blob/master/avro-serializer/src/main/java/io/confluent/kafka/serializers/KafkaAvroDeserializer.java">KafkaAvroDeserializer</a> </li>
-<li><a href="https://github.com/confluentinc/schema-registry/blob/master/json-serializer/src/main/java/io/confluent/kafka/serializers/KafkaJsonDeserializer.java">KafkaJsonDeserializer</a></li>
+<li><a href="https://github.com/confluentinc/schema-registry/blob/master/avro-serializer/src/main/java/io/confluent/kafka/serializers/KafkaAvroDeserializer.java" rel="nofollow">KafkaAvroDeserializer</a> </li>
+<li><a href="https://github.com/confluentinc/schema-registry/blob/master/json-serializer/src/main/java/io/confluent/kafka/serializers/KafkaJsonDeserializer.java" rel="nofollow">KafkaJsonDeserializer</a></li>
</ul>
-<p>With regards to Gobblin, these classes are useful if Confluent's <a href="https://github.com/confluentinc/schema-registry/blob/master/avro-serializer/src/main/java/io/confluent/kafka/serializers/KafkaAvroSerializer.java">KafkaAvroSerializer</a> or <a href="https://github.com/confluentinc/schema-registry/blob/master/json-serializer/src/main/java/io/confluent/kafka/serializers/KafkaJsonSerializer.java">KafkaJsonSerializer</a> is used to write data to Kafka.</p>
+<p>With regards to Gobblin, these classes are useful if Confluent's <a href="https://github.com/confluentinc/schema-registry/blob/master/avro-serializer/src/main/java/io/confluent/kafka/serializers/KafkaAvroSerializer.java" rel="nofollow">KafkaAvroSerializer</a> or <a href="https://github.com/confluentinc/schema-registry/blob/master/json-serializer/src/main/java/io/confluent/kafka/serializers/KafkaJsonSerializer.java" rel="nofollow">KafkaJsonSerializer</a> is used to write data to Kafka.</p>
<p>The <a href="https://kafka.apache.org/0100/javadoc/org/apache/kafka/common/serialization/Serializer.html">Serializer</a> class is a Kafka interface that is the converse of the <code>Deserializer</code> class. The <code>Serializer</code> provides a generic way of taking Java Objects and converting them to <code>byte[]</code> that are written to Kafka by a <code>KafkaProducer</code>.</p>
<h3 id="kafkaavrodeserializer">KafkaAvroDeserializer</h3>
-<p>Documentation for the <code>KafkaAvroDeserializer</code> can be found <a href="http://docs.confluent.io/2.0.1/schema-registry/docs/serializer-formatter.html#serializer">here</a>.</p>
+<p>Documentation for the <code>KafkaAvroDeserializer</code> can be found <a href="http://docs.confluent.io/2.0.1/schema-registry/docs/serializer-formatter.html#serializer" rel="nofollow">here</a>.</p>
<p>If data is written to a Kafka cluster using Confluent's <code>KafkaAvroSerializer</code>, then the <code>KafkaAvroDeserializer</code> should be used in Gobblin. Setting this up simply requires a setting the config key <code>kafka.deserializer.type</code> to <code>CONFLUENT_AVRO</code> (see the section on <a href="#kafka-deserializer-integration">Kafka Deserializer Integration</a> for more information).</p>
<h3 id="kafkajsondeserializer">KafkaJsonDeserializer</h3>
-<p>The <code>KafkaJsonDeserializer</code> class uses <a href="https://fasterxml.github.io/jackson-databind/javadoc/2.7/com/fasterxml/jackson/databind/ObjectMapper.html">Jackson's Object Mapper</a> to convert <code>byte[]</code> to Java Objects. In order to <code>KafkaJsonDeserializer</code> to know which class the <code>byte[]</code> array should be converted to, the config property <code>json.value.type</code> needs to be set to the fully-qualified class name of the Java Object that the <code>Deserializer</code> should return. For more information about how the Jackson works, check out the docs <a href="https://github.com/FasterXML/jackson-databind">here</a>.</p>
+<p>The <code>KafkaJsonDeserializer</code> class uses <a href="https://fasterxml.github.io/jackson-databind/javadoc/2.7/com/fasterxml/jackson/databind/ObjectMapper.html" rel="nofollow">Jackson's Object Mapper</a> to convert <code>byte[]</code> to Java Objects. In order to <code>KafkaJsonDeserializer</code> to know which class the <code>byte[]</code> array should be converted to, the config property <code>json.value.type</code> needs to be set to the fully-qualified class name of the Java Object that the <code>Deserializer</code> should return. For more information about how the Jackson works, check out the docs <a href="https://github.com/FasterXML/jackson-databind" rel="nofollow">here</a>.</p>
<p>Using the <code>KafkaJsonDeserializer</code> simply requires setting the config key <code>kafka.deserializer.type</code> to <code>CONFLUENT_JSON</code> (see the section on <a href="#kafka-deserializer-integration">Kafka Deserializer Integration</a> for more information).</p>
</div>
@@ -903,7 +903,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/case-studies/Publishing-Data-to-S3/index.html b/docs/case-studies/Publishing-Data-to-S3/index.html
index 57cce09..8a98581 100644
--- a/docs/case-studies/Publishing-Data-to-S3/index.html
+++ b/docs/case-studies/Publishing-Data-to-S3/index.html
@@ -542,7 +542,7 @@
<li>Publishing Data to S3</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/case-studies/Publishing-Data-to-S3.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/case-studies/Publishing-Data-to-S3.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -588,17 +588,17 @@
</ul>
</div>
<h1 id="introduction">Introduction</h1>
-<p>While Gobblin is not tied to any specific cloud provider, <a href="https://aws.amazon.com/">Amazon Web Services</a> is a popular choice. This document will outline how Gobblin can publish data to <a href="https://aws.amazon.com/s3/">S3</a>. Specifically, it will provide a step by step guide to help setup Gobblin on Amazon <a href="https://aws.amazon.com/ec2/">EC2</a>, run Gobblin on EC2, and publish data from EC2 to S3.</p>
-<p>It is recommended to configure Gobblin to first write data to <a href="https://aws.amazon.com/ebs/">EBS</a>, and then publish the data to S3. This is the recommended approach because there are a few caveats when working with with S3. See the <a href="#hadoop-and-s3">Hadoop and S3</a> section for more details.</p>
-<p>This document will also provide a step by step guide for launching and configuring an EC2 instance and creating a S3 bucket. However, it is by no means a source of truth guide to working with AWS, it will only provide high level steps. The best place to learn about how to use AWS is through the <a href="https://aws.amazon.com/documentation/">Amazon documentation</a>.</p>
+<p>While Gobblin is not tied to any specific cloud provider, <a href="https://aws.amazon.com/" rel="nofollow">Amazon Web Services</a> is a popular choice. This document will outline how Gobblin can publish data to <a href="https://aws.amazon.com/s3/" rel="nofollow">S3</a>. Specifically, it will provide a step by step guide to help setup Gobblin on Amazon <a href="https://aws.amazon.com/ec2/" rel="nofollow">EC2</a>, run Gobblin on EC2, and publish data from EC2 to S3.</p>
+<p>It is recommended to configure Gobblin to first write data to <a href="https://aws.amazon.com/ebs/" rel="nofollow">EBS</a>, and then publish the data to S3. This is the recommended approach because there are a few caveats when working with with S3. See the <a href="#hadoop-and-s3">Hadoop and S3</a> section for more details.</p>
+<p>This document will also provide a step by step guide for launching and configuring an EC2 instance and creating a S3 bucket. However, it is by no means a source of truth guide to working with AWS, it will only provide high level steps. The best place to learn about how to use AWS is through the <a href="https://aws.amazon.com/documentation/" rel="nofollow">Amazon documentation</a>.</p>
<h1 id="hadoop-and-s3">Hadoop and S3</h1>
<p>A majority of Gobblin's code base uses Hadoop's <a href="https://hadoop.apache.org/docs/r2.4.1/api/org/apache/hadoop/fs/FileSystem.html">FileSystem</a> object to read and write data. The <code>FileSystem</code> object is an abstract class, and typical implementations either write to the local file system, or write to HDFS. There has been significant work to create an implementation of the <code>FileSystem</code> object that reads and writes to S3. The best guide to read about the different S3 <code>FileSystem</code> implementations is <a href="https://wiki.apache.org/hadoop/AmazonS3">here</a>.</p>
<p>There are a few different S3 <code>FileSystem</code> implementations, the two of note are the <code>s3a</code> and the <code>s3</code> file systems. The <code>s3a</code> file system is relatively new and is only available in Hadoop 2.6.0 (see the original <a href="https://issues.apache.org/jira/browse/HADOOP-10400">JIRA</a> for more information). The <code>s3</code> filesystem has been around for a while.</p>
<h2 id="the-s3a-file-system">The <code>s3a</code> File System</h2>
<p>The <code>s3a</code> file system uploads files to a specified bucket. The data uploaded to S3 via this file system is interoperable with other S3 tools. However, there are a few caveats when working with this file system:</p>
<ul>
-<li>Since S3 does not support renaming of files in a bucket, the <code>S3AFileSystem.rename(Path, Path)</code> operation will actually copy data from the source <code>Path</code> to the destination <code>Path</code>, and then delete the source <code>Path</code> (see the <a href="http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hadoop/hadoop-aws/2.6.0/org/apache/hadoop/fs/s3a/S3AFileSystem.java">source code</a> for more information)</li>
-<li>When creating a file using <code>S3AFileSystem.create(...)</code> data will be first written to a staging file on the local file system, and when the file is closed, the staging file will be uploaded to S3 (see the <a href="http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hadoop/hadoop-aws/2.6.0/org/apache/hadoop/fs/s3a/S3AOutputStream.java">source code</a> for more information)</li>
+<li>Since S3 does not support renaming of files in a bucket, the <code>S3AFileSystem.rename(Path, Path)</code> operation will actually copy data from the source <code>Path</code> to the destination <code>Path</code>, and then delete the source <code>Path</code> (see the <a href="http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hadoop/hadoop-aws/2.6.0/org/apache/hadoop/fs/s3a/S3AFileSystem.java" rel="nofollow">source code</a> for more information)</li>
+<li>When creating a file using <code>S3AFileSystem.create(...)</code> data will be first written to a staging file on the local file system, and when the file is closed, the staging file will be uploaded to S3 (see the <a href="http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hadoop/hadoop-aws/2.6.0/org/apache/hadoop/fs/s3a/S3AOutputStream.java" rel="nofollow">source code</a> for more information)</li>
</ul>
<p>Thus, when using the <code>s3a</code> file system with Gobblin it is recommended that one configures Gobblin to first write its staging data to the local filesystem, and then to publish the data to S3. The reason this is the recommended approach is that each Gobblin <code>Task</code> will write data to a staging file, and once the file has been completely written it publishes the file to a output directory (it does this by using a rename function). Finally, the <code>DataPublisher</code> moves the files from the staging directory to its final directory (again done using a rename function). This requires two renames operations and would be very inefficient if a <code>Task</code> wrote directly to S3.</p>
<p>Furthermore, writing directly to S3 requires creating a staging file on the local file system, and then creating a <code>PutObjectRequest</code> to upload the data to S3. This is logically equivalent to just configuring Gobblin to write to a local file and then publishing it to S3.</p>
@@ -608,14 +608,14 @@
<p>This section will provide a step by step guide to setting up an EC2 instance, a S3 bucket, installing Gobblin on EC2, and configuring Gobblin to publish data to S3.</p>
<p>This guide will use the free-tier provided by AWS to setup EC2 and S3.</p>
<h2 id="signing-up-for-aws">Signing Up For AWS</h2>
-<p>In order to use EC2 and S3, one first needs to sign up for an AWS account. The easiest way to get started with AWS is to use their <a href="https://aws.amazon.com/free/">free tier</a>.</p>
+<p>In order to use EC2 and S3, one first needs to sign up for an AWS account. The easiest way to get started with AWS is to use their <a href="https://aws.amazon.com/free/" rel="nofollow">free tier</a>.</p>
<h2 id="setting-up-ec2">Setting Up EC2</h2>
<h3 id="launching-an-ec2-instance">Launching an EC2 Instance</h3>
-<p>Once you have an AWS account, login to the AWS <a href="https://console.aws.amazon.com/console/home">console</a>. Select the EC2 link, which will bring you to the <a href="https://console.aws.amazon.com/ec2/">EC2 dashboard</a>.</p>
+<p>Once you have an AWS account, login to the AWS <a href="https://console.aws.amazon.com/console/home" rel="nofollow">console</a>. Select the EC2 link, which will bring you to the <a href="https://console.aws.amazon.com/ec2/" rel="nofollow">EC2 dashboard</a>.</p>
<p>Click on <code>Launch Instance</code> to create a new EC2 instance. Before the instance actually starts to run, there area a few more configuration steps necessary:</p>
<ul>
-<li>Choose an Amazon Machine Image (<a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html">AMI</a>)<ul>
-<li>For this walkthrough we will pick Red Hat Enterprise Linux (<a href="https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux">RHEL</a>) AMI</li>
+<li>Choose an Amazon Machine Image (<a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html" rel="nofollow">AMI</a>)<ul>
+<li>For this walkthrough we will pick Red Hat Enterprise Linux (<a href="https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux" rel="nofollow">RHEL</a>) AMI</li>
</ul>
</li>
<li>Choose an Instance Type<ul>
@@ -623,18 +623,18 @@
<li>This instance provides us with 1 vCPU and 1 GiB of RAM</li>
</ul>
</li>
-<li>For more information on other instance types, check out the AWS <a href="https://aws.amazon.com/ec2/instance-types/">docs</a></li>
+<li>For more information on other instance types, check out the AWS <a href="https://aws.amazon.com/ec2/instance-types/" rel="nofollow">docs</a></li>
</ul>
</li>
<li>Click Review and Launch<ul>
<li>We will use the defaults for all other setting options</li>
<li>When reviewing your instance, you will most likely get a warning saying access to your EC2 instance is open to the world</li>
-<li>If you want to fix this you have to edit the <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html">Security Groups</a>; how to do that is out of the scope of this document</li>
+<li>If you want to fix this you have to edit the <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html" rel="nofollow">Security Groups</a>; how to do that is out of the scope of this document</li>
</ul>
</li>
<li>Set Up SSH Keys<ul>
<li>After reviewing your instance, click <code>Launch</code></li>
-<li>You should be prompted to setup <a href="https://en.wikipedia.org/wiki/Secure_Shell">SSH</a> keys</li>
+<li>You should be prompted to setup <a href="https://en.wikipedia.org/wiki/Secure_Shell" rel="nofollow">SSH</a> keys</li>
<li>Use an existing key pair if you have one, otherwise create a new one and download it</li>
</ul>
</li>
@@ -651,9 +651,9 @@
</ul>
</li>
</ul>
-<p>After following the above steps, you should be able to freely SSH into the launched EC2 instance, and monitor / control the instance from the <a href="https://console.aws.amazon.com/ec2/">EC2 dashboard</a>.</p>
+<p>After following the above steps, you should be able to freely SSH into the launched EC2 instance, and monitor / control the instance from the <a href="https://console.aws.amazon.com/ec2/" rel="nofollow">EC2 dashboard</a>.</p>
<h3 id="ec2-package-installations">EC2 Package Installations</h3>
-<p>Before setting up Gobblin, you need to install <a href="https://en.wikipedia.org/wiki/Java_(programming_language)">Java</a> first. Depending on the AMI instance you are running Java may or may not already be installed (you can check if Java is already installed by executing <code>java -version</code>).</p>
+<p>Before setting up Gobblin, you need to install <a href="https://en.wikipedia.org/wiki/Java_(programming_language)" rel="nofollow">Java</a> first. Depending on the AMI instance you are running Java may or may not already be installed (you can check if Java is already installed by executing <code>java -version</code>).</p>
<h4 id="installing-java">Installing Java</h4>
<ul>
<li>Execute <code>sudo yum install java-1.8.0-openjdk*</code> to install Open JDK 8</li>
@@ -664,18 +664,18 @@
</li>
</ul>
<h2 id="setting-up-s3">Setting Up S3</h2>
-<p>Go to the <a href="https://console.aws.amazon.com/s3">S3 dashboard</a></p>
+<p>Go to the <a href="https://console.aws.amazon.com/s3" rel="nofollow">S3 dashboard</a></p>
<ul>
<li>Click on <code>Create Bucket</code><ul>
<li>Enter a name for the bucket (e.g. <code>gobblin-demo-bucket</code>)</li>
-<li>Enter a <a href="http://docs.aws.amazon.com/general/latest/gr/rande.html">Region</a> for the bucket (e.g. <code>US Standard</code>)</li>
+<li>Enter a <a href="http://docs.aws.amazon.com/general/latest/gr/rande.html" rel="nofollow">Region</a> for the bucket (e.g. <code>US Standard</code>)</li>
</ul>
</li>
</ul>
<h2 id="setting-up-gobblin-on-ec2">Setting Up Gobblin on EC2</h2>
<ul>
<li>Download and Build Gobblin Locally<ul>
-<li>On your local machine, clone the <a href="https://github.com/apache/incubator-gobblin">Gobblin repository</a>: <code>git clone git@github.com:apache/incubator-gobblin.git</code> (this assumes you have <a href="https://en.wikipedia.org/wiki/Git_(software)">Git</a> installed locally)</li>
+<li>On your local machine, clone the <a href="https://github.com/apache/incubator-gobblin" rel="nofollow">Gobblin repository</a>: <code>git clone git@github.com:apache/incubator-gobblin.git</code> (this assumes you have <a href="https://en.wikipedia.org/wiki/Git_(software)" rel="nofollow">Git</a> installed locally)</li>
<li>Build Gobblin using the following commands (it is important to use Hadoop version 2.6.0 as it includes the <code>s3a</code> file system implementation):</li>
</ul>
</li>
@@ -717,22 +717,22 @@
</li>
<li>Add the AWS Access Key Id and Secret Access Key<ul>
<li>Set the keys <code>fs.s3a.access.key</code> and <code>fs.s3a.secret.key</code> to the appropriate values</li>
-<li>These keys correspond to <a href="http://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html">AWS security credentials</a></li>
-<li>For information on how to get these credentials, check out the AWS documentation <a href="http://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html">here</a></li>
-<li>The AWS documentation recommends using <a href="http://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html">IAM roles</a>; how to set this up is out of the scope of this document; for this walkthrough we will use root access credentials</li>
+<li>These keys correspond to <a href="http://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html" rel="nofollow">AWS security credentials</a></li>
+<li>For information on how to get these credentials, check out the AWS documentation <a href="http://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html" rel="nofollow">here</a></li>
+<li>The AWS documentation recommends using <a href="http://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html" rel="nofollow">IAM roles</a>; how to set this up is out of the scope of this document; for this walkthrough we will use root access credentials</li>
</ul>
</li>
</ul>
<h2 id="launching-gobblin-on-ec2">Launching Gobblin on EC2</h2>
<p>Assuming we want Gobblin to run in standalone mode, follow the usual steps for <a href="../user-guide/Gobblin-Deployment#Standalone-Deployment">standalone deployment</a>.</p>
-<p>For the sake of this walkthrough, we will launch the Gobblin <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull">wikipedia example</a>. Directions on how to run this example can be found <a href="../Getting-Started">here</a>. The command to launch Gobblin should look similar to:</p>
+<p>For the sake of this walkthrough, we will launch the Gobblin <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull" rel="nofollow">wikipedia example</a>. Directions on how to run this example can be found <a href="../Getting-Started">here</a>. The command to launch Gobblin should look similar to:</p>
<pre><code>sh bin/gobblin standalone start --conf-dir /home/ec2-user/gobblin-dist/config
</code></pre>
<p>If you are running on the Amazon free tier, you will probably get an error in the <code>nohup.out</code> file saying there is insufficient memory for the JVM. To fix this add <code>--jvmflags "-Xms256m -Xmx512m"</code> to the <code>start</code> command.</p>
-<p>Data should be written to S3 during the publishing phase of Gobblin. One can confirm data was successfully written to S3 by looking at the <a href="https://console.aws.amazon.com/s3">S3 dashboard</a>.</p>
+<p>Data should be written to S3 during the publishing phase of Gobblin. One can confirm data was successfully written to S3 by looking at the <a href="https://console.aws.amazon.com/s3" rel="nofollow">S3 dashboard</a>.</p>
<h3 id="writing-to-s3-outside-ec2">Writing to S3 Outside EC2</h3>
-<p>It is possible to write to an S3 bucket outside of an EC2 instance. The setup steps are similar to walkthrough outlined above. For more information on writing to S3 outside of AWS, check out <a href="https://aws.amazon.com/articles/5050">this article</a>.</p>
+<p>It is possible to write to an S3 bucket outside of an EC2 instance. The setup steps are similar to walkthrough outlined above. For more information on writing to S3 outside of AWS, check out <a href="https://aws.amazon.com/articles/5050" rel="nofollow">this article</a>.</p>
<h2 id="configuration-properties-for-s3a">Configuration Properties for <code>s3a</code></h2>
<p>The <code>s3a</code> FileSystem has a number of configuration properties that can be set to tune the behavior and performance of the <code>s3a</code> FileSystem. A complete list of the properties can be found here: https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html</p>
<h1 id="faqs">FAQs</h1>
@@ -760,7 +760,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/case-studies/Writing-ORC-Data/index.html b/docs/case-studies/Writing-ORC-Data/index.html
index 773b6a2..fa378a8 100644
--- a/docs/case-studies/Writing-ORC-Data/index.html
+++ b/docs/case-studies/Writing-ORC-Data/index.html
@@ -516,7 +516,7 @@
<li>Writing ORC Data</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/case-studies/Writing-ORC-Data.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/case-studies/Writing-ORC-Data.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -539,7 +539,7 @@
</ul>
</div>
<h1 id="introduction">Introduction</h1>
-<p>Gobblin is capable of writing data to ORC files by leveraging Hive's SerDe library. Gobblin has native integration with Hive SerDe's library via the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/HiveSerDeManager.java">HiveSerDeWrapper</a> class.</p>
+<p>Gobblin is capable of writing data to ORC files by leveraging Hive's SerDe library. Gobblin has native integration with Hive SerDe's library via the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/HiveSerDeManager.java" rel="nofollow">HiveSerDeWrapper</a> class.</p>
<p>This document will briefly explain how Gobblin integrates with Hive's SerDe library, and show an example of writing ORC files.</p>
<h1 id="hive-serde-integration">Hive SerDe Integration</h1>
<p><a href="https://cwiki.apache.org/confluence/display/Hive/SerDe">Hive's SerDe library</a> defines the interface Hive uses for serialization and deserialization of data. The Hive SerDe library has out of the box SerDe support for Avro, ORC, Parquet, CSV, and JSON SerDes. However, users are free to define custom SerDes.</p>
@@ -550,7 +550,7 @@
<li><code>HiveWritableHdfsDataWriter</code> writes a <code>Writable</code> object to a specific file, typically this writes the output of a <code>HiveSerDeConverter</code></li>
</ul>
<h1 id="writing-to-an-orc-file">Writing to an ORC File</h1>
-<p>An end-to-end example of writing to an ORC file is provided in the configuration found <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia-orc.pull">here</a>. This <code>.pull</code> file is almost identical to the Wikipedia example discussed in the <a href="../../Getting-Started/">Getting Started Guide</a>. The only difference is that the output is written in ORC instead of Avro. The configuration file mentioned above can be directly used as a template for writing data to ORC files, below is a detailed explanation of the configuration options that need to be changed, and why they need to be changed.</p>
+<p>An end-to-end example of writing to an ORC file is provided in the configuration found <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia-orc.pull" rel="nofollow">here</a>. This <code>.pull</code> file is almost identical to the Wikipedia example discussed in the <a href="../../Getting-Started/">Getting Started Guide</a>. The only difference is that the output is written in ORC instead of Avro. The configuration file mentioned above can be directly used as a template for writing data to ORC files, below is a detailed explanation of the configuration options that need to be changed, and why they need to be changed.</p>
<ul>
<li><code>converter.classes</code> requires two additional converters: <code>gobblin.converter.avro.AvroRecordToAvroWritableConverter</code> and <code>gobblin.converter.serde.HiveSerDeConverter</code><ul>
<li>The output of the first converter (the <code>WikipediaConverter</code>) returns Avro <code>GenericRecord</code>s</li>
@@ -610,7 +610,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/data-management/DistcpNgEvents/index.html b/docs/data-management/DistcpNgEvents/index.html
index cabbf55..9c5e625 100644
--- a/docs/data-management/DistcpNgEvents/index.html
+++ b/docs/data-management/DistcpNgEvents/index.html
@@ -502,7 +502,7 @@
<li>Distcp-NG events</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/data-management/DistcpNgEvents.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/data-management/DistcpNgEvents.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -577,7 +577,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/data-management/Gobblin-Retention/index.html b/docs/data-management/Gobblin-Retention/index.html
index 9242ef9..dd4f402 100644
--- a/docs/data-management/Gobblin-Retention/index.html
+++ b/docs/data-management/Gobblin-Retention/index.html
@@ -517,7 +517,7 @@
<li>Retention</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/data-management/Gobblin-Retention.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/data-management/Gobblin-Retention.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -596,7 +596,7 @@
<h3 id="retentionaction">RetentionAction</h3>
<p>An abstraction for the kind of action to be performed on all the versions discoverd by the <code>VersionFinder</code> or a subset of versions filtered by the <code>VersionSelectionPolicy</code>. Delete is the default action on selected versions. Gobblin also supports <code>AccessControlAction</code> which sets permissions on selected versions.</p>
<h2 id="retention-configuration">Retention Configuration</h2>
-<p>Gobblin Retention is configured through Gobblin config management. All dataset configs are stored in a config store that can be accessed through a <code>ConfigClient</code>. The gobblin config management uses <a href="https://github.com/typesafehub/config">TypeSafe Config</a>. The language used is <a href="https://github.com/typesafehub/config/blob/master/HOCON.md#hocon-human-optimized-config-object-notation">HOCON</a>, a more readable JSON superset.</p>
+<p>Gobblin Retention is configured through Gobblin config management. All dataset configs are stored in a config store that can be accessed through a <code>ConfigClient</code>. The gobblin config management uses <a href="https://github.com/typesafehub/config" rel="nofollow">TypeSafe Config</a>. The language used is <a href="https://github.com/typesafehub/config/blob/master/HOCON.md#hocon-human-optimized-config-object-notation" rel="nofollow">HOCON</a>, a more readable JSON superset.</p>
<p>The gobblin config management library allows any implementation of config store but for the scope of this document we assume a HDFS based ConfigStore that stores dataset configs in files on HDFS.</p>
<p>Let us take an example ConfigStore instance on HDFS as below.</p>
<pre>
@@ -910,7 +910,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/developer-guide/CodingStyle/index.html b/docs/developer-guide/CodingStyle/index.html
index 317b575..79ddd56 100644
--- a/docs/developer-guide/CodingStyle/index.html
+++ b/docs/developer-guide/CodingStyle/index.html
@@ -506,7 +506,7 @@
<li>Code Style Guide</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/developer-guide/CodingStyle.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/developer-guide/CodingStyle.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -516,7 +516,7 @@
<div class="section">
<h2 id="overview">Overview</h2>
-<p>The code formatting standard in this project is based on the <a href="http://www.oracle.com/technetwork/java/codeconventions-150003.pdf">Oracle/Sun Code Convention</a> and <a href="http://google-styleguide.googlecode.com/svn/trunk/javaguide.html">Google Java Style</a>. </p>
+<p>The code formatting standard in this project is based on the <a href="http://www.oracle.com/technetwork/java/codeconventions-150003.pdf" rel="nofollow">Oracle/Sun Code Convention</a> and <a href="http://google-styleguide.googlecode.com/svn/trunk/javaguide.html" rel="nofollow">Google Java Style</a>. </p>
<h2 id="guideline">Guideline</h2>
<p>The coding style is consistent with most of the open source projects with the following callout:</p>
<ol>
@@ -610,7 +610,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/developer-guide/Contributing/index.html b/docs/developer-guide/Contributing/index.html
index 44eb763..b6a9aea 100644
--- a/docs/developer-guide/Contributing/index.html
+++ b/docs/developer-guide/Contributing/index.html
@@ -511,7 +511,7 @@
<li>Contributing</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/developer-guide/Contributing.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/developer-guide/Contributing.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -534,7 +534,7 @@
<h1 id="contributing-to-gobblin">Contributing to Gobblin</h1>
<p>You can contribute to Gobblin in multiple ways. For resources and guides, please refer <a href="/contributor/">here</a>.</p>
<h2 id="code-contributions">Code Contributions</h2>
-<p>We observe standard Apache practices for code contributions. For code changes, we recommend forking the repository and making your local changes on a feature branch, then updating the Jira, and opening a pull request (PR). A committer will review the changes and merge it in once it is approved. For first time contributors to Gobblin, we do request that you fill out a <a href="https://docs.google.com/a/linkedin.com/forms/d/e/1FAIpQLSeH-8so0m68et6kPvxEiCNqezL7k6cyOlz9W-6eXnk7LEkwiA/viewform">one-time survey</a>, so that we can identify and credit you properly in the future. </p>
+<p>We observe standard Apache practices for code contributions. For code changes, we recommend forking the repository and making your local changes on a feature branch, then updating the Jira, and opening a pull request (PR). A committer will review the changes and merge it in once it is approved. For first time contributors to Gobblin, we do request that you fill out a <a href="https://docs.google.com/a/linkedin.com/forms/d/e/1FAIpQLSeH-8so0m68et6kPvxEiCNqezL7k6cyOlz9W-6eXnk7LEkwiA/viewform" rel="nofollow">one-time survey</a>, so that we can identify and credit you properly in the future. </p>
<h2 id="documentation-contributions">Documentation Contributions</h2>
<p>To make changes to the documentation modify the files under <code>gobblin-docs</code> as you would any other version controlled file. All documentation is checked into GitHub, so the process for making documentation changes is similar to how code changes are made (creating Pull Requests). If one wants to see what the rendered documentation looks like they simply need to take the following steps:</p>
<ol>
@@ -565,7 +565,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/developer-guide/Customization-for-Converter-and-Operator/index.html b/docs/developer-guide/Customization-for-Converter-and-Operator/index.html
index 208e94c..66afddc 100644
--- a/docs/developer-guide/Customization-for-Converter-and-Operator/index.html
+++ b/docs/developer-guide/Customization-for-Converter-and-Operator/index.html
@@ -497,7 +497,7 @@
<li>Customization for Converter and Operator</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/developer-guide/Customization-for-Converter-and-Operator.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/developer-guide/Customization-for-Converter-and-Operator.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -529,7 +529,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/developer-guide/Customization-for-New-Source/index.html b/docs/developer-guide/Customization-for-New-Source/index.html
index 3c3408e..a47da1e 100644
--- a/docs/developer-guide/Customization-for-New-Source/index.html
+++ b/docs/developer-guide/Customization-for-New-Source/index.html
@@ -497,7 +497,7 @@
<li>Customization for New Source</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/developer-guide/Customization-for-New-Source.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/developer-guide/Customization-for-New-Source.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -529,7 +529,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/developer-guide/Documentation-Architecture/index.html b/docs/developer-guide/Documentation-Architecture/index.html
index f231681..f97871c 100644
--- a/docs/developer-guide/Documentation-Architecture/index.html
+++ b/docs/developer-guide/Documentation-Architecture/index.html
@@ -515,7 +515,7 @@
<li>Documentation Architecture</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/developer-guide/Documentation-Architecture.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/developer-guide/Documentation-Architecture.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -536,7 +536,7 @@
</ul>
</div>
<h1 id="documentation-overview">Documentation Overview</h1>
-<p>The documentation for Gobblin is based on <a href="https://readthedocs.org/">ReadTheDocs</a> and <a href="http://www.mkdocs.org/">MkDocs</a>. MkDocs is used to convert MarkDown files to HTML, and ReadTheDocs is used to host the documentation.</p>
+<p>The documentation for Gobblin is based on <a href="https://readthedocs.org/" rel="nofollow">ReadTheDocs</a> and <a href="http://www.mkdocs.org/" rel="nofollow">MkDocs</a>. MkDocs is used to convert MarkDown files to HTML, and ReadTheDocs is used to host the documentation.</p>
<h1 id="github-wiki-limitations">GitHub Wiki Limitations</h1>
<p>Historically, documentation was hosted using the GitHub wiki. The problem is that only Gobblin committers can modify the wiki; any external contributors who want to update or add documentation cannot do so. Editing the Gobblin Wiki is also not PR based, so any committer can make changes without going through a review process.</p>
<h1 id="mkdocs">MkDocs</h1>
@@ -567,7 +567,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/developer-guide/Gobblin-Compliance-Design/index.html b/docs/developer-guide/Gobblin-Compliance-Design/index.html
index 3bddd25..cde4c38 100644
--- a/docs/developer-guide/Gobblin-Compliance-Design/index.html
+++ b/docs/developer-guide/Gobblin-Compliance-Design/index.html
@@ -514,7 +514,7 @@
<li>Gobblin Compliance Design</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/developer-guide/Gobblin-Compliance-Design.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/developer-guide/Gobblin-Compliance-Design.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -649,7 +649,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/developer-guide/GobblinModules/index.html b/docs/developer-guide/GobblinModules/index.html
index 910f2b9..8dc9822 100644
--- a/docs/developer-guide/GobblinModules/index.html
+++ b/docs/developer-guide/GobblinModules/index.html
@@ -519,7 +519,7 @@
<li>Gobblin Modules</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/developer-guide/GobblinModules.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/developer-guide/GobblinModules.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -680,7 +680,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/developer-guide/HighLevelConsumer/index.html b/docs/developer-guide/HighLevelConsumer/index.html
index f6ca58e..475c478 100644
--- a/docs/developer-guide/HighLevelConsumer/index.html
+++ b/docs/developer-guide/HighLevelConsumer/index.html
@@ -503,7 +503,7 @@
<li>High Level Consumer</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/developer-guide/HighLevelConsumer.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/developer-guide/HighLevelConsumer.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -513,12 +513,12 @@
<div class="section">
<h1 id="problem-statement">Problem Statement</h1>
-<p>Current Gobblin Kafka <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/kafka/HighLevelConsumer.java"><code>High Level Consumer</code></a> uses Kafka Consumer (0.8) APIs and gobblin support for them will be deprecated. The Re-design's primary goal is to replace old kafka consumer APIs like <a href="https://archive.apache.org/dist/kafka/0.8.2.2/scaladoc/index.html#kafka.consumer.ConsumerConnector"><code>ConsumerConnector</code></a> and <a href="https://archive.apache.org/dist/kafka/0.8.2.2/scaladoc/index.html#kafka.message.MessageAndMetadata"><code>MessageAndMetadata</code></a> with a consumer abstraction <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/kafka/client/GobblinKafkaConsumerClient.java"><code>GobblinKafkaConsumerClient</code></a>.
+<p>Current Gobblin Kafka <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/kafka/HighLevelConsumer.java"><code>High Level Consumer</code></a> uses Kafka Consumer (0.8) APIs and gobblin support for them will be deprecated. The Re-design's primary goal is to replace old kafka consumer APIs like <a href="https://archive.apache.org/dist/kafka/0.8.2.2/scaladoc/index.html#kafka.consumer.ConsumerConnector"><code>ConsumerConnector</code></a> and <a href="https://archive.apache.org/dist/kafka/0.8.2.2/scaladoc/index.html#kafka.message.MessageAndMetadata"><code>MessageAndMetadata</code></a> with a consumer abstraction <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/kafka/client/GobblinKafkaConsumerClient.java" rel="nofollow"><code>GobblinKafkaConsumerClient</code></a>.
Additionally, the old design uses kafka auto commit feature which can cause potential loss of messages when offsets are committed and the system fails before messages are processed.</p>
<p>Detailed design and implementation details can be found <a href="https://cwiki.apache.org/confluence/display/GOBBLIN/GIP+5%3A+High+Level+Consumer+Re-design">here</a></p>
<h1 id="new-design-details">New Design & Details</h1>
<p>GobblinKafkaConsumerClient</p>
-<p>The new design uses <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/kafka/client/GobblinKafkaConsumerClient.java"><code>GobblinKafkaConsumerClient</code></a> which is a simplified, generic wrapper client to communicate with Kafka. This class does not depend on classes defined in kafka-clients library. This allows the high level consumer to work with different versions of kafka. Concrete classes implementing this interface use a specific version of kafka-client library. See <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-09/src/main/java/org/apache/gobblin/kafka/client/Kafka09ConsumerClient.java"><code>Kafka09ConsumerClient</code></a></p>
+<p>The new design uses <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/kafka/client/GobblinKafkaConsumerClient.java" rel="nofollow"><code>GobblinKafkaConsumerClient</code></a> which is a simplified, generic wrapper client to communicate with Kafka. This class does not depend on classes defined in kafka-clients library. This allows the high level consumer to work with different versions of kafka. Concrete classes implementing this interface use a specific version of kafka-client library. See <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-09/src/main/java/org/apache/gobblin/kafka/client/Kafka09ConsumerClient.java" rel="nofollow"><code>Kafka09ConsumerClient</code></a></p>
<p>Manual Offset Commit</p>
<p><code>GobblinKafkaConsumerClient</code> API has been enhanced to allow manual committing of offsets.</p>
<pre><code class="java"> /**
@@ -546,7 +546,7 @@
</code></pre>
<p>High level consumer records topic partitions and their offsets AFTER the messages are processed and commits them periodically to kafka. This ensures at-least once delivery in case of a failure.</p>
-<p>Additionally, APIs are provided to subscribe to a topic along with a <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/kafka/client/GobblinConsumerRebalanceListener.java"><code>GobblinKafkaRebalanceListener</code></a> that provides hooks to when a consumer joins/leaves a consumer group.
+<p>Additionally, APIs are provided to subscribe to a topic along with a <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/kafka/client/GobblinConsumerRebalanceListener.java" rel="nofollow"><code>GobblinKafkaRebalanceListener</code></a> that provides hooks to when a consumer joins/leaves a consumer group.
In this case, we commit remaining offsets and clear offset caches.</p>
<pre><code class="java"> /**
* Subscribe to a topic
@@ -586,7 +586,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/developer-guide/IDE-setup/index.html b/docs/developer-guide/IDE-setup/index.html
index 238474d..1f9d4da 100644
--- a/docs/developer-guide/IDE-setup/index.html
+++ b/docs/developer-guide/IDE-setup/index.html
@@ -511,7 +511,7 @@
<li>IDE setup</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/developer-guide/IDE-setup.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/developer-guide/IDE-setup.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -531,7 +531,7 @@
</ul>
</div>
<h1 id="introduction">Introduction</h1>
-<p>This document is for users who want to import the Gobblin code base into an <a href="https://en.wikipedia.org/wiki/Integrated_development_environment">IDE</a> and directly modify that Gobblin code base. This is not for users who want to just setup Gobblin as a Maven dependency.</p>
+<p>This document is for users who want to import the Gobblin code base into an <a href="https://en.wikipedia.org/wiki/Integrated_development_environment" rel="nofollow">IDE</a> and directly modify that Gobblin code base. This is not for users who want to just setup Gobblin as a Maven dependency.</p>
<h1 id="intellij-integration">IntelliJ Integration</h1>
<p>Gobblin uses standard build tools to import code into an IntelliJ project. Execute the following command to build the necessary <code>*.iml</code> files:</p>
<pre><code>./gradlew clean idea
@@ -547,7 +547,7 @@
<p>Once the command finishes, use standard practices to import the project into Eclipse.
If you plan to write code, remember to read the <a href="CodingStyle">coding style guide</a> and import the coding style template file. </p>
<h1 id="lombok">Lombok</h1>
-<p>Gobblin uses <a href="https://projectlombok.org/">Lombok</a> for reducing boilerplate code. Lombok auto generates boilerplate code at runtime if you are building gobblin from command line.If you are using an IDE, you will see compile errors in some of the classes that use Lombok. Please follow the <a href="https://projectlombok.org/download.html">IDE setup instructions</a> for your IDE to setup lombok.</p>
+<p>Gobblin uses <a href="https://projectlombok.org/" rel="nofollow">Lombok</a> for reducing boilerplate code. Lombok auto generates boilerplate code at runtime if you are building gobblin from command line.If you are using an IDE, you will see compile errors in some of the classes that use Lombok. Please follow the <a href="https://projectlombok.org/download.html" rel="nofollow">IDE setup instructions</a> for your IDE to setup lombok.</p>
</div>
</div>
@@ -570,7 +570,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/developer-guide/Monitoring-Design/index.html b/docs/developer-guide/Monitoring-Design/index.html
index 07885ce..c647596 100644
--- a/docs/developer-guide/Monitoring-Design/index.html
+++ b/docs/developer-guide/Monitoring-Design/index.html
@@ -499,7 +499,7 @@
<li>Monitoring Design</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/developer-guide/Monitoring-Design.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/developer-guide/Monitoring-Design.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -532,7 +532,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/index.html b/docs/index.html
index ebf4605..16c1f81 100644
--- a/docs/index.html
+++ b/docs/index.html
@@ -488,7 +488,7 @@
<li>Home</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/index.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/index.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -504,8 +504,8 @@
<p>Over the years, LinkedIn's data infrastructure team built custom solutions for ingesting diverse data entities into our Hadoop eco-system. At one point, we were running 15 types of ingestion pipelines which created significant data quality, metadata management, development, and operation challenges.</p>
<p>Our experiences and challenges motivated us to build <em>Gobblin</em>. Gobblin is a universal data ingestion framework for extracting, transforming, and loading large volume of data from a variety of data sources, e.g., databases, rest APIs, FTP/SFTP servers, filers, etc., onto Hadoop. Gobblin handles the common routine tasks required for all data ingestion ETLs, including job/task scheduling, task partitioning, error handling, state management, data quality checking, data publishing, etc. Gobblin ingests data from different data sources in the same execution framework, and manages metadata of different sources all in one place. This, combined with other features such as auto scalability, fault tolerance, data quality assurance, extensibility, and the ability of handling data model evolution, makes Gobblin an easy-to-use, self-serving, and efficient data ingestion framework.</p>
<p>You can find a lot of useful resources in our wiki pages, including <a href="Getting-Started">how to get started with Gobblin</a>, an <a href="Gobblin-Architecture">architecture overview of Gobblin</a>, and
-the <a href="user-guide/Gobblin-Deployment">Gobblin user guide</a>. We also provide a discussion group: <a href="https://groups.google.com/forum/#!forum/gobblin-users">Google Gobblin-Users Group</a>. Please feel free to post any questions or comments.</p>
-<p>For a detailed overview, please take a look at the <a href="http://www.vldb.org/pvldb/vol8/p1764-qiao.pdf">VLDB 2015 paper</a> and the <a href="https://engineering.linkedin.com/data-ingestion/gobblin-big-data-ease">LinkedIn's Gobblin blog post</a>.</p>
+the <a href="user-guide/Gobblin-Deployment">Gobblin user guide</a>. We also provide a discussion group: <a href="https://groups.google.com/forum/#!forum/gobblin-users" rel="nofollow">Google Gobblin-Users Group</a>. Please feel free to post any questions or comments.</p>
+<p>For a detailed overview, please take a look at the <a href="http://www.vldb.org/pvldb/vol8/p1764-qiao.pdf" rel="nofollow">VLDB 2015 paper</a> and the <a href="https://engineering.linkedin.com/data-ingestion/gobblin-big-data-ease" rel="nofollow">LinkedIn's Gobblin blog post</a>.</p>
</div>
</div>
@@ -519,7 +519,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/metrics/Existing-Reporters/index.html b/docs/metrics/Existing-Reporters/index.html
index 91f615e..c212bba 100644
--- a/docs/metrics/Existing-Reporters/index.html
+++ b/docs/metrics/Existing-Reporters/index.html
@@ -505,7 +505,7 @@
<li>Existing Reporters</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/metrics/Existing-Reporters.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/metrics/Existing-Reporters.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -524,18 +524,18 @@
</div>
<h1 id="metric-reporters">Metric Reporters</h1>
<ul>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/OutputStreamReporter.java">Output Stream Reporter</a>: allows printing metrics to any OutputStream, including STDOUT and files.</li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/metrics/kafka/KafkaReporter.java">Kafka Reporter</a>: emits metrics to Kafka topic as Json messages.</li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/metrics/kafka/KafkaAvroReporter.java">Kafka Avro Reporter</a>: emits metrics to Kafka topic as Avro messages with schema <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/MetricReport.avsc">MetricReport</a>.</li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-metrics-graphite/src/main/java/org/apache/gobblin/metrics/graphite/GraphiteReporter.java">Graphite Reporter</a>: emits metrics to Graphite. This reporter has a different, deprecated construction API included in its javadoc.</li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-metrics-influxdb/src/main/java/org/apache/gobblin/metrics/influxdb/InfluxDBReporter.java">Influx DB Reporter</a>: emits metrics to Influx DB. This reporter has a different, deprecated construction API included in its javadoc.</li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-metrics-hadoop/src/main/java/org/apache/gobblin/metrics/hadoop/HadoopCounterReporter.java">Hadoop Counter Reporter</a>: emits metrics as Hadoop counters at the end of the execution. Available for old and new Hadoop API. This reporter has a different, deprecated construction API included in its javadoc. Due to limits on the number of Hadoop counters that can be created, this reporter is not recommended except for applications with very few metrics.</li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/OutputStreamReporter.java" rel="nofollow">Output Stream Reporter</a>: allows printing metrics to any OutputStream, including STDOUT and files.</li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/metrics/kafka/KafkaReporter.java" rel="nofollow">Kafka Reporter</a>: emits metrics to Kafka topic as Json messages.</li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/metrics/kafka/KafkaAvroReporter.java" rel="nofollow">Kafka Avro Reporter</a>: emits metrics to Kafka topic as Avro messages with schema <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/MetricReport.avsc" rel="nofollow">MetricReport</a>.</li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-metrics-graphite/src/main/java/org/apache/gobblin/metrics/graphite/GraphiteReporter.java" rel="nofollow">Graphite Reporter</a>: emits metrics to Graphite. This reporter has a different, deprecated construction API included in its javadoc.</li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-metrics-influxdb/src/main/java/org/apache/gobblin/metrics/influxdb/InfluxDBReporter.java" rel="nofollow">Influx DB Reporter</a>: emits metrics to Influx DB. This reporter has a different, deprecated construction API included in its javadoc.</li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-metrics-hadoop/src/main/java/org/apache/gobblin/metrics/hadoop/HadoopCounterReporter.java" rel="nofollow">Hadoop Counter Reporter</a>: emits metrics as Hadoop counters at the end of the execution. Available for old and new Hadoop API. This reporter has a different, deprecated construction API included in its javadoc. Due to limits on the number of Hadoop counters that can be created, this reporter is not recommended except for applications with very few metrics.</li>
</ul>
<h1 id="event-reporters">Event Reporters</h1>
<ul>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/OutputStreamEventReporter.java">Output Stream Event Reporter</a>: Emits events to any output stream, including STDOUT and files.</li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/metrics/kafka/KafkaEventReporter.java">Kafka Event Reporter</a>: Emits events to Kafka topic as Json messages.</li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/metrics/kafka/KafkaAvroEventReporter.java">Kafka Avro Event Reporter</a>: Emits events to Kafka topic as Avro messages using the schema <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/GobblinTrackingEvent.avsc">GobblinTrackingEvent</a>.</li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/OutputStreamEventReporter.java" rel="nofollow">Output Stream Event Reporter</a>: Emits events to any output stream, including STDOUT and files.</li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/metrics/kafka/KafkaEventReporter.java" rel="nofollow">Kafka Event Reporter</a>: Emits events to Kafka topic as Json messages.</li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/metrics/kafka/KafkaAvroEventReporter.java" rel="nofollow">Kafka Avro Event Reporter</a>: Emits events to Kafka topic as Avro messages using the schema <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/GobblinTrackingEvent.avsc" rel="nofollow">GobblinTrackingEvent</a>.</li>
</ul>
</div>
@@ -559,7 +559,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/metrics/Gobblin-Metrics-Architecture/index.html b/docs/metrics/Gobblin-Metrics-Architecture/index.html
index daa19b1..459686b 100644
--- a/docs/metrics/Gobblin-Metrics-Architecture/index.html
+++ b/docs/metrics/Gobblin-Metrics-Architecture/index.html
@@ -525,7 +525,7 @@
<li>Gobblin Metrics Architecture</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/metrics/Gobblin-Metrics-Architecture.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/metrics/Gobblin-Metrics-Architecture.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -561,31 +561,31 @@
<li>A <code>String</code> <code>name</code>. The name is not used by the core metrics engine, but can be accessed by users to identify the context.</li>
<li>A reference to the parent metric context, or null if it has no parent.</li>
<li>A list of children metric context references, stored as soft references.</li>
-<li>An object of type <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/Tagged.java">Tagged</a> containing the tags for this metric context.</li>
-<li>A <code>Set</code> of notification targets. Notification targets are objects of type <a href="https://google.github.io/guava/releases/15.0/api/docs/com/google/common/base/Function.html">Function</a><<a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/notification/Notification.java">Notification</a>, Void> which are all called every time there is a new notification. Notifications can be submitted to the Metric Context using the method <code>sendNotification(Notification)</code>. Notification targets can be added using <code>addNotificationTarget(Function<Notification, Void>)</code>.</li>
+<li>An object of type <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/Tagged.java" rel="nofollow">Tagged</a> containing the tags for this metric context.</li>
+<li>A <code>Set</code> of notification targets. Notification targets are objects of type <a href="https://google.github.io/guava/releases/15.0/api/docs/com/google/common/base/Function.html" rel="nofollow">Function</a><<a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/notification/Notification.java" rel="nofollow">Notification</a>, Void> which are all called every time there is a new notification. Notifications can be submitted to the Metric Context using the method <code>sendNotification(Notification)</code>. Notification targets can be added using <code>addNotificationTarget(Function<Notification, Void>)</code>.</li>
<li>A lazily instantiated <code>ExecutorService</code> used for asynchronously executing the notification targets. The executor service will only be started the first time there is a notification and the number of notification targets is positive.</li>
<li>A <code>ConcurrentMap</code> from metric names to <code>Metric</code> for all metrics registered in this Metric Context. Metrics can be added to this map using the <code>register(Metric)</code>, <code>register(String, Metric)</code>, or <code>registerAll(MetricSet)</code>, although it is recommended to instead use the methods to create and register the metrics. Metric Context implements getter methods for all metrics, as well as for each type of metric individually (<code>getMetrics</code>, <code>getGauges</code>, <code>getCounters</code>, <code>getHistograms</code>, <code>getMeters</code>, <code>getTimers</code>).</li>
</ul>
<h1 id="metrics">Metrics</h1>
-<p>All metrics extend the interface <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/ContextAwareMetric.java">ContextAwareMetric</a>. Each metric type in Dropwizard Metrics is extended to a Context Aware type: <code>ContextAwareCounter</code>, <code>ContextAwareGauge</code>, <code>ContextAwareHistogram</code>, <code>ContextAwareMeter</code>, <code>ContextAwareTimer</code>.</p>
+<p>All metrics extend the interface <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/ContextAwareMetric.java" rel="nofollow">ContextAwareMetric</a>. Each metric type in Dropwizard Metrics is extended to a Context Aware type: <code>ContextAwareCounter</code>, <code>ContextAwareGauge</code>, <code>ContextAwareHistogram</code>, <code>ContextAwareMeter</code>, <code>ContextAwareTimer</code>.</p>
<p>Context Aware metrics all always created from the Metric Context where they will be registered. For example, to get a counter under Metric Context <code>context</code>, the user would call <code>context.counter("counter.name")</code>. This method first checks all registered metrics in the Metric Context to find a counter with that name, if it succeeds, it simply returns that counter. If a counter with that name has not been registered in <code>context</code>, then a new <code>ContextAwareCounter</code> is created and registered in <code>context</code>.</p>
<p>On creation, each Context Aware metric (except Gauges) checks if its parent Metric Context has parents itself. If so, then it automatically creates a metric of the same type, with the same name, in that parent. This will be repeated recursively until, at the end, all ancestor Metric Contexts will all contain a context aware metric of the same type and with the same name. Every time the context aware metric is updated, the metric will automatically call the same update method, with the same update value, for its parent metric. Again, this will continue recursively until the corresponding metrics in all ancestor metric contexts are updated by the same value. If multiple children of a metric context <code>context</code> all have metrics with the same name, when either of them is updated, the corresponding metric in <code>context</code> will also get updated. In this way, the corresponding metric in <code>context</code> will aggregate all updated to the metrics in the children context.</p>
<p>Users can also register objects of type <code>com.codahale.metrics.Metric</code> with any Metric Context, but they will not be auto-aggregated.</p>
<h1 id="events">Events</h1>
-<p>Events are objects of type <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/GobblinTrackingEvent.avsc">GobblinTrackingEvent</a>, which is a type generated from an Avro schema. Events have:</p>
+<p>Events are objects of type <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/GobblinTrackingEvent.avsc" rel="nofollow">GobblinTrackingEvent</a>, which is a type generated from an Avro schema. Events have:</p>
<ul>
<li>A <code>namespace</code>.</li>
<li>A <code>name</code>.</li>
<li>A <code>timestamp</code>.</li>
<li>A <code>Map<String,String></code> of <code>metadata</code>.</li>
</ul>
-<p>Events are submitted using the <code>MetricContext#submitEvent(GobblinTrackingEvent)</code> method. When called, this method packages the event into an <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/notification/EventNotification.java">EventNotification</a> and submits it to the metric context using the method <code>MetricContext#sendNotification(Notification)</code>. This notification is passed to all metrics context ancestors. Each notification target of each ancestor metric context will receive the EventNotification. Events are not stored by any Metric Context, so the notification targets need to handle these events appropriately.</p>
-<p>Events can be created manually using Avro constructors, and using the method <code>context.submitEvent(GobblinTrackinEvent)</code>, but this is unfriendly when trying to build events incrementally, especially when using metadata. To address this, users can instead use <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/event/EventSubmitter.java">EventSubmitter</a> which is an abstraction around the Avro constructor for GobblinTrackingEvent.</p>
+<p>Events are submitted using the <code>MetricContext#submitEvent(GobblinTrackingEvent)</code> method. When called, this method packages the event into an <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/notification/EventNotification.java" rel="nofollow">EventNotification</a> and submits it to the metric context using the method <code>MetricContext#sendNotification(Notification)</code>. This notification is passed to all metrics context ancestors. Each notification target of each ancestor metric context will receive the EventNotification. Events are not stored by any Metric Context, so the notification targets need to handle these events appropriately.</p>
+<p>Events can be created manually using Avro constructors, and using the method <code>context.submitEvent(GobblinTrackinEvent)</code>, but this is unfriendly when trying to build events incrementally, especially when using metadata. To address this, users can instead use <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/event/EventSubmitter.java" rel="nofollow">EventSubmitter</a> which is an abstraction around the Avro constructor for GobblinTrackingEvent.</p>
<h2 id="event-submitter">Event Submitter</h2>
<p>An event submitter is created using an <code>EventSubmitter.Builder</code>. It is associated with a Metric Context where it will submit all events, and it contains a <code>namespace</code> and default <code>metadata</code> that will be applied to all events generated through the event submitter. The user can then call <code>EventSubmitter#submit</code> which will package the event with the provided metadata and submit it to the Metric Context.</p>
<h1 id="reporters">Reporters</h1>
<p>Reporters export the metrics and/or events of a metric context to a sink. Reporters extend the interface <code>com.codahale.metrics.Reporter</code>. Most reporters will attach themselves to a Metric Context. The reporter can then navigate the Metric Context tree where the Metric Context belongs, get tags and metrics, get notified of events, and export them to the sink.</p>
-<p>The two best entry points for developing reporters are <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/RecursiveScheduledMetricReporter.java">RecursiveScheduledMetricReporter</a> and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/EventReporter.java">EventReporter</a>. These classes do most of the heavy lifting for reporting metrics and events respectively. They are both scheduled reporters, meaning the will export their metrics / events following a configurable schedule.</p>
+<p>The two best entry points for developing reporters are <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/RecursiveScheduledMetricReporter.java" rel="nofollow">RecursiveScheduledMetricReporter</a> and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/EventReporter.java" rel="nofollow">EventReporter</a>. These classes do most of the heavy lifting for reporting metrics and events respectively. They are both scheduled reporters, meaning the will export their metrics / events following a configurable schedule.</p>
<h2 id="recursiveschedulemetricreporter">RecursiveScheduleMetricReporter</h2>
<p>This abstract reporter base is used for emitting metrics on a schedule. The reporter, on creation, is attached to a particular Metric Report. Every time the reporter is required to emit events, the reporter selects the attached Metric Context and all descendant Metric Contexts. For each of these metric contexts, it queries the Metric Context for all metrics, filtered by an optional user supplied filter, and then calls <code>RecursiveScheduledMetricReporter#report</code>, providing the method with all appropriate metrics and tags. Developers need only implement the report method.</p>
<h2 id="eventreporter">EventReporter</h2>
@@ -613,7 +613,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/metrics/Gobblin-Metrics-Performance/index.html b/docs/metrics/Gobblin-Metrics-Performance/index.html
index 8a4b316..833f390 100644
--- a/docs/metrics/Gobblin-Metrics-Performance/index.html
+++ b/docs/metrics/Gobblin-Metrics-Performance/index.html
@@ -524,7 +524,7 @@
<li>Gobblin Metrics Performance</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/metrics/Gobblin-Metrics-Performance.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/metrics/Gobblin-Metrics-Performance.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -642,7 +642,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/metrics/Gobblin-Metrics-next-generation-instrumentation-for-applications/index.html b/docs/metrics/Gobblin-Metrics-next-generation-instrumentation-for-applications/index.html
index d3a1bca..b1d7244 100644
--- a/docs/metrics/Gobblin-Metrics-next-generation-instrumentation-for-applications/index.html
+++ b/docs/metrics/Gobblin-Metrics-next-generation-instrumentation-for-applications/index.html
@@ -489,7 +489,7 @@
<li>Gobblin Metrics next generation instrumentation for applications</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/metrics/Gobblin-Metrics-next-generation-instrumentation-for-applications.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/metrics/Gobblin-Metrics-next-generation-instrumentation-for-applications.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -513,7 +513,7 @@
<li> Easily switch between sinks where reports and events are emitted.
<li> Generate queriable reports.
</ul>
-Among existing solutions, we found <a href="http://metrics.dropwizard.io/">Dropwizard Metrics</a> to be the closest to what we needed, but it was not enough, so we developed Gobblin Metrics.
+Among existing solutions, we found <a href="http://metrics.dropwizard.io/" rel="nofollow">Dropwizard Metrics</a> to be the closest to what we needed, but it was not enough, so we developed Gobblin Metrics.
</p>
<p>
@@ -534,7 +534,7 @@
</p>
<p>
- To learn more about Gobblin Metrics, check out the <a href="https://gobblin.readthedocs.io/en/latest/metrics/Gobblin-Metrics/">Wiki</a> and the <a href="https://github.com/apache/incubator-gobblin">Gobblin project</a> in Github.
+ To learn more about Gobblin Metrics, check out the <a href="https://gobblin.readthedocs.io/en/latest/metrics/Gobblin-Metrics/" rel="nofollow">Wiki</a> and the <a href="https://github.com/apache/incubator-gobblin" rel="nofollow">Gobblin project</a> in Github.
</p>
</div>
@@ -549,7 +549,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/metrics/Gobblin-Metrics/index.html b/docs/metrics/Gobblin-Metrics/index.html
index d965346..ee75f25 100644
--- a/docs/metrics/Gobblin-Metrics/index.html
+++ b/docs/metrics/Gobblin-Metrics/index.html
@@ -514,7 +514,7 @@
<li>Quick Start</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/metrics/Gobblin-Metrics.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/metrics/Gobblin-Metrics.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -536,7 +536,7 @@
</div>
<p>Gobblin Metrics is a metrics library for emitting metrics and events instrumenting java applications.
Metrics and events are easy to use and enriched with tags. Metrics allow full granularity, auto-aggregation, and configurable
-reporting schedules. Gobblin Metrics is based on <a href="http://metrics.dropwizard.io/">Dropwizard Metrics</a>, enhanced to better support
+reporting schedules. Gobblin Metrics is based on <a href="http://metrics.dropwizard.io/" rel="nofollow">Dropwizard Metrics</a>, enhanced to better support
modular applications (by providing hierarchical, auto-aggregated metrics) and their monitoring / auditing.</p>
<h1 id="quick-start">Quick Start</h1>
<p>The following code excerpt shows the functionality of Gobblin Metrics.</p>
@@ -592,7 +592,7 @@
<h1 id="metrics">Metrics</h1>
<p>Metrics are used to monitor the progress of an application. Metrics are emitted regularly following a schedule and represent
the current state of the application. The metrics supported by Gobblin Metrics are the same ones as those supported
-by <a href="http://metrics.dropwizard.io/3.1.0/manual/core/">Dropwizard Metrics Core</a>, adapted for tagging and auto-aggregation.
+by <a href="http://metrics.dropwizard.io/3.1.0/manual/core/" rel="nofollow">Dropwizard Metrics Core</a>, adapted for tagging and auto-aggregation.
The types supported are:</p>
<ul>
<li>Counter: simple long counter.</li>
@@ -644,7 +644,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/metrics/Implementing-New-Reporters/index.html b/docs/metrics/Implementing-New-Reporters/index.html
index ced9857..f8a73c6 100644
--- a/docs/metrics/Implementing-New-Reporters/index.html
+++ b/docs/metrics/Implementing-New-Reporters/index.html
@@ -512,7 +512,7 @@
<li>Implementing New Reporters</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/metrics/Implementing-New-Reporters.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/metrics/Implementing-New-Reporters.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -531,7 +531,7 @@
<li><a href="#other-reporters">Other Reporters</a></li>
</ul>
</div>
-<p>The two best entry points for implementing custom reporters are <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/RecursiveScheduledMetricReporter.java">RecursiveScheduledMetricReporter</a> and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/EventReporter.java">EventReporter</a>. Each of these classes automatically schedules reporting, extracts the correct metrics, and calls a single method that must be implemented by the developer. These methods also implement builder patterns that can be extended by the developer.</p>
+<p>The two best entry points for implementing custom reporters are <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/RecursiveScheduledMetricReporter.java" rel="nofollow">RecursiveScheduledMetricReporter</a> and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/EventReporter.java" rel="nofollow">EventReporter</a>. Each of these classes automatically schedules reporting, extracts the correct metrics, and calls a single method that must be implemented by the developer. These methods also implement builder patterns that can be extended by the developer.</p>
<p>In the interest of giving more control to the users, metric and event reporters are kept separate, allowing users to more easily specify separate sinks for events and metrics. However, it is possible to implement a single report that handles both events and metrics.</p>
<blockquote>
<p>It is recommended that each reporter has a constructor with signature <code><init>(Properties)</code>. In the near future we are planning to implement auto-starting, file-configurable reporting similar to Log4j architecture, and compliant reporters will be required to have such a constructor.</p>
@@ -638,7 +638,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/metrics/Metrics-for-Gobblin-ETL/index.html b/docs/metrics/Metrics-for-Gobblin-ETL/index.html
index d3ae6df..0b1e231 100644
--- a/docs/metrics/Metrics-for-Gobblin-ETL/index.html
+++ b/docs/metrics/Metrics-for-Gobblin-ETL/index.html
@@ -541,7 +541,7 @@
<li>Metrics for Gobblin ETL</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/metrics/Metrics-for-Gobblin-ETL.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/metrics/Metrics-for-Gobblin-ETL.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -747,10 +747,10 @@
</ul>
<h1 id="customizing-instrumentation">Customizing Instrumentation</h1>
<h2 id="custom-constructs">Custom constructs</h2>
-<p>When using a custom construct (for example a custom extractor for your data source), you will get the above mentioned instrumentation for free. However, you may want to implement additional metrics. To aid with this, instead of extending the usual class Extractor, you can extend the class <code>gobblin.instrumented.extractor.InstrumentedExtractor</code>. Similarly, for each construct there is an instrumented version that allows extension of the default metrics (<a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/instrumented/extractor/InstrumentedExtractor.java">InstrumentedExtractor</a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/instrumented/converter/InstrumentedConverter.java">InstrumentedConverter</a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/instrumented/fork/InstrumentedForkOperator.java">InstrumentedForkOperator</a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/instrumented/qualitychecker/InstrumentedRowLevelPolicy.java">InstrumentedRowLevelPolicy</a>, and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/instrumented/writer/InstrumentedDataWriter.java">InstrumentedDataWriter</a>).</p>
+<p>When using a custom construct (for example a custom extractor for your data source), you will get the above mentioned instrumentation for free. However, you may want to implement additional metrics. To aid with this, instead of extending the usual class Extractor, you can extend the class <code>gobblin.instrumented.extractor.InstrumentedExtractor</code>. Similarly, for each construct there is an instrumented version that allows extension of the default metrics (<a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/instrumented/extractor/InstrumentedExtractor.java" rel="nofollow">InstrumentedExtractor</a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/instrumented/converter/InstrumentedConverter.java" rel="nofollow">InstrumentedConverter</a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/instrumented/fork/InstrumentedForkOperator.java" rel="nofollow">InstrumentedForkOperator</a>, <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/instrumented/qualitychecker/InstrumentedRowLevelPolicy.java" rel="nofollow">InstrumentedRowLevelPolicy</a>, and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/instrumented/writer/InstrumentedDataWriter.java" rel="nofollow">InstrumentedDataWriter</a>).</p>
<p>All of the instrumented constructs have Javadoc providing with additional information. In general, when extending an instrumented construct, you will have to implement a different method. For example, when extending an InstrumentedExtractor, instead of implementing <code>readRecord</code>, you will implement <code>readRecordImpl</code>. To make this clearer for the user, implementing <code>readRecord</code> will throw a compilation error, and the javadoc of each method specifies the method that should be implemented.</p>
<h3 id="instrumentable-interface">Instrumentable Interface</h3>
-<p>Instrumented constructs extend the interface <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/instrumented/Instrumentable.java">Instrumentable</a>. It contains the following methods:</p>
+<p>Instrumented constructs extend the interface <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/instrumented/Instrumentable.java" rel="nofollow">Instrumentable</a>. It contains the following methods:</p>
<ul>
<li><code>getMetricContext()</code>: get the default metric context generated for that instance of the construct, with all the appropriate tags. Use this metric context to create any additional metrics.</li>
<li><code>isInstrumentationEnabled()</code>: returns true if instrumentation is enabled.</li>
@@ -762,7 +762,7 @@
<h3 id="callback-methods">Callback Methods</h3>
<p>Instrumented constructs have a set of callback methods that are called at different points in the processing of each record, and which can be used to update metrics. For example, the <code>InstrumentedExtractor</code> has the callbacks <code>beforeRead()</code>, <code>afterRead(D, long)</code>, and <code>onException(Exception)</code>. The javadoc for the instrumented constructs has further descriptions for each callback. Users should always call <code>super()</code> when overriding this callbacks, as default metrics depend on that.</p>
<h2 id="custom-reporters">Custom Reporters</h2>
-<p>Besides the reporters implemented by default (file, Kafka, and JMX), users can add custom reporters to the classpath and instruct Gobblin to use these reporters. To do this, users should extend the interface <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/CustomReporterFactory.java">CustomReporterFactory</a>, and specify a comma-separated list of CustomReporterFactory classes in the configuration key <code>metrics.reporting.custom.builders</code>.</p>
+<p>Besides the reporters implemented by default (file, Kafka, and JMX), users can add custom reporters to the classpath and instruct Gobblin to use these reporters. To do this, users should extend the interface <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/CustomReporterFactory.java" rel="nofollow">CustomReporterFactory</a>, and specify a comma-separated list of CustomReporterFactory classes in the configuration key <code>metrics.reporting.custom.builders</code>.</p>
<p>Gobblin will automatically search for these CustomReporterFactory implementations, instantiate each one with a parameter-less constructor, and then call the method <code>newScheduledReporter(MetricContext, Properties)</code>, where the properties contain all of the input configurations supplied to Gobblin. Gobblin will then manage this <code>ScheduledReporter</code>.</p>
</div>
@@ -786,7 +786,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/miscellaneous/Camus-to-Gobblin-Migration/index.html b/docs/miscellaneous/Camus-to-Gobblin-Migration/index.html
index 7d715dc..a79b429 100644
--- a/docs/miscellaneous/Camus-to-Gobblin-Migration/index.html
+++ b/docs/miscellaneous/Camus-to-Gobblin-Migration/index.html
@@ -519,7 +519,7 @@
<li>Camus to Gobblin Migration</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/miscellaneous/Camus-to-Gobblin-Migration.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/miscellaneous/Camus-to-Gobblin-Migration.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -542,7 +542,7 @@
<li><a href="#migrating-from-camus-to-gobblin-in-production">Migrating from Camus to Gobblin in Production</a></li>
</ul>
</div>
-<p>This page is a guide for <a href="https://github.com/linkedin/camus">Camus</a> → Gobblin migration, intended for users and organizations currently using Camus. Camus is LinkedIn's previous-generation Kafka-HDFS pipeline.</p>
+<p>This page is a guide for <a href="https://github.com/linkedin/camus" rel="nofollow">Camus</a> → Gobblin migration, intended for users and organizations currently using Camus. Camus is LinkedIn's previous-generation Kafka-HDFS pipeline.</p>
<p>It is recommended that one read <a href="../case-studies/Kafka-HDFS-Ingestion">Kafka-HDFS Ingestion</a> before reading this page. This page focuses on the Kafka-related configuration properties in Gobblin vs Camus.</p>
<h2 id="advantages-of-migrating-to-gobblin">Advantages of Migrating to Gobblin</h2>
<p><strong>Operability</strong>: Gobblin is a generic data ingestion pipeline that supports not only Kafka but several other data sources, and new data sources can be easily added. If you have multiple data sources, using a single tool to ingest data from these sources is a lot more pleasant operationally than deploying a separate tool for each source.</p>
@@ -781,7 +781,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/miscellaneous/Exactly-Once-Support/index.html b/docs/miscellaneous/Exactly-Once-Support/index.html
index 856d592..620fc07 100644
--- a/docs/miscellaneous/Exactly-Once-Support/index.html
+++ b/docs/miscellaneous/Exactly-Once-Support/index.html
@@ -508,7 +508,7 @@
<li>Exactly Once Support</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/miscellaneous/Exactly-Once-Support.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/miscellaneous/Exactly-Once-Support.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -696,7 +696,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/project/Feature-List/index.html b/docs/project/Feature-List/index.html
index dd9dd3c..2def23a 100644
--- a/docs/project/Feature-List/index.html
+++ b/docs/project/Feature-List/index.html
@@ -499,7 +499,7 @@
<li>Feature List</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/project/Feature-List.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/project/Feature-List.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -581,7 +581,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/project/Posts/index.html b/docs/project/Posts/index.html
index bf6a87b..3578bde 100644
--- a/docs/project/Posts/index.html
+++ b/docs/project/Posts/index.html
@@ -496,7 +496,7 @@
<li>Posts</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/project/Posts.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/project/Posts.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -530,7 +530,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/project/Talks-and-Tech-Blogs/index.html b/docs/project/Talks-and-Tech-Blogs/index.html
index c2fa141..99cafd6 100644
--- a/docs/project/Talks-and-Tech-Blogs/index.html
+++ b/docs/project/Talks-and-Tech-Blogs/index.html
@@ -496,7 +496,7 @@
<li>Talks and Tech Blog Posts</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/project/Talks-and-Tech-Blogs.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/project/Talks-and-Tech-Blogs.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -507,19 +507,19 @@
<p>Gobblin Talks and Tech Blogs</p>
<ul>
-<li>QCon presentation <small><b>| <a href="http://www.slideshare.net/LinQiao1/gobblin-big-data-with-ease">Presentation</a> | (Nov 5th, 2014)</b></small></li>
-<li>Engineering Blog Post <small><b>| <a href="http://engineering.linkedin.com/data-ingestion/gobblin-big-data-ease">Presentation</a> | (Nov 25th, 2014)</b></small></li>
-<li>Bigger, Faster, Easier: Building a Real-Time Self Service Data Analytics Ecosystem at LinkedIn <small><b>| <a href="http://www.slideshare.net/Hadoop_Summit/bigger-faster-easier-building-a-realtime-self-service-data-analytics-ecosystem-at-linkedin?qid=9c8f8c33-0083-495b-a6e2-572ac45f7f2c&v=qf1&b=&from_search=9">Presentation</a> | (Hadoop Summit 2015)</b></small></li>
-<li>Gobblin: Unifying Data Ingestion for Hadoop <small><b>| <a href="http://www.vldb.org/pvldb/vol8/p1764-qiao.pdf">Presentation</a> | (VLDB 2015)</b></small></li>
-<li>Gobblin: Unifying Data Ingestion for Hadoop <small><b>| <a href="http://www.slideshare.net/YinanLi/gobblin-unifying-data-ingestion-for-hadoop">Presentation</a> | (VLDB 2015 slides)</b></small></li>
-<li>Ingestion from Kafka using Gobblin <small><b>| <a href="http://www.slideshare.net/ZiyangLiu1/ingestion-from-kafka-using-gobblin?qid=b7dce13f-85f6-49f2-94df-feedd6057cbe&v=qf1&b=&from_search=4">Presentation</a> | (Gobblin Meetup November 2015)</b></small></li>
-<li>Gobblin on Yarn: A Preview <small><b>| <a href="http://www.slideshare.net/YinanLi/gobblinmeetupyarn?qid=bda2e238-f302-402b-8c02-9dca1a3b7f4e&v=qf1&b=&from_search=6">Presentation</a> | (Gobblin Meetup November 2015)</b></small></li>
-<li>Gobblin@NerdWallet External Use Case 1 <small><b>| <a href="http://www.slideshare.net/NerdWalletHQ/gobblin-nerdwallet-nov-2015?qid=33ba50e5-8122-4668-89d5-bbf3302adb31&v=default&b=&from_search=2">Presentation</a> | (Gobblin Meetup November 2015)</b></small></li>
-<li>Gobblin@Intel External Use Case 2 <small><b>| <a href="http://www.slideshare.net/IntelITCenter/gobblin-for-data-analytics">Presentation</a> | (Gobblin Meetup November 2015)</b></small></li>
-<li>Gobblin: Beyond ingest to big data management <small><b>| <a href="https://www.youtube.com/watch?v=MvohU8rSFqU">Video</a> | <a href="http://www.slideshare.net/VasanthRajamani/gobblin-meetupwhats-new-in-07">Presentation</a> | (Gobblin Meetup June 2016)</b></small></li>
-<li>Gobblin: Inter cluster replication <small><b>| <a href="https://www.youtube.com/watch?v=o1BnaovUObE">Video</a> | <a href="http://www.slideshare.net/VasanthRajamani/distcp-gobblin">Presentation</a> | (Gobblin Meetup June 2016)</b></small></li>
-<li>Gobblin: Configuration and Orchestration <small><b>| <a href="https://www.youtube.com/watch?v=O9KbPDLsy_c">Video</a> | <a href="http://www.slideshare.net/VasanthRajamani/gobbin-configmeetupjune2016">Presentation</a> | (Gobblin Meetup June 2016)</b></small></li>
-<li>Gobblin on AWS <small><b>| <a href="https://youtu.be/_c7agtS5bI8?t=107">Video</a> | <a href="http://www.slideshare.net/VasanthRajamani/gobblin-onaws-63970489">Presentation</a> | (Gobblin Meetup June 2016)</b></small></li>
+<li>QCon presentation <small><b>| <a href="http://www.slideshare.net/LinQiao1/gobblin-big-data-with-ease" rel="nofollow">Presentation</a> | (Nov 5th, 2014)</b></small></li>
+<li>Engineering Blog Post <small><b>| <a href="http://engineering.linkedin.com/data-ingestion/gobblin-big-data-ease" rel="nofollow">Presentation</a> | (Nov 25th, 2014)</b></small></li>
+<li>Bigger, Faster, Easier: Building a Real-Time Self Service Data Analytics Ecosystem at LinkedIn <small><b>| <a href="http://www.slideshare.net/Hadoop_Summit/bigger-faster-easier-building-a-realtime-self-service-data-analytics-ecosystem-at-linkedin?qid=9c8f8c33-0083-495b-a6e2-572ac45f7f2c&v=qf1&b=&from_search=9" rel="nofollow">Presentation</a> | (Hadoop Summit 2015)</b></small></li>
+<li>Gobblin: Unifying Data Ingestion for Hadoop <small><b>| <a href="http://www.vldb.org/pvldb/vol8/p1764-qiao.pdf" rel="nofollow">Presentation</a> | (VLDB 2015)</b></small></li>
+<li>Gobblin: Unifying Data Ingestion for Hadoop <small><b>| <a href="http://www.slideshare.net/YinanLi/gobblin-unifying-data-ingestion-for-hadoop" rel="nofollow">Presentation</a> | (VLDB 2015 slides)</b></small></li>
+<li>Ingestion from Kafka using Gobblin <small><b>| <a href="http://www.slideshare.net/ZiyangLiu1/ingestion-from-kafka-using-gobblin?qid=b7dce13f-85f6-49f2-94df-feedd6057cbe&v=qf1&b=&from_search=4" rel="nofollow">Presentation</a> | (Gobblin Meetup November 2015)</b></small></li>
+<li>Gobblin on Yarn: A Preview <small><b>| <a href="http://www.slideshare.net/YinanLi/gobblinmeetupyarn?qid=bda2e238-f302-402b-8c02-9dca1a3b7f4e&v=qf1&b=&from_search=6" rel="nofollow">Presentation</a> | (Gobblin Meetup November 2015)</b></small></li>
+<li>Gobblin@NerdWallet External Use Case 1 <small><b>| <a href="http://www.slideshare.net/NerdWalletHQ/gobblin-nerdwallet-nov-2015?qid=33ba50e5-8122-4668-89d5-bbf3302adb31&v=default&b=&from_search=2" rel="nofollow">Presentation</a> | (Gobblin Meetup November 2015)</b></small></li>
+<li>Gobblin@Intel External Use Case 2 <small><b>| <a href="http://www.slideshare.net/IntelITCenter/gobblin-for-data-analytics" rel="nofollow">Presentation</a> | (Gobblin Meetup November 2015)</b></small></li>
+<li>Gobblin: Beyond ingest to big data management <small><b>| <a href="https://www.youtube.com/watch?v=MvohU8rSFqU" rel="nofollow">Video</a> | <a href="http://www.slideshare.net/VasanthRajamani/gobblin-meetupwhats-new-in-07" rel="nofollow">Presentation</a> | (Gobblin Meetup June 2016)</b></small></li>
+<li>Gobblin: Inter cluster replication <small><b>| <a href="https://www.youtube.com/watch?v=o1BnaovUObE" rel="nofollow">Video</a> | <a href="http://www.slideshare.net/VasanthRajamani/distcp-gobblin" rel="nofollow">Presentation</a> | (Gobblin Meetup June 2016)</b></small></li>
+<li>Gobblin: Configuration and Orchestration <small><b>| <a href="https://www.youtube.com/watch?v=O9KbPDLsy_c" rel="nofollow">Video</a> | <a href="http://www.slideshare.net/VasanthRajamani/gobbin-configmeetupjune2016" rel="nofollow">Presentation</a> | (Gobblin Meetup June 2016)</b></small></li>
+<li>Gobblin on AWS <small><b>| <a href="https://youtu.be/_c7agtS5bI8?t=107" rel="nofollow">Video</a> | <a href="http://www.slideshare.net/VasanthRajamani/gobblin-onaws-63970489" rel="nofollow">Presentation</a> | (Gobblin Meetup June 2016)</b></small></li>
</ul>
</div>
@@ -543,7 +543,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/search.html b/docs/search.html
index 08d499b..02ea840 100644
--- a/docs/search.html
+++ b/docs/search.html
@@ -512,7 +512,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sinks/AvroHdfsDataWriter/index.html b/docs/sinks/AvroHdfsDataWriter/index.html
index 05413f6..4c12069 100644
--- a/docs/sinks/AvroHdfsDataWriter/index.html
+++ b/docs/sinks/AvroHdfsDataWriter/index.html
@@ -505,7 +505,7 @@
<li>Avro HDFS</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sinks/AvroHdfsDataWriter.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sinks/AvroHdfsDataWriter.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -520,7 +520,7 @@
<pre><code>writer.builder.class=org.apache.gobblin.writer.AvroDataWriterBuilder
writer.destination.type=HDFS
</code></pre>
-<p>For more info, see <a href="https://github.com/apache/incubator-gobblin/search?utf8=%E2%9C%93&q=AvroHdfsDataWriter"><code>AvroHdfsDataWriter</code></a></p>
+<p>For more info, see <a href="https://github.com/apache/incubator-gobblin/search?utf8=%E2%9C%93&q=AvroHdfsDataWriter" rel="nofollow"><code>AvroHdfsDataWriter</code></a></p>
<h1 id="configuration">Configuration</h1>
<table>
<thead>
@@ -568,7 +568,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sinks/ConsoleWriter/index.html b/docs/sinks/ConsoleWriter/index.html
index 592ff75..e6174ea 100644
--- a/docs/sinks/ConsoleWriter/index.html
+++ b/docs/sinks/ConsoleWriter/index.html
@@ -502,7 +502,7 @@
<li>Console</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sinks/ConsoleWriter.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sinks/ConsoleWriter.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -539,7 +539,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sinks/CouchbaseWriter/index.html b/docs/sinks/CouchbaseWriter/index.html
index 410653d..40a4ba1 100644
--- a/docs/sinks/CouchbaseWriter/index.html
+++ b/docs/sinks/CouchbaseWriter/index.html
@@ -515,7 +515,7 @@
<li>Couchbase</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sinks/CouchbaseWriter.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sinks/CouchbaseWriter.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -546,7 +546,7 @@
</ul>
</div>
<h1 id="introduction">Introduction</h1>
-<p>The <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-couchbase/src/main/java/org/apache/gobblin/couchbase/writer/CouchbaseWriter.java">CouchbaseWriter</a> supports writing documents to a Couchbase bucket though the <a href="https://docs.couchbase.com/java-sdk/current/start-using-sdk.html">Couchbase Java SDK</a>. Note that CouchbaseWriter only supports writing to a single bucket as there should be only 1 CouchbaseEnvironment per JVM.</p>
+<p>The <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-couchbase/src/main/java/org/apache/gobblin/couchbase/writer/CouchbaseWriter.java" rel="nofollow">CouchbaseWriter</a> supports writing documents to a Couchbase bucket though the <a href="https://docs.couchbase.com/java-sdk/current/start-using-sdk.html" rel="nofollow">Couchbase Java SDK</a>. Note that CouchbaseWriter only supports writing to a single bucket as there should be only 1 CouchbaseEnvironment per JVM.</p>
<h1 id="record-format">Record format</h1>
<p>Couchbase writer currently support <code>AVRO</code> and <code>JSON</code> as data inputs. On both of them it requires the following structured schema:</p>
<table>
@@ -559,7 +559,7 @@
<tbody>
<tr>
<td><code>key</code></td>
-<td>Unique key used to store the document on the bucket. For more info view <a href="https://developer.couchbase.com/documentation/server/3.x/developer/dev-guide-3.0/keys-values.html">Couchbase docs</a></td>
+<td>Unique key used to store the document on the bucket. For more info view <a href="https://developer.couchbase.com/documentation/server/3.x/developer/dev-guide-3.0/keys-values.html" rel="nofollow">Couchbase docs</a></td>
</tr>
<tr>
<td><code>data.data</code></td>
@@ -567,7 +567,7 @@
</tr>
<tr>
<td><code>data.flags</code></td>
-<td><a href="https://docs.couchbase.com/server/4.1/developer-guide/transcoders.html">Couchbase flags</a> To store JSON on <code>data.data</code> use <code>0x02 << 24</code> for UTF-8 <code>0x04 << 24</code> .</td>
+<td><a href="https://docs.couchbase.com/server/4.1/developer-guide/transcoders.html" rel="nofollow">Couchbase flags</a> To store JSON on <code>data.data</code> use <code>0x02 << 24</code> for UTF-8 <code>0x04 << 24</code> .</td>
</tr>
</tbody>
</table>
@@ -644,7 +644,7 @@
<tr>
<td><code>writer.couchbase.dnsSrvEnabled</code></td>
<td><code>"false"</code></td>
-<td>Enable DNS SRV bootstrapping <a href="https://docs.couchbase.com/java-sdk/current/managing-connections.html">docs</a></td>
+<td>Enable DNS SRV bootstrapping <a href="https://docs.couchbase.com/java-sdk/current/managing-connections.html" rel="nofollow">docs</a></td>
</tr>
<tr>
<td><code>writer.couchbase.bootstrapServers |</code>localhost<code>| URL to bootstrap servers. If using DNS SRV set</code>writer.couchbase.dnsSrvEnabled` to true</td>
@@ -694,7 +694,7 @@
<tr>
<td><code>writer.couchbase.documentTTLUnits</code></td>
<td><code>SECONDS</code></td>
-<td>Unit for <code>writer.couchbase.documentTTL</code>. Must be one of <a href="https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/TimeUnit.html">java.util.concurrent.TimeUnit</a>. Case insensitive</td>
+<td>Unit for <code>writer.couchbase.documentTTL</code>. Must be one of <a href="https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/TimeUnit.html" rel="nofollow">java.util.concurrent.TimeUnit</a>. Case insensitive</td>
</tr>
<tr>
<td><code>writer.couchbase.documentTTLOriginField</code></td>
@@ -704,7 +704,7 @@
<tr>
<td><code>writer.couchbase.documentTTLOriginUnits</code></td>
<td><code>MILLISECONDS</code></td>
-<td>Unit for <code>writer.couchbase.documentTTL</code>. Must be one of <a href="https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/TimeUnit.html">java.util.concurrent.TimeUnit</a>. Case insensitive. As an example a <code>writer.couchbase.documentTTLOriginField</code> value of <code>1568240399000</code> and <code>writer.couchbase.documentTTLOriginUnits</code> value of <code>MILLISECONDS</code> timeunit would be <code>Wed Sep 11 15:19:59 PDT 2019</code></td>
+<td>Unit for <code>writer.couchbase.documentTTL</code>. Must be one of <a href="https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/TimeUnit.html" rel="nofollow">java.util.concurrent.TimeUnit</a>. Case insensitive. As an example a <code>writer.couchbase.documentTTLOriginField</code> value of <code>1568240399000</code> and <code>writer.couchbase.documentTTLOriginUnits</code> value of <code>MILLISECONDS</code> timeunit would be <code>Wed Sep 11 15:19:59 PDT 2019</code></td>
</tr>
<tr>
<td><code>writer.couchbase.retriesEnabled</code></td>
@@ -738,7 +738,7 @@
<h3 id="using-bucket-password">Using bucket password</h3>
<p>Set <code>writer.couchbase.password</code></p>
<h2 id="document-level-expiration">Document level expiration</h2>
-<p>Couchbase writer allows to set expiration at the document level using the <a href="https://docs.couchbase.com/java-sdk/current/document-operations.html">expiry</a> property of the couchbase document. PLease note that current couchbase implementation using timestamps limits it to January 19, 2038 03:14:07 GM given the type of expiry is set to int. CouchbaseWriter only works with global timestamps and does not use relative expiration in seconds (<30 days) for simplicity.
+<p>Couchbase writer allows to set expiration at the document level using the <a href="https://docs.couchbase.com/java-sdk/current/document-operations.html" rel="nofollow">expiry</a> property of the couchbase document. PLease note that current couchbase implementation using timestamps limits it to January 19, 2038 03:14:07 GM given the type of expiry is set to int. CouchbaseWriter only works with global timestamps and does not use relative expiration in seconds (<30 days) for simplicity.
Currently three modes are supported:</p>
<h3 id="1-expiration-from-write-time">1 - Expiration from write time</h3>
<p>Define only <code>writer.couchbase.documentTTL</code> and <code>writer.couchbase.documentTTLUnits</code>. For example for a 2 days expiration configs would look like:</p>
@@ -827,7 +827,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sinks/Gobblin-JDBC-Writer/index.html b/docs/sinks/Gobblin-JDBC-Writer/index.html
index 4866143..b8df574 100644
--- a/docs/sinks/Gobblin-JDBC-Writer/index.html
+++ b/docs/sinks/Gobblin-JDBC-Writer/index.html
@@ -521,7 +521,7 @@
<li>JDBC</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sinks/Gobblin-JDBC-Writer.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sinks/Gobblin-JDBC-Writer.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -714,7 +714,7 @@
<h3 id="concrete-implementations">Concrete implementations</h3>
<p>To configure a concrete writer, please refer the <a href="Configuration-Properties-Glossary#JdbcWriter-Properties">JDBC Writer Properties</a> section in the <a href="Configuration-Properties-Glossary">Configuration Glossary</a>.</p>
<h4 id="mysql-writer">MySQL Writer</h4>
-<p>The MySQL writer uses <a href="http://dev.mysql.com/doc/refman/5.0/en/insert-speed.html">buffered inserts</a> to increase performance.<br />
+<p>The MySQL writer uses <a href="http://dev.mysql.com/doc/refman/5.0/en/insert-speed.html" rel="nofollow">buffered inserts</a> to increase performance.<br />
The sink configuration for MySQL in a Gobblin job is as follows:</p>
<pre><code>writer.destination.type=MYSQL
writer.builder.class=org.apache.gobblin.writer.JdbcWriterBuilder
@@ -732,7 +732,7 @@
<p>Similarly to the MySQL Writer, this writer also inserts data in batches, configured by <code>writer.jdbc.batch_size</code>.
Ideally, for performance reasons the target table is advised to be set to type MULTISET, without a primary index.<br />
Please note, that the Teradata JDBC drivers are <em>not</em> part of Gobblin, one needs to obtain them from
-<a href="http://downloads.teradata.com/download/connectivity/jdbc-driver">Teradata</a> and pass them as job specific jars to the
+<a href="http://downloads.teradata.com/download/connectivity/jdbc-driver" rel="nofollow">Teradata</a> and pass them as job specific jars to the
Gobblin submitter scripts. Teradata may use the FASTLOAD option during the insert if conditions are met.<br />
The sink configuration for Teradata in a Gobblin job is as follows:</p>
<pre><code>writer.destination.type=TERADATA
@@ -836,7 +836,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sinks/Http/index.html b/docs/sinks/Http/index.html
index 80e8def..6e8697b 100644
--- a/docs/sinks/Http/index.html
+++ b/docs/sinks/Http/index.html
@@ -537,7 +537,7 @@
<li>HTTP</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sinks/Http.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sinks/Http.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -576,8 +576,8 @@
builds a http or rest request from multiple http records, sends the request with a client that knows the server,
and handles the response.</p>
<h2 id="note">Note</h2>
-<p>The old http write framework under <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/writer/http/AbstractHttpWriter.java"><code>AbstractHttpWriter</code></a>
-and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/writer/http/AbstractHttpWriterBuilder.java"><code>AbstractHttpWriterBuilder</code></a>
+<p>The old http write framework under <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/writer/http/AbstractHttpWriter.java" rel="nofollow"><code>AbstractHttpWriter</code></a>
+and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/writer/http/AbstractHttpWriterBuilder.java" rel="nofollow"><code>AbstractHttpWriterBuilder</code></a>
is deprecated (Deprecation date: 05/15/2018)! Use <code>AsyncHttpWriter</code> and <code>AsyncHttpWriterBuilder</code> instead</p>
<h1 id="constructs">Constructs</h1>
<p><p align="center">
@@ -640,9 +640,9 @@
<p>An <code>AvroHttpWriterBuilder</code> builds an <code>AsyncHttpWriter</code> on top of the <a href="https://hc.apache.org/">apache httpcomponents framework</a>, sending vanilla http request.
The 3 major components are:</p>
<ul>
-<li><code>ApacheHttpClient</code>. It uses <a href="https://github.com/apache/httpcomponents-client/blob/master/httpclient5/src/main/java/org/apache/hc/client5/http/impl/classic/CloseableHttpClient.java"><code>CloseableHttpClient</code></a> to
- send <a href="https://github.com/apache/httpcomponents-client/blob/master/httpclient5/src/main/java/org/apache/hc/client5/http/classic/methods/HttpUriRequest.java"><code>HttpUriRequest</code></a>
- and receive <a href="https://github.com/apache/httpcomponents-client/blob/master/httpclient5/src/main/java/org/apache/hc/client5/http/impl/classic/CloseableHttpResponse.java"><code>CloseableHttpResponse</code></a></li>
+<li><code>ApacheHttpClient</code>. It uses <a href="https://github.com/apache/httpcomponents-client/blob/master/httpclient5/src/main/java/org/apache/hc/client5/http/impl/classic/CloseableHttpClient.java" rel="nofollow"><code>CloseableHttpClient</code></a> to
+ send <a href="https://github.com/apache/httpcomponents-client/blob/master/httpclient5/src/main/java/org/apache/hc/client5/http/classic/methods/HttpUriRequest.java" rel="nofollow"><code>HttpUriRequest</code></a>
+ and receive <a href="https://github.com/apache/httpcomponents-client/blob/master/httpclient5/src/main/java/org/apache/hc/client5/http/impl/classic/CloseableHttpResponse.java" rel="nofollow"><code>CloseableHttpResponse</code></a></li>
<li><code>ApacheHttpRequestBuilder</code>. It builds a <code>ApacheHttpRequest</code>, which is an <code>AsyncRequest</code> that wraps the <code>HttpUriRequest</code>, from one <code>HttpOperation</code></li>
<li><code>ApacheHttpResponseHandler</code>. It handles a <code>HttpResponse</code></li>
</ul>
@@ -663,7 +663,7 @@
</tr>
<tr>
<td><code>gobblin.writer.http.verb</code></td>
-<td>Required, <a href="http://www.restapitutorial.com/lessons/httpmethods.html">http verbs</a></td>
+<td>Required, <a href="http://www.restapitutorial.com/lessons/httpmethods.html" rel="nofollow">http verbs</a></td>
<td>get, update, delete, etc</td>
</tr>
<tr>
@@ -684,16 +684,16 @@
</tbody>
</table>
<h2 id="r2restwriterbuilder"><code>R2RestWriterBuilder</code></h2>
-<p>A <code>R2RestWriterBuilder</code> builds an <code>AsyncHttpWriter</code> on top of <a href="https://github.com/linkedin/rest.li/wiki/Request---Response-API-(R2)">restli r2 framework</a>, sending
+<p>A <code>R2RestWriterBuilder</code> builds an <code>AsyncHttpWriter</code> on top of <a href="https://github.com/linkedin/rest.li/wiki/Request---Response-API-(R2)" rel="nofollow">restli r2 framework</a>, sending
rest request. The 3 major components are:</p>
<ul>
-<li><code>R2Client</code>. It uses a R2 <a href="https://github.com/linkedin/rest.li/blob/master/r2-core/src/main/java/com/linkedin/r2/transport/common/Client.java"><code>Client</code></a> to
- send <a href="https://github.com/linkedin/rest.li/blob/master/r2-core/src/main/java/com/linkedin/r2/message/rest/RestRequest.java"><code>RestRequest</code></a> and
- receive <a href="https://github.com/linkedin/rest.li/blob/master/r2-core/src/main/java/com/linkedin/r2/message/rest/RestResponse.java"><code>RestResponse</code></a></li>
+<li><code>R2Client</code>. It uses a R2 <a href="https://github.com/linkedin/rest.li/blob/master/r2-core/src/main/java/com/linkedin/r2/transport/common/Client.java" rel="nofollow"><code>Client</code></a> to
+ send <a href="https://github.com/linkedin/rest.li/blob/master/r2-core/src/main/java/com/linkedin/r2/message/rest/RestRequest.java" rel="nofollow"><code>RestRequest</code></a> and
+ receive <a href="https://github.com/linkedin/rest.li/blob/master/r2-core/src/main/java/com/linkedin/r2/message/rest/RestResponse.java" rel="nofollow"><code>RestResponse</code></a></li>
<li><code>R2RestRequestBuilder</code>. It builds a <code>R2Request</code>, which is an <code>AsyncRequest</code> that wraps the <code>RestRequest</code>, from one <code>HttpOperation</code></li>
<li><code>R2RestResponseHandler</code>. It handles a <code>RestResponse</code></li>
</ul>
-<p><code>R2RestWriterBuilder</code> has <a href="https://github.com/linkedin/rest.li/wiki/Dynamic-Discovery">d2</a> and ssl support. Configurations(<code>(d2.)</code> part should be added in d2 mode) for the builder are:</p>
+<p><code>R2RestWriterBuilder</code> has <a href="https://github.com/linkedin/rest.li/wiki/Dynamic-Discovery" rel="nofollow">d2</a> and ssl support. Configurations(<code>(d2.)</code> part should be added in d2 mode) for the builder are:</p>
<table>
<thead>
<tr>
@@ -710,7 +710,7 @@
</tr>
<tr>
<td><code>gobblin.writer.http.verb</code></td>
-<td>Required, <a href="https://github.com/linkedin/rest.li/wiki/Rest.li-User-Guide#resource-methods">rest(rest.li) verbs</a></td>
+<td>Required, <a href="https://github.com/linkedin/rest.li/wiki/Rest.li-User-Guide#resource-methods" rel="nofollow">rest(rest.li) verbs</a></td>
<td>get, update, put, delete, etc</td>
</tr>
<tr>
@@ -791,7 +791,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sinks/Kafka/index.html b/docs/sinks/Kafka/index.html
index fd771b4..9420c4c 100644
--- a/docs/sinks/Kafka/index.html
+++ b/docs/sinks/Kafka/index.html
@@ -514,7 +514,7 @@
<li>Kafka</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sinks/Kafka.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sinks/Kafka.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -547,7 +547,7 @@
</ul>
<h1 id="steps">Steps</h1>
<ul>
-<li>Edit the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia-kafka.pull">wikipedia-kafka.pull</a> example to get started with setting up ingestion into Kafka. This is a very similar pipeline to the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull">wikipedia.pull</a> example which pulls pages from 5 titles from Wikipedia to HDFS. The main differences to note are: <ul>
+<li>Edit the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia-kafka.pull" rel="nofollow">wikipedia-kafka.pull</a> example to get started with setting up ingestion into Kafka. This is a very similar pipeline to the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull" rel="nofollow">wikipedia.pull</a> example which pulls pages from 5 titles from Wikipedia to HDFS. The main differences to note are: <ul>
<li>The <code>writer.builder.class</code> is set to <code>gobblin.kafka.writer.KafkaDataWriterBuilder</code>. This is the class that creates a Kafka writer.</li>
<li>The <code>writer.kafka.topic</code> is set to <code>WikipediaExample</code>. This is the topic that the writer will write the records to.</li>
<li>The <code>writer.kafka.producerConfig.bootstrap.servers</code> is set to <code>localhost:9092</code>. This is the address of the kafka broker(s) that the writer must write to.</li>
@@ -586,7 +586,7 @@
</code></pre>
<ul>
-<li>To verify that the records have indeed been ingested into Kafka, you can run a kafka console consumer or run Gobblin's <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/kafka-console.pull">kafka-console pull file</a> which prints the events from Kafka onto the console.</li>
+<li>To verify that the records have indeed been ingested into Kafka, you can run a kafka console consumer or run Gobblin's <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/kafka-console.pull" rel="nofollow">kafka-console pull file</a> which prints the events from Kafka onto the console.</li>
</ul>
<h1 id="configuration-details">Configuration Details</h1>
<p>At this time, Gobblin supports integration with Kafka 0.8 and 0.9. The Kafka writer supports all the configuration parameters supported by the version-specific Kafka Producer (e.g. <a href="https://kafka.apache.org/documentation/#producerconfigs">Latest Producer Configs</a>). All you have to do is prefix <code>writer.kafka.producerConfig.</code> to each configuration property that the producer supports. For example, if you want to set the <code>acks</code> parameter to <code>all</code> to ensure full acknowledgement of writes, you would set <code>writer.kafka.producerConfig.acks=all</code> in your pull file. For a comprehensive list of all the configuration properties supported by the producers, go through the <a href="http://kafka.apache.org/082/documentation.html#newproducerconfigs">official documentation here</a>. Note: Since Gobblin is currently built against Kafka 0.8.2, the configuration options apply to the new 0.8.2 java producer.</p>
@@ -653,7 +653,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sinks/ParquetHdfsDataWriter/index.html b/docs/sinks/ParquetHdfsDataWriter/index.html
index f606e2b..35891ea 100644
--- a/docs/sinks/ParquetHdfsDataWriter/index.html
+++ b/docs/sinks/ParquetHdfsDataWriter/index.html
@@ -511,7 +511,7 @@
<li>Parquet HDFS</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sinks/ParquetHdfsDataWriter.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sinks/ParquetHdfsDataWriter.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -521,7 +521,7 @@
<div class="section">
<h1 id="description">Description</h1>
-<p>An extension to <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/writer/FsDataWriter.java"><code>FsDataWriter</code></a> that writes in Parquet format in the form of either Avro, Protobuf or <a href="https://github.com/apache/parquet-mr/blob/master/parquet-column/src/main/java/org/apache/parquet/example/data/Group.java"><code>ParquetGroup</code></a>. This implementation allows users to specify the CodecFactory to use through the configuration property <a href="https://gobblin.readthedocs.io/en/latest/user-guide/Configuration-Properties-Glossary/#writercodectype"><code>writer.codec.type</code></a>. By default, the snappy codec is used. See <a href="#developer-notes">Developer Notes</a> to make sure you are using the right Gobblin jar.</p>
+<p>An extension to <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/writer/FsDataWriter.java" rel="nofollow"><code>FsDataWriter</code></a> that writes in Parquet format in the form of either Avro, Protobuf or <a href="https://github.com/apache/parquet-mr/blob/master/parquet-column/src/main/java/org/apache/parquet/example/data/Group.java" rel="nofollow"><code>ParquetGroup</code></a>. This implementation allows users to specify the CodecFactory to use through the configuration property <a href="https://gobblin.readthedocs.io/en/latest/user-guide/Configuration-Properties-Glossary/#writercodectype" rel="nofollow"><code>writer.codec.type</code></a>. By default, the snappy codec is used. See <a href="#developer-notes">Developer Notes</a> to make sure you are using the right Gobblin jar.</p>
<h1 id="usage">Usage</h1>
<pre><code>writer.builder.class=org.apache.gobblin.writer.ParquetDataWriterBuilder
writer.destination.type=HDFS
@@ -530,7 +530,7 @@
<h1 id="example-pipeline-configuration">Example Pipeline Configuration</h1>
<ul>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/example-parquet.pull"><code>example-parquet.pull</code></a> contains an example of generating test data and writing to Parquet files.</li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/example-parquet.pull" rel="nofollow"><code>example-parquet.pull</code></a> contains an example of generating test data and writing to Parquet files.</li>
</ul>
<h1 id="configuration">Configuration</h1>
<table>
@@ -563,7 +563,7 @@
</tr>
<tr>
<td>writer.parquet.validate</td>
-<td>To turn on validation using the schema. This validation is done by <a href="https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetWriter.java"><code>ParquetWriter</code></a> not by Gobblin.</td>
+<td>To turn on validation using the schema. This validation is done by <a href="https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetWriter.java" rel="nofollow"><code>ParquetWriter</code></a> not by Gobblin.</td>
<td>false</td>
<td>No</td>
</tr>
@@ -575,7 +575,7 @@
</tr>
<tr>
<td>writer.parquet.format</td>
-<td>In-memory format of the record being written to Parquet. <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-parquet-common/src/main/java/org/apache/gobblin/parquet/writer/ParquetRecordFormat.java"><code>Options</code></a> are AVRO, PROTOBUF and GROUP</td>
+<td>In-memory format of the record being written to Parquet. <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-parquet-common/src/main/java/org/apache/gobblin/parquet/writer/ParquetRecordFormat.java" rel="nofollow"><code>Options</code></a> are AVRO, PROTOBUF and GROUP</td>
<td>GROUP</td>
<td>No</td>
</tr>
@@ -593,13 +593,13 @@
</thead>
<tbody>
<tr>
-<td><a href="https://mvnrepository.com/artifact/org.apache.gobblin/gobblin-parquet"><code>gobblin-parquet</code></a></td>
-<td><a href="https://mvnrepository.com/artifact/com.twitter/parquet-hadoop-bundle"><code>com.twitter:parquet-hadoop-bundle</code></a></td>
+<td><a href="https://mvnrepository.com/artifact/org.apache.gobblin/gobblin-parquet" rel="nofollow"><code>gobblin-parquet</code></a></td>
+<td><a href="https://mvnrepository.com/artifact/com.twitter/parquet-hadoop-bundle" rel="nofollow"><code>com.twitter:parquet-hadoop-bundle</code></a></td>
<td>>= 0.12.0</td>
</tr>
<tr>
-<td><a href="https://mvnrepository.com/artifact/org.apache.gobblin/gobblin-parquet-apache"><code>gobblin-parquet-apache</code></a></td>
-<td><a href="https://mvnrepository.com/artifact/org.apache.parquet/parquet-hadoop"><code>org.apache.parquet:parquet-hadoop</code></a></td>
+<td><a href="https://mvnrepository.com/artifact/org.apache.gobblin/gobblin-parquet-apache" rel="nofollow"><code>gobblin-parquet-apache</code></a></td>
+<td><a href="https://mvnrepository.com/artifact/org.apache.parquet/parquet-hadoop" rel="nofollow"><code>org.apache.parquet:parquet-hadoop</code></a></td>
<td>>= 0.15.0</td>
</tr>
</tbody>
@@ -615,19 +615,19 @@
<tbody>
<tr>
<td>gobblin-parquet</td>
-<td><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-parquet/src/main/java/org/apache/gobblin/writer/ParquetHdfsDataWriter.java"><code>ParquetHdfsDataWriter</code></a></td>
+<td><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-parquet/src/main/java/org/apache/gobblin/writer/ParquetHdfsDataWriter.java" rel="nofollow"><code>ParquetHdfsDataWriter</code></a></td>
</tr>
<tr>
<td>gobblin-parquet</td>
-<td><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-parquet/src/main/java/org/apache/gobblin/writer/ParquetDataWriterBuilder.java"><code>ParquetDataWriterBuilder</code></a></td>
+<td><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-parquet/src/main/java/org/apache/gobblin/writer/ParquetDataWriterBuilder.java" rel="nofollow"><code>ParquetDataWriterBuilder</code></a></td>
</tr>
<tr>
<td>gobblin-parquet-apache</td>
-<td><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-parquet-apache/src/main/java/org/apache/gobblin/writer/ParquetHdfsDataWriter.java"><code>ParquetHdfsDataWriter</code></a></td>
+<td><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-parquet-apache/src/main/java/org/apache/gobblin/writer/ParquetHdfsDataWriter.java" rel="nofollow"><code>ParquetHdfsDataWriter</code></a></td>
</tr>
<tr>
<td>gobblin-parquet-apache</td>
-<td><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-parquet-apache/src/main/java/org/apache/gobblin/writer/ParquetDataWriterBuilder.java"><code>ParquetDataWriterBuilder</code></a></td>
+<td><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-parquet-apache/src/main/java/org/apache/gobblin/writer/ParquetDataWriterBuilder.java" rel="nofollow"><code>ParquetDataWriterBuilder</code></a></td>
</tr>
</tbody>
</table>
@@ -653,7 +653,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sinks/SimpleBytesWriter/index.html b/docs/sinks/SimpleBytesWriter/index.html
index 576d464..960850f 100644
--- a/docs/sinks/SimpleBytesWriter/index.html
+++ b/docs/sinks/SimpleBytesWriter/index.html
@@ -506,7 +506,7 @@
<li>HDFS Byte array</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sinks/SimpleBytesWriter.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sinks/SimpleBytesWriter.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -567,7 +567,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sources/AvroFileSource/index.html b/docs/sources/AvroFileSource/index.html
index 805e7f8..8819a8d 100644
--- a/docs/sources/AvroFileSource/index.html
+++ b/docs/sources/AvroFileSource/index.html
@@ -506,7 +506,7 @@
<li>Avro files</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/AvroFileSource.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/AvroFileSource.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -543,7 +543,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sources/CopySource/index.html b/docs/sources/CopySource/index.html
index 9349904..a8c28e7 100644
--- a/docs/sources/CopySource/index.html
+++ b/docs/sources/CopySource/index.html
@@ -506,7 +506,7 @@
<li>File copy</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/CopySource.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/CopySource.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -543,7 +543,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sources/GoogleAnalyticsSource/index.html b/docs/sources/GoogleAnalyticsSource/index.html
index bf53674..2a5e619 100644
--- a/docs/sources/GoogleAnalyticsSource/index.html
+++ b/docs/sources/GoogleAnalyticsSource/index.html
@@ -505,7 +505,7 @@
<li>Google Analytics</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/GoogleAnalyticsSource.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/GoogleAnalyticsSource.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -542,7 +542,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sources/GoogleDriveSource/index.html b/docs/sources/GoogleDriveSource/index.html
index b9b3351..e3e6e10 100644
--- a/docs/sources/GoogleDriveSource/index.html
+++ b/docs/sources/GoogleDriveSource/index.html
@@ -506,7 +506,7 @@
<li>Google Drive</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/GoogleDriveSource.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/GoogleDriveSource.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -543,7 +543,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sources/GoogleWebmaster/index.html b/docs/sources/GoogleWebmaster/index.html
index b4efb77..625f6be 100644
--- a/docs/sources/GoogleWebmaster/index.html
+++ b/docs/sources/GoogleWebmaster/index.html
@@ -519,7 +519,7 @@
<li>Google Webmaster</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/GoogleWebmaster.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/GoogleWebmaster.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -760,7 +760,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sources/HadoopTextInputSource/index.html b/docs/sources/HadoopTextInputSource/index.html
index 76d1816..2570901 100644
--- a/docs/sources/HadoopTextInputSource/index.html
+++ b/docs/sources/HadoopTextInputSource/index.html
@@ -506,7 +506,7 @@
<li>Hadoop Text Input</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/HadoopTextInputSource.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/HadoopTextInputSource.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -543,7 +543,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sources/HelloWorldSource/index.html b/docs/sources/HelloWorldSource/index.html
index 4f511ce..0a3fd52 100644
--- a/docs/sources/HelloWorldSource/index.html
+++ b/docs/sources/HelloWorldSource/index.html
@@ -505,7 +505,7 @@
<li>Hello World</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/HelloWorldSource.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/HelloWorldSource.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -542,7 +542,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sources/HiveAvroToOrcSource/index.html b/docs/sources/HiveAvroToOrcSource/index.html
index 226c09d..5e94e41 100644
--- a/docs/sources/HiveAvroToOrcSource/index.html
+++ b/docs/sources/HiveAvroToOrcSource/index.html
@@ -506,7 +506,7 @@
<li>Hive Avro-to-ORC</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/HiveAvroToOrcSource.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/HiveAvroToOrcSource.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -543,7 +543,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sources/HivePurgerSource/index.html b/docs/sources/HivePurgerSource/index.html
index dde1d8b..0c28865 100644
--- a/docs/sources/HivePurgerSource/index.html
+++ b/docs/sources/HivePurgerSource/index.html
@@ -506,7 +506,7 @@
<li>Hive compliance purging</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/HivePurgerSource.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/HivePurgerSource.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -543,7 +543,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sources/HiveSource/index.html b/docs/sources/HiveSource/index.html
index 8ac3faf..a9bd3ce 100644
--- a/docs/sources/HiveSource/index.html
+++ b/docs/sources/HiveSource/index.html
@@ -489,7 +489,7 @@
<li>Description</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/HiveSource.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/HiveSource.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -517,7 +517,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sources/KafkaSource/index.html b/docs/sources/KafkaSource/index.html
index b21ed9e..7016d0e 100644
--- a/docs/sources/KafkaSource/index.html
+++ b/docs/sources/KafkaSource/index.html
@@ -505,7 +505,7 @@
<li>Kafka</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/KafkaSource.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/KafkaSource.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -542,7 +542,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sources/MySQLSource/index.html b/docs/sources/MySQLSource/index.html
index 5e6c1d1..2d913b4 100644
--- a/docs/sources/MySQLSource/index.html
+++ b/docs/sources/MySQLSource/index.html
@@ -506,7 +506,7 @@
<li>MySQL</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/MySQLSource.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/MySQLSource.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -543,7 +543,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sources/OracleSource/index.html b/docs/sources/OracleSource/index.html
index 282edea..a14e091 100644
--- a/docs/sources/OracleSource/index.html
+++ b/docs/sources/OracleSource/index.html
@@ -506,7 +506,7 @@
<li>Oracle</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/OracleSource.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/OracleSource.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -543,7 +543,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sources/QueryBasedSource/index.html b/docs/sources/QueryBasedSource/index.html
index baa05f2..5768dae 100644
--- a/docs/sources/QueryBasedSource/index.html
+++ b/docs/sources/QueryBasedSource/index.html
@@ -513,7 +513,7 @@
<li>Query based</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/QueryBasedSource.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/QueryBasedSource.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -538,11 +538,11 @@
</ul>
</div>
<h1 id="introduction">Introduction</h1>
-<p><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/QueryBasedSource.java"><code>QueryBasedSource</code></a>
+<p><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/QueryBasedSource.java" rel="nofollow"><code>QueryBasedSource</code></a>
represents a category of sources whose data is pulled by sending queries. A dataset of a source is identified as a
-<a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/QueryBasedSource.java#L96"><code>SourceEntity</code></a>.
+<a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/QueryBasedSource.java#L96" rel="nofollow"><code>SourceEntity</code></a>.
Query can be done by sending HTTP requests or SQL commands. A source often, but not always, has a corresponding
-<a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/QueryBasedExtractor.java"><code>QueryBasedExtractor</code></a>,
+<a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/QueryBasedExtractor.java" rel="nofollow"><code>QueryBasedExtractor</code></a>,
which defines the way and implements common routines to extract data from the source.</p>
<h1 id="constructs">Constructs</h1>
<h2 id="querybasedsource"><code>QueryBasedSource</code></h2>
@@ -560,7 +560,7 @@
<li>pick up previously failed work units.</li>
</ul>
<p>At last, it will group several work units as
-<a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/workunit/MultiWorkUnit.java"><code>MultiWorkUnit</code></a>
+<a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/workunit/MultiWorkUnit.java" rel="nofollow"><code>MultiWorkUnit</code></a>
according to the <code>mr.job.max.mappers</code> configuration (Note: other categories of source might have a different approach to group work units into <code>MultiWorkUnit</code>). </p>
<h2 id="querybasedextractor"><code>QueryBasedExtractor</code></h2>
<p><p align="center">
@@ -570,13 +570,13 @@
</figure>
</p></p>
<p>Currently in Gobblin, depending on how an extractor communicates with a source
-(or <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/ProtocolSpecificLayer.java">different communication protocols</a>),
+(or <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/ProtocolSpecificLayer.java" rel="nofollow">different communication protocols</a>),
a <code>QueryBasedExtractor</code> falls into 2 categories:
-<a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/restapi/RestApiExtractor.java"><code>RestApiExtractor</code></a>
+<a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/restapi/RestApiExtractor.java" rel="nofollow"><code>RestApiExtractor</code></a>
and
-<a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-sql/src/main/java/org/apache/gobblin/source/jdbc/JdbcExtractor.java"><code>JdbcExtractor</code></a>.
+<a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-sql/src/main/java/org/apache/gobblin/source/jdbc/JdbcExtractor.java" rel="nofollow"><code>JdbcExtractor</code></a>.
A specific extractor has to provide some
-<a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/SourceSpecificLayer.java">source specific logic</a>
+<a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/SourceSpecificLayer.java" rel="nofollow">source specific logic</a>
in order to successfully extract information from the source.</p>
<h3 id="build"><code>build</code></h3>
<p>Building a query based extractor may involve three queries:</p>
@@ -710,7 +710,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sources/RestApiSource/index.html b/docs/sources/RestApiSource/index.html
index 9144c34..1ce2590 100644
--- a/docs/sources/RestApiSource/index.html
+++ b/docs/sources/RestApiSource/index.html
@@ -513,7 +513,7 @@
<li>Rest Api</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/RestApiSource.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/RestApiSource.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -534,9 +534,9 @@
</ul>
</div>
<h1 id="introduction">Introduction</h1>
-<p>A RestApiSource is a <a href="../QueryBasedSource/">QueryBasedSource</a> which uses <a href="https://en.wikipedia.org/wiki/Representational_state_transfer">RESTful</a>
+<p>A RestApiSource is a <a href="../QueryBasedSource/">QueryBasedSource</a> which uses <a href="https://en.wikipedia.org/wiki/Representational_state_transfer" rel="nofollow">RESTful</a>
Api for query. <code>RestApiExtractor</code> is a <code>QueryBasedExtractor</code> that uses REST to communicate with the source. To establish the communication,
-a <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/restapi/RestApiConnector.java"><code>RestApiConnector</code></a> is
+a <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/restapi/RestApiConnector.java" rel="nofollow"><code>RestApiConnector</code></a> is
required.</p>
<h1 id="constructs">Constructs</h1>
<h2 id="restapisource"><code>RestApiSource</code></h2>
@@ -593,7 +593,7 @@
</table>
<p>There are other interactions between the <code>RestApiExtractor</code> layer and <code>SourceSpecificLayer</code>. The key points are:</p>
<ul>
-<li>A <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/ProtocolSpecificLayer.java"><code>ProtocolSpecificLayer</code></a>, such as
+<li>A <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/ProtocolSpecificLayer.java" rel="nofollow"><code>ProtocolSpecificLayer</code></a>, such as
<code>RestApiExtractor</code>, understands the protocol and sets up a routine to communicate with the source</li>
<li>A <code>SourceSpecificLayer</code>, such as <code>SalesforceExtractor</code>, knows the source and fits into the routine by providing and analyzing source specific information</li>
</ul>
@@ -651,7 +651,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sources/SalesforceSource/index.html b/docs/sources/SalesforceSource/index.html
index 49a242e..d0d07bb 100644
--- a/docs/sources/SalesforceSource/index.html
+++ b/docs/sources/SalesforceSource/index.html
@@ -505,7 +505,7 @@
<li>Salesforce</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/SalesforceSource.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/SalesforceSource.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -542,7 +542,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sources/SftpSource/index.html b/docs/sources/SftpSource/index.html
index e909747..eb133a0 100644
--- a/docs/sources/SftpSource/index.html
+++ b/docs/sources/SftpSource/index.html
@@ -505,7 +505,7 @@
<li>SFTP</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/SftpSource.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/SftpSource.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -542,7 +542,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sources/SimpleJsonSource/index.html b/docs/sources/SimpleJsonSource/index.html
index 16e6e98..e64dfe9 100644
--- a/docs/sources/SimpleJsonSource/index.html
+++ b/docs/sources/SimpleJsonSource/index.html
@@ -505,7 +505,7 @@
<li>JSON</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/SimpleJsonSource.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/SimpleJsonSource.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -542,7 +542,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sources/SqlServerSource/index.html b/docs/sources/SqlServerSource/index.html
index 490c4ae..77672cf 100644
--- a/docs/sources/SqlServerSource/index.html
+++ b/docs/sources/SqlServerSource/index.html
@@ -505,7 +505,7 @@
<li>SQL Server</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/SqlServerSource.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/SqlServerSource.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -542,7 +542,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sources/TeradataSource/index.html b/docs/sources/TeradataSource/index.html
index a7eab4c..d8a2913 100644
--- a/docs/sources/TeradataSource/index.html
+++ b/docs/sources/TeradataSource/index.html
@@ -505,7 +505,7 @@
<li>Teradata</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/TeradataSource.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/TeradataSource.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -542,7 +542,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/sources/WikipediaSource/index.html b/docs/sources/WikipediaSource/index.html
index 5f27277..0a92238 100644
--- a/docs/sources/WikipediaSource/index.html
+++ b/docs/sources/WikipediaSource/index.html
@@ -505,7 +505,7 @@
<li>Wikipedia</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/WikipediaSource.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/sources/WikipediaSource.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -542,7 +542,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/user-guide/Azure-Kubernetes-Deployment/index.html b/docs/user-guide/Azure-Kubernetes-Deployment/index.html
index 586aadc..8c2f07c 100644
--- a/docs/user-guide/Azure-Kubernetes-Deployment/index.html
+++ b/docs/user-guide/Azure-Kubernetes-Deployment/index.html
@@ -489,7 +489,7 @@
<li>GaaS on Azure Deployment Steps</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Azure-Kubernetes-Deployment.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Azure-Kubernetes-Deployment.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -536,7 +536,7 @@
<pre><code class="bash">helm install stable/nginx-ingress
</code></pre>
-<p>If this is the first time deploying helm (v2.0), you will need to set up the tiller, which is a helm serviceaccount with sudo permissions that lives inside of the cluster. Otherwise you'll run into this <a href="https://github.com/helm/helm/issues/2224">issue</a>.</p>
+<p>If this is the first time deploying helm (v2.0), you will need to set up the tiller, which is a helm serviceaccount with sudo permissions that lives inside of the cluster. Otherwise you'll run into this <a href="https://github.com/helm/helm/issues/2224" rel="nofollow">issue</a>.</p>
<blockquote>
<p>Error: configmaps is forbidden: User "system:serviceaccount:kube-system:default" cannot list configmaps in the namespace "kube-system"</p>
</blockquote>
@@ -568,7 +568,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/user-guide/Building-Gobblin-as-a-Service/index.html b/docs/user-guide/Building-Gobblin-as-a-Service/index.html
index dcdc0ee..c0a7554 100644
--- a/docs/user-guide/Building-Gobblin-as-a-Service/index.html
+++ b/docs/user-guide/Building-Gobblin-as-a-Service/index.html
@@ -489,7 +489,7 @@
<li>Building Gobblin as a Service</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Building-Gobblin-as-a-Service.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Building-Gobblin-as-a-Service.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -534,7 +534,7 @@
<p>The docker container exposes the endpoints from Gobblin as a Service which can be accessed on <code>localhost:6956</code></p>
<h1 id="running-gobblin-as-a-service-with-kubernetes">Running Gobblin as a Service with Kubernetes</h1>
<p>Gobblin as a service also has a kubernetes cluster, which can be deployed to any K8s environment.</p>
-<p>Currently, the yamls use <a href="https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/">Kustomize</a> for configuration management. In the future, we may utilise Helm instead.</p>
+<p>Currently, the yamls use <a href="https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/" rel="nofollow">Kustomize</a> for configuration management. In the future, we may utilise Helm instead.</p>
<p>To cluster is split into 3 environments
1) base-cluster (deploys one pod of GaaS and Gobblin standalone, where GaaS writes jobSpecs to a folder tracked by the standalone instance)
2) mysql-cluster (utilises MySQL for storing specStores instead of FS, future work may involve writing to a job queue to be picked by gobblin standalone)
@@ -559,7 +559,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/user-guide/Building-Gobblin/index.html b/docs/user-guide/Building-Gobblin/index.html
index 07e8b58..44ca193 100644
--- a/docs/user-guide/Building-Gobblin/index.html
+++ b/docs/user-guide/Building-Gobblin/index.html
@@ -527,7 +527,7 @@
<li>Building Gobblin</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Building-Gobblin.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Building-Gobblin.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -613,7 +613,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/user-guide/Compaction/index.html b/docs/user-guide/Compaction/index.html
index c45c94e..f803ac0 100644
--- a/docs/user-guide/Compaction/index.html
+++ b/docs/user-guide/Compaction/index.html
@@ -528,7 +528,7 @@
<li>Compaction</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Compaction.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Compaction.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -583,7 +583,7 @@
<li>A non-deduping compaction runs daily, consumes data in <code>hourly</code> and publish data into <code>daily</code>, e.g., <code>/data/kafka_topics/PageViewEvent/daily/2015/10/29...</code></li>
</ul>
<h2 id="basic-usage">Basic Usage</h2>
-<p><code>MRCompactor.compact()</code> is the entry point for MapReduce-based compaction. The compaction unit is <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-compaction/src/main/java/org/apache/gobblin/compaction/dataset/Dataset.java"><code>Dataset</code></a>. <code>MRCompactor</code> uses a <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-compaction/src/main/java/org/apache/gobblin/compaction/dataset/DatasetsFinder.java"><code>DatasetsFinder</code></a> to find all datasets eligible for compaction. Implementations of <code>DatasetsFinder</code> include <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-compaction/src/main/java/org/apache/gobblin/compaction/dataset/SimpleDatasetsFinder.java"><code>SimpleDatasetsFinder</code></a> and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-compaction/src/main/java/org/apache/gobblin/compaction/dataset/TimeBasedSubDirDatasetsFinder.java"><code>TimeBasedSubDirDatasetsFinder</code></a>.</p>
+<p><code>MRCompactor.compact()</code> is the entry point for MapReduce-based compaction. The compaction unit is <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-compaction/src/main/java/org/apache/gobblin/compaction/dataset/Dataset.java" rel="nofollow"><code>Dataset</code></a>. <code>MRCompactor</code> uses a <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-compaction/src/main/java/org/apache/gobblin/compaction/dataset/DatasetsFinder.java" rel="nofollow"><code>DatasetsFinder</code></a> to find all datasets eligible for compaction. Implementations of <code>DatasetsFinder</code> include <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-compaction/src/main/java/org/apache/gobblin/compaction/dataset/SimpleDatasetsFinder.java" rel="nofollow"><code>SimpleDatasetsFinder</code></a> and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-compaction/src/main/java/org/apache/gobblin/compaction/dataset/TimeBasedSubDirDatasetsFinder.java" rel="nofollow"><code>TimeBasedSubDirDatasetsFinder</code></a>.</p>
<p>In the above example use case, for hourly compaction, each dataset contains an hour's data in the <code>hourly_staging</code> folder, e.g., <code>/data/kafka_topics/PageViewEvent/hourly_staging/2015/10/29/08</code>; for daily compaction, each dataset contains 24 hourly folder of a day, e.g., <code>/data/kafka_topics/PageViewEvent/hourly/2015/10/29</code>. In hourly compaction, you may use the following config properties:</p>
<pre><code>compaction.datasets.finder=org.apache.gobblin.compaction.dataset.TimeBasedSubDirDatasetsFinder
compaction.input.dir=/data/kafka_topics
@@ -827,7 +827,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/user-guide/Config-Management/index.html b/docs/user-guide/Config-Management/index.html
index ebf1dec..7a013b2 100644
--- a/docs/user-guide/Config-Management/index.html
+++ b/docs/user-guide/Config-Management/index.html
@@ -532,7 +532,7 @@
<li>Config Management</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Config-Management.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Config-Management.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -620,7 +620,7 @@
<li>The URI path is significant so that configs can be associated at every level. For example, for the dataset URI hdfs_store:/data/databases/DB/Table , we should be able to associate config at every level: /data/databases/DB/Table, /data/databases/DB/, /data/databases/, etc.</li>
</ul>
<h1 id="current-dataset-config-management-implementation">Current Dataset Config Management Implementation</h1>
-<p>At a very high-level, we extend <a href="https://github.com/typesafehub/config">typesafe config</a> with:</p>
+<p>At a very high-level, we extend <a href="https://github.com/typesafehub/config" rel="nofollow">typesafe config</a> with:</p>
<ul>
<li>Support for logical include URIs</li>
<li>Abstraction of a Config Store</li>
@@ -711,7 +711,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/user-guide/Configuration-Properties-Glossary/index.html b/docs/user-guide/Configuration-Properties-Glossary/index.html
index 307789a..83ffa6b 100644
--- a/docs/user-guide/Configuration-Properties-Glossary/index.html
+++ b/docs/user-guide/Configuration-Properties-Glossary/index.html
@@ -608,7 +608,7 @@
<li>Configuration Glossary</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Configuration-Properties-Glossary.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Configuration-Properties-Glossary.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -618,7 +618,7 @@
<div class="section">
<p>Configuration properties are key/value pairs that are set in text files. They include system properties that control how Gobblin will pull data, and control what source Gobblin will pull the data from. Configuration files end in some user-specified suffix (by default text files ending in <code>.pull</code> or <code>.job</code> are recognized as configs files, although this is configurable). Each file represents some unit of work that needs to be done in Gobblin. For example, there will typically be a separate configuration file for each table that needs to be pulled from a database. </p>
-<p>The first section of this document contains all the required properties needed to run a basic Gobblin job. The rest of the document is dedicated to other properties that can be used to configure Gobbin jobs. The description of each configuration parameter will often refer to core Gobblin concepts and terms. If any of these terms are confusing, check out the <a href="../Gobblin-Architecture">Gobblin Architecture</a> page for a more detailed explanation of how Gobblin works. The GitHub repo also contains sample config files for specific sources. For example, there are sample config files to connect to <a href="https://github.com/apache/incubator-gobblin/tree/master/gobblin-core/src/main/resources/mysql">MySQL databases</a> and <a href="https://github.com/apache/incubator-gobblin/tree/master/gobblin-core/src/main/resources/sftp">SFTP servers</a>. </p>
+<p>The first section of this document contains all the required properties needed to run a basic Gobblin job. The rest of the document is dedicated to other properties that can be used to configure Gobbin jobs. The description of each configuration parameter will often refer to core Gobblin concepts and terms. If any of these terms are confusing, check out the <a href="../Gobblin-Architecture">Gobblin Architecture</a> page for a more detailed explanation of how Gobblin works. The GitHub repo also contains sample config files for specific sources. For example, there are sample config files to connect to <a href="https://github.com/apache/incubator-gobblin/tree/master/gobblin-core/src/main/resources/mysql" rel="nofollow">MySQL databases</a> and <a href="https://github.com/apache/incubator-gobblin/tree/master/gobblin-core/src/main/resources/sftp" rel="nofollow">SFTP servers</a>. </p>
<p>Gobblin also allows you to specify a global configuration file that contains common properties that are shared across all jobs. The <a href="#Job-Launcher-Properties">Job Launcher Properties</a> section has more information on how to specify a global properties file. </p>
<h1 id="table-of-contents">Table of Contents</h1>
<ul>
@@ -709,10 +709,10 @@
</ul>
<p>For more information on each property, check out the comprehensive list below. </p>
<p>If only these properties are set, then by default, Gobblin will run in Local mode, as opposed to running on Hadoop M/R. This means Gobblin will write Avro data to the local filesystem. In order to write to HDFS, set the <code>writer.fs.uri</code> property to the URI of the HDFS NameNode that data should be written to. Since the default version of Gobblin writes data in Avro format, the writer expects Avro records to be passed to it. Thus, any data pulled from an external source must be converted to Avro before it can be written out to the filesystem. </p>
-<p>The <code>source.class</code> property is one of the most important properties in Gobblin. It specifies what Source class to use. The Source class is responsible for determining what work needs to be done during each run of the job, and specifies what Extractor to use in order to read over each sub-unit of data. Examples of Source classes are <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/java/org/apache/gobblin/example/wikipedia/WikipediaSource.java">WikipediaSource</a> and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/java/org/apache/gobblin/example/simplejson/SimpleJsonSource.java">SimpleJsonSource</a>, which can be found in the GitHub repository. For more information on Sources and Extractors, check out the <a href="../Gobblin-Architecture">Architecture</a> page. </p>
+<p>The <code>source.class</code> property is one of the most important properties in Gobblin. It specifies what Source class to use. The Source class is responsible for determining what work needs to be done during each run of the job, and specifies what Extractor to use in order to read over each sub-unit of data. Examples of Source classes are <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/java/org/apache/gobblin/example/wikipedia/WikipediaSource.java" rel="nofollow">WikipediaSource</a> and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/java/org/apache/gobblin/example/simplejson/SimpleJsonSource.java" rel="nofollow">SimpleJsonSource</a>, which can be found in the GitHub repository. For more information on Sources and Extractors, check out the <a href="../Gobblin-Architecture">Architecture</a> page. </p>
<p>Typically, Gobblin jobs will be launched using the launch scripts in the <code>bin</code> folder. These scripts allow jobs to be launched on the local machine (e.g. SchedulerDaemon) or on Hadoop (e.g. CliMRJobLauncher). Check out the Job Launcher section below, to see the configuration difference between each launch mode. The <a href="Gobblin-Deployment">Deployment</a> page also has more information on the different ways a job can be launched. </p>
<h1 id="job-launcher-properties">Job Launcher Properties <a name="Job-Launcher-Properties"></a></h1>
-<p>Gobblin jobs can be launched and scheduled in a variety of ways. They can be scheduled via a Quartz scheduler or through <a href="https://github.com/azkaban/azkaban">Azkaban</a>. Jobs can also be run without a scheduler via the Command Line. For more information on launching Gobblin jobs, check out the <a href="Gobblin-Deployment">Deployment</a> page.</p>
+<p>Gobblin jobs can be launched and scheduled in a variety of ways. They can be scheduled via a Quartz scheduler or through <a href="https://github.com/azkaban/azkaban" rel="nofollow">Azkaban</a>. Jobs can also be run without a scheduler via the Command Line. For more information on launching Gobblin jobs, check out the <a href="Gobblin-Deployment">Deployment</a> page.</p>
<h2 id="common-job-launcher-properties">Common Job Launcher Properties <a name="Common-Launcher-Properties"></a></h2>
<p>These properties are common to both the Job Launcher and the Command Line.</p>
<table>
@@ -818,7 +818,7 @@
<h2 id="climrjoblauncher-properties">CliMRJobLauncher Properties <a name="CliMRJobLauncher-Properties"></a></h2>
<p>There are no configuration parameters specific to CliMRJobLauncher. This class is used to launch Gobblin jobs on Hadoop from the command line, the jobs are not scheduled. Common properties are set using the <code>--sysconfig</code> option when launching jobs via the command line. For more information on how to set the configuration parameters for jobs launched through the command line, check out the <a href="Gobblin-Deployment">Deployment</a> page.</p>
<h2 id="azkabanjoblauncher-properties">AzkabanJobLauncher Properties <a name="AzkabanJobLauncher-Properties"></a></h2>
-<p>There are no configuration parameters specific to AzkabanJobLauncher. This class is used to schedule Gobblin jobs on Azkaban. Common properties can be set through Azkaban by creating a <code>.properties</code> file, check out the <a href="http://azkaban.github.io/">Azkaban Documentation</a> for more information. For more information on how to set the configuration parameters for jobs scheduled through the Azkaban, check out the <a href="Gobblin-Deployment">Deployment</a> page.</p>
+<p>There are no configuration parameters specific to AzkabanJobLauncher. This class is used to schedule Gobblin jobs on Azkaban. Common properties can be set through Azkaban by creating a <code>.properties</code> file, check out the <a href="http://azkaban.github.io/" rel="nofollow">Azkaban Documentation</a> for more information. For more information on how to set the configuration parameters for jobs scheduled through the Azkaban, check out the <a href="Gobblin-Deployment">Deployment</a> page.</p>
<h1 id="job-type-properties">Job Type Properties <a name="Job-Type-Properties"></a></h1>
<h2 id="common-job-type-properties">Common Job Type Properties <a name="Common-Job-Type-Properties"></a></h2>
<table>
@@ -2266,7 +2266,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/user-guide/Docker-Integration/index.html b/docs/user-guide/Docker-Integration/index.html
index 6531146..e80dd56 100644
--- a/docs/user-guide/Docker-Integration/index.html
+++ b/docs/user-guide/Docker-Integration/index.html
@@ -519,7 +519,7 @@
<li>Docker Integration</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Docker-Integration.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Docker-Integration.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -547,12 +547,12 @@
<h1 id="docker">Docker</h1>
<p>For more information on Docker, including how to install it, check out the documentation at: https://docs.docker.com/</p>
<h1 id="docker-repositories">Docker Repositories</h1>
-<p>Gobblin currently has four different repositories, and all are on Docker Hub <a href="https://hub.docker.com/u/gobblin/">here</a>.</p>
+<p>Gobblin currently has four different repositories, and all are on Docker Hub <a href="https://hub.docker.com/u/gobblin/" rel="nofollow">here</a>.</p>
<p>The <code>gobblin/gobblin-wikipedia</code> repository contains images that run the Gobblin Wikipedia job found in the <a href="../Getting-Started">getting started guide</a>. These images are useful for users new to Docker or Gobblin, they primarily act as a "Hello World" example for the Gobblin Docker integration.</p>
<p>The <code>gobblin/gobblin-standalone</code> repository contains images that run a <a href="Gobblin-Deployment#standalone-architecture">Gobblin standalone service</a> inside a Docker container. These images provide an easy and simple way to setup a Gobblin standalone service on any Docker compatible machine.</p>
<p>The <code>gobblin/gobblin-base</code> and <code>gobblin/gobblin-distributions</code> repositories are for internal use only, and are primarily useful for Gobblin developers.</p>
<h2 id="gobblin-wikipedia-repository">Gobblin-Wikipedia Repository</h2>
-<p>The Docker images for this repository can be found on Docker Hub <a href="https://hub.docker.com/r/gobblin/gobblin-wikipedia/">here</a>. These images are mainly meant to act as a "Hello World" example for the Gobblin-Docker integration, and to provide a sanity check to see if the Gobblin-Docker integration is working on a given machine. The image contains the Gobblin configuration files to run the <a href="../Getting-Started">Gobblin Wikipedia job</a>. When a container is launched using the <code>gobblin-wikipedia</code> image, Gobblin starts up, runs the Wikipedia example, and then exits.</p>
+<p>The Docker images for this repository can be found on Docker Hub <a href="https://hub.docker.com/r/gobblin/gobblin-wikipedia/" rel="nofollow">here</a>. These images are mainly meant to act as a "Hello World" example for the Gobblin-Docker integration, and to provide a sanity check to see if the Gobblin-Docker integration is working on a given machine. The image contains the Gobblin configuration files to run the <a href="../Getting-Started">Gobblin Wikipedia job</a>. When a container is launched using the <code>gobblin-wikipedia</code> image, Gobblin starts up, runs the Wikipedia example, and then exits.</p>
<p>Running the <code>gobblin-wikipedia</code> image requires taking following steps (lets assume we want to an Ubuntu based image):</p>
<ul>
<li>Download the images from the <code>gobblin/gobblin-wikipedia</code> repository</li>
@@ -568,14 +568,14 @@
<p>The logs are printed to the console, and no errors should pop up. This should provide a nice sanity check to ensure that everything is working as expected. The output of the job will be written to a directory inside the container. When the container exits that data will be lost. In order to preserve the output of the job, continue to the next step.</p>
<ul>
-<li>Preserving the output of a Docker container requires using a <a href="https://docs.docker.com/engine/tutorials/dockervolumes/">data volume</a>. To do this, run the below command:</li>
+<li>Preserving the output of a Docker container requires using a <a href="https://docs.docker.com/engine/tutorials/dockervolumes/" rel="nofollow">data volume</a>. To do this, run the below command:</li>
</ul>
<pre><code>docker run -v /home/gobblin/work-dir:/home/gobblin/work-dir gobblin-wikipedia
</code></pre>
-<p>The output of the Gobblin-Wikipedia job should now be written to <code>/home/gobblin/work-dir/job-output</code>. The <code>-v</code> command in Docker uses a feature of Docker called <a href="https://docs.docker.com/engine/tutorials/dockervolumes/">data volumes</a>. The <code>-v</code> option mounts a host directory into a container and is of the form <code>[host-directory]:[container-directory]</code>. Now any modifications to the host directory can be seen inside the container-directory, and any modifications to the container-directory can be seen inside the host-directory. This is a standard way to ensure data persists even after a Docker container finishes. It's important to note that the <code>[host-directory]</code> in the <code>-v</code> option can be changed to any directory (on OSX it must be under the <code>/Users/</code> directory), but the <code>[container-directory]</code> must remain <code>/home/gobblin/work-dir</code> (at least for now).</p>
+<p>The output of the Gobblin-Wikipedia job should now be written to <code>/home/gobblin/work-dir/job-output</code>. The <code>-v</code> command in Docker uses a feature of Docker called <a href="https://docs.docker.com/engine/tutorials/dockervolumes/" rel="nofollow">data volumes</a>. The <code>-v</code> option mounts a host directory into a container and is of the form <code>[host-directory]:[container-directory]</code>. Now any modifications to the host directory can be seen inside the container-directory, and any modifications to the container-directory can be seen inside the host-directory. This is a standard way to ensure data persists even after a Docker container finishes. It's important to note that the <code>[host-directory]</code> in the <code>-v</code> option can be changed to any directory (on OSX it must be under the <code>/Users/</code> directory), but the <code>[container-directory]</code> must remain <code>/home/gobblin/work-dir</code> (at least for now).</p>
<h2 id="gobblin-standalone-repository">Gobblin-Standalone Repository</h2>
-<p>The Docker images for this repository can be found on Docker Hub <a href="https://hub.docker.com/r/gobblin/gobblin-standalone/">here</a>. These images run a Gobblin standalone service inside a Docker container. The Gobblin standalone service is a long running process that can run Gobblin jobs defined in a <code>.job</code> or <code>.pull</code> file. The job / pull files are submitted to the standalone service by placing them in a directory on the local filesystem. The standalone service monitors this directory for any new job / pull files and runs them either immediately or on a scheduled basis (more information on how this works can be found <a href="Working-with-Job-Configuration-Files#adding-or-changing-job-configuration-files">here</a>). Running the Gobblin standalone service inside a Docker container allows Gobblin to pick up job / pull files from a directory on the host filesystem, run the job, and write the output back the host filesystem. All the heavy lifting is done inside a Docker container, the user just needs to worry about defining and submitting job / pull files. The goal is to provide a easy to setup environment for the Gobblin standalone service.</p>
+<p>The Docker images for this repository can be found on Docker Hub <a href="https://hub.docker.com/r/gobblin/gobblin-standalone/" rel="nofollow">here</a>. These images run a Gobblin standalone service inside a Docker container. The Gobblin standalone service is a long running process that can run Gobblin jobs defined in a <code>.job</code> or <code>.pull</code> file. The job / pull files are submitted to the standalone service by placing them in a directory on the local filesystem. The standalone service monitors this directory for any new job / pull files and runs them either immediately or on a scheduled basis (more information on how this works can be found <a href="Working-with-Job-Configuration-Files#adding-or-changing-job-configuration-files">here</a>). Running the Gobblin standalone service inside a Docker container allows Gobblin to pick up job / pull files from a directory on the host filesystem, run the job, and write the output back the host filesystem. All the heavy lifting is done inside a Docker container, the user just needs to worry about defining and submitting job / pull files. The goal is to provide a easy to setup environment for the Gobblin standalone service.</p>
<p>Running the <code>gobblin-standalone</code> image requires taking the following steps:</p>
<ul>
<li>Download the images from the <code>gobblin/gobblin-standalone</code> repository</li>
@@ -601,7 +601,7 @@
<li>Create <code>gobblin-dev</code> images that provide an development environment for Gobblin contributors</li>
<li>Create <code>gobblin-kafka</code> images that provide an end-to-end service for writing to Kafka and ingesting the Kafka data through Gobblin</li>
<li>Test and write a tutorial on using <code>gobblin-standalone</code> images to write to a HDFS cluster</li>
-<li>Create images based on <a href="https://hub.docker.com/_/alpine/">Linux Alpine</a> (lightweight Linux distro)</li>
+<li>Create images based on <a href="https://hub.docker.com/_/alpine/" rel="nofollow">Linux Alpine</a> (lightweight Linux distro)</li>
</ul>
</div>
@@ -625,7 +625,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/user-guide/FAQs/index.html b/docs/user-guide/FAQs/index.html
index 925c870..a7dc111 100644
--- a/docs/user-guide/FAQs/index.html
+++ b/docs/user-guide/FAQs/index.html
@@ -519,7 +519,7 @@
<li>FAQs</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/FAQs.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/FAQs.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -583,7 +583,7 @@
<h5 id="why-does-gobblin-on-hadoop-stall-for-a-long-time-between-adding-files-to-the-distrbutedcache-and-launching-the-actual-job">Why does Gobblin on Hadoop stall for a long time between adding files to the DistrbutedCache, and launching the actual job?</h5>
<p>Gobblin takes all WorkUnits created by the Source class and serializes each one into a file on Hadoop. These files are read by each map task, and are deserialized into Gobblin Tasks. These Tasks are then run by the map-task. The reason the job stalls is that Gobblin is writing all these files to HDFS, which can take a while especially if there are a lot of tasks to run. See the <a href="Gobblin Deployment">Gobblin Deployment</a> page for more information.</p>
<h5 id="how-do-i-fix-unsupportedfilesystemexception-no-abstractfilesystem-for-scheme-null">How do I fix <code>UnsupportedFileSystemException: No AbstractFileSystem for scheme: null</code>?</h5>
-<p>This error typically occurs due to Hadoop version conflict issues. If Gobblin is compiled against a specific Hadoop version, but then deployed on a different Hadoop version or installation, this error may be thrown. For example, if you simply compile Gobblin using <code>./gradlew clean build</code>, but deploy Gobblin to a cluster with <a href="https://www.cloudera.com/content/www/en-us/products/apache-hadoop/key-cdh-components.html">CDH</a> installed, you may hit this error.</p>
+<p>This error typically occurs due to Hadoop version conflict issues. If Gobblin is compiled against a specific Hadoop version, but then deployed on a different Hadoop version or installation, this error may be thrown. For example, if you simply compile Gobblin using <code>./gradlew clean build</code>, but deploy Gobblin to a cluster with <a href="https://www.cloudera.com/content/www/en-us/products/apache-hadoop/key-cdh-components.html" rel="nofollow">CDH</a> installed, you may hit this error.</p>
<p>It is important to realize that the the <code>gobblin-dist.tar.gz</code> file produced by <code>./gradlew clean build</code> will include all the Hadoop jar dependencies; and if one follows the <a href="Gobblin-Deployment#Hadoop-MapReduce-Deployment">MR deployment guide</a>, Gobblin will be launched with these dependencies on the classpath.</p>
<p>To fix this take the following steps:</p>
<ul>
@@ -591,18 +591,18 @@
<li>Ensure that the environment variable <code>HADOOP_CLASSPATH</code> is set and points to a directory containing the Hadoop libraries for the cluster</li>
</ul>
<h5 id="how-do-i-compile-gobblin-against-cdh">How do I compile Gobblin against CDH?</h5>
-<p><a href="https://www.cloudera.com/content/www/en-us/products/apache-hadoop/key-cdh-components.html">Cloudera Distributed Hadoop</a> (often abbreviated as CDH) is a popular Hadoop distribution. Typically, when running Gobblin on a CDH cluster it is recommended that one also compile Gobblin against the same CDH version. Not doing so may cause unexpected runtime behavior. To compile against a specific CDH version simply use the <code>hadoopVersion</code> parameter. For example, to compile against version <code>2.5.0-cdh5.3.0</code> run <code>./gradlew clean build -PhadoopVersion=2.5.0-cdh5.3.0</code>.</p>
+<p><a href="https://www.cloudera.com/content/www/en-us/products/apache-hadoop/key-cdh-components.html" rel="nofollow">Cloudera Distributed Hadoop</a> (often abbreviated as CDH) is a popular Hadoop distribution. Typically, when running Gobblin on a CDH cluster it is recommended that one also compile Gobblin against the same CDH version. Not doing so may cause unexpected runtime behavior. To compile against a specific CDH version simply use the <code>hadoopVersion</code> parameter. For example, to compile against version <code>2.5.0-cdh5.3.0</code> run <code>./gradlew clean build -PhadoopVersion=2.5.0-cdh5.3.0</code>.</p>
<h5 id="resolve-gobblin-on-mr-exception-ioexception-not-all-tasks-running-in-mapper-attempt_id-completed-successfully">Resolve Gobblin-on-MR Exception <code>IOException: Not all tasks running in mapper attempt_id completed successfully</code></h5>
<p>This exception usually just means that a Hadoop Map Task running Gobblin Tasks threw some exception. Unfortunately, the exception isn't truly indicative of the underlying problem, all it is really saying is that something went wrong in the Gobblin Task. Each Hadoop Map Task has its own log file and it is often easiest to look at the logs of the Map Task when debugging this problem. There are multiple ways to do this, but one of the easiest ways is to execute <code>yarn logs -applicationId <application ID> [OPTIONS]</code></p>
<h5 id="gradle-build-fails-with-cannot-invoke-method-geturls-on-null-object">Gradle Build Fails With <code>Cannot invoke method getURLs on null object</code></h5>
-<p>Add <code>-x test</code> to build the project without running the tests; this will make the exception go away. If one needs to run the tests then make sure <a href="https://en.wikipedia.org/wiki/Java_Cryptography_Extension">Java Cryptography Extension</a> is installed.</p>
+<p>Add <code>-x test</code> to build the project without running the tests; this will make the exception go away. If one needs to run the tests then make sure <a href="https://en.wikipedia.org/wiki/Java_Cryptography_Extension" rel="nofollow">Java Cryptography Extension</a> is installed.</p>
<h1 id="gradle">Gradle</h1>
<h2 id="technical-questions_1">Technical Questions</h2>
<h4 id="how-do-i-add-a-new-external-dependency">How do I add a new external dependency?</h4>
-<p>Say I want to add <a href="http://mvnrepository.com/artifact/org.apache.oozie/oozie-core/4.2.0"><code>oozie-core-4.2.0.jar</code></a> as a dependency to the <code>gobblin-scheduler</code> subproject. I would first open the file <code>build.gradle</code> and add the following entry to the <code>ext.externalDependency</code> array: <code>"oozieCore": "org.apache.oozie:oozie-core:4.2.0"</code>.</p>
+<p>Say I want to add <a href="http://mvnrepository.com/artifact/org.apache.oozie/oozie-core/4.2.0" rel="nofollow"><code>oozie-core-4.2.0.jar</code></a> as a dependency to the <code>gobblin-scheduler</code> subproject. I would first open the file <code>build.gradle</code> and add the following entry to the <code>ext.externalDependency</code> array: <code>"oozieCore": "org.apache.oozie:oozie-core:4.2.0"</code>.</p>
<p>Then in the <code>gobblin-scheduler/build.gradle</code> file I would add the following line to the dependency block: <code>compile externalDependency.oozieCore</code>.</p>
<h4 id="how-do-i-add-a-new-maven-repository-to-pull-artifacts-from">How do I add a new Maven Repository to pull artifacts from?</h4>
-<p>Often times, one may have important artifacts stored in a local or private Maven repository. As of 01/21/2016 Gobblin only pulls artifacts from the following Maven Repositories: <a href="http://repo1.maven.org/maven/">Maven Central</a>, <a href="http://conjars.org/repo">Conjars</a>, and <a href="https://repository.cloudera.com/artifactory/cloudera-repos/">Cloudera</a>.</p>
+<p>Often times, one may have important artifacts stored in a local or private Maven repository. As of 01/21/2016 Gobblin only pulls artifacts from the following Maven Repositories: <a href="http://repo1.maven.org/maven/" rel="nofollow">Maven Central</a>, <a href="http://conjars.org/repo" rel="nofollow">Conjars</a>, and <a href="https://repository.cloudera.com/artifactory/cloudera-repos/" rel="nofollow">Cloudera</a>.</p>
<p>In order to add another Maven Repository modify the <code>defaultEnvironment.gradle</code> file and the new repository using the same pattern as the existing ones.</p>
</div>
@@ -626,7 +626,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/user-guide/Gobblin-CLI/index.html b/docs/user-guide/Gobblin-CLI/index.html
index b857b41..a9058c8 100644
--- a/docs/user-guide/Gobblin-CLI/index.html
+++ b/docs/user-guide/Gobblin-CLI/index.html
@@ -523,7 +523,7 @@
<li>Gobblin CLI</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Gobblin-CLI.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Gobblin-CLI.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -587,7 +587,7 @@
<p>Argument details:
<em> <code>--conf-dir</code>: specifies the path to directory containing gobblin system configuration files, like <code>application.conf</code> or <code>reference.conf</code>, <code>log4j.properties</code> and <code>quartz.properties</code>.
-</em> <code>--log4j-conf</code>: specify the path of log4j config file to override the one in config directory (default is <code><conf>/<gobblin-mode>/log4j.properties</code>. Gobblin uses <a href="http://www.slf4j.org/">SLF4J</a> and the <a href="http://mvnrepository.com/artifact/org.slf4j/slf4j-log4j12">slf4j-log4j12</a> binding for logging.
+</em> <code>--log4j-conf</code>: specify the path of log4j config file to override the one in config directory (default is <code><conf>/<gobblin-mode>/log4j.properties</code>. Gobblin uses <a href="http://www.slf4j.org/" rel="nofollow">SLF4J</a> and the <a href="http://mvnrepository.com/artifact/org.slf4j/slf4j-log4j12" rel="nofollow">slf4j-log4j12</a> binding for logging.
<em> <code>--jvmopts</code>: to specify any JVM parameters, default is <code>-Xmx1g -Xms512m</code>.
</em> <code>--enable-gc-logs</code>: adds GC options to JVM parameters: <code>-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+UseCompressedOops -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$GOBBLIN_LOGS/ -Xloggc:$GOBBLIN_LOGS/gobblin-$GOBBLIN_MODE-gc.log</code>
<em> <code>--show-classpath</code>: It prints the full value of the classpath that gobblin uses.
@@ -661,7 +661,7 @@
# e.g file:///gobblin-conf/my-job/wikipedia.pull or hdfs:///gobblin-conf/my-job/kafka-hdfs.pull
</code></pre>
-<p>The <strong>oneShot</strong> app comes with certain hardcoded defaults (that it inherits from EmbeddedGobblin <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/resources/embedded/embedded.conf">here</a>), that you may not be expecting. Make sure you understand what they do and override them in your baseConf or appConf files if needed.</p>
+<p>The <strong>oneShot</strong> app comes with certain hardcoded defaults (that it inherits from EmbeddedGobblin <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/resources/embedded/embedded.conf" rel="nofollow">here</a>), that you may not be expecting. Make sure you understand what they do and override them in your baseConf or appConf files if needed.</p>
<p>Notable differences at the time of this writing include:</p>
<ul>
<li>state.store.enabled = false (set this to true in your appConfig or baseConfig if you want state storage for repeated oneshot runs)</li>
@@ -774,7 +774,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/user-guide/Gobblin-Compliance/index.html b/docs/user-guide/Gobblin-Compliance/index.html
index 28f47f1..9f3c06d 100644
--- a/docs/user-guide/Gobblin-Compliance/index.html
+++ b/docs/user-guide/Gobblin-Compliance/index.html
@@ -508,7 +508,7 @@
<li>Gobblin Compliance</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Gobblin-Compliance.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Gobblin-Compliance.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -619,7 +619,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/user-guide/Gobblin-Deployment/index.html b/docs/user-guide/Gobblin-Deployment/index.html
index 3c372b3..dd04ac2 100644
--- a/docs/user-guide/Gobblin-Deployment/index.html
+++ b/docs/user-guide/Gobblin-Deployment/index.html
@@ -520,7 +520,7 @@
<li>Deployment</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Gobblin-Deployment.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Gobblin-Deployment.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -544,7 +544,7 @@
</div>
<h2 id="gobblin-execution-modes-overview">Gobblin Execution Modes Overview <a name="gobblin-execution-modes-Overview"></a></h2>
<p>One important feature of Gobblin is that it can be run on different platforms. Currently, Gobblin can run in standalone mode (which runs on a single machine), and on Hadoop MapReduce mode (which runs on a Hadoop cluster). This page summarizes the different deployment modes of Gobblin. It is important to understand the architecture of Gobblin in a specific deployment mode, so this page also describes the architecture of each deployment mode. </p>
-<p>Gobblin supports Java 7 and up, but can only run on Hadoop 2.x. By default, Gobblin will build against Hadoop 2.x, run <code>./gradlew clean build</code>. More information on how to build Gobblin can be found <a href="https://github.com/apache/incubator-gobblin/blob/master/README.md">here</a>. All directories/paths referred below are relative to <code>gobblin-dist</code>.</p>
+<p>Gobblin supports Java 7 and up, but can only run on Hadoop 2.x. By default, Gobblin will build against Hadoop 2.x, run <code>./gradlew clean build</code>. More information on how to build Gobblin can be found <a href="https://github.com/apache/incubator-gobblin/blob/master/README.md" rel="nofollow">here</a>. All directories/paths referred below are relative to <code>gobblin-dist</code>.</p>
<p>To run gobblin in any of the following executuon mode using <code>gobblin.sh</code>, refer <a href="/gobblin-docs/user-guide/Gobblin-CLI.md">Gobblin-CLI</a> for the usage.</p>
<h2 id="standalone-architecture">Standalone Architecture <a name="Standalone-Architecture"></a></h2>
<p>The following diagram illustrates the Gobblin standalone architecture. In the standalone mode, a Gobblin instance runs in a single JVM and tasks run in a thread pool, the size of which is configurable. The standalone mode is good for light-weight data sources such as small databases. The standalone mode is also the default mode for trying and testing Gobblin. </p>
@@ -609,7 +609,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/user-guide/Gobblin-Schedulers/index.html b/docs/user-guide/Gobblin-Schedulers/index.html
index 540a8bf..0801e3b 100644
--- a/docs/user-guide/Gobblin-Schedulers/index.html
+++ b/docs/user-guide/Gobblin-Schedulers/index.html
@@ -522,7 +522,7 @@
<li>Schedulers</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Gobblin-Schedulers.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Gobblin-Schedulers.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -559,21 +559,21 @@
</ul>
</div>
<h1 id="introduction">Introduction</h1>
-<p>Gobblin jobs can be scheduled on a recurring basis using a few different tools. Gobblin ships with a built in <a href="https://quartz-scheduler.org/">Quartz Scheduler</a>. Gobblin also integrates with a few other third party tools.</p>
+<p>Gobblin jobs can be scheduled on a recurring basis using a few different tools. Gobblin ships with a built in <a href="https://quartz-scheduler.org/" rel="nofollow">Quartz Scheduler</a>. Gobblin also integrates with a few other third party tools.</p>
<h1 id="quartz">Quartz</h1>
-<p>Gobblin has a built in Quartz scheduler as part of the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/scheduler/JobScheduler.java"><code>JobScheduler</code></a> class. This class integrates with the Gobblin <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/scheduler/SchedulerDaemon.java"><code>SchedulerDaemon</code></a>, which can be run using the Gobblin <a href="https://github.com/apache/incubator-gobblin/blob/master/bin/gobblin-standalone.sh">`bin/gobblin-standalone.sh</a> script.</p>
+<p>Gobblin has a built in Quartz scheduler as part of the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/scheduler/JobScheduler.java" rel="nofollow"><code>JobScheduler</code></a> class. This class integrates with the Gobblin <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/scheduler/SchedulerDaemon.java" rel="nofollow"><code>SchedulerDaemon</code></a>, which can be run using the Gobblin <a href="https://github.com/apache/incubator-gobblin/blob/master/bin/gobblin-standalone.sh" rel="nofollow">`bin/gobblin-standalone.sh</a> script.</p>
<p>So in order to take advantage of the Quartz scheduler two steps need to be taken:</p>
<ul>
<li>Use the <code>bin/gobblin-standalone.sh</code> script</li>
<li>Add the property <code>job.schedule</code> to the <code>.pull</code> file<ul>
-<li>The value for this property should be a <a href="http://quartz-scheduler.org/api/2.2.0/org/quartz/CronTrigger.html">CRONTrigger</a></li>
+<li>The value for this property should be a <a href="http://quartz-scheduler.org/api/2.2.0/org/quartz/CronTrigger.html" rel="nofollow">CRONTrigger</a></li>
</ul>
</li>
</ul>
<h1 id="azkaban">Azkaban</h1>
-<p>Gobblin can be launched via <a href="https://azkaban.github.io/">Azkaban</a>, and open-source Workflow Manager for scheduling and launching Hadoop jobs. Gobblin's <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-azkaban/src/main/java/org/apache/gobblin/azkaban/AzkabanJobLauncher.java"><code>AzkabanJobLauncher</code></a> can be used to launch a Gobblin job through Azkaban.</p>
-<p>One has to follow the typical setup to create a zip file that can be uploaded to Azkaban (it should include all dependent jars, which can be found in <code>gobblin-dist.tar.gz</code>). The <code>.job</code> file for the Azkaban Job should contain all configuration properties that would be put in a <code>.pull</code> file (for example, the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull">Wikipedia Example</a> <code>.pull</code> file). All Gobblin system dependent properties (e.g. <a href="https://github.com/apache/incubator-gobblin/blob/master/conf/gobblin-mapreduce.properties"><code>conf/gobblin-mapreduce.properties</code></a> or <a href="https://github.com/apache/incubator-gobblin/blob/master/conf/gobblin-standalone-v2.properties"><code>conf/gobblin-standalone.properties</code></a>) should also be in the zip file.</p>
-<p>In the Azkaban <code>.job</code> file, the <code>type</code> parameter should be set to <code>hadoopJava</code> (see <a href="http://azkaban.github.io/azkaban/docs/latest/#hadoopjava-type">here</a> for more information about the <code>hadoopJava</code> Job Type). The <code>job.class</code> parameter should be set to <code>gobblin.azkaban.AzkabanJobLauncher</code>.</p>
+<p>Gobblin can be launched via <a href="https://azkaban.github.io/" rel="nofollow">Azkaban</a>, and open-source Workflow Manager for scheduling and launching Hadoop jobs. Gobblin's <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-azkaban/src/main/java/org/apache/gobblin/azkaban/AzkabanJobLauncher.java" rel="nofollow"><code>AzkabanJobLauncher</code></a> can be used to launch a Gobblin job through Azkaban.</p>
+<p>One has to follow the typical setup to create a zip file that can be uploaded to Azkaban (it should include all dependent jars, which can be found in <code>gobblin-dist.tar.gz</code>). The <code>.job</code> file for the Azkaban Job should contain all configuration properties that would be put in a <code>.pull</code> file (for example, the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull" rel="nofollow">Wikipedia Example</a> <code>.pull</code> file). All Gobblin system dependent properties (e.g. <a href="https://github.com/apache/incubator-gobblin/blob/master/conf/gobblin-mapreduce.properties" rel="nofollow"><code>conf/gobblin-mapreduce.properties</code></a> or <a href="https://github.com/apache/incubator-gobblin/blob/master/conf/gobblin-standalone-v2.properties" rel="nofollow"><code>conf/gobblin-standalone.properties</code></a>) should also be in the zip file.</p>
+<p>In the Azkaban <code>.job</code> file, the <code>type</code> parameter should be set to <code>hadoopJava</code> (see <a href="http://azkaban.github.io/azkaban/docs/latest/#hadoopjava-type" rel="nofollow">here</a> for more information about the <code>hadoopJava</code> Job Type). The <code>job.class</code> parameter should be set to <code>gobblin.azkaban.AzkabanJobLauncher</code>.</p>
<h1 id="oozie">Oozie</h1>
<p><a href="https://oozie.apache.org/">Oozie</a> is a very popular scheduler for the Hadoop environment. It allows users to define complex workflows using XML files. A workflow can be composed of a series of actions, such as Java Jobs, Pig Jobs, Spark Jobs, etc. Gobblin has two integration points with Oozie. It can be run as a stand-alone Java process via Oozie's <code><java></code> tag, or it can be run as an Map Reduce job via Oozie.</p>
<p>The following guides assume Oozie is already setup and running on some machine, if this is not the case consult the Oozie documentation for getting everything setup.</p>
@@ -581,10 +581,10 @@
<h3 id="launching-gobblin-in-local-mode">Launching Gobblin in Local Mode</h3>
<p>This guide focuses on getting Gobblin to run in as a stand alone Java Process. This means it will not launch a separate MR job to distribute its workload. It is important to understand how the current version of Oozie will launch a Java process. It will first start an MapReduce job and will run the Gobblin as a Java process inside a single map task. The Gobblin job will then ingest all data it is configured to pull and then it will shutdown.</p>
<h4 id="example-config-files">Example Config Files</h4>
-<p><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-oozie/src/test/resources/local"><code>gobblin-oozie/src/main/resources/local</code></a> contains sample configuration files for launching Gobblin Oozie. There are a number of important files in this directory:</p>
-<p><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-oozie/src/test/resources/local/gobblin-oozie-example-system.properties"><code>gobblin-oozie-example-system.properties</code></a> contains default system level properties for Gobblin. When launched with Oozie, Gobblin will run inside a map task; it is thus recommended to configure Gobblin to write directly to HDFS rather than the local file system. The property <code>fs.uri</code> in this file should be changed to point to the NameNode of the Hadoop File System the job should write to. By default, all data is written under a folder called <code>gobblin-out</code>; to change this modify the <code>gobblin.work.dir</code> parameter in this file.</p>
-<p><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-oozie/src/test/resources/local/gobblin-oozie-example-workflow.properties"><code>gobblin-oozie-example-workflow.properties</code></a> contains default Oozie properties for any job launched. It is also the entry point for launching an Oozie job (e.g. to launch an Oozie job from the command line you execute <code>oozie job -config gobblin-oozie-example-workflow.properties -run</code>). In this file one needs to update the <code>name.node</code> and <code>resource.manager</code> to the values specific to their environment. Another important property in this file is <code>oozie.wf.application.path</code>; it points to a folder on HDFS that contains any workflows to be run. It is important to note, that the <code>workflow.xml</code> files must be on HDFS in order for Oozie to pick them up (this is because Oozie typically runs on a separate machine as any client process).</p>
-<p><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-oozie/src/test/resources/local/gobblin-oozie-example-workflow.xml"><code>gobblin-oozie-example-workflow.xml</code></a> contains an example Oozie workflow. This example simply launches a Java process that invokes the main method of the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/local/CliLocalJobLauncher.java"><code>CliLocalJobLauncher</code></a>. The main method of this class expects two file paths to be passed to it (once again these files need to be on HDFS). The <code>jobconfig</code> arg should point to a file on HDFS containing all job configuration parameters. An example <code>jobconfig</code> file can be found <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull">here</a>. The <code>sysconfig</code> arg should point to a file on HDFS containing all system configuration parameters. An example <code>sysconfig</code> file for Oozie can be found <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-oozie/src/test/resources/local/gobblin-oozie-example-system.properties">here</a>.</p>
+<p><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-oozie/src/test/resources/local" rel="nofollow"><code>gobblin-oozie/src/main/resources/local</code></a> contains sample configuration files for launching Gobblin Oozie. There are a number of important files in this directory:</p>
+<p><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-oozie/src/test/resources/local/gobblin-oozie-example-system.properties" rel="nofollow"><code>gobblin-oozie-example-system.properties</code></a> contains default system level properties for Gobblin. When launched with Oozie, Gobblin will run inside a map task; it is thus recommended to configure Gobblin to write directly to HDFS rather than the local file system. The property <code>fs.uri</code> in this file should be changed to point to the NameNode of the Hadoop File System the job should write to. By default, all data is written under a folder called <code>gobblin-out</code>; to change this modify the <code>gobblin.work.dir</code> parameter in this file.</p>
+<p><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-oozie/src/test/resources/local/gobblin-oozie-example-workflow.properties" rel="nofollow"><code>gobblin-oozie-example-workflow.properties</code></a> contains default Oozie properties for any job launched. It is also the entry point for launching an Oozie job (e.g. to launch an Oozie job from the command line you execute <code>oozie job -config gobblin-oozie-example-workflow.properties -run</code>). In this file one needs to update the <code>name.node</code> and <code>resource.manager</code> to the values specific to their environment. Another important property in this file is <code>oozie.wf.application.path</code>; it points to a folder on HDFS that contains any workflows to be run. It is important to note, that the <code>workflow.xml</code> files must be on HDFS in order for Oozie to pick them up (this is because Oozie typically runs on a separate machine as any client process).</p>
+<p><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-oozie/src/test/resources/local/gobblin-oozie-example-workflow.xml" rel="nofollow"><code>gobblin-oozie-example-workflow.xml</code></a> contains an example Oozie workflow. This example simply launches a Java process that invokes the main method of the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/local/CliLocalJobLauncher.java" rel="nofollow"><code>CliLocalJobLauncher</code></a>. The main method of this class expects two file paths to be passed to it (once again these files need to be on HDFS). The <code>jobconfig</code> arg should point to a file on HDFS containing all job configuration parameters. An example <code>jobconfig</code> file can be found <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull" rel="nofollow">here</a>. The <code>sysconfig</code> arg should point to a file on HDFS containing all system configuration parameters. An example <code>sysconfig</code> file for Oozie can be found <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-oozie/src/test/resources/local/gobblin-oozie-example-system.properties" rel="nofollow">here</a>.</p>
<!---Ying Do you think we can add some descriptions about launching through MR mode? The simplest way is to use the <shell> tag and invoke `gobblin-mapreduce.sh`. I've tested it before.-->
<h4 id="uploading-files-to-hdfs">Uploading Files to HDFS</h4>
@@ -596,7 +596,7 @@
<h3 id="launching-gobblin-in-mapreduce-mode">Launching Gobblin in MapReduce Mode</h3>
<p>Launching Gobblin in mapreduce Mode works quite similar to the local mode. In this mode, the oozie launcher action will spawn a second mapreduce process where gobblin will process its tasks in distributed mode across the cluster. Since each of the Mappers needs access to the gobblin libraries, we need to provide the jars via the <code>job.hdfs.jars</code> variable</p>
<h4 id="example-config-files_1">Example Config Files</h4>
-<p><a href="https://github.com/apache/incubator-gobblin/tree/master/gobblin-oozie/src/test/resources/mapreduce"><code>gobblin-oozie/src/main/resources/mapreduce</code></a> contains sample configuration files for launching Gobblin Oozie in Mapreduce mode. The main difference to launching Gobblin Oozie in Local mode are a view extra MapReduce related configuration variables in the sysconfig.properties file and launching CliMRJobLauncher instead CliLocalJobLauncher.</p>
+<p><a href="https://github.com/apache/incubator-gobblin/tree/master/gobblin-oozie/src/test/resources/mapreduce" rel="nofollow"><code>gobblin-oozie/src/main/resources/mapreduce</code></a> contains sample configuration files for launching Gobblin Oozie in Mapreduce mode. The main difference to launching Gobblin Oozie in Local mode are a view extra MapReduce related configuration variables in the sysconfig.properties file and launching CliMRJobLauncher instead CliLocalJobLauncher.</p>
<h4 id="further-steps">Further steps</h4>
<p>Everything else should be working the same way as in Local mode (see above)</p>
<h3 id="debugging-tips">Debugging Tips</h3>
@@ -624,7 +624,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/user-guide/Gobblin-as-a-Library/index.html b/docs/user-guide/Gobblin-as-a-Library/index.html
index 85bd150..75225f5 100644
--- a/docs/user-guide/Gobblin-as-a-Library/index.html
+++ b/docs/user-guide/Gobblin-as-a-Library/index.html
@@ -514,7 +514,7 @@
<li>Gobblin as a Library</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Gobblin-as-a-Library.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Gobblin-as-a-Library.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -649,7 +649,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/user-guide/Gobblin-genericLoad/index.html b/docs/user-guide/Gobblin-genericLoad/index.html
index a7776f5..380bd17 100644
--- a/docs/user-guide/Gobblin-genericLoad/index.html
+++ b/docs/user-guide/Gobblin-genericLoad/index.html
@@ -506,7 +506,7 @@
<li>Generic Configuration Loading</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Gobblin-genericLoad.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Gobblin-genericLoad.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -553,7 +553,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/user-guide/Gobblin-on-Yarn/index.html b/docs/user-guide/Gobblin-on-Yarn/index.html
index 40cb9ea..f37c230 100644
--- a/docs/user-guide/Gobblin-on-Yarn/index.html
+++ b/docs/user-guide/Gobblin-on-Yarn/index.html
@@ -557,7 +557,7 @@
<li>Gobblin on Yarn</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Gobblin-on-Yarn.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Gobblin-on-Yarn.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -644,17 +644,17 @@
<p>Helix also supports a way of doing messaging between different components of a cluster, e.g., between the controller to the participants, or between the client and the controller. The Gobblin Yarn application uses this messaging mechanism to implement graceful shutdown initiated by the client as well as delegation token renew notifications from the client to the ApplicationMaster and the WorkUnitRunner containers.</p>
<p>Heiix relies on ZooKeeper for its operations, and particularly for maintaining the state of the cluster and the resources (tasks in this case). Both the Helix controller and participants connect to ZooKeeper during their entire lifetime. The ApplicationMaster serves as the Helix controller and the worker containers serve as the Helix participants, respectively, as discussed in details below. </p>
<h2 id="gobblin-yarn-application-launcher">Gobblin Yarn Application Launcher</h2>
-<p>The Gobblin Yarn Application Launcher (implemented by class <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java"><code>GobblinYarnAppLauncher</code></a>) is the client/driver of a Gobblin Yarn application. The first thing the <code>GobblinYarnAppLauncher</code> does when it starts is to register itself with Helix as a <em>spectator</em> and creates a new Helix cluster with name specified through the configuration property <code>gobblin.yarn.helix.cluster.name</code>, if no cluster with the name exists. </p>
+<p>The Gobblin Yarn Application Launcher (implemented by class <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java" rel="nofollow"><code>GobblinYarnAppLauncher</code></a>) is the client/driver of a Gobblin Yarn application. The first thing the <code>GobblinYarnAppLauncher</code> does when it starts is to register itself with Helix as a <em>spectator</em> and creates a new Helix cluster with name specified through the configuration property <code>gobblin.yarn.helix.cluster.name</code>, if no cluster with the name exists. </p>
<p>The <code>GobblinYarnAppLauncher</code> then sets up the Gobblin Yarn application and submits it to run on Yarn. Once the Yarn application successfully starts running, it starts an application state monitor that periodically checks the state of the Gobblin Yarn application. If the state is one of the exit states (<code>FINISHED</code>, <code>FAILED</code>, or <code>KILLED</code>), the <code>GobblinYarnAppLauncher</code> shuts down itself. </p>
<p>Upon successfully submitting the application to run on Yarn, the <code>GobblinYarnAppLauncher</code> also starts a <code>ServiceManager</code> that manages the following services that auxiliate the running of the application:</p>
<h3 id="yarnappsecuritymanager">YarnAppSecurityManager</h3>
-<p>The <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnAppSecurityManager.java"><code>YarnAppSecurityManager</code></a> works with the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnContainerSecurityManager.java"><code>YarnContainerSecurityManager</code></a> running in the ApplicationMaster and the WorkUnitRunner for a complete solution for security and delegation token management. The <code>YarnAppSecurityManager</code> is responsible for periodically logging in through a Kerberos keytab and getting the delegation token refreshed regularly after each login. Each time the delegation token is refreshed, the <code>YarnContainerSecurityManager</code> writes the new token to a file on HDFS and sends a message to the ApplicationMaster and each WorkUnitRunner, notifying them the refresh of the delegation token. Checkout <a href="#yarncontainersecuritymanager"><code>YarnContainerSecurityManager</code></a> on how the other side of this system works.</p>
+<p>The <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnAppSecurityManager.java" rel="nofollow"><code>YarnAppSecurityManager</code></a> works with the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnContainerSecurityManager.java" rel="nofollow"><code>YarnContainerSecurityManager</code></a> running in the ApplicationMaster and the WorkUnitRunner for a complete solution for security and delegation token management. The <code>YarnAppSecurityManager</code> is responsible for periodically logging in through a Kerberos keytab and getting the delegation token refreshed regularly after each login. Each time the delegation token is refreshed, the <code>YarnContainerSecurityManager</code> writes the new token to a file on HDFS and sends a message to the ApplicationMaster and each WorkUnitRunner, notifying them the refresh of the delegation token. Checkout <a href="#yarncontainersecuritymanager"><code>YarnContainerSecurityManager</code></a> on how the other side of this system works.</p>
<h3 id="logcopier">LogCopier</h3>
-<p>The service <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-utility/src/main/java/org/apache/gobblin/util/logs/LogCopier.java"><code>LogCopier</code></a> in <code>GobblinYarnAppLauncher</code> streams the ApplicationMaster and WorkUnitRunner logs in near real-time from the central location on HDFS where the logs are streamed to from the ApplicationMaster and WorkUnitRunner containers, to the local directory specified through the configuration property <code>gobblin.yarn.logs.sink.root.dir</code> on the machine where the <code>GobblinYarnAppLauncher</code> runs. More details on this can be found in <a href="#log-aggregation">Log Aggregation</a>.</p>
+<p>The service <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-utility/src/main/java/org/apache/gobblin/util/logs/LogCopier.java" rel="nofollow"><code>LogCopier</code></a> in <code>GobblinYarnAppLauncher</code> streams the ApplicationMaster and WorkUnitRunner logs in near real-time from the central location on HDFS where the logs are streamed to from the ApplicationMaster and WorkUnitRunner containers, to the local directory specified through the configuration property <code>gobblin.yarn.logs.sink.root.dir</code> on the machine where the <code>GobblinYarnAppLauncher</code> runs. More details on this can be found in <a href="#log-aggregation">Log Aggregation</a>.</p>
<h2 id="gobblin-applicationmaster">Gobblin ApplicationMaster</h2>
-<p>The ApplicationMaster process runs the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinApplicationMaster.java"><code>GobblinApplicationMaster</code></a>, which uses a <code>ServiceManager</code> to manage the services supporting the operation of the ApplicationMaster process. The services running in <code>GobblinApplicationMaster</code> will be discussed later. When it starts, the first thing <code>GobblinApplicationMaster</code> does is to connect to ZooKeeper and register itself as a Helix <em>controller</em>. It then starts the <code>ServiceManager</code>, which in turn starts the services it manages, as described below. </p>
+<p>The ApplicationMaster process runs the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinApplicationMaster.java" rel="nofollow"><code>GobblinApplicationMaster</code></a>, which uses a <code>ServiceManager</code> to manage the services supporting the operation of the ApplicationMaster process. The services running in <code>GobblinApplicationMaster</code> will be discussed later. When it starts, the first thing <code>GobblinApplicationMaster</code> does is to connect to ZooKeeper and register itself as a Helix <em>controller</em>. It then starts the <code>ServiceManager</code>, which in turn starts the services it manages, as described below. </p>
<h3 id="yarnservice">YarnService</h3>
-<p>The service <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnService.java"><code>YarnService</code></a> handles all Yarn-related task including the following:</p>
+<p>The service <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnService.java" rel="nofollow"><code>YarnService</code></a> handles all Yarn-related task including the following:</p>
<ul>
<li>Registering and un-registering the ApplicationMaster with the Yarn ResourceManager.</li>
<li>Requesting the initial set of containers from the Yarn ResourceManager.</li>
@@ -662,22 +662,22 @@
</ul>
<p>This design makes it switch to a different resource manager, e.g., Mesos, by replacing the service <code>YarnService</code> with something else specific to the resource manager, e.g., <code>MesosService</code>.</p>
<h3 id="gobblinhelixjobscheduler">GobblinHelixJobScheduler</h3>
-<p><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinApplicationMaster.java"><code>GobblinApplicationMaster</code></a> runs the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinHelixJobScheduler.java"><code>GobblinHelixJobScheduler</code></a> that schedules jobs to run through the Helix <a href="http://helix.apache.org/0.7.1-docs/recipes/task_dag_execution.html">Distributed Task Execution Framework</a>. For each Gobblin job run, the <code>GobblinHelixJobScheduler</code> starts a <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinHelixJobLauncher.java"><code>GobblinHelixJobLauncher</code></a> that wraps the Gobblin job into a <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinHelixJob.java"><code>GobblinHelixJob</code></a> and each Gobblin <code>Task</code> into a <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinHelixTask.java"><code>GobblinHelixTask</code></a>, which implements the Helix's <code>Task</code> interface so Helix knows how to execute it. The <code>GobblinHelixJobLauncher</code> then submits the job to a Helix job queue named after the Gobblin job name, from which the Helix Distributed Task Execution Framework picks up the job and runs its tasks through the live participants (available containers).</p>
-<p>Like the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/local/LocalJobLauncher.java"><code>LocalJobLauncher</code></a> and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/mapreduce/MRJobLauncher.java"><code>MRJobLauncher</code></a>, the <code>GobblinHelixJobLauncher</code> handles output data commit and job state persistence. </p>
+<p><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinApplicationMaster.java"><code>GobblinApplicationMaster</code></a> runs the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinHelixJobScheduler.java"><code>GobblinHelixJobScheduler</code></a> that schedules jobs to run through the Helix <a href="http://helix.apache.org/0.7.1-docs/recipes/task_dag_execution.html">Distributed Task Execution Framework</a>. For each Gobblin job run, the <code>GobblinHelixJobScheduler</code> starts a <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinHelixJobLauncher.java" rel="nofollow"><code>GobblinHelixJobLauncher</code></a> that wraps the Gobblin job into a <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinHelixJob.java" rel="nofollow"><code>GobblinHelixJob</code></a> and each Gobblin <code>Task</code> into a <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinHelixTask.java" rel="nofollow"><code>GobblinHelixTask</code></a>, which implements the Helix's <code>Task</code> interface so Helix knows how to execute it. The <code>GobblinHelixJobLauncher</code> then submits the job to a Helix job queue named after the Gobblin job name, from which the Helix Distributed Task Execution Framework picks up the job and runs its tasks through the live participants (available containers).</p>
+<p>Like the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/local/LocalJobLauncher.java" rel="nofollow"><code>LocalJobLauncher</code></a> and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/mapreduce/MRJobLauncher.java" rel="nofollow"><code>MRJobLauncher</code></a>, the <code>GobblinHelixJobLauncher</code> handles output data commit and job state persistence. </p>
<h3 id="logcopier_1">LogCopier</h3>
-<p>The service <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-utility/src/main/java/org/apache/gobblin/util/logs/LogCopier.java"><code>LogCopier</code></a> in <code>GobblinApplicationMaster</code> streams the ApplicationMaster logs in near real-time from the machine running the ApplicationMaster container to a central location on HDFS so the logs can be accessed at runtime. More details on this can be found in <a href="#log-aggregation">Log Aggregation</a>.</p>
+<p>The service <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-utility/src/main/java/org/apache/gobblin/util/logs/LogCopier.java" rel="nofollow"><code>LogCopier</code></a> in <code>GobblinApplicationMaster</code> streams the ApplicationMaster logs in near real-time from the machine running the ApplicationMaster container to a central location on HDFS so the logs can be accessed at runtime. More details on this can be found in <a href="#log-aggregation">Log Aggregation</a>.</p>
<h3 id="yarncontainersecuritymanager">YarnContainerSecurityManager</h3>
-<p>The <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnContainerSecurityManager.java"><code>YarnContainerSecurityManager</code></a> runs in both the ApplicationMaster and the WorkUnitRunner. When it starts, it registers a message handler with the <code>HelixManager</code> for handling messages on refreshes of the delegation token. Once such a message is received, the <code>YarnContainerSecurityManager</code> gets the path to the token file on HDFS from the message, and updated the the current login user with the new token read from the file.</p>
+<p>The <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnContainerSecurityManager.java" rel="nofollow"><code>YarnContainerSecurityManager</code></a> runs in both the ApplicationMaster and the WorkUnitRunner. When it starts, it registers a message handler with the <code>HelixManager</code> for handling messages on refreshes of the delegation token. Once such a message is received, the <code>YarnContainerSecurityManager</code> gets the path to the token file on HDFS from the message, and updated the the current login user with the new token read from the file.</p>
<h2 id="gobblin-workunitrunner">Gobblin WorkUnitRunner</h2>
-<p>The WorkUnitRunner process runs as part of <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinTaskRunner.java"><code>GobblinTaskRunner</code></a>, which uses a <code>ServiceManager</code> to manage the services supporting the operation of the WorkUnitRunner process. The services running in <code>GobblinWorkUnitRunner</code> will be discussed later. When it starts, the first thing <code>GobblinWorkUnitRunner</code> does is to connect to ZooKeeper and register itself as a Helix <em>participant</em>. It then starts the <code>ServiceManager</code>, which in turn starts the services it manages, as discussed below. </p>
+<p>The WorkUnitRunner process runs as part of <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinTaskRunner.java" rel="nofollow"><code>GobblinTaskRunner</code></a>, which uses a <code>ServiceManager</code> to manage the services supporting the operation of the WorkUnitRunner process. The services running in <code>GobblinWorkUnitRunner</code> will be discussed later. When it starts, the first thing <code>GobblinWorkUnitRunner</code> does is to connect to ZooKeeper and register itself as a Helix <em>participant</em>. It then starts the <code>ServiceManager</code>, which in turn starts the services it manages, as discussed below. </p>
<h3 id="taskexecutor">TaskExecutor</h3>
-<p>The <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/TaskExecutor.java"><code>TaskExecutor</code></a> remains the same as in the standalone and MR modes, and is purely responsible for running tasks assigned to a WorkUnitRunner. </p>
+<p>The <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/TaskExecutor.java" rel="nofollow"><code>TaskExecutor</code></a> remains the same as in the standalone and MR modes, and is purely responsible for running tasks assigned to a WorkUnitRunner. </p>
<h3 id="gobblinhelixtaskstatetracker">GobblinHelixTaskStateTracker</h3>
-<p>The <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinHelixTaskStateTracker.java"><code>GobblinHelixTaskStateTracker</code></a> has a similar responsibility as the <code>LocalTaskStateTracker</code> and <code>MRTaskStateTracker</code>: keeping track of the state of running tasks including operational metrics, e.g., total records pulled, records pulled per second, total bytes pulled, bytes pulled per second, etc.</p>
+<p>The <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinHelixTaskStateTracker.java" rel="nofollow"><code>GobblinHelixTaskStateTracker</code></a> has a similar responsibility as the <code>LocalTaskStateTracker</code> and <code>MRTaskStateTracker</code>: keeping track of the state of running tasks including operational metrics, e.g., total records pulled, records pulled per second, total bytes pulled, bytes pulled per second, etc.</p>
<h3 id="logcopier_2">LogCopier</h3>
-<p>The service <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-utility/src/main/java/org/apache/gobblin/util/logs/LogCopier.java"><code>LogCopier</code></a> in <code>GobblinWorkUnitRunner</code> streams the WorkUnitRunner logs in near real-time from the machine running the WorkUnitRunner container to a central location on HDFS so the logs can be accessed at runtime. More details on this can be found in <a href="#log-aggregation">Log Aggregation</a>.</p>
+<p>The service <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-utility/src/main/java/org/apache/gobblin/util/logs/LogCopier.java" rel="nofollow"><code>LogCopier</code></a> in <code>GobblinWorkUnitRunner</code> streams the WorkUnitRunner logs in near real-time from the machine running the WorkUnitRunner container to a central location on HDFS so the logs can be accessed at runtime. More details on this can be found in <a href="#log-aggregation">Log Aggregation</a>.</p>
<h3 id="yarncontainersecuritymanager_1">YarnContainerSecurityManager</h3>
-<p>The <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnContainerSecurityManager.java"><code>YarnContainerSecurityManager</code></a> in <code>GobblinWorkUnitRunner</code> works in the same way as it in <code>GobblinApplicationMaster</code>. </p>
+<p>The <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnContainerSecurityManager.java" rel="nofollow"><code>YarnContainerSecurityManager</code></a> in <code>GobblinWorkUnitRunner</code> works in the same way as it in <code>GobblinApplicationMaster</code>. </p>
<h2 id="failure-handling">Failure Handling</h2>
<h3 id="applicationmaster-failure-handling">ApplicationMaster Failure Handling</h3>
<p>Under normal operation, the Gobblin ApplicationMaster stays alive unless being asked to stop through a message sent from the launcher (the <code>GobblinYarnAppLauncher</code>) as part of the orderly shutdown process. It may, however, fail or get killed by the Yarn ResourceManager for various reasons. For example, the container running the ApplicationMaster may fail and exit due to node failures, or get killed because of using more memory than claimed. When a shutdown of the ApplicationMaster is triggered (e.g., when the shutdown hook is triggered) for any reason, it does so gracefully, i.e., it attempts to stop every services it manages, stop all the running containers, and unregister itself with the ResourceManager. Shutting down the ApplicationMaster shuts down the Yarn application and the application launcher will eventually know that the application completes through a periodic check on the application status. </p>
@@ -876,7 +876,7 @@
<h2 id="job-lock">Job Lock</h2>
<p>It is recommended to use zookeeper for maintaining job locks. See <a href="Configuration-Properties-Glossary#ZookeeperBasedJobLock-Properties">ZookeeperBasedJobLock Properties</a> for the relevant configuration properties.</p>
<h2 id="configuration-system">Configuration System</h2>
-<p>The Gobblin Yarn application uses the <a href="https://github.com/typesafehub/config">Typesafe Config</a> library to handle the application configuration. Following <a href="https://github.com/typesafehub/config">Typesafe Config</a>'s model, the Gobblin Yarn application uses a single file named <code>application.conf</code> for all configuration properties and another file named <code>reference.conf</code> for default values. A sample <code>application.conf</code> is shown below: </p>
+<p>The Gobblin Yarn application uses the <a href="https://github.com/typesafehub/config" rel="nofollow">Typesafe Config</a> library to handle the application configuration. Following <a href="https://github.com/typesafehub/config" rel="nofollow">Typesafe Config</a>'s model, the Gobblin Yarn application uses a single file named <code>application.conf</code> for all configuration properties and another file named <code>reference.conf</code> for default values. A sample <code>application.conf</code> is shown below: </p>
<pre><code># Yarn/Helix configuration properties
gobblin.yarn.helix.cluster.name=GobblinYarnTest
gobblin.yarn.app.name=GobblinYarnTest
@@ -975,9 +975,9 @@
<h2 id="supporting-existing-gobblin-jobs">Supporting Existing Gobblin Jobs</h2>
<p>Gobblin on Yarn is backward compatible and supports existing Gobblin jobs running in the standalone and MR modes. To run existing Gobblin jobs, simply put the job configuration files into a directory on the local file system of the driver and setting the configuration property <code>gobblin.yarn.job.conf.path</code> to point to the directory. When the Gobblin Yarn application starts, Yarn will package the configuration files as a tarball and make sure the tarball gets copied to the ApplicationMaster and properly uncompressed. The <code>GobblinHelixJobScheduler</code> then loads the job configuration files and schedule the jobs to run.</p>
<h1 id="monitoring">Monitoring</h1>
-<p>Gobblin Yarn uses the <a href="../metrics/Gobblin-Metrics">Gobblin Metrics</a> library for collecting and reporting metrics at the container, job, and task levels. Each <code>GobblinWorkUnitRunner</code> maintains a <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/ContainerMetrics.java"><code>ContainerMetrics</code></a> that is the parent of the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/util/JobMetrics.java"><code>JobMetrics</code></a> of each job run the container is involved, which is the parent of the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/util/TaskMetrics.java"><code>TaskMetrics</code></a> of each task of the job run. This hierarchical structure allows us to do pre-aggregation in the containers before reporting the metrics to the backend. </p>
+<p>Gobblin Yarn uses the <a href="../metrics/Gobblin-Metrics">Gobblin Metrics</a> library for collecting and reporting metrics at the container, job, and task levels. Each <code>GobblinWorkUnitRunner</code> maintains a <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/ContainerMetrics.java" rel="nofollow"><code>ContainerMetrics</code></a> that is the parent of the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/util/JobMetrics.java" rel="nofollow"><code>JobMetrics</code></a> of each job run the container is involved, which is the parent of the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/util/TaskMetrics.java" rel="nofollow"><code>TaskMetrics</code></a> of each task of the job run. This hierarchical structure allows us to do pre-aggregation in the containers before reporting the metrics to the backend. </p>
<p>Collected metrics can be reported to various sinks such as Kafka, files, and JMX, depending on the configuration. Specifically, <code>metrics.enabled</code> controls whether metrics collecting and reporting are enabled or not. <code>metrics.reporting.kafka.enabled</code>, <code>metrics.reporting.file.enabled</code>, and <code>metrics.reporting.jmx.enabled</code> control whether collected metrics should be reported or not to Kafka, files, and JMX, respectively. Please refer to <a href="Configuration-Properties-Glossary#Metrics-Properties">Metrics Properties</a> for the available configuration properties related to metrics collecting and reporting. </p>
-<p>In addition to metric collecting and reporting, Gobblin Yarn also supports writing job execution information to a MySQL-backed job execution history store, which keeps track of job execution information. Please refer to the <a href="https://github.com/apache/incubator-gobblin/tree/master/gobblin-metastore/src/main/resources/db/migration">DDL</a> for the relevant MySQL tables. Detailed information on the job execution history store including how to configure it can be found <a href="Job-Execution-History-Store">here</a>. </p>
+<p>In addition to metric collecting and reporting, Gobblin Yarn also supports writing job execution information to a MySQL-backed job execution history store, which keeps track of job execution information. Please refer to the <a href="https://github.com/apache/incubator-gobblin/tree/master/gobblin-metastore/src/main/resources/db/migration" rel="nofollow">DDL</a> for the relevant MySQL tables. Detailed information on the job execution history store including how to configure it can be found <a href="Job-Execution-History-Store">here</a>. </p>
</div>
</div>
@@ -1000,7 +1000,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/user-guide/Gobblin-template/index.html b/docs/user-guide/Gobblin-template/index.html
index cc8cb06..81ffeda 100644
--- a/docs/user-guide/Gobblin-template/index.html
+++ b/docs/user-guide/Gobblin-template/index.html
@@ -515,7 +515,7 @@
<li>Template</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Gobblin-template.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Gobblin-template.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -607,7 +607,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/user-guide/Hive-Registration/index.html b/docs/user-guide/Hive-Registration/index.html
index e331efe..cda7419 100644
--- a/docs/user-guide/Hive-Registration/index.html
+++ b/docs/user-guide/Hive-Registration/index.html
@@ -510,7 +510,7 @@
<li>Hive Registration</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Hive-Registration.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Hive-Registration.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -538,12 +538,12 @@
</div>
<p>Gobblin has the ability to register the ingested/compacted data in Hive. This allows registering data in Hive immediately after data is published at the destination, offering much lower latency compared to doing data ingestion and Hive registration separately.</p>
<h2 id="how-hive-registration-works-in-gobblin">How Hive Registration Works in Gobblin</h2>
-<p>Hive registration is done in <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/HiveRegister.java"><code>HiveRegister</code></a>. After the data is published, the publisher or compaction runner will create an instance of <code>HiveRegister</code>, and for each path that should be registered in Hive, the publisher or compaction runner will use a specific <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/policy/HiveRegistrationPolicy.java"><code>HiveRegistrationPolicy</code></a> to create a list of <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/spec/HiveSpec.java"><code>HiveSpec</code></a>s for the path. It creates a list of <code>HiveSpec</code>s rather than a single <code>HiveSpec</code> for each path, so that the same path can be registered in multiple tables or databases.</p>
+<p>Hive registration is done in <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/HiveRegister.java" rel="nofollow"><code>HiveRegister</code></a>. After the data is published, the publisher or compaction runner will create an instance of <code>HiveRegister</code>, and for each path that should be registered in Hive, the publisher or compaction runner will use a specific <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/policy/HiveRegistrationPolicy.java" rel="nofollow"><code>HiveRegistrationPolicy</code></a> to create a list of <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/spec/HiveSpec.java" rel="nofollow"><code>HiveSpec</code></a>s for the path. It creates a list of <code>HiveSpec</code>s rather than a single <code>HiveSpec</code> for each path, so that the same path can be registered in multiple tables or databases.</p>
<h3 id="hivespec"><code>HiveSpec</code></h3>
-<p>A <code>HiveSpec</code> specifies how a path should be registered in Hive, i.e., which database, which table, which partition should the path be registered. An example is <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/spec/SimpleHiveSpec.java"><code>SimpleHiveSpec</code></a>.</p>
+<p>A <code>HiveSpec</code> specifies how a path should be registered in Hive, i.e., which database, which table, which partition should the path be registered. An example is <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/spec/SimpleHiveSpec.java" rel="nofollow"><code>SimpleHiveSpec</code></a>.</p>
<h3 id="hiveregistrationpolicy"><code>HiveRegistrationPolicy</code></h3>
<p><code>HiveRegistrationPolicy</code> is responsible for generating <code>HiveSpec</code>s given a path. For example, if you want paths ending with a date (e.g., <code>/(something)/2016/05/22</code>) to be registered in the corresponding daily partition (e.g., <code>daily-2016-05-22</code>), you can create an implementation of <code>HiveRegistrationPolicy</code> that contains the logic of converting such a path into a Hive partition. </p>
-<p>An example is <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/policy/HiveRegistrationPolicyBase.java"><code>HiveRegistrationPolicyBase</code></a>, which provides base implementation for getting database names and table names for a path:</p>
+<p>An example is <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/policy/HiveRegistrationPolicyBase.java" rel="nofollow"><code>HiveRegistrationPolicyBase</code></a>, which provides base implementation for getting database names and table names for a path:</p>
<ul>
<li>A database/table name can be specified explicitly in <code>hive.database.name</code> or <code>hive.table.name</code>.</li>
<li>Alternatively, a database/table regex can be provided in <code>hive.database.regex</code> or <code>hive.table.regex</code>. The regex will be matched against the path to be registered, and if they match, the first group is considered the database/table name.</li>
@@ -552,10 +552,10 @@
</ul>
<p>One should in general extend <code>HiveRegistrationPolicyBase</code> when implementing a new <code>HiveRegistrationPolicy</code>.</p>
<h3 id="hiveserdemanager"><code>HiveSerDeManager</code></h3>
-<p>If the data to be registered is in a format other than plain text (CSV, TSV, etc.), you often need to use a SerDe and specify some SerDe properties including the type of SerDe, input format, output format, schema, etc. This is done in <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/HiveSerDeManager.java"><code>HiveSerDeManager</code></a>, which is part of a <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/HiveRegistrationUnit.java"><code>HiveRegistrationUnit</code></a> (i.e., <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-compaction/src/main/java/org/apache/gobblin/compaction/hive/HiveTable.java"><code>HiveTable</code></a> or <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/HivePartition.java"><code>HivePartition</code></a>). An example is <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/avro/HiveAvroSerDeManager.java"><code>HiveAvroSerDeManager</code></a>.</p>
+<p>If the data to be registered is in a format other than plain text (CSV, TSV, etc.), you often need to use a SerDe and specify some SerDe properties including the type of SerDe, input format, output format, schema, etc. This is done in <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/HiveSerDeManager.java" rel="nofollow"><code>HiveSerDeManager</code></a>, which is part of a <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/HiveRegistrationUnit.java" rel="nofollow"><code>HiveRegistrationUnit</code></a> (i.e., <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-compaction/src/main/java/org/apache/gobblin/compaction/hive/HiveTable.java" rel="nofollow"><code>HiveTable</code></a> or <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/HivePartition.java" rel="nofollow"><code>HivePartition</code></a>). An example is <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/avro/HiveAvroSerDeManager.java" rel="nofollow"><code>HiveAvroSerDeManager</code></a>.</p>
<h3 id="predicate-and-activity">Predicate and Activity</h3>
-<p>One or more <code>Predicate</code>s can be attached to a <code>HiveSpec</code>. If a <code>HiveSpec</code> contains <code>Predicate</code>s, unless <code>Predicate</code>s return <code>true</code>, the <code>HiveSpec</code> will not be registered. This is useful in cases where, for example, one only wants to register a path in Hive if a particular Hive table or partition doesn't already exist. An example is <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/spec/predicate/TableNotExistPredicate.java"><code>TableNotExistPredicate</code></a>.</p>
-<p>One or more <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/spec/activity/Activity.java"><code>Activity</code></a>s can be attached to a <code>HiveSpec</code>. There are two types of activities: pre-activities and post-activities, which will be executed before and after a <code>HiveSpec</code> is registered, respectively. This is useful, for example, when you need to drop/alter a table/partition before or after a path is registered. An example is <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/spec/activity/DropTableActivity.java"><code>DropTableActivity</code></a>.</p>
+<p>One or more <code>Predicate</code>s can be attached to a <code>HiveSpec</code>. If a <code>HiveSpec</code> contains <code>Predicate</code>s, unless <code>Predicate</code>s return <code>true</code>, the <code>HiveSpec</code> will not be registered. This is useful in cases where, for example, one only wants to register a path in Hive if a particular Hive table or partition doesn't already exist. An example is <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/spec/predicate/TableNotExistPredicate.java" rel="nofollow"><code>TableNotExistPredicate</code></a>.</p>
+<p>One or more <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/spec/activity/Activity.java" rel="nofollow"><code>Activity</code></a>s can be attached to a <code>HiveSpec</code>. There are two types of activities: pre-activities and post-activities, which will be executed before and after a <code>HiveSpec</code> is registered, respectively. This is useful, for example, when you need to drop/alter a table/partition before or after a path is registered. An example is <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/spec/activity/DropTableActivity.java" rel="nofollow"><code>DropTableActivity</code></a>.</p>
<h2 id="how-to-use-hive-registration-in-your-gobblin-job">How to Use Hive Registration in Your Gobblin Job</h2>
<p>First, is to implement a <code>HiveRegistrationPolicy</code> (or reuse an existing one), then specify its class name in config property <code>hive.registration.policy</code>.</p>
<p>Then, specify the appropriate table/partition properties in <code>hive.table.partition.props</code>, storage descriptor properties in
@@ -563,10 +563,10 @@
<p>Example table/partition properties are "owner" and "retention", example storage descriptor properties are "location", "compressed", "numBuckets", example SerDe properties are "serializationLib", "avro.schema.url".</p>
<p>If you are running a Gobblin ingestion job:</p>
<ul>
-<li>If data is published in the job (which is the default case), use a job-level data publisher that can perform Hive registration, such as <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/publisher/BaseDataPublisherWithHiveRegistration.java"><code>BaseDataPublisherWithHiveRegistration</code></a>. If you need to do Hive registration with a different publisher than <code>BaseDataPublisher</code>, you will need to extend that publisher to do Hive registration, which will be similar as how <code>BaseDataPublisher</code> is extended into <code>BaseDataPublisherWithHiveRegistration</code>.</li>
-<li>If data is published in the tasks, use <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/publisher/HiveRegistrationPublisher.java"><code>HiveRegistrationPublisher</code></a> as the job-level data publisher. This publisher does not publish any data; it only does Hive registration.</li>
+<li>If data is published in the job (which is the default case), use a job-level data publisher that can perform Hive registration, such as <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/publisher/BaseDataPublisherWithHiveRegistration.java" rel="nofollow"><code>BaseDataPublisherWithHiveRegistration</code></a>. If you need to do Hive registration with a different publisher than <code>BaseDataPublisher</code>, you will need to extend that publisher to do Hive registration, which will be similar as how <code>BaseDataPublisher</code> is extended into <code>BaseDataPublisherWithHiveRegistration</code>.</li>
+<li>If data is published in the tasks, use <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/publisher/HiveRegistrationPublisher.java" rel="nofollow"><code>HiveRegistrationPublisher</code></a> as the job-level data publisher. This publisher does not publish any data; it only does Hive registration.</li>
</ul>
-<p>If you are running a Gobblin compaction job: add <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-compaction/src/main/java/org/apache/gobblin/compaction/hive/registration/HiveRegistrationCompactorListener.java"><code>HiveRegistrationCompactorListener</code></a> to the list of compaction listeners by adding the class name to property <code>compaction.listeners</code>.</p>
+<p>If you are running a Gobblin compaction job: add <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-compaction/src/main/java/org/apache/gobblin/compaction/hive/registration/HiveRegistrationCompactorListener.java" rel="nofollow"><code>HiveRegistrationCompactorListener</code></a> to the list of compaction listeners by adding the class name to property <code>compaction.listeners</code>.</p>
<h2 id="hive-registration-config-properties">Hive Registration Config Properties</h2>
<table>
<thead>
@@ -676,7 +676,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/user-guide/Job-Execution-History-Store/index.html b/docs/user-guide/Job-Execution-History-Store/index.html
index 23b0975..eaab0aa 100644
--- a/docs/user-guide/Job-Execution-History-Store/index.html
+++ b/docs/user-guide/Job-Execution-History-Store/index.html
@@ -526,7 +526,7 @@
<li>Job Execution History Store</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Job-Execution-History-Store.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Job-Execution-History-Store.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -684,7 +684,7 @@
</tbody>
</table>
<h2 id="default-implementation">Default Implementation</h2>
-<p>The default implementation of the Job Execution History Store stores job execution information into a MySQL database in a few different tables. Specifically, the following tables are used and should be created before writing to the store is enabled. Checkout the MySQL <a href="https://github.com/apache/incubator-gobblin/tree/master/gobblin-metastore/src/main/resources/db/migration">DDLs</a> of the tables for detailed columns of each table.</p>
+<p>The default implementation of the Job Execution History Store stores job execution information into a MySQL database in a few different tables. Specifically, the following tables are used and should be created before writing to the store is enabled. Checkout the MySQL <a href="https://github.com/apache/incubator-gobblin/tree/master/gobblin-metastore/src/main/resources/db/migration" rel="nofollow">DDLs</a> of the tables for detailed columns of each table.</p>
<ul>
<li>Table <code>gobblin_job_executions</code> stores basic information about a job execution including the start and end times, job running state, number of launched and completed tasks, etc. </li>
<li>Table <code>gobblin_task_executions</code> stores basic information on task executions of a job, including the start and end times, task running state, task failure message if any, etc, of each task. </li>
@@ -814,7 +814,7 @@
}
</code></pre>
-<p>The API is built with <a href="http://www.rest.li">rest.li</a>, which generates documentation on compilation and can be found at <code>http://<hostname:port>/restli/docs</code>.</p>
+<p>The API is built with <a href="http://www.rest.li" rel="nofollow">rest.li</a>, which generates documentation on compilation and can be found at <code>http://<hostname:port>/restli/docs</code>.</p>
<h3 id="example-queries">Example Queries</h3>
<p><em>Fetch the 10 most recent job executions with a job name <code>TestJobName</code></em></p>
<pre><code class="bash">curl "http://<hostname:port>/jobExecutions/idType=JOB_NAME&id.string=TestJobName&limit=10"
@@ -849,7 +849,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/user-guide/Monitoring/index.html b/docs/user-guide/Monitoring/index.html
index 4f52c84..007319e 100644
--- a/docs/user-guide/Monitoring/index.html
+++ b/docs/user-guide/Monitoring/index.html
@@ -526,7 +526,7 @@
<li>Monitoring</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Monitoring.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Monitoring.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -557,7 +557,7 @@
<h2 id="metrics-reporting">Metrics Reporting</h2>
<p>Out-of-the-box, Gobblin reports metrics though:</p>
<ul>
-<li><em>JMX</em> : used in the standalone deployment. Metrics reported to JMX can be checked using using tools such as <a href="http://visualvm.java.net/">VisualVM</a> or JConsole. </li>
+<li><em>JMX</em> : used in the standalone deployment. Metrics reported to JMX can be checked using using tools such as <a href="http://visualvm.java.net/" rel="nofollow">VisualVM</a> or JConsole. </li>
<li><em>Metric log files</em>: Files are stored in a root directory defined by the property <code>metrics.log.dir</code>. Each Gobblin job has its own subdirectory under the root directory and each run of the job has its own metric log file named after the job ID as <code>${job_id}.metrics.log</code>.</li>
<li><em>Hadoop counters</em> : used for M/R deployments. Gobblin-specific metrics are reported in the "JOB" or "TASK" groups for job- and task- level metrics. By default, task-level metrics are not reported through Hadoop counters as doing so may cause the number of Hadoop counters to go beyond the system-wide limit. However, users can choose to turn on reporting task-level metrics as Hadoop counters by setting <code>mr.include.task.counters=true</code>. </li>
</ul>
@@ -570,7 +570,7 @@
<li><code>jvm.threads</code>: this covers metrics related to thread states, e.g., thread count and thread deadlocks.</li>
<li><code>jvm.fileDescriptorRatio</code>: this measures the ratio of open file descriptors.</li>
</ul>
-<p>All JVM metrics are reported via JMX and can be checked using tools such as <a href="http://visualvm.java.net/">VisualVM</a> or JConsole. </p>
+<p>All JVM metrics are reported via JMX and can be checked using tools such as <a href="http://visualvm.java.net/" rel="nofollow">VisualVM</a> or JConsole. </p>
<h3 id="pre-defined-job-execution-metrics">Pre-defined Job Execution Metrics</h3>
<p>Internally, Gobblin pre-defines a minimum set of metrics listed below in two metric groups: <code>JOB</code> and <code>TASK</code> for job-level metrics and task-level metrics, respectively. Those metrics are useful in keeping track of the progress and performance of job executions.</p>
<ul>
@@ -581,7 +581,7 @@
</ul>
<p>Among the above metrics, <code>${metric_group}.${id}.records</code> and <code>${metric_group}.${id}.bytes</code> are reported as Hadoop MapReduce counters for Gobblin jobs running on Hadoop.</p>
<h2 id="job-execution-history-store">Job Execution History Store</h2>
-<p>Gobblin also supports writing job execution information to a job execution history store backed by a database of choice. Gobblin uses MySQL by default and it ships with the SQL <a href="https://github.com/apache/incubator-gobblin/tree/master/gobblin-metastore/src/main/resources/db/migration">DDLs</a> of the relevant MySQL tables, although it still allows users to choose which database to use as long as the schema of the tables is compatible. Users can use the properties <code>job.history.store.url</code> and <code>job.history.store.jdbc.driver</code> to specify the database URL and the JDBC driver to work with the database of choice. The user name and password used to access the database can be specified using the properties <code>job.history.store.user</code> and <code>job.history.store.password</code>. An example configuration is shown below:</p>
+<p>Gobblin also supports writing job execution information to a job execution history store backed by a database of choice. Gobblin uses MySQL by default and it ships with the SQL <a href="https://github.com/apache/incubator-gobblin/tree/master/gobblin-metastore/src/main/resources/db/migration" rel="nofollow">DDLs</a> of the relevant MySQL tables, although it still allows users to choose which database to use as long as the schema of the tables is compatible. Users can use the properties <code>job.history.store.url</code> and <code>job.history.store.jdbc.driver</code> to specify the database URL and the JDBC driver to work with the database of choice. The user name and password used to access the database can be specified using the properties <code>job.history.store.user</code> and <code>job.history.store.password</code>. An example configuration is shown below:</p>
<pre><code>job.history.store.url=jdbc:mysql://localhost/gobblin
job.history.store.jdbc.driver=com.mysql.jdbc.Driver
job.history.store.user=gobblin
@@ -635,7 +635,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/user-guide/Partitioned-Writers/index.html b/docs/user-guide/Partitioned-Writers/index.html
index d52013f..faefc3c 100644
--- a/docs/user-guide/Partitioned-Writers/index.html
+++ b/docs/user-guide/Partitioned-Writers/index.html
@@ -514,7 +514,7 @@
<li>Partitioned Writers</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Partitioned-Writers.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Partitioned-Writers.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -613,7 +613,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/user-guide/Source-schema-and-Converters/index.html b/docs/user-guide/Source-schema-and-Converters/index.html
index 322e802..8084760 100644
--- a/docs/user-guide/Source-schema-and-Converters/index.html
+++ b/docs/user-guide/Source-schema-and-Converters/index.html
@@ -525,7 +525,7 @@
<li>Source schema and Converters</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Source-schema-and-Converters.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Source-schema-and-Converters.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -560,7 +560,7 @@
<p>A source schema has to be declared before extracting the data from the source.
To define the source schema <code>source.schema</code> property is available which takes a JSON value defining the source schema.
This schema is used by Converters to perform data type or data format conversions.
-The java class representation of a source schema can be found here <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/schema/Schema.java">Schema.java</a>.</p>
+The java class representation of a source schema can be found here <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/schema/Schema.java" rel="nofollow">Schema.java</a>.</p>
<h2 id="converters">Converters</h2>
<p>In Gobblin library a Converter is an interface for classes that implement data transformations, e.g., data type conversions,
schema projections, data manipulations, data filtering, etc. This interface is responsible for
@@ -614,32 +614,32 @@
<code>converter.classes="org.apache.gobblin.converter.json.JsonStringToJsonIntermediateConverter,org.apache.gobblin.converter.avro.JsonIntermediateToAvroConverter"</code></p>
<h2 id="converters-available-in-gobblin">Converters available in Gobblin</h2>
<ul>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/avro/AvroFieldRetrieverConverter.java">AvroFieldRetrieverConverter.java</a></li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/avro/AvroRecordToAvroWritableConverter.java">AvroRecordToAvroWritableConverter.java</a></li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/avro/AvroToAvroCopyableConverter.java">AvroToAvroCopyableConverter.java</a></li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/avro/AvroToBytesConverter.java">AvroToBytesConverter.java</a></li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/avro/BytesToAvroConverter.java">BytesToAvroConverter.java</a></li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/avro/FlattenNestedKeyConverter.java">FlattenNestedKeyConverter.java</a></li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/avro/JsonIntermediateToAvroConverter.java">JsonIntermediateToAvroConverter.java</a></li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/avro/JsonRecordAvroSchemaToAvroConverter.java">JsonRecordAvroSchemaToAvroConverter.java</a></li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/csv/CsvToJsonConverter.java">CsvToJsonConverter.java</a></li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/csv/CsvToJsonConverterV2.java">CsvToJsonConverterV2.java</a></li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/filter/AvroFieldsPickConverter.java">AvroFieldsPickConverter.java</a></li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/filter/AvroFilterConverter.java">AvroFilterConverter.java</a></li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/http/AvroToRestJsonEntryConverter.java">AvroToRestJsonEntryConverter.java</a></li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/json/BytesToJsonConverter.java">BytesToJsonConverter.java</a></li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/json/JsonStringToJsonIntermediateConverter.java">JsonStringToJsonIntermediateConverter.java</a></li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/json/JsonToStringConverter.java">JsonToStringConverter.java</a></li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/objectstore/ObjectStoreConverter.java">ObjectStoreConverter.java</a></li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/objectstore/ObjectStoreDeleteConverter.java">ObjectStoreDeleteConverter.java</a></li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/serde/HiveSerDeConverter.java">HiveSerDeConverter.java</a></li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/string/ObjectToStringConverter.java">ObjectToStringConverter.java</a></li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/string/StringFilterConverter.java">StringFilterConverter.java</a></li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/string/StringSplitterConverter.java">StringSplitterConverter.java</a></li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/string/StringSplitterToListConverter.java">StringSplitterToListConverter.java</a></li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/string/StringToBytesConverter.java">StringToBytesConverter.java</a></li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/string/TextToStringConverter.java">TextToStringConverter.java</a></li>
-<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/GobblinMetricsPinotFlattenerConverter.java">GobblinMetricsPinotFlattenerConverter.java</a></li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/avro/AvroFieldRetrieverConverter.java" rel="nofollow">AvroFieldRetrieverConverter.java</a></li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/avro/AvroRecordToAvroWritableConverter.java" rel="nofollow">AvroRecordToAvroWritableConverter.java</a></li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/avro/AvroToAvroCopyableConverter.java" rel="nofollow">AvroToAvroCopyableConverter.java</a></li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/avro/AvroToBytesConverter.java" rel="nofollow">AvroToBytesConverter.java</a></li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/avro/BytesToAvroConverter.java" rel="nofollow">BytesToAvroConverter.java</a></li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/avro/FlattenNestedKeyConverter.java" rel="nofollow">FlattenNestedKeyConverter.java</a></li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/avro/JsonIntermediateToAvroConverter.java" rel="nofollow">JsonIntermediateToAvroConverter.java</a></li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/avro/JsonRecordAvroSchemaToAvroConverter.java" rel="nofollow">JsonRecordAvroSchemaToAvroConverter.java</a></li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/csv/CsvToJsonConverter.java" rel="nofollow">CsvToJsonConverter.java</a></li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/csv/CsvToJsonConverterV2.java" rel="nofollow">CsvToJsonConverterV2.java</a></li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/filter/AvroFieldsPickConverter.java" rel="nofollow">AvroFieldsPickConverter.java</a></li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/filter/AvroFilterConverter.java" rel="nofollow">AvroFilterConverter.java</a></li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/http/AvroToRestJsonEntryConverter.java" rel="nofollow">AvroToRestJsonEntryConverter.java</a></li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/json/BytesToJsonConverter.java" rel="nofollow">BytesToJsonConverter.java</a></li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/json/JsonStringToJsonIntermediateConverter.java" rel="nofollow">JsonStringToJsonIntermediateConverter.java</a></li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/json/JsonToStringConverter.java" rel="nofollow">JsonToStringConverter.java</a></li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/objectstore/ObjectStoreConverter.java" rel="nofollow">ObjectStoreConverter.java</a></li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/objectstore/ObjectStoreDeleteConverter.java" rel="nofollow">ObjectStoreDeleteConverter.java</a></li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/serde/HiveSerDeConverter.java" rel="nofollow">HiveSerDeConverter.java</a></li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/string/ObjectToStringConverter.java" rel="nofollow">ObjectToStringConverter.java</a></li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/string/StringFilterConverter.java" rel="nofollow">StringFilterConverter.java</a></li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/string/StringSplitterConverter.java" rel="nofollow">StringSplitterConverter.java</a></li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/string/StringSplitterToListConverter.java" rel="nofollow">StringSplitterToListConverter.java</a></li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/string/StringToBytesConverter.java" rel="nofollow">StringToBytesConverter.java</a></li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/string/TextToStringConverter.java" rel="nofollow">TextToStringConverter.java</a></li>
+<li><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/GobblinMetricsPinotFlattenerConverter.java" rel="nofollow">GobblinMetricsPinotFlattenerConverter.java</a></li>
</ul>
<h2 id="schema-specification">Schema specification</h2>
<p>The following section discusses the specification to define source schema using a JSON format.</p>
@@ -725,11 +725,11 @@
</thead>
<tbody>
<tr>
-<td><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/avro/JsonIntermediateToAvroConverter.java">JsonIntermediateToAvroConverter.java</a></td>
+<td><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/converter/avro/JsonIntermediateToAvroConverter.java" rel="nofollow">JsonIntermediateToAvroConverter.java</a></td>
<td><ul><li>DATE</li><li>TIMESTAMP</li><li>TIME</li><li>STRING</li><li>BYTES</li><li>INT</li><li>LONG</li><li>FLOAT</li><li>DOUBLE</li><li>BOOLEAN</li><li>ARRAY</li><li>MAP</li><li>ENUM</li></ul></td>
</tr>
<tr>
-<td><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-parquet/src/main/java/org/apache/gobblin/converter/parquet/JsonIntermediateToParquetGroupConverter.java">JsonIntermediateToParquetGroupConverter.java</a></td>
+<td><a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-parquet/src/main/java/org/apache/gobblin/converter/parquet/JsonIntermediateToParquetGroupConverter.java" rel="nofollow">JsonIntermediateToParquetGroupConverter.java</a></td>
<td><ul><li>DATE</li><li>TIMESTAMP</li><li>TIME</li><li>STRING</li><li>BYTES</li><li>INT</li><li>LONG</li><li>FLOAT</li><li>DOUBLE</li><li>BOOLEAN</li><li>ARRAY</li><li>MAP</li><li>ENUM</li></ul></td>
</tr>
</tbody>
@@ -992,7 +992,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/user-guide/State-Management-and-Watermarks/index.html b/docs/user-guide/State-Management-and-Watermarks/index.html
index 7cee9ae..97f227e 100644
--- a/docs/user-guide/State-Management-and-Watermarks/index.html
+++ b/docs/user-guide/State-Management-and-Watermarks/index.html
@@ -524,7 +524,7 @@
<li>State Management and Watermarks</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/State-Management-and-Watermarks.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/State-Management-and-Watermarks.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -560,11 +560,11 @@
<p>When a task finishes extracting data, it should write the actual high watermark into its <code>WorkUnitState</code>. To do so, the <code>Extractor</code> may maintain a <code>nextWatermark</code> field, and in <code>Extractor.close()</code>, call <code>this.workUnitState.setActualHighWatermark(this.nextWatermark)</code>. The actual high Watermark is normally the same as the expected high Watermark if the task completes successfully, and may be smaller than the expected high Watermark if the task failed or timed-out. In some cases, the expected high watermark may not be available so the actual high watermark is the only stat available that tells where the previous run left off. </p>
<p>In the next run, the <code>Source</code> will call <code>SourceState.getPreviousWorkUnitStates()</code> which should contain the actual high watermarks the last run checked in, to be used as the low watermarks of the new run.</p>
<p><strong>watermark type</strong></p>
-<p>A watermark can be of any custom type by implementing the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/extractor/Watermark.java"><code>Watermark</code></a> interface. For example, for Kafka-HDFS ingestion, if each <code>WorkUnit</code> is responsible for pulling a single Kafka topic partition, a watermark is a single <code>long</code> value representing a Kafka offset. If each <code>WorkUnit</code> is responsible for pulling multiple Kafka topic partitions, a watermark can be a list of <code>long</code> values, such as <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/MultiLongWatermark.java"><code>MultiLongWatermark</code></a>.</p>
+<p>A watermark can be of any custom type by implementing the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/extractor/Watermark.java" rel="nofollow"><code>Watermark</code></a> interface. For example, for Kafka-HDFS ingestion, if each <code>WorkUnit</code> is responsible for pulling a single Kafka topic partition, a watermark is a single <code>long</code> value representing a Kafka offset. If each <code>WorkUnit</code> is responsible for pulling multiple Kafka topic partitions, a watermark can be a list of <code>long</code> values, such as <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/MultiLongWatermark.java" rel="nofollow"><code>MultiLongWatermark</code></a>.</p>
<h3 id="task-failures">Task Failures</h3>
<p>A task may pull some data and then fail. If a task fails and job commit policy specified by configuration property <code>job.commit.policy</code> is set to <code>full</code>, the data it pulled won't be published. In this case, it doesn't matter what value <code>Extractor.nextWatermark</code> is, the actual high watermark will be automatically rolled back to the low watermark by Gobblin internally. On the other hand, if the commit policy is set to <code>partial</code>, the failed task may get committed and the data may get published. In this case the <code>Extractor</code> is responsible for setting the correct actual high watermark in <code>Extractor.close()</code>. Therefore, it is recommended that the <code>Extractor</code> update <code>nextWatermark</code> every time it pulls a record, so that <code>nextWatermark</code> is always up to date (unless you are OK with the next run re-doing the work which may cause some data to be published twice).</p>
<h3 id="multi-dataset-jobs">Multi-Dataset Jobs</h3>
-<p>Currently the only state store implementation Gobblin provides is <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metastore/src/main/java/org/apache/gobblin/metastore/FsStateStore.java"><code>FsStateStore</code></a> which uses Hadoop SequenceFiles to store the states. By default, each job run reads the SequenceFile created by the previous run, and generates a new SequenceFile. This creates a pitfall when a job pulls data from multiple datasets: if a data set is skipped in a job run for whatever reason (e.g., it is blacklisted), its watermark will be unavailable for the next run.</p>
+<p>Currently the only state store implementation Gobblin provides is <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-metastore/src/main/java/org/apache/gobblin/metastore/FsStateStore.java" rel="nofollow"><code>FsStateStore</code></a> which uses Hadoop SequenceFiles to store the states. By default, each job run reads the SequenceFile created by the previous run, and generates a new SequenceFile. This creates a pitfall when a job pulls data from multiple datasets: if a data set is skipped in a job run for whatever reason (e.g., it is blacklisted), its watermark will be unavailable for the next run.</p>
<p><strong>Example</strong>: suppose we schedule a Gobblin job to pull a Kafka topic from a Kafka broker, which has 10 partitions. In this case each partition is a dataset. In one of the job runs, a partition is skipped due to either being blacklisted or some failure. If no <code>WorkUnit</code> is created for this partition, this partition's watermark will not be checked in to the state store, and will not be available for the next run.</p>
<p>The are two solutions to the above problem (three if you count the one that implements a different state store that behaves differently and doesn't have this problem).</p>
<p><strong>Solution 1</strong>: make sure to create a <code>WorkUnit</code> for every dataset. Even if a dataset should be skipped, an empty <code>WorkUnit</code> should still be created for the dataset ('empty' means low watermark = expected high watermark).</p>
@@ -572,7 +572,7 @@
<p>If different <code>WorkUnit</code>s have different values of <code>dataset.urn</code>, the job will create one state store SequenceFile for each <code>dataset.urn</code>. In the next run, instead of calling <code>SourceState.getPreviousWorkUnitStates()</code>, one should use <code>SourceState.getPreviousWorkUnitStatesByDatasetUrns()</code>. In this way, each run will look for the most recent state store SequenceFile for each dataset, and therefore, even if a dataset is not processed by a job run, its watermark won't be lost.</p>
<p>Note that when using Dataset URNs, <strong>each <code>WorkUnit</code> can only have one <code>dataset.urn</code></strong>, which means, for example, in the Kafka ingestion case, each <code>WorkUnit</code> can only process one partition. This is usually not a big problem except that it may output too many small files (as explained in <a href="../case-studies/Kafka-HDFS-Ingestion">Kafka HDFS ingestion</a>, by having a <code>WorkUnit</code> pull multiple partitions of the same topic, these partitions can share output files). On the other hand, different <code>WorkUnit</code>s may have the same <code>dataset.urn</code>.</p>
<h2 id="gobblin-state-deep-dive">Gobblin State Deep Dive</h2>
-<p>Gobblin involves several types of states during a job run, such as <code>JobState</code>, <code>TaskState</code>, <code>WorkUnit</code>, etc. They all extend the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/configuration/State.java"><code>State</code></a> class, which is a wrapper around <a href="https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html"><code>Properties</code></a> and provides some useful utility functions. </p>
+<p>Gobblin involves several types of states during a job run, such as <code>JobState</code>, <code>TaskState</code>, <code>WorkUnit</code>, etc. They all extend the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/configuration/State.java" rel="nofollow"><code>State</code></a> class, which is a wrapper around <a href="https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html" rel="nofollow"><code>Properties</code></a> and provides some useful utility functions. </p>
<h3 id="state-class-hierarchy"><code>State</code> class hierarchy</h3>
<p align="left">
<figure>
@@ -608,7 +608,7 @@
<li>
<p>The job launcher prepares to run the <code>WorkUnit</code>s.</p>
</li>
-<li>In standalone mode, the job launcher will add properties in the <code>JobState</code> into each <code>WorkUnit</code> (if a property in <code>JobState</code> already exists in the <code>WorkUnit</code>, it will NOT be overwritten, i.e., the value in the <code>WorkUnit</code> takes precedence). Then for each <code>WorkUnit</code> it creates a <code>Task</code> to run the <code>WorkUnit</code>, and submits all these Tasks to a <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/TaskExecutor.java"><code>TaskExecutor</code></a> which will run these <code>Task</code>s in a thread pool.</li>
+<li>In standalone mode, the job launcher will add properties in the <code>JobState</code> into each <code>WorkUnit</code> (if a property in <code>JobState</code> already exists in the <code>WorkUnit</code>, it will NOT be overwritten, i.e., the value in the <code>WorkUnit</code> takes precedence). Then for each <code>WorkUnit</code> it creates a <code>Task</code> to run the <code>WorkUnit</code>, and submits all these Tasks to a <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/TaskExecutor.java" rel="nofollow"><code>TaskExecutor</code></a> which will run these <code>Task</code>s in a thread pool.</li>
<li>In MR mode, the job launcher will serialize the <code>JobState</code> and each <code>WorkUnit</code> into a file, which will be picked up by the mappers. It then creates, configures and submits a Hadoop job.</li>
</ul>
<p>After this step, the job launcher will be waiting till all tasks finish.</p>
@@ -642,7 +642,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/user-guide/Troubleshooting/index.html b/docs/user-guide/Troubleshooting/index.html
index d1fe2ed..db82828 100644
--- a/docs/user-guide/Troubleshooting/index.html
+++ b/docs/user-guide/Troubleshooting/index.html
@@ -503,7 +503,7 @@
<li>Troubleshooting</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Troubleshooting.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Troubleshooting.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -617,7 +617,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/user-guide/Working-with-Job-Configuration-Files/index.html b/docs/user-guide/Working-with-Job-Configuration-Files/index.html
index c72b8f0..2a02393 100644
--- a/docs/user-guide/Working-with-Job-Configuration-Files/index.html
+++ b/docs/user-guide/Working-with-Job-Configuration-Files/index.html
@@ -521,7 +521,7 @@
<li>Job Configuration Files</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Working-with-Job-Configuration-Files.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Working-with-Job-Configuration-Files.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -544,7 +544,7 @@
</ul>
</div>
<h2 id="job-configuration-basics">Job Configuration Basics</h2>
-<p>A Job configuration file is a text file with extension <code>.pull</code> or <code>.job</code> that defines the job properties that can be loaded into a Java <a href="http://docs.oracle.com/javase/7/docs/api/java/util/Properties.html">Properties</a> object. Gobblin uses <a href="http://commons.apache.org/proper/commons-configuration/">commons-configuration</a> to allow variable substitutions in job configuration files. You can find some example Gobblin job configuration files <a href="https://github.com/apache/incubator-gobblin/tree/master/gobblin-core/src/main/resources">here</a>. </p>
+<p>A Job configuration file is a text file with extension <code>.pull</code> or <code>.job</code> that defines the job properties that can be loaded into a Java <a href="http://docs.oracle.com/javase/7/docs/api/java/util/Properties.html">Properties</a> object. Gobblin uses <a href="http://commons.apache.org/proper/commons-configuration/">commons-configuration</a> to allow variable substitutions in job configuration files. You can find some example Gobblin job configuration files <a href="https://github.com/apache/incubator-gobblin/tree/master/gobblin-core/src/main/resources" rel="nofollow">here</a>. </p>
<p>A Job configuration file typically includes the following properties, in additional to any mandatory configuration properties required by the custom <a href="Gobblin-Architecture#gobblin-constructs">Gobblin Constructs</a> classes. For a complete reference of all configuration properties supported by Gobblin, please refer to <a href="Configuration-Properties-Glossary">Configuration Properties Glossary</a>.</p>
<ul>
<li><code>job.name</code>: job name.</li>
@@ -579,8 +579,8 @@
<p>In this example, <code>common.properties</code> will be included when loading <code>foo1.job</code>, <code>foo2.job</code>, <code>bar1.job</code>, <code>bar2.job</code>, <code>baz1.pull</code>, and <code>baz2.pull</code>. <code>foo.properties</code> will be included when loading <code>foo1.job</code> and <code>foo2.job</code> and properties set here are considered more special and will overwrite the same properties defined in <code>common.properties</code>. Similarly, <code>bar.properties</code> will be included when loading <code>bar1.job</code> and <code>bar2.job</code>, as well as <code>baz1.pull</code> and <code>baz2.pull</code>. <code>baz.properties</code> will be included when loading <code>baz1.pull</code> and <code>baz2.pull</code> and will overwrite the same properties defined in <code>bar.properties</code> and <code>common.properties</code>.</p>
<h2 id="password-encryption">Password Encryption</h2>
<p>To avoid storing passwords in configuration files in plain text, Gobblin supports encryption of the password configuration properties. All such properties can be encrypted (and decrypted) using a master password. The master password is stored in a file available at runtime. The file can be on a local file system or HDFS and has restricted access.</p>
-<p>The URI of the master password file is controlled by the configuration option <code>encrypt.key.loc</code> . By default, Gobblin will use <a href="http://www.jasypt.org/api/jasypt/1.8/org/jasypt/util/password/BasicPasswordEncryptor.html">org.jasypt.util.password.BasicPasswordEncryptor</a>. If you have installed the <a href="http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html">JCE Unlimited Strength Policy</a>, you can set
-<code>encrypt.use.strong.encryptor=true</code> which will configure Gobblin to use <a href="http://www.jasypt.org/api/jasypt/1.8/org/jasypt/util/password/StrongPasswordEncryptor.html">org.jasypt.util.password.StrongPasswordEncryptor</a>.</p>
+<p>The URI of the master password file is controlled by the configuration option <code>encrypt.key.loc</code> . By default, Gobblin will use <a href="http://www.jasypt.org/api/jasypt/1.8/org/jasypt/util/password/BasicPasswordEncryptor.html" rel="nofollow">org.jasypt.util.password.BasicPasswordEncryptor</a>. If you have installed the <a href="http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html" rel="nofollow">JCE Unlimited Strength Policy</a>, you can set
+<code>encrypt.use.strong.encryptor=true</code> which will configure Gobblin to use <a href="http://www.jasypt.org/api/jasypt/1.8/org/jasypt/util/password/StrongPasswordEncryptor.html" rel="nofollow">org.jasypt.util.password.StrongPasswordEncryptor</a>.</p>
<p>Encrypted passwords can be generated using the <code>CLIPasswordEncryptor</code> tool.</p>
<pre><code>$ gradle :gobblin-utility:assemble
$ cd build/gobblin-utility/distributions/
@@ -607,7 +607,7 @@
</ul>
<p>Note that this job configuration file change monitoring mechanism uses the <code>FileAlterationMonitor</code> of Apache's <a href="http://commons.apache.org/proper/commons-io/">commons-io</a> with a custom <code>FileAlterationListener</code>. Regardless of how close two adjacent file system checks are, there are still chances that more than one files are changed between two file system checks. In case more than one file including at least one common properties file are changed between two adjacent checks, the reloading of affected job configuration files may be intermixed and applied in an order that is not desirable. This is because the order the listener is called on the changes is not controlled by Gobblin, but instead by the monitor itself. So the best practice to use this feature is to avoid making multiple changes together in a short period of time. </p>
<h2 id="scheduled-jobs">Scheduled Jobs</h2>
-<p>Gobblin ships with a job scheduler backed by a <a href="http://quartz-scheduler.org/">Quartz</a> scheduler and supports Quartz's <a href="http://quartz-scheduler.org/generated/2.2.1/html/qs-all/#page/Quartz_Scheduler_Documentation_Set%2Fco-trg_crontriggers.html%23">cron triggers</a>. A job that is to be scheduled should have a cron schedule defined using the property <code>job.schedule</code>. Here is an example cron schedule that triggers every two minutes:</p>
+<p>Gobblin ships with a job scheduler backed by a <a href="http://quartz-scheduler.org/" rel="nofollow">Quartz</a> scheduler and supports Quartz's <a href="http://quartz-scheduler.org/generated/2.2.1/html/qs-all/#page/Quartz_Scheduler_Documentation_Set%2Fco-trg_crontriggers.html%23" rel="nofollow">cron triggers</a>. A job that is to be scheduled should have a cron schedule defined using the property <code>job.schedule</code>. Here is an example cron schedule that triggers every two minutes:</p>
<pre><code>job.schedule=0 0/2 * * * ?
</code></pre>
@@ -637,7 +637,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/docs/user-guide/Working-with-the-ForkOperator/index.html b/docs/user-guide/Working-with-the-ForkOperator/index.html
index 0fa5108..edcacec 100644
--- a/docs/user-guide/Working-with-the-ForkOperator/index.html
+++ b/docs/user-guide/Working-with-the-ForkOperator/index.html
@@ -531,7 +531,7 @@
<li>Fork Operator</li>
<li class="wy-breadcrumbs-aside">
- <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Working-with-the-ForkOperator.md"> Edit on Gobblin</a>
+ <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/Working-with-the-ForkOperator.md" rel="nofollow"> Edit on Gobblin</a>
</li>
</ul>
@@ -560,7 +560,7 @@
</ul>
</div>
<h2 id="overview-of-the-forkoperator">Overview of the ForkOperator</h2>
-<p>The <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/fork/ForkOperator.java"><code>ForkOperator</code></a> is a type of control operators that allow a task flow to branch into multiple streams (or forked branches) as represented by a <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/fork/Fork.java"><code>Fork</code></a>, each of which goes to a separately configured sink with its own data writer. The <code>ForkOperator</code> gives users more flexibility in terms of controlling where and how ingested data should be output. This is useful for situations, e.g., that data records need to be written into multiple different storages, or that data records need to be written out to the same storage (say, HDFS) but in different forms for different downstream consumers. The best practices of using the <code>ForkOperator</code> that we recommend, though, are discussed below. The diagram below illustrates how the <code>ForkOperator</code> in a Gobblin task flow allows an input stream to be forked into multiple output streams, each of which can have its own converters, quality checkers, and writers.</p>
+<p>The <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/fork/ForkOperator.java" rel="nofollow"><code>ForkOperator</code></a> is a type of control operators that allow a task flow to branch into multiple streams (or forked branches) as represented by a <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/fork/Fork.java" rel="nofollow"><code>Fork</code></a>, each of which goes to a separately configured sink with its own data writer. The <code>ForkOperator</code> gives users more flexibility in terms of controlling where and how ingested data should be output. This is useful for situations, e.g., that data records need to be written into multiple different storages, or that data records need to be written out to the same storage (say, HDFS) but in different forms for different downstream consumers. The best practices of using the <code>ForkOperator</code> that we recommend, though, are discussed below. The diagram below illustrates how the <code>ForkOperator</code> in a Gobblin task flow allows an input stream to be forked into multiple output streams, each of which can have its own converters, quality checkers, and writers.</p>
<p align="center">
<figure>
<img src=../../img/Gobblin-Task-Flow.png alt="Gobblin Task Flow" width="500">
@@ -570,15 +570,15 @@
<h2 id="using-the-forkoperator">Using the ForkOperator</h2>
<h3 id="basics-of-usage">Basics of Usage</h3>
-<p>The <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/fork/ForkOperator.java"><code>ForkOperator</code></a>, like most other operators in a Gobblin task flow, is pluggable through the configuration, or more specifically , the configuration property <code>fork.operator.class</code> that points to a class that implements the <code>ForkOperator</code> interface. For instance:</p>
+<p>The <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/fork/ForkOperator.java" rel="nofollow"><code>ForkOperator</code></a>, like most other operators in a Gobblin task flow, is pluggable through the configuration, or more specifically , the configuration property <code>fork.operator.class</code> that points to a class that implements the <code>ForkOperator</code> interface. For instance:</p>
<pre><code>fork.operator.class=org.apache.gobblin.fork.IdentityForkOperator
</code></pre>
-<p>By default, if no <code>ForkOperator</code> class is specified, internally Gobblin uses the default implementation <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/fork/IdentityForkOperator.java"><code>IdentityForkOperator</code></a> with a single forked branch (although it does supports multiple forked branches). The <code>IdentityForkOperator</code> simply unconditionally forwards the schema and ingested data records to all the forked branches, the number of which is specified through the configuration property <code>fork.branches</code> with a default value of 1. When an <code>IdentityForkOperator</code> instance is initialized, it will read the value of <code>fork.branches</code> and use that as the return value of <code>getBranches</code>. </p>
+<p>By default, if no <code>ForkOperator</code> class is specified, internally Gobblin uses the default implementation <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/fork/IdentityForkOperator.java" rel="nofollow"><code>IdentityForkOperator</code></a> with a single forked branch (although it does supports multiple forked branches). The <code>IdentityForkOperator</code> simply unconditionally forwards the schema and ingested data records to all the forked branches, the number of which is specified through the configuration property <code>fork.branches</code> with a default value of 1. When an <code>IdentityForkOperator</code> instance is initialized, it will read the value of <code>fork.branches</code> and use that as the return value of <code>getBranches</code>. </p>
<p>The <em>expected</em> number of forked branches is given by the method <code>getBranches</code> of the <code>ForkOperator</code>. This number must match the size of the list of <code>Boolean</code>s returned by <code>forkSchema</code> as well as the size of the list of <code>Boolean</code>s returned by <code>forkDataRecords</code>. Otherwise, <code>ForkBranchMismatchException</code> will be thrown. Note that the <code>ForkOperator</code> itself <em>is not making and returning a copy</em> for the input schema and data records, but rather just providing a <code>Boolean</code> for each forked branch telling if the schema or data records should be in each particular branch. Each forked branch has a branch index starting at 0. So if there are three forked branches, the branches will have indices 0, 1, and 2, respectively. Branch indices are useful to tell which branch the Gobblin task flow is in. Each branch also has a name associated with it that can be specified using the configuration property <code>fork.branch.name.<branch index></code>. Note that the branch index is added as a suffix to the property name in this case. More on this later. If the user does not specify a name for the branches, the names in the form <code>fork_<branch index></code> will be used. </p>
<p>The use of the <code>ForkOperator</code> with <em>the possibility that the schema and/or data records may be forwarded to more than one forked branches</em> has some special requirement on the input schema and data records to the <code>ForkOperator</code>. Specifically, because the same schema or data records may be forwarded to more than branches that may alter the schema or data records in place, it is necessary for the Gobblin task flow to make a copy of the input schema or data records for each forked branch so any modification within one branch won't affect any other branches. </p>
-<p>To guarantee that it is always able to make a copy in such a case, Gobblin requires the input schema and data records to be of type <code>Copyable</code> when there are more than one forked branch. <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/fork/Copyable.java"><code>Copyable</code></a> is an interface that defines a method <code>copy</code> for making a copy of an instance of a given type. The Gobblin task flow will check if the input schema and data records are instances of <code>Copyable</code> and throw a <code>CopyNotSupportedException</code> if not. This check is performed independently on the schema first and data records subsequently. Note that this requirement is enforced <em>if and only if the schema or data records are to be forwarded to more than one branches</em>, which is the case if <code>forkSchema</code> or <code>forkDataRecord</code> returns a list containing more than one <code>TRUE</code>. Having more than one branch does not necessarily mean the schema and/or data records need to be <code>Copyable</code>.</p>
-<p>Gobblin ships with some built-in <code>Copyable</code> implementations, e.g., <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/fork/CopyableSchema.java"><code>CopyableSchema</code></a> and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/fork/CopyableGenericRecord.java"><code>CopyableGenericRecord</code></a> for Avro's <code>Schema</code> and <code>GenericRecord</code>. </p>
+<p>To guarantee that it is always able to make a copy in such a case, Gobblin requires the input schema and data records to be of type <code>Copyable</code> when there are more than one forked branch. <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/fork/Copyable.java" rel="nofollow"><code>Copyable</code></a> is an interface that defines a method <code>copy</code> for making a copy of an instance of a given type. The Gobblin task flow will check if the input schema and data records are instances of <code>Copyable</code> and throw a <code>CopyNotSupportedException</code> if not. This check is performed independently on the schema first and data records subsequently. Note that this requirement is enforced <em>if and only if the schema or data records are to be forwarded to more than one branches</em>, which is the case if <code>forkSchema</code> or <code>forkDataRecord</code> returns a list containing more than one <code>TRUE</code>. Having more than one branch does not necessarily mean the schema and/or data records need to be <code>Copyable</code>.</p>
+<p>Gobblin ships with some built-in <code>Copyable</code> implementations, e.g., <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/fork/CopyableSchema.java" rel="nofollow"><code>CopyableSchema</code></a> and <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/fork/CopyableGenericRecord.java" rel="nofollow"><code>CopyableGenericRecord</code></a> for Avro's <code>Schema</code> and <code>GenericRecord</code>. </p>
<h3 id="per-fork-configuration">Per-Fork Configuration</h3>
<p>Since each forked branch may have it's own converters, quality checkers, and writers, in addition to the ones in the pre-fork stream (which does not have a writer apparently), there must be a way to tell the converter, quality checker, and writer classes of one branch from another and from the pre-fork stream. Gobblin uses a pretty straightforward approach: if a configuration property is used to specify something for a branch in a multi-branch use case, <em>the branch index should be appended as a suffix</em> to the property name. The original configuration name without the suffix is <em>generally reserved for the pre-fork stream</em>. For example, <code>converter.classes.0</code> and <code>converter.classes.1</code> are used to specify the list of converter classes for branch 0 and 1, respectively, whereas <code>converter.classes</code> is reserved for the pre-fork stream. If there's only a single branch (the default case), then the index suffix is not applicable. Without being a comprehensive list, the following groups of built-in configuration properties may be used with branch indices as suffices to specify things for forked branches:</p>
<ul>
@@ -592,12 +592,12 @@
<li>The failure of any forked branch means the failure of the whole task flow, i.e., the task succeeds if and only if all the forked branches succeed.</li>
<li>A forked branch stops processing any outstanding incoming data records in the queue if it fails in the middle of processing the data. </li>
<li>The failure and subsequent stop/completion of any forked branch does not prevent other branches from processing their copies of the ingested data records. The task will wait until all the branches to finish, regardless if they succeed or fail. </li>
-<li>The commit of output data of forks is determined by the job commit policy (see <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/source/extractor/JobCommitPolicy.java"><code>JobCommitPolicy</code></a>) specified. If <code>JobCommitPolicy.COMMIT_ON_FULL_SUCCESS</code> (or <code>full</code> in short) is used, the output data of the entire job will be discarded if any forked branch fails, which will fail the task and consequently the job. If instead <code>JobCommitPolicy.COMMIT_SUCCESSFUL_TASKS</code> (or <code>successful</code> in short) is used, output data of tasks whose forked branches all succeed will be committed. Output data of any task that has <em>at least one failed forked branch</em> will not be committed since the the task is considered failed in this case. This also means output data of the successful forked branches of the task won't be committed either.</li>
+<li>The commit of output data of forks is determined by the job commit policy (see <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/source/extractor/JobCommitPolicy.java" rel="nofollow"><code>JobCommitPolicy</code></a>) specified. If <code>JobCommitPolicy.COMMIT_ON_FULL_SUCCESS</code> (or <code>full</code> in short) is used, the output data of the entire job will be discarded if any forked branch fails, which will fail the task and consequently the job. If instead <code>JobCommitPolicy.COMMIT_SUCCESSFUL_TASKS</code> (or <code>successful</code> in short) is used, output data of tasks whose forked branches all succeed will be committed. Output data of any task that has <em>at least one failed forked branch</em> will not be committed since the the task is considered failed in this case. This also means output data of the successful forked branches of the task won't be committed either.</li>
</ul>
<h3 id="performance-tuning">Performance Tuning</h3>
-<p>Internally, each forked branch as represented by a <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/fork/Fork.java"><code>Fork</code></a> maintains a bounded record queue (implemented by <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/BoundedBlockingRecordQueue.java"><code>BoundedBlockingRecordQueue</code></a>), which serves as a buffer between the pre-fork stream and the forked stream of the particular branch. The size if this bounded record queue can be configured through the property <code>fork.record.queue.capacity</code>. A larger queue allows for more data records to be buffered therefore giving the producer (the pre-fork stream) more head room to move forward. On the other hand, a larger queue requires more memory. The bounded record queue imposes a timeout time on all blocking operations such as putting a new record to the tail and polling a record off the head of the queue. Tuning the queue size and timeout time together offers a lot of flexibility and a tradeoff between queuing performance vs. memory consumption.</p>
+<p>Internally, each forked branch as represented by a <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/fork/Fork.java" rel="nofollow"><code>Fork</code></a> maintains a bounded record queue (implemented by <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/BoundedBlockingRecordQueue.java" rel="nofollow"><code>BoundedBlockingRecordQueue</code></a>), which serves as a buffer between the pre-fork stream and the forked stream of the particular branch. The size if this bounded record queue can be configured through the property <code>fork.record.queue.capacity</code>. A larger queue allows for more data records to be buffered therefore giving the producer (the pre-fork stream) more head room to move forward. On the other hand, a larger queue requires more memory. The bounded record queue imposes a timeout time on all blocking operations such as putting a new record to the tail and polling a record off the head of the queue. Tuning the queue size and timeout time together offers a lot of flexibility and a tradeoff between queuing performance vs. memory consumption.</p>
<p>In terms of the number of forked branches, we have seen use cases with a half dozen forked branches, and we are anticipating uses cases with much larger numbers. Again, when using a large number of forked branches, the size of the record queues and the timeout time need to be carefully tuned. </p>
-<p>The <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/BoundedBlockingRecordQueue.java"><code>BoundedBlockingRecordQueue</code></a> in each <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/fork/Fork.java"><code>Fork</code></a> keeps trach of the following queue statistics that can be output to the logs if the <code>DEBUG</code> logging level is turned on. Those statistics provide good indications on the performance of the forks.</p>
+<p>The <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/BoundedBlockingRecordQueue.java" rel="nofollow"><code>BoundedBlockingRecordQueue</code></a> in each <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/fork/Fork.java" rel="nofollow"><code>Fork</code></a> keeps trach of the following queue statistics that can be output to the logs if the <code>DEBUG</code> logging level is turned on. Those statistics provide good indications on the performance of the forks.</p>
<ul>
<li>Queue size, i.e., the number of records in queue.</li>
<li>Queue fill ratio, i.e., a ratio of the number of records in queue over the queue capacity.</li>
@@ -607,7 +607,7 @@
<li>Total get attempt count. </li>
</ul>
<h3 id="comparison-with-partitioneddatawriter">Comparison with PartitionedDataWriter</h3>
-<p>Gobblin ships with a special type of <code>DataWriter</code>s called <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/writer/PartitionedDataWriter.java"><code>PartitionedDataWriter</code></a> that allow ingested records to be written in a partitioned fashion using a <code>WriterPartitioner</code> into different locations in the same sink. The <code>WriterPartitioner</code> determines the specific partition for each data record. So there's certain overlap in terms of functionality between the <code>ForkOperator</code> and <code>PartitionedDataWriter</code>. The question is which one should be used under which circumstances? Below is a summary of the major differences between the two operators.</p>
+<p>Gobblin ships with a special type of <code>DataWriter</code>s called <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/writer/PartitionedDataWriter.java" rel="nofollow"><code>PartitionedDataWriter</code></a> that allow ingested records to be written in a partitioned fashion using a <code>WriterPartitioner</code> into different locations in the same sink. The <code>WriterPartitioner</code> determines the specific partition for each data record. So there's certain overlap in terms of functionality between the <code>ForkOperator</code> and <code>PartitionedDataWriter</code>. The question is which one should be used under which circumstances? Below is a summary of the major differences between the two operators.</p>
<ul>
<li>The <code>ForkOperator</code> requires the number of forked branches to be known and returned through <code>getBranches</code> before the task starts, whereas the <code>PartitionedDataWriter</code> does not have this requirement.</li>
<li>The <code>PartitionedDataWriter</code> writes each data record to a single partition, whereas the <code>ForkOperator</code> allows data records to be forwarded to any number of forked branches.</li>
@@ -616,7 +616,7 @@
<li>The <code>PartitionedDataWriter</code> writes data records sequentially in a single thread, whereas use of the <code>ForkOperator</code> allows forked branches to write independently in parallel since <code>Fork</code>s are executed in a thread pool. </li>
</ul>
<h2 id="writing-your-own-forkoperator">Writing your Own ForkOperator</h2>
-<p>Since the built-in default implementation <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/fork/IdentityForkOperator.java"><code>IdentityForkOperator</code></a> simply blindly forks the input schema and data records to every branches, it's often necessary to have a custom implementation of the <code>ForkOperator</code> interface for more fine-grained control on the actual branching. Checkout the interface <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/fork/ForkOperator.java"><code>ForkOperator</code></a> for the methods that need to be implemented. You will also find the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-utility/src/main/java/org/apache/gobblin/util/ForkOperatorUtils.java"><code>ForkOperatorUtils</code></a> to be handy when writing your own <code>ForkOperator</code> implementations.</p>
+<p>Since the built-in default implementation <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/fork/IdentityForkOperator.java" rel="nofollow"><code>IdentityForkOperator</code></a> simply blindly forks the input schema and data records to every branches, it's often necessary to have a custom implementation of the <code>ForkOperator</code> interface for more fine-grained control on the actual branching. Checkout the interface <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/fork/ForkOperator.java" rel="nofollow"><code>ForkOperator</code></a> for the methods that need to be implemented. You will also find the <a href="https://github.com/apache/incubator-gobblin/blob/master/gobblin-utility/src/main/java/org/apache/gobblin/util/ForkOperatorUtils.java" rel="nofollow"><code>ForkOperatorUtils</code></a> to be handy when writing your own <code>ForkOperator</code> implementations.</p>
<h2 id="best-practices">Best Practices</h2>
<p>The <code>ForkOperator</code> can have many potential use cases and we have seen the following common ones:</p>
<ul>
@@ -736,7 +736,7 @@
</div>
- Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+ Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>.
</footer>
</div>
diff --git a/download/index.html b/download/index.html
index f5370a7..84d5483 100644
--- a/download/index.html
+++ b/download/index.html
@@ -75,7 +75,7 @@
[ <a href="https://www.apache.org/dist/incubator/gobblin/apache-gobblin-incubating-0.15.0/apache-gobblin-incubating-sources-0.15.0.tgz.asc" target="_blank">ASC</a> ]
</p>
<p>
- <a href="https://github.com/apache/incubator-gobblin/blob/release-0.15.0/CHANGELOG.md" target="_blank">CHANGELOG</a>
+ <a href="https://github.com/apache/incubator-gobblin/blob/release-0.15.0/CHANGELOG.md" target="_blank" rel="nofollow">CHANGELOG</a>
</p>
<!-- 0.15.0 Release end -->
@@ -88,7 +88,7 @@
[ <a href="https://www.apache.org/dist/incubator/gobblin/apache-gobblin-incubating-0.14.0/apache-gobblin-incubating-sources-0.14.0.tgz.asc" target="_blank">ASC</a> ]
</p>
<p>
- <a href="https://github.com/apache/incubator-gobblin/blob/release-0.14.0/CHANGELOG.md" target="_blank">CHANGELOG</a>
+ <a href="https://github.com/apache/incubator-gobblin/blob/release-0.14.0/CHANGELOG.md" target="_blank" rel="nofollow">CHANGELOG</a>
</p>
<!-- 0.14.0 Release end -->
@@ -101,7 +101,7 @@
[ <a href="https://www.apache.org/dist/incubator/gobblin/apache-gobblin-incubating-0.13.0/apache-gobblin-incubating-sources-0.13.0.tgz.asc" target="_blank">ASC</a> ]
</p>
<p>
- <a href="https://github.com/apache/incubator-gobblin/blob/release-0.13.0/CHANGELOG.md" target="_blank">CHANGELOG</a>
+ <a href="https://github.com/apache/incubator-gobblin/blob/release-0.13.0/CHANGELOG.md" target="_blank" rel="nofollow">CHANGELOG</a>
</p>
<!-- 0.13.0 Release end -->
@@ -121,7 +121,7 @@
</p>
-->
<p>
- <a href="https://github.com/apache/incubator-gobblin/blob/release-0.12.0/CHANGELOG.md" target="_blank">CHANGELOG</a>
+ <a href="https://github.com/apache/incubator-gobblin/blob/release-0.12.0/CHANGELOG.md" target="_blank" rel="nofollow">CHANGELOG</a>
</p>
<!-- 0.12.0 Release end -->
@@ -131,15 +131,15 @@
<h3 id="latest-label-label-primary-">0.11.0 (Pre-Apache)</h3>
<p>The last release prior to Apache incubation is version 0.11.0. <br>It is not supported or maintained anymore and can be found on Github</p>
<p>
- Official <a href="https://github.com/apache/incubator-gobblin/archive/gobblin_0.11.0.zip" target="_blank">source release</a>
+ Official <a href="https://github.com/apache/incubator-gobblin/archive/gobblin_0.11.0.zip" target="_blank" rel="nofollow">source release</a>
[ <a href="#">SHA512</a> ]
</p>
<p>
- <a href="https://github.com/apache/incubator-gobblin/archive/gobblin_0.11.0.tar.gz" target="_blank">Binary distribution</a>
+ <a href="https://github.com/apache/incubator-gobblin/archive/gobblin_0.11.0.tar.gz" target="_blank" rel="nofollow">Binary distribution</a>
[ <a href="#">SHA512</a> ]
</p>
<p>
- <a href="https://github.com/apache/incubator-gobblin/blob/gobblin_0.11.0/CHANGELOG.md" target="_blank">CHANGELOG</a>
+ <a href="https://github.com/apache/incubator-gobblin/blob/gobblin_0.11.0/CHANGELOG.md" target="_blank" rel="nofollow">CHANGELOG</a>
</p>
-->
<!-- 0.11.0 Release end -->
@@ -150,7 +150,7 @@
<!--
<h2 id="older-releases">Older Pre-Apache releases<a class="header-link" href="#older-releases"><i class="fa fa-link"></i></a></h2>
- <p>Older releases can be found in the <a href="https://github.com/apache/incubator-gobblin/releases" target="_blank">archives</a>.</p>
+ <p>Older releases can be found in the <a href="https://github.com/apache/incubator-gobblin/releases" target="_blank" rel="nofollow">archives</a>.</p>
-->
</div>
</div>
diff --git a/get-involved/index.html b/get-involved/index.html
index f32fd18..92e63b3 100644
--- a/get-involved/index.html
+++ b/get-involved/index.html
@@ -107,7 +107,7 @@
<h2 id="community-interaction">Community Interaction<a class="header-link" href="#community-interaction"><i class="fa fa-link"></i></a></h2>
- <p>Learn about <a href="http://www.betaversion.org/~stefano/papers/ac2006.2.pdf">building open source communities</a>.</p>
+ <p>Learn about <a href="http://www.betaversion.org/~stefano/papers/ac2006.2.pdf" rel="nofollow">building open source communities</a>.</p>
<h2 id="decision-making">Decision Making<a class="header-link" href="#decision-making"><i class="fa fa-link"></i></a></h2>
diff --git a/index.html b/index.html
index ed9a163..0f1408e 100644
--- a/index.html
+++ b/index.html
@@ -83,7 +83,7 @@
<p class="lead">A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both <b>streaming</b> and <b>batch</b> data ecosystems.</p>
<p class="lead">
<a class="btn btn-primary btn-animated btn-outline" href="/download">Download</a>
- <!-- <a class="btn btn-primary btn-animated btn-outline" href="https://www.gobblin.io" target="_blank">Try Gobblin</a> -->
+ <!-- <a class="btn btn-primary btn-animated btn-outline" href="https://www.gobblin.io" target="_blank" rel="nofollow">Try Gobblin</a> -->
</p>
</div>
<!--===============================================
@@ -192,21 +192,21 @@
<div class="col-sm-12">
<h2 class="sectiontile" style="text-align: center">Powered By Gobblin</h2>
<div class="row">
- <a href="http://www.linkedin.com/" style="margin-left: 30px;" target="_blank"><img style="width: 120px" src="assets/images/linkedin.png"></a>
- <a href="http://www.intel.com/" style="margin-left: 30px;" target="_blank"><img style="width: 85px" src="assets/images/intel.png"></a>
- <a href="http://www.paypal.com/" style="margin-left: 30px;" target="_blank"><img style="width: 95px" src="assets/images/paypal.png" ></a>
- <a href="http://www.microsoft.com/" style="margin-left: 30px;" target="_blank"><img style="width: 155px" src="assets/images/microsoft.png" class=""></a>
- <a href="http://www.ibm.com/" style="margin-left: 30px;" target="_blank"><img style="width: 85px" src="assets/images/ibm.png" ></a>
- <a href="http://www.home.cern/" style="margin-left: 30px;" target="_blank"><img style="width: 90px" src="assets/images/cern.png" ></a>
- <a href="http://www.apple.com/" style="margin-left: 30px;" target="_blank"><img style="width: 60px" src="assets/images/apple.png" ></a>
- <a href="http://www.swisscom.com/" style="margin-left: 30px;" target="_blank"><img style="width: 170px" src="assets/images/swisscom.png" ></a>
- <a href="http://www.sandia.gov/" style="margin-left: 30px;" target="_blank"><img style="width: 125px" src="assets/images/sandia.png" ></a>
- <a href="http://www.nerdwallet.com/" style="margin-left: 30px;" target="_blank"><img style="width: 180px" src="assets/images/nerdwallet.png" ></a>
- <a href="http://www.prezi.com/" style="margin-left: 30px;" target="_blank"><img style="width: 125px" src="assets/images/prezi.png" class=""></a>
- <a href="http://www.cleverdata.ru/" style="margin-left: 30px;" target="_blank"><img style="width: 145px" src="assets/images/cleverdata.png" ></a>
- <a href="http://www.applift.com/" style="margin-left: 30px;" target="_blank"><img style="width: 95px" src="assets/images/applift.png" ></a>
- <a href="http://www.bpuholdings.com/" style="margin-left: 30px;" target="_blank"><img style="width: 95px" src="assets/images/bpu.png" ></a>
- <a href="http://www.stunlockstudios.com/" style="margin-left: 30px;" target="_blank"><img style="width: 85px" src="assets/images/stunlock-studios.png" ></a>
+ <a href="http://www.linkedin.com/" rel="nofollow" style="margin-left: 30px;" target="_blank"><img style="width: 120px" src="assets/images/linkedin.png"></a>
+ <a href="http://www.intel.com/" style="margin-left: 30px;" target="_blank" rel="nofollow"><img style="width: 85px" src="assets/images/intel.png"></a>
+ <a href="http://www.paypal.com/" style="margin-left: 30px;" target="_blank" rel="nofollow"><img style="width: 95px" src="assets/images/paypal.png" ></a>
+ <a href="http://www.microsoft.com/" style="margin-left: 30px;" target="_blank" rel="nofollow"><img style="width: 155px" src="assets/images/microsoft.png" class=""></a>
+ <a href="http://www.ibm.com/" style="margin-left: 30px;" target="_blank" rel="nofollow"><img style="width: 85px" src="assets/images/ibm.png" ></a>
+ <a href="http://www.home.cern/" style="margin-left: 30px;" target="_blank" rel="nofollow"><img style="width: 90px" src="assets/images/cern.png" ></a>
+ <a href="http://www.apple.com/" style="margin-left: 30px;" target="_blank" rel="nofollow"><img style="width: 60px" src="assets/images/apple.png" ></a>
+ <a href="http://www.swisscom.com/" style="margin-left: 30px;" target="_blank" rel="nofollow"><img style="width: 170px" src="assets/images/swisscom.png" ></a>
+ <a href="http://www.sandia.gov/" style="margin-left: 30px;" target="_blank" rel="nofollow"><img style="width: 125px" src="assets/images/sandia.png" ></a>
+ <a href="http://www.nerdwallet.com/" style="margin-left: 30px;" target="_blank" rel="nofollow"><img style="width: 180px" src="assets/images/nerdwallet.png" ></a>
+ <a href="http://www.prezi.com/" style="margin-left: 30px;" target="_blank" rel="nofollow"><img style="width: 125px" src="assets/images/prezi.png" class=""></a>
+ <a href="http://www.cleverdata.ru/" style="margin-left: 30px;" target="_blank" rel="nofollow"><img style="width: 145px" src="assets/images/cleverdata.png" ></a>
+ <a href="http://www.applift.com/" style="margin-left: 30px;" target="_blank" rel="nofollow"><img style="width: 95px" src="assets/images/applift.png" ></a>
+ <a href="http://www.bpuholdings.com/" style="margin-left: 30px;" target="_blank" rel="nofollow"><img style="width: 95px" src="assets/images/bpu.png" ></a>
+ <a href="http://www.stunlockstudios.com/" style="margin-left: 30px;" target="_blank" rel="nofollow"><img style="width: 85px" src="assets/images/stunlock-studios.png" ></a>
</div>
</div>
</div>
diff --git a/navigation.html b/navigation.html
index cc2203b..b2e7163 100644
--- a/navigation.html
+++ b/navigation.html
@@ -36,8 +36,8 @@
<!--
<li class="divider"></li>
<li class="dropdown-header">Pre Apache</li>
- <li><a href="https://github.com/apache/incubator-gobblin/releases/tag/gobblin_0.11.0" target="_blank">0.11.0 <i class="fa fa-external-link"></i></a></li>
- <li><a href="https://github.com/apache/incubator-gobblin/releases" target="_blank">Archive <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://github.com/apache/incubator-gobblin/releases/tag/gobblin_0.11.0" target="_blank" rel="nofollow">0.11.0 <i class="fa fa-external-link"></i></a></li>
+ <li><a href="https://github.com/apache/incubator-gobblin/releases" target="_blank" rel="nofollow">Archive <i class="fa fa-external-link"></i></a></li>
-->
</ul>
</li>
@@ -64,7 +64,7 @@
<li class="divider"></li>
<li class="dropdown-header">Engagement</li>
<li><a href="/mailing-lists">Mailing Lists</a></li>
- <!-- <li><a href="https://gitter.im/gobblin/Lobby" target="_blank">Gitter <i class="fa fa-external-link"></i></a></li> -->
+ <!-- <li><a href="https://gitter.im/gobblin/Lobby" target="_blank" rel="nofollow">Gitter <i class="fa fa-external-link"></i></a></li> -->
<li><a href="https://cwiki.apache.org/confluence/display/GOBBLIN/Meetups" target="_blank">Meetups <i class="fa fa-external-link"></i></a></li>
<li class="divider"></li>
<li class="dropdown-header">Issue / Feature Tracking</li>
diff --git a/people/index.html b/people/index.html
index d81f05b..5d8933a 100644
--- a/people/index.html
+++ b/people/index.html
@@ -67,8 +67,8 @@
<td>
Abhishek Tiwari<br>
Committer, and PPMC member<br>
- <a href="http://www.linkedin.com/in/findabti" target="_blank">/in/findabti</a><br>
- <a href="http://twitter.com/findabti">@findabti</a><br>
+ <a href="http://www.linkedin.com/in/findabti" target="_blank" rel="nofollow">/in/findabti</a><br>
+ <a href="http://twitter.com/findabti" rel="nofollow">@findabti</a><br>
<b>LinkedIn</b>
</td>
<td>
@@ -77,7 +77,7 @@
<td>
Hung Tran<br>
Committer, and PPMC member<br>
- <a href="https://www.linkedin.com/in/hung-tran-30a8101/" target="_blank">/in/hung-tran</a><br>
+ <a href="https://www.linkedin.com/in/hung-tran-30a8101/" target="_blank" rel="nofollow">/in/hung-tran</a><br>
<b>LinkedIn</b>
</td>
</tr>
@@ -88,7 +88,7 @@
<td>
Issac Buenrostro<br>
Committer, and PPMC member<br>
- <a href="https://www.linkedin.com/in/ibuenros/" target="_blank">/in/ibuenros</a><br>
+ <a href="https://www.linkedin.com/in/ibuenros/" target="_blank" rel="nofollow">/in/ibuenros</a><br>
<b>LinkedIn</b>
</td>
<td>
@@ -97,7 +97,7 @@
<td>
Jean-Baptiste Onofré<br>
Mentor<br>
- <a href="https://www.linkedin.com/in/jean-baptiste-onofr%C3%A9-a0739317/" target="_blank">/in/jean-baptiste</a><br>
+ <a href="https://www.linkedin.com/in/jean-baptiste-onofr%C3%A9-a0739317/" target="_blank" rel="nofollow">/in/jean-baptiste</a><br>
<b>Talend</b>
</td>
</tr>
@@ -108,7 +108,7 @@
<td>
Joel Baranick<br>
Committer<br>
- <a href="https://www.linkedin.com/in/joelbaranick/" target="_blank">/in/joelbaranick</a><br>
+ <a href="https://www.linkedin.com/in/joelbaranick/" target="_blank" rel="nofollow">/in/joelbaranick</a><br>
<b>Ensighten</b>
</td>
<td>
@@ -117,7 +117,7 @@
<td>
Kishore Gopalakrishna<br>
Committer, and PPMC member<br>
- <a href="https://www.linkedin.com/in/kgopalak/" target="_blank">/in/kgopalak</a><br>
+ <a href="https://www.linkedin.com/in/kgopalak/" target="_blank" rel="nofollow">/in/kgopalak</a><br>
<b>Founder, Stealth Startup</b>
</td>
</tr>
@@ -128,7 +128,7 @@
<td>
Kuai Yu<br>
Committer<br>
- <a href="https://www.linkedin.com/in/yukuai518/" target="_blank">/in/yukuai518</a><br>
+ <a href="https://www.linkedin.com/in/yukuai518/" target="_blank" rel="nofollow">/in/yukuai518</a><br>
<b>Google</b>
</td>
<td>
@@ -137,7 +137,7 @@
<td>
Lei Sun<br>
Committer<br>
- <a href="https://www.linkedin.com/in/lei-s-a93138a0/" target="_blank">/in/lei-s-a93138a0</a><br> <b>LinkedIn</b>
+ <a href="https://www.linkedin.com/in/lei-s-a93138a0/" target="_blank" rel="nofollow">/in/lei-s-a93138a0</a><br> <b>LinkedIn</b>
</td>
</tr>
<tr>
@@ -147,7 +147,7 @@
<td>
Lorand Bendig<br>
Committer, and PPMC member<br>
- <a href="https://www.linkedin.com/in/lbendig/" target="_blank">/in/lbendig</a><br>
+ <a href="https://www.linkedin.com/in/lbendig/" target="_blank" rel="nofollow">/in/lbendig</a><br>
<b>Swisscom</b>
</td>
<td>
@@ -156,7 +156,7 @@
<td>
Olivier Lamy<br>
Champion<br>
- <a href="https://www.linkedin.com/in/olamy/" target="_blank">/in/olamy</a><br>
+ <a href="https://www.linkedin.com/in/olamy/" target="_blank" rel="nofollow">/in/olamy</a><br>
</td>
</tr>
<tr>
@@ -166,7 +166,7 @@
<td>
Owen O'Malley<br>
Mentor<br>
- <a href="https://www.linkedin.com/in/owenomalley/" target="_blank">/in/owenomalley</a><br>
+ <a href="https://www.linkedin.com/in/owenomalley/" target="_blank" rel="nofollow">/in/owenomalley</a><br>
<b>LinkedIn</b>
</td>
<td>
@@ -175,7 +175,7 @@
<td>
Sahil Takiar<br>
Committer, and PPMC member<br>
- <a href="https://www.linkedin.com/in/sahil-takiar-94186216/" target="_blank">/in/sahil-takiar</a><br>
+ <a href="https://www.linkedin.com/in/sahil-takiar-94186216/" target="_blank" rel="nofollow">/in/sahil-takiar</a><br>
<b>Cloudera</b>
</td>
@@ -187,7 +187,7 @@
<td>
Shirshanka Das<br>
Committer, and PPMC member<br>
- <a href="https://www.linkedin.com/in/shirshankadas/" target="_blank">/in/shirshankadas</a><br>
+ <a href="https://www.linkedin.com/in/shirshankadas/" target="_blank" rel="nofollow">/in/shirshankadas</a><br>
<b>LinkedIn</b>
</td>
<td>
@@ -196,7 +196,7 @@
<td>
Sudarshan Vasudevan<br>
Committer, and PPMC member<br>
- <a href="https://www.linkedin.com/in/suddu/" target="_blank">/in/suddu</a><br>
+ <a href="https://www.linkedin.com/in/suddu/" target="_blank" rel="nofollow">/in/suddu</a><br>
<b>LinkedIn</b>
</td>
</tr>
@@ -207,7 +207,7 @@
<td>
Tamas Nemeth<br>
Committer, and PPMC member<br>
- <a href="https://www.linkedin.com/in/nemetht/" target="_blank">/in/nemetht</a><br>
+ <a href="https://www.linkedin.com/in/nemetht/" target="_blank" rel="nofollow">/in/nemetht</a><br>
<b>Prezi</b>
</td>
<td>
@@ -216,7 +216,7 @@
<td>
Yinan Li<br>
Committer, and PPMC member<br>
- <a href="https://www.linkedin.com/in/yinan-li-91a3b214/" target="_blank">/in/yinan-li</a><br>
+ <a href="https://www.linkedin.com/in/yinan-li-91a3b214/" target="_blank" rel="nofollow">/in/yinan-li</a><br>
<b>Google</b>
</td>
</tr>
