| <!DOCTYPE html> |
| <!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> |
| <!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| |
| <meta name="author" content="Apache Software Foundation"> |
| <link rel="shortcut icon" href="../../img/favicon.ico"> |
| <title>FAQs - Apache Gobblin</title> |
| <link href='https://fonts.googleapis.com/css?family=Lato:400,700|Roboto+Slab:400,700|Inconsolata:400,700' rel='stylesheet' type='text/css'> |
| |
| <link rel="stylesheet" href="../../css/theme.css" type="text/css" /> |
| <link rel="stylesheet" href="../../css/theme_extra.css" type="text/css" /> |
| <link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/github.min.css"> |
| <link href="../../css/extra.css" rel="stylesheet"> |
| |
| <script> |
| // Current page data |
| var mkdocs_page_name = "FAQs"; |
| var mkdocs_page_input_path = "user-guide/FAQs.md"; |
| var mkdocs_page_url = null; |
| </script> |
| |
| <script src="../../js/jquery-2.1.1.min.js" defer></script> |
| <script src="../../js/modernizr-2.8.3.min.js" defer></script> |
| <script src="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/highlight.min.js"></script> |
| <script>hljs.initHighlightingOnLoad();</script> |
| |
| |
| </head> |
| |
| <body class="wy-body-for-nav" role="document"> |
| |
| <div class="wy-grid-for-nav"> |
| |
| |
| <nav data-toggle="wy-nav-shift" class="wy-nav-side stickynav"> |
| <div class="wy-side-nav-search"> |
| <a href="../.." class="icon icon-home"> Apache Gobblin</a> |
| <div role="search"> |
| <form id ="rtd-search-form" class="wy-form" action="../../search.html" method="get"> |
| <input type="text" name="q" placeholder="Search docs" title="Type search term here" /> |
| </form> |
| </div> |
| </div> |
| |
| <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> |
| <ul class="current"> |
| |
| |
| <li class="toctree-l1"> |
| |
| <a class="" href="/">Home</a> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <a class="" href="../../Powered-By/">Companies Powered By Gobblin</a> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <a class="" href="../../Getting-Started/">Getting Started</a> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <a class="" href="../../Gobblin-Architecture/">Architecture</a> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">User Guide</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../Working-with-Job-Configuration-Files/">Job Configuration Files</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Gobblin-Deployment/">Deployment</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Gobblin-as-a-Library/">Gobblin as a Library</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Gobblin-CLI/">Gobblin CLI</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Gobblin-Compliance/">Gobblin Compliance</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Gobblin-on-Yarn/">Gobblin on Yarn</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Compaction/">Compaction</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../State-Management-and-Watermarks/">State Management and Watermarks</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Working-with-the-ForkOperator/">Fork Operator</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Configuration-Properties-Glossary/">Configuration Glossary</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Source-schema-and-Converters/">Source schema and Converters</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Partitioned-Writers/">Partitioned Writers</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Monitoring/">Monitoring</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Gobblin-template/">Template</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Gobblin-Schedulers/">Schedulers</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Job-Execution-History-Store/">Job Execution History Store</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Building-Gobblin/">Building Gobblin</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Gobblin-genericLoad/">Generic Configuration Loading</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Hive-Registration/">Hive Registration</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Config-Management/">Config Management</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Docker-Integration/">Docker Integration</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../Troubleshooting/">Troubleshooting</a> |
| </li> |
| <li class=" current"> |
| |
| <a class="current" href="./">FAQs</a> |
| <ul class="subnav"> |
| |
| <li class="toctree-l3"><a href="#table-of-contents">Table of Contents</a></li> |
| |
| |
| <li class="toctree-l3"><a href="#gobblin">Gobblin</a></li> |
| |
| <ul> |
| |
| <li><a class="toctree-l4" href="#general-questions">General Questions </a></li> |
| |
| <li><a class="toctree-l4" href="#technical-questions">Technical Questions </a></li> |
| |
| </ul> |
| |
| |
| <li class="toctree-l3"><a href="#gradle">Gradle</a></li> |
| |
| <ul> |
| |
| <li><a class="toctree-l4" href="#technical-questions_1">Technical Questions</a></li> |
| |
| </ul> |
| |
| |
| </ul> |
| </li> |
| </ul> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">Sources</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../../sources/AvroFileSource/">Avro files</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/CopySource/">File copy</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/QueryBasedSource/">Query based</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/RestApiSource/">Rest Api</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/GoogleAnalyticsSource/">Google Analytics</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/GoogleDriveSource/">Google Drive</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/GoogleWebmaster/">Google Webmaster</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/HadoopTextInputSource/">Hadoop Text Input</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/HelloWorldSource/">Hello World</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/HiveAvroToOrcSource/">Hive Avro-to-ORC</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/HivePurgerSource/">Hive compliance purging</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/SimpleJsonSource/">JSON</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/KafkaSource/">Kafka</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/MySQLSource/">MySQL</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/OracleSource/">Oracle</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/SalesforceSource/">Salesforce</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/SftpSource/">SFTP</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/SqlServerSource/">SQL Server</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/TeradataSource/">Teradata</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sources/WikipediaSource/">Wikipedia</a> |
| </li> |
| </ul> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">Sinks (Writers)</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../../sinks/AvroHdfsDataWriter/">Avro HDFS</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sinks/ParquetHdfsDataWriter/">Parquet HDFS</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sinks/SimpleBytesWriter/">HDFS Byte array</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sinks/ConsoleWriter/">Console</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sinks/CouchbaseWriter/">Couchbase</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sinks/Http/">HTTP</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sinks/Gobblin-JDBC-Writer/">JDBC</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../sinks/Kafka/">Kafka</a> |
| </li> |
| </ul> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">Gobblin Adaptors</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../../adaptors/Gobblin-Distcp/">Gobblin Distcp</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../adaptors/Hive-Avro-To-ORC-Converter/">Hive Avro-To-Orc Converter</a> |
| </li> |
| </ul> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">Case Studies</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../../case-studies/Kafka-HDFS-Ingestion/">Kafka-HDFS Ingestion</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../case-studies/Publishing-Data-to-S3/">Publishing Data to S3</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../case-studies/Writing-ORC-Data/">Writing ORC Data</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../case-studies/Hive-Distcp/">Hive Distcp</a> |
| </li> |
| </ul> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">Gobblin Data Management</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../../data-management/Gobblin-Retention/">Retention</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../data-management/DistcpNgEvents/">Distcp-NG events</a> |
| </li> |
| </ul> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">Gobblin Metrics</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../../metrics/Gobblin-Metrics/">Quick Start</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../metrics/Existing-Reporters/">Existing Reporters</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../metrics/Metrics-for-Gobblin-ETL/">Metrics for Gobblin ETL</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../metrics/Gobblin-Metrics-Architecture/">Gobblin Metrics Architecture</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../metrics/Implementing-New-Reporters/">Implementing New Reporters</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../metrics/Gobblin-Metrics-Performance/">Gobblin Metrics Performance</a> |
| </li> |
| </ul> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">Developer Guide</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../../developer-guide/Customization-for-New-Source/">Customization for New Source</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../developer-guide/Customization-for-Converter-and-Operator/">Customization for Converter and Operator</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../developer-guide/CodingStyle/">Code Style Guide</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../developer-guide/Gobblin-Compliance-Design/">Gobblin Compliance Design</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../developer-guide/IDE-setup/">IDE setup</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../developer-guide/Monitoring-Design/">Monitoring Design</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../developer-guide/Documentation-Architecture/">Documentation Architecture</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../developer-guide/Contributing/">Contributing</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../developer-guide/GobblinModules/">Gobblin Modules</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../developer-guide/HighLevelConsumer/">High Level Consumer</a> |
| </li> |
| </ul> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">Project</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../../project/Feature-List/">Feature List</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="/people">Contributors and Team</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../project/Talks-and-Tech-Blogs/">Talks and Tech Blog Posts</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../project/Posts/">Posts</a> |
| </li> |
| </ul> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">Miscellaneous</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../../miscellaneous/Camus-to-Gobblin-Migration/">Camus to Gobblin Migration</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../miscellaneous/Exactly-Once-Support/">Exactly Once Support</a> |
| </li> |
| </ul> |
| </li> |
| |
| </ul> |
| </div> |
| |
| </nav> |
| |
| <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> |
| |
| |
| <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> |
| <i data-toggle="wy-nav-top" class="fa fa-bars"></i> |
| <a href="../..">Apache Gobblin</a> |
| </nav> |
| |
| |
| <div class="wy-nav-content"> |
| <div class="rst-content"> |
| <div role="navigation" aria-label="breadcrumbs navigation"> |
| <ul class="wy-breadcrumbs"> |
| <li><a href="../..">Docs</a> »</li> |
| |
| |
| |
| <li>User Guide »</li> |
| |
| |
| |
| <li>FAQs</li> |
| <li class="wy-breadcrumbs-aside"> |
| |
| <a href="https://github.com/apache/incubator-gobblin/edit/master/docs/user-guide/FAQs.md" rel="nofollow"> Edit on Gobblin</a> |
| |
| </li> |
| </ul> |
| <hr/> |
| </div> |
| <div role="main"> |
| <div class="section"> |
| |
| <h2 id="table-of-contents">Table of Contents</h2> |
| <div class="toc"> |
| <ul> |
| <li><a href="#table-of-contents">Table of Contents</a></li> |
| <li><a href="#gobblin">Gobblin</a><ul> |
| <li><a href="#general-questions">General Questions <a name="General-Questions"></a></a><ul> |
| <li><a href="#what-is-gobblin">What is Gobblin?</a></li> |
| <li><a href="#what-programming-languages-does-gobblin-support">What programming languages does Gobblin support?</a></li> |
| <li><a href="#does-gobblin-require-any-external-software-to-be-installed">Does Gobblin require any external software to be installed?</a></li> |
| <li><a href="#what-hadoop-versions-can-gobblin-run-on">What Hadoop versions can Gobblin run on?</a></li> |
| <li><a href="#how-do-i-run-and-schedule-a-gobblin-job">How do I run and schedule a Gobblin job?</a></li> |
| <li><a href="#how-is-gobblin-different-from-sqoop">How is Gobblin different from Sqoop?</a></li> |
| </ul> |
| </li> |
| <li><a href="#technical-questions">Technical Questions <a name="Technical-Questions"></a></a><ul> |
| <li><a href="#when-running-on-hadoop-each-map-task-quickly-reaches-100-percent-completion-but-then-stalls-for-a-long-time-why-does-this-happen">When running on Hadoop, each map task quickly reaches 100 Percent completion, but then stalls for a long time. Why does this happen?</a></li> |
| <li><a href="#why-does-gobblin-on-hadoop-stall-for-a-long-time-between-adding-files-to-the-distrbutedcache-and-launching-the-actual-job">Why does Gobblin on Hadoop stall for a long time between adding files to the DistrbutedCache, and launching the actual job?</a></li> |
| <li><a href="#how-do-i-fix-unsupportedfilesystemexception-no-abstractfilesystem-for-scheme-null">How do I fix UnsupportedFileSystemException: No AbstractFileSystem for scheme: null?</a></li> |
| <li><a href="#how-do-i-compile-gobblin-against-cdh">How do I compile Gobblin against CDH?</a></li> |
| <li><a href="#resolve-gobblin-on-mr-exception-ioexception-not-all-tasks-running-in-mapper-attempt_id-completed-successfully">Resolve Gobblin-on-MR Exception IOException: Not all tasks running in mapper attempt_id completed successfully</a></li> |
| <li><a href="#gradle-build-fails-with-cannot-invoke-method-geturls-on-null-object">Gradle Build Fails With Cannot invoke method getURLs on null object</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li><a href="#gradle">Gradle</a><ul> |
| <li><a href="#technical-questions_1">Technical Questions</a><ul> |
| <li><a href="#how-do-i-add-a-new-external-dependency">How do I add a new external dependency?</a></li> |
| <li><a href="#how-do-i-add-a-new-maven-repository-to-pull-artifacts-from">How do I add a new Maven Repository to pull artifacts from?</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| </div> |
| <h1 id="gobblin">Gobblin</h1> |
| <h2 id="general-questions">General Questions <a name="General-Questions"></a></h2> |
| <h5 id="what-is-gobblin">What is Gobblin?</h5> |
| <p>Gobblin is a universal ingestion framework. It's goal is to pull data from any source into an arbitrary data store. One major use case for Gobblin is pulling data into Hadoop. Gobblin can pull data from file systems, SQL stores, and data that is exposed by a REST API. See the Gobblin <a href="../index">Home</a> page for more information.</p> |
| <h5 id="what-programming-languages-does-gobblin-support">What programming languages does Gobblin support?</h5> |
| <p>Gobblin currently only supports Java 7 and up.</p> |
| <h5 id="does-gobblin-require-any-external-software-to-be-installed">Does Gobblin require any external software to be installed?</h5> |
| <p>The machine that Gobblin is built on must have Java installed, and the <code>$JAVA_HOME</code> environment variable must be set.</p> |
| <h5 id="what-hadoop-versions-can-gobblin-run-on">What Hadoop versions can Gobblin run on?</h5> |
| <p>Gobblin can only be run on Hadoop 2.x. By default, Gobblin compiles against Hadoop 2.3.0.</p> |
| <h5 id="how-do-i-run-and-schedule-a-gobblin-job">How do I run and schedule a Gobblin job?</h5> |
| <p>Check out the <a href="Gobblin Deployment">Deployment</a> page for information on how to run and schedule Gobblin jobs. Check out the <a href="Configuration Properties Glossary">Configuration</a> page for information on how to set proper configuration properties for a job.</p> |
| <h5 id="how-is-gobblin-different-from-sqoop">How is Gobblin different from Sqoop?</h5> |
| <p>Sqoop main focus bulk import and export of data from relational databases to HDFS, it lacks the ETL functionality of data cleansing, data transformation, and data quality checks that Gobblin provides. Gobblin is also capable of pulling from any data source (e.g. file systems, RDMS, REST APIs).</p> |
| <h2 id="technical-questions">Technical Questions <a name="Technical-Questions"></a></h2> |
| <h5 id="when-running-on-hadoop-each-map-task-quickly-reaches-100-percent-completion-but-then-stalls-for-a-long-time-why-does-this-happen">When running on Hadoop, each map task quickly reaches 100 Percent completion, but then stalls for a long time. Why does this happen?</h5> |
| <p>Gobblin currently uses Hadoop map tasks as a container for running Gobblin tasks. Each map task runs 1 or more Gobblin workunits, and the progress of each workunit is not hooked into the progress of each map task. Even though the Hadoop job reports 100% completion, Gobblin is still doing work. See the <a href="Gobblin Deployment">Gobblin Deployment</a> page for more information.</p> |
| <h5 id="why-does-gobblin-on-hadoop-stall-for-a-long-time-between-adding-files-to-the-distrbutedcache-and-launching-the-actual-job">Why does Gobblin on Hadoop stall for a long time between adding files to the DistrbutedCache, and launching the actual job?</h5> |
| <p>Gobblin takes all WorkUnits created by the Source class and serializes each one into a file on Hadoop. These files are read by each map task, and are deserialized into Gobblin Tasks. These Tasks are then run by the map-task. The reason the job stalls is that Gobblin is writing all these files to HDFS, which can take a while especially if there are a lot of tasks to run. See the <a href="Gobblin Deployment">Gobblin Deployment</a> page for more information.</p> |
| <h5 id="how-do-i-fix-unsupportedfilesystemexception-no-abstractfilesystem-for-scheme-null">How do I fix <code>UnsupportedFileSystemException: No AbstractFileSystem for scheme: null</code>?</h5> |
| <p>This error typically occurs due to Hadoop version conflict issues. If Gobblin is compiled against a specific Hadoop version, but then deployed on a different Hadoop version or installation, this error may be thrown. For example, if you simply compile Gobblin using <code>./gradlew clean build</code>, but deploy Gobblin to a cluster with <a href="https://www.cloudera.com/content/www/en-us/products/apache-hadoop/key-cdh-components.html" rel="nofollow">CDH</a> installed, you may hit this error.</p> |
| <p>It is important to realize that the the <code>gobblin-dist.tar.gz</code> file produced by <code>./gradlew clean build</code> will include all the Hadoop jar dependencies; and if one follows the <a href="Gobblin-Deployment#Hadoop-MapReduce-Deployment">MR deployment guide</a>, Gobblin will be launched with these dependencies on the classpath.</p> |
| <p>To fix this take the following steps:</p> |
| <ul> |
| <li>Delete all the Hadoop jars from the Gobblin <code>lib</code> folder</li> |
| <li>Ensure that the environment variable <code>HADOOP_CLASSPATH</code> is set and points to a directory containing the Hadoop libraries for the cluster</li> |
| </ul> |
| <h5 id="how-do-i-compile-gobblin-against-cdh">How do I compile Gobblin against CDH?</h5> |
| <p><a href="https://www.cloudera.com/content/www/en-us/products/apache-hadoop/key-cdh-components.html" rel="nofollow">Cloudera Distributed Hadoop</a> (often abbreviated as CDH) is a popular Hadoop distribution. Typically, when running Gobblin on a CDH cluster it is recommended that one also compile Gobblin against the same CDH version. Not doing so may cause unexpected runtime behavior. To compile against a specific CDH version simply use the <code>hadoopVersion</code> parameter. For example, to compile against version <code>2.5.0-cdh5.3.0</code> run <code>./gradlew clean build -PhadoopVersion=2.5.0-cdh5.3.0</code>.</p> |
| <h5 id="resolve-gobblin-on-mr-exception-ioexception-not-all-tasks-running-in-mapper-attempt_id-completed-successfully">Resolve Gobblin-on-MR Exception <code>IOException: Not all tasks running in mapper attempt_id completed successfully</code></h5> |
| <p>This exception usually just means that a Hadoop Map Task running Gobblin Tasks threw some exception. Unfortunately, the exception isn't truly indicative of the underlying problem, all it is really saying is that something went wrong in the Gobblin Task. Each Hadoop Map Task has its own log file and it is often easiest to look at the logs of the Map Task when debugging this problem. There are multiple ways to do this, but one of the easiest ways is to execute <code>yarn logs -applicationId <application ID> [OPTIONS]</code></p> |
| <h5 id="gradle-build-fails-with-cannot-invoke-method-geturls-on-null-object">Gradle Build Fails With <code>Cannot invoke method getURLs on null object</code></h5> |
| <p>Add <code>-x test</code> to build the project without running the tests; this will make the exception go away. If one needs to run the tests then make sure <a href="https://en.wikipedia.org/wiki/Java_Cryptography_Extension" rel="nofollow">Java Cryptography Extension</a> is installed.</p> |
| <h1 id="gradle">Gradle</h1> |
| <h2 id="technical-questions_1">Technical Questions</h2> |
| <h4 id="how-do-i-add-a-new-external-dependency">How do I add a new external dependency?</h4> |
| <p>Say I want to add <a href="http://mvnrepository.com/artifact/org.apache.oozie/oozie-core/4.2.0" rel="nofollow"><code>oozie-core-4.2.0.jar</code></a> as a dependency to the <code>gobblin-scheduler</code> subproject. I would first open the file <code>build.gradle</code> and add the following entry to the <code>ext.externalDependency</code> array: <code>"oozieCore": "org.apache.oozie:oozie-core:4.2.0"</code>.</p> |
| <p>Then in the <code>gobblin-scheduler/build.gradle</code> file I would add the following line to the dependency block: <code>compile externalDependency.oozieCore</code>.</p> |
| <h4 id="how-do-i-add-a-new-maven-repository-to-pull-artifacts-from">How do I add a new Maven Repository to pull artifacts from?</h4> |
| <p>Often times, one may have important artifacts stored in a local or private Maven repository. As of 01/21/2016 Gobblin only pulls artifacts from the following Maven Repositories: <a href="http://repo1.maven.org/maven/" rel="nofollow">Maven Central</a>, <a href="http://conjars.org/repo" rel="nofollow">Conjars</a>, and <a href="https://repository.cloudera.com/artifactory/cloudera-repos/" rel="nofollow">Cloudera</a>.</p> |
| <p>In order to add another Maven Repository modify the <code>defaultEnvironment.gradle</code> file and the new repository using the same pattern as the existing ones.</p> |
| |
| </div> |
| </div> |
| <footer> |
| |
| <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> |
| |
| <a href="../../sources/AvroFileSource/" class="btn btn-neutral float-right" title="Avro files">Next <span class="icon icon-circle-arrow-right"></span></a> |
| |
| |
| <a href="../Troubleshooting/" class="btn btn-neutral" title="Troubleshooting"><span class="icon icon-circle-arrow-left"></span> Previous</a> |
| |
| </div> |
| |
| |
| <hr/> |
| |
| <div role="contentinfo"> |
| <!-- Copyright etc --> |
| |
| </div> |
| |
| Built with <a href="http://www.mkdocs.org" rel="nofollow">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme" rel="nofollow">theme</a> provided by <a href="https://readthedocs.org" rel="nofollow">Read the Docs</a>. |
| </footer> |
| |
| </div> |
| </div> |
| |
| </section> |
| |
| </div> |
| |
| <div class="rst-versions" role="note" style="cursor: pointer"> |
| <span class="rst-current-version" data-toggle="rst-current-version"> |
| |
| |
| <span><a href="../Troubleshooting/" style="color: #fcfcfc;">« Previous</a></span> |
| |
| |
| <span style="margin-left: 15px"><a href="../../sources/AvroFileSource/" style="color: #fcfcfc">Next »</a></span> |
| |
| </span> |
| </div> |
| <script>var base_url = '../..';</script> |
| <script src="../../js/theme.js" defer></script> |
| <script src="../../js/extra.js" defer></script> |
| <script src="../../search/main.js" defer></script> |
| |
| </body> |
| </html> |
