| <!DOCTYPE html> |
| <!-- |
| | Generated by Apache Maven Doxia at 2018-03-12 |
| | Rendered using Apache Maven Fluido Skin 1.3.0 |
| --> |
| <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> |
| <head> |
| <meta charset="UTF-8" /> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> |
| <meta name="Date-Revision-yyyymmdd" content="20180312" /> |
| <meta http-equiv="Content-Language" content="en" /> |
| <title>Falcon - Building & Installing Falcon</title> |
| <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> |
| <link rel="stylesheet" href="./css/site.css" /> |
| <link rel="stylesheet" href="./css/print.css" media="print" /> |
| |
| |
| <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> |
| |
| |
| |
| <script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> |
| |
| </head> |
| <body class="topBarDisabled"> |
| |
| |
| |
| <div class="container-fluid"> |
| <div id="banner"> |
| <div class="pull-left"> |
| <a href="../index.html" id="bannerLeft"> |
| <img src="images/falcon-logo.png" alt="Apache Falcon" width="200px" height="45px"/> |
| </a> |
| </div> |
| <div class="pull-right"> <a href="http://www.apache.org" id="bannerRight"> |
| <img src="images/apache-feather-tm.gif" alt="Falcon" height="45px"/> |
| </a> |
| </div> |
| <div class="clear"><hr/></div> |
| </div> |
| |
| <div id="breadcrumbs"> |
| <ul class="breadcrumb"> |
| |
| |
| <li class=""> |
| <a href="http://www.apache.org" class="externalLink" title="Apache"> |
| Apache</a> |
| </li> |
| <li class="divider ">/</li> |
| <li class=""> |
| <a href="index.html" title="Falcon"> |
| Falcon</a> |
| </li> |
| <li class="divider ">/</li> |
| <li class="">Building & Installing Falcon</li> |
| |
| |
| |
| <li id="publishDate" class="pull-right">Last Published: 2018-03-12</li> |
| |
| </ul> |
| </div> |
| |
| |
| <div class="row-fluid"> |
| <div id="leftColumn" class="span3"> |
| <div class="well sidebar-nav"> |
| |
| |
| <ul class="nav nav-list"> |
| <li class="nav-header">Falcon</li> |
| |
| <li> |
| |
| <a href="index.html" title="About"> |
| <i class="none"></i> |
| About</a> |
| </li> |
| |
| <li> |
| |
| <a href="slides/falcon-overview.html" title="Overview"> |
| <i class="none"></i> |
| Overview</a> |
| </li> |
| |
| <li> |
| |
| <a href="slides/falcon-user-guide.html" title="User Guide"> |
| <i class="none"></i> |
| User Guide</a> |
| </li> |
| |
| <li> |
| |
| <a href="GettingStarted.html" title="Getting Started"> |
| <i class="none"></i> |
| Getting Started</a> |
| </li> |
| |
| <li> |
| |
| <a href="FalconDocumentation.html" title="Architecture"> |
| <i class="none"></i> |
| Architecture</a> |
| </li> |
| |
| <li class="active"> |
| |
| <a href="#"><i class="none"></i>Installation</a> |
| </li> |
| |
| <li> |
| |
| <a href="OnBoarding.html" title="On Boarding"> |
| <i class="none"></i> |
| On Boarding</a> |
| </li> |
| |
| <li> |
| |
| <a href="MigrationInstructions.html" title="Migrate to 0.10"> |
| <i class="none"></i> |
| Migrate to 0.10</a> |
| </li> |
| |
| <li> |
| |
| <a href="Operability.html" title="Operability"> |
| <i class="none"></i> |
| Operability</a> |
| </li> |
| |
| <li> |
| |
| <a href="EntitySpecification.html" title="Entity Specification"> |
| <i class="none"></i> |
| Entity Specification</a> |
| </li> |
| |
| <li> |
| |
| <a href="falconcli/FalconCLI.html" title="Client (Falcon CLI)"> |
| <i class="none"></i> |
| Client (Falcon CLI)</a> |
| </li> |
| |
| <li> |
| |
| <a href="restapi/ResourceList.html" title="Rest API"> |
| <i class="icon-chevron-right"></i> |
| Rest API</a> |
| </li> |
| |
| <li> |
| |
| <a href="HiveIntegration.html" title="Hive Integration"> |
| <i class="none"></i> |
| Hive Integration</a> |
| </li> |
| |
| <li> |
| |
| <a href="Extensions.html" title="Server side Extensions"> |
| <i class="none"></i> |
| Server side Extensions</a> |
| </li> |
| |
| <li> |
| |
| <a href="Security.html" title="Security"> |
| <i class="none"></i> |
| Security</a> |
| </li> |
| <li class="nav-header">Project Information</li> |
| |
| <li> |
| |
| <a href="project-info.html" title="Summary"> |
| <i class="none"></i> |
| Summary</a> |
| </li> |
| |
| <li> |
| |
| <a href="mail-lists.html" title="Mailing Lists"> |
| <i class="none"></i> |
| Mailing Lists</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://webchat.freenode.net?channels=apachefalcon&uio=d4" class="externalLink" title="IRC"> |
| <i class="none"></i> |
| IRC</a> |
| </li> |
| |
| <li> |
| |
| <a href="team-list.html" title="Team"> |
| <i class="none"></i> |
| Team</a> |
| </li> |
| |
| <li> |
| |
| <a href="issue-tracking.html" title="Issue Tracking"> |
| <i class="none"></i> |
| Issue Tracking</a> |
| </li> |
| |
| <li> |
| |
| <a href="source-repository.html" title="Source Repository"> |
| <i class="none"></i> |
| Source Repository</a> |
| </li> |
| |
| <li> |
| |
| <a href="https://cwiki.apache.org/confluence/display/FALCON/Index" class="externalLink" title="Wiki"> |
| <i class="none"></i> |
| Wiki</a> |
| </li> |
| |
| <li> |
| |
| <a href="license.html" title="License"> |
| <i class="none"></i> |
| License</a> |
| </li> |
| |
| <li> |
| |
| <a href="https://cwiki.apache.org/confluence/display/FALCON/News" class="externalLink" title="News"> |
| <i class="none"></i> |
| News</a> |
| </li> |
| |
| <li> |
| |
| <a href="https://cwiki.apache.org/confluence/display/FALCON/PoweredBy" class="externalLink" title="Powered by"> |
| <i class="none"></i> |
| Powered by</a> |
| </li> |
| |
| <li> |
| |
| <a href="https://cwiki.apache.org/confluence/display/FALCON/Acknowledgements" class="externalLink" title="Acknowledgements"> |
| <i class="none"></i> |
| Acknowledgements</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://blogs.apache.org/falcon/" class="externalLink" title="Blog"> |
| <i class="none"></i> |
| Blog</a> |
| </li> |
| <li class="nav-header">Releases</li> |
| |
| <li> |
| |
| <a href="http://www.apache.org/dyn/closer.lua/falcon/0.11" class="externalLink" title="0.11"> |
| <i class="none"></i> |
| 0.11</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://www.apache.org/dyn/closer.lua/falcon/0.10" class="externalLink" title="0.10"> |
| <i class="none"></i> |
| 0.10</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://www.apache.org/dyn/closer.lua/falcon/0.9" class="externalLink" title="0.9"> |
| <i class="none"></i> |
| 0.9</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://www.apache.org/dyn/closer.lua/falcon/0.8" class="externalLink" title="0.8"> |
| <i class="none"></i> |
| 0.8</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://www.apache.org/dyn/closer.lua/falcon/0.7" class="externalLink" title="0.7"> |
| <i class="none"></i> |
| 0.7</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://archive.apache.org/dist/falcon/0.6.1" class="externalLink" title="0.6.1"> |
| <i class="none"></i> |
| 0.6.1</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://archive.apache.org/dist/incubator/falcon/0.6-incubating" class="externalLink" title="0.6-incubating"> |
| <i class="none"></i> |
| 0.6-incubating</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://archive.apache.org/dist/incubator/falcon/0.5-incubating" class="externalLink" title="0.5-incubating"> |
| <i class="none"></i> |
| 0.5-incubating</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://archive.apache.org/dist/incubator/falcon/0.4-incubating" class="externalLink" title="0.4-incubating"> |
| <i class="none"></i> |
| 0.4-incubating</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://archive.apache.org/dist/incubator/falcon/0.3-incubating" class="externalLink" title="0.3-incubating"> |
| <i class="none"></i> |
| 0.3-incubating</a> |
| </li> |
| |
| <li> |
| |
| <a href="https://cwiki.apache.org/confluence/display/FALCON/Roadmap" class="externalLink" title="Coming soon"> |
| <i class="none"></i> |
| Coming soon</a> |
| </li> |
| <li class="nav-header">Documentation</li> |
| |
| <li> |
| |
| <a href="0.11/index.html" title="0.11 (Current)"> |
| <i class="none"></i> |
| 0.11 (Current)</a> |
| </li> |
| |
| <li> |
| |
| <a href="0.10/index.html" title="0.10"> |
| <i class="none"></i> |
| 0.10</a> |
| </li> |
| |
| <li> |
| |
| <a href="0.9/index.html" title="0.9"> |
| <i class="none"></i> |
| 0.9</a> |
| </li> |
| |
| <li> |
| |
| <a href="0.8/index.html" title="0.8"> |
| <i class="none"></i> |
| 0.8</a> |
| </li> |
| |
| <li> |
| |
| <a href="0.7/index.html" title="0.7"> |
| <i class="none"></i> |
| 0.7</a> |
| </li> |
| |
| <li> |
| |
| <a href="0.6.1/index.html" title="0.6.1"> |
| <i class="none"></i> |
| 0.6.1</a> |
| </li> |
| |
| <li> |
| |
| <a href="0.6-incubating/index.html" title="0.6-incubating"> |
| <i class="none"></i> |
| 0.6-incubating</a> |
| </li> |
| |
| <li> |
| |
| <a href="0.5-incubating/index.html" title="0.5-incubating"> |
| <i class="none"></i> |
| 0.5-incubating</a> |
| </li> |
| |
| <li> |
| |
| <a href="0.4-incubating/index.html" title="0.4-incubating"> |
| <i class="none"></i> |
| 0.4-incubating</a> |
| </li> |
| |
| <li> |
| |
| <a href="0.3-incubating/index.html" title="0.3-incubating"> |
| <i class="none"></i> |
| 0.3-incubating</a> |
| </li> |
| <li class="nav-header">ASF</li> |
| |
| <li> |
| |
| <a href="http://www.apache.org/foundation/how-it-works.html" class="externalLink" title="How Apache Works"> |
| <i class="none"></i> |
| How Apache Works</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://www.apache.org/foundation/" class="externalLink" title="Foundation"> |
| <i class="none"></i> |
| Foundation</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://www.apache.org/foundation/sponsorship.html" class="externalLink" title="Sponsoring Apache"> |
| <i class="none"></i> |
| Sponsoring Apache</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://www.apache.org/foundation/thanks.html" class="externalLink" title="Thanks"> |
| <i class="none"></i> |
| Thanks</a> |
| </li> |
| </ul> |
| |
| |
| |
| <hr class="divider" /> |
| |
| <div id="poweredBy"> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"> |
| <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> |
| </a> |
| </div> |
| </div> |
| </div> |
| |
| |
| <div id="bodyColumn" class="span9" > |
| |
| <div class="section"> |
| <h2>Building & Installing Falcon<a name="Building__Installing_Falcon"></a></h2></div> |
| <div class="section"> |
| <h3>Building Falcon<a name="Building_Falcon"></a></h3></div> |
| <div class="section"> |
| <h4>Prerequisites<a name="Prerequisites"></a></h4> |
| <p></p> |
| <ul> |
| <li>JDK 1.7/1.8</li> |
| <li>Maven 3.2.x</li></ul></div> |
| <div class="section"> |
| <h4>Step 1 - Clone the Falcon repository<a name="Step_1_-_Clone_the_Falcon_repository"></a></h4> |
| <div class="source"> |
| <pre> |
| $git clone https://git-wip-us.apache.org/repos/asf/falcon.git falcon |
| |
| </pre></div></div> |
| <div class="section"> |
| <h4>Step 2 - Build Falcon<a name="Step_2_-_Build_Falcon"></a></h4> |
| <div class="source"> |
| <pre> |
| $ cd falcon |
| $ export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=256m -noverify" |
| $ mvn clean install |
| |
| |
| </pre></div> |
| <p>It builds and installs the package into the local repository, for use as a dependency in other projects locally.</p> |
| <p>[optionally -Dhadoop.version=<<hadoop.version>> can be appended to build for a specific version of hadoop]</p> |
| <p><b>Note 1:</b> Falcon drops support for Hadoop-1 and only supports Hadoop-2 from Falcon 0.6 onwards Falcon build with JDK 1.7 using -noverify option</p> |
| <p><b>Note 2:</b> To compile Falcon with addon extensions, append additional profiles to build command using syntax -P<<profile1,profile2>> For Hive Mirroring extension, use profile"hivedr". Hive >= 1.2.0 and Oozie >= 4.2.0 is required For HDFS Snapshot mirroring extension, use profile "hdfs-snapshot-mirroring". Hadoop >= 2.7.0 is required For ADF integration, use profile "adf"</p></div> |
| <div class="section"> |
| <h4>Step 3 - Package and Deploy Falcon<a name="Step_3_-_Package_and_Deploy_Falcon"></a></h4> |
| <p>Once the build successfully completes, artifacts can be packaged for deployment using the assembly plugin. The Assembly Plugin for Maven is primarily intended to allow users to aggregate the project output along with its dependencies, modules, site documentation, and other files into a single distributable archive. There are two basic ways in which you can deploy Falcon - Embedded mode(also known as Stand Alone Mode) and Distributed mode. Your next steps will vary based on the mode in which you want to deploy Falcon.</p> |
| <p><b>NOTE</b> : Oozie is being extended by Falcon (particularly on el-extensions) and hence the need for Falcon to build & re-package Oozie, so that users of Falcon can work with the right Oozie setup. Though Oozie is packaged by Falcon, it needs to be deployed separately by the administrator and is not auto deployed along with Falcon.</p></div> |
| <div class="section"> |
| <h5>Embedded/Stand Alone Mode<a name="EmbeddedStand_Alone_Mode"></a></h5> |
| <p>Embedded mode is useful when the Hadoop jobs and relevant data processing involve only one Hadoop cluster. In this mode there is a single Falcon server that contacts the scheduler to schedule jobs on Hadoop. All the process/feed requests like submit, schedule, suspend, kill etc. are sent to this server. For running Falcon in this mode one should use the Falcon which has been built using standalone option. You can find the instructions for Embedded mode setup <a href="./Embedded-mode.html">here</a>.</p></div> |
| <div class="section"> |
| <h5>Distributed Mode<a name="Distributed_Mode"></a></h5> |
| <p>Distributed mode is for multiple (colos) instances of Hadoop clusters, and multiple workflow schedulers to handle them. In this mode Falcon has 2 components: Prism and Server(s). Both Prism and Server(s) have their own their own config locations(startup and runtime properties). In this mode Prism acts as a contact point for Falcon servers. While all commands are available through Prism, only read and instance api's are available through Server. You can find the instructions for Distributed Mode setup <a href="./Distributed-mode.html">here</a>.</p></div> |
| <div class="section"> |
| <h4>Preparing Oozie and Falcon packages for deployment<a name="Preparing_Oozie_and_Falcon_packages_for_deployment"></a></h4> |
| <div class="source"> |
| <pre> |
| $cd <<project home>> |
| $src/bin/package.sh <<hadoop-version>> <<oozie-version>> |
| |
| >> ex. src/bin/package.sh 1.1.2 4.0.1 or src/bin/package.sh 0.20.2-cdh3u5 4.0.1 |
| >> ex. src/bin/package.sh 2.5.0 4.0.0 |
| >> Falcon package is available in <<falcon home>>/target/apache-falcon-<<version>>-bin.tar.gz |
| >> Oozie package is available in <<falcon home>>/target/oozie-4.0.1-distro.tar.gz |
| >> __IMPORTANT: You need to download the je-5.0.73 version from http://download.oracle.com/otn/berkeley-db/je-5.0.73.zip and extract je-5.0.73 under the Falcon webapp directory or provision an HBase cluster for use as Falcon graphdb backend DB. |
| Depending on the Graphdb backend choice, update the startup.properties appropriately.__ |
| |
| </pre></div> |
| <p><b>NOTE:</b> If you have a separate Apache Oozie installation, you will need to follow some additional steps:</p> |
| <ol style="list-style-type: decimal"> |
| <li>Once you have setup the Falcon Server, copy libraries under {falcon-server-dir}/oozie/libext/ to {oozie-install-dir}/libext.</li> |
| <li>Modify Oozie's configuration file. Copy all Falcon related properties from {falcon-server-dir}/oozie/conf/oozie-site.xml to {oozie-install-dir}/conf/oozie-site.xml</li> |
| <li>Restart oozie: |
| <ol style="list-style-type: decimal"> |
| <li>cd {oozie-install-dir}</li> |
| <li>sudo -u oozie ./bin/oozie-stop.sh</li> |
| <li>sudo -u oozie ./bin/oozie-setup.sh prepare-war</li> |
| <li>sudo -u oozie ./bin/oozie-start.sh</li></ol></li></ol></div> |
| </div> |
| </div> |
| </div> |
| |
| <hr/> |
| |
| <footer> |
| <div class="container-fluid"> |
| <div class="row span12">Copyright © 2013-2018 |
| <a href="http://www.apache.org">Apache Software Foundation</a>. |
| All Rights Reserved. |
| |
| </div> |
| |
| |
| |
| </div> |
| </footer> |
| </body> |
| </html> |
