| <!DOCTYPE html> |
| <!-- |
| | Generated by Apache Maven Doxia at 2018-03-12 |
| | Rendered using Apache Maven Fluido Skin 1.3.0 |
| --> |
| <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> |
| <head> |
| <meta charset="UTF-8" /> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> |
| <meta name="Date-Revision-yyyymmdd" content="20180312" /> |
| <meta http-equiv="Content-Language" content="en" /> |
| <title>Falcon - Hive Disaster Recovery</title> |
| <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> |
| <link rel="stylesheet" href="./css/site.css" /> |
| <link rel="stylesheet" href="./css/print.css" media="print" /> |
| |
| |
| <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> |
| |
| |
| |
| <script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> |
| |
| </head> |
| <body class="topBarDisabled"> |
| |
| |
| |
| <div class="container-fluid"> |
| <div id="banner"> |
| <div class="pull-left"> |
| <a href="../index.html" id="bannerLeft"> |
| <img src="images/falcon-logo.png" alt="Apache Falcon" width="200px" height="45px"/> |
| </a> |
| </div> |
| <div class="pull-right"> <a href="http://www.apache.org" id="bannerRight"> |
| <img src="images/apache-feather-tm.gif" alt="Falcon" height="45px"/> |
| </a> |
| </div> |
| <div class="clear"><hr/></div> |
| </div> |
| |
| <div id="breadcrumbs"> |
| <ul class="breadcrumb"> |
| |
| |
| <li class=""> |
| <a href="http://www.apache.org" class="externalLink" title="Apache"> |
| Apache</a> |
| </li> |
| <li class="divider ">/</li> |
| <li class=""> |
| <a href="index.html" title="Falcon"> |
| Falcon</a> |
| </li> |
| <li class="divider ">/</li> |
| <li class="">Hive Disaster Recovery</li> |
| |
| |
| |
| <li id="publishDate" class="pull-right">Last Published: 2018-03-12</li> |
| |
| </ul> |
| </div> |
| |
| |
| <div class="row-fluid"> |
| <div id="leftColumn" class="span3"> |
| <div class="well sidebar-nav"> |
| |
| |
| <ul class="nav nav-list"> |
| <li class="nav-header">Falcon</li> |
| |
| <li> |
| |
| <a href="index.html" title="About"> |
| <i class="none"></i> |
| About</a> |
| </li> |
| |
| <li> |
| |
| <a href="slides/falcon-overview.html" title="Overview"> |
| <i class="none"></i> |
| Overview</a> |
| </li> |
| |
| <li> |
| |
| <a href="slides/falcon-user-guide.html" title="User Guide"> |
| <i class="none"></i> |
| User Guide</a> |
| </li> |
| |
| <li> |
| |
| <a href="GettingStarted.html" title="Getting Started"> |
| <i class="none"></i> |
| Getting Started</a> |
| </li> |
| |
| <li> |
| |
| <a href="FalconDocumentation.html" title="Architecture"> |
| <i class="none"></i> |
| Architecture</a> |
| </li> |
| |
| <li> |
| |
| <a href="InstallationSteps.html" title="Installation"> |
| <i class="none"></i> |
| Installation</a> |
| </li> |
| |
| <li> |
| |
| <a href="OnBoarding.html" title="On Boarding"> |
| <i class="none"></i> |
| On Boarding</a> |
| </li> |
| |
| <li> |
| |
| <a href="MigrationInstructions.html" title="Migrate to 0.10"> |
| <i class="none"></i> |
| Migrate to 0.10</a> |
| </li> |
| |
| <li> |
| |
| <a href="Operability.html" title="Operability"> |
| <i class="none"></i> |
| Operability</a> |
| </li> |
| |
| <li> |
| |
| <a href="EntitySpecification.html" title="Entity Specification"> |
| <i class="none"></i> |
| Entity Specification</a> |
| </li> |
| |
| <li> |
| |
| <a href="falconcli/FalconCLI.html" title="Client (Falcon CLI)"> |
| <i class="none"></i> |
| Client (Falcon CLI)</a> |
| </li> |
| |
| <li> |
| |
| <a href="restapi/ResourceList.html" title="Rest API"> |
| <i class="icon-chevron-right"></i> |
| Rest API</a> |
| </li> |
| |
| <li> |
| |
| <a href="HiveIntegration.html" title="Hive Integration"> |
| <i class="none"></i> |
| Hive Integration</a> |
| </li> |
| |
| <li> |
| |
| <a href="Extensions.html" title="Server side Extensions"> |
| <i class="none"></i> |
| Server side Extensions</a> |
| </li> |
| |
| <li> |
| |
| <a href="Security.html" title="Security"> |
| <i class="none"></i> |
| Security</a> |
| </li> |
| <li class="nav-header">Project Information</li> |
| |
| <li> |
| |
| <a href="project-info.html" title="Summary"> |
| <i class="none"></i> |
| Summary</a> |
| </li> |
| |
| <li> |
| |
| <a href="mail-lists.html" title="Mailing Lists"> |
| <i class="none"></i> |
| Mailing Lists</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://webchat.freenode.net?channels=apachefalcon&uio=d4" class="externalLink" title="IRC"> |
| <i class="none"></i> |
| IRC</a> |
| </li> |
| |
| <li> |
| |
| <a href="team-list.html" title="Team"> |
| <i class="none"></i> |
| Team</a> |
| </li> |
| |
| <li> |
| |
| <a href="issue-tracking.html" title="Issue Tracking"> |
| <i class="none"></i> |
| Issue Tracking</a> |
| </li> |
| |
| <li> |
| |
| <a href="source-repository.html" title="Source Repository"> |
| <i class="none"></i> |
| Source Repository</a> |
| </li> |
| |
| <li> |
| |
| <a href="https://cwiki.apache.org/confluence/display/FALCON/Index" class="externalLink" title="Wiki"> |
| <i class="none"></i> |
| Wiki</a> |
| </li> |
| |
| <li> |
| |
| <a href="license.html" title="License"> |
| <i class="none"></i> |
| License</a> |
| </li> |
| |
| <li> |
| |
| <a href="https://cwiki.apache.org/confluence/display/FALCON/News" class="externalLink" title="News"> |
| <i class="none"></i> |
| News</a> |
| </li> |
| |
| <li> |
| |
| <a href="https://cwiki.apache.org/confluence/display/FALCON/PoweredBy" class="externalLink" title="Powered by"> |
| <i class="none"></i> |
| Powered by</a> |
| </li> |
| |
| <li> |
| |
| <a href="https://cwiki.apache.org/confluence/display/FALCON/Acknowledgements" class="externalLink" title="Acknowledgements"> |
| <i class="none"></i> |
| Acknowledgements</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://blogs.apache.org/falcon/" class="externalLink" title="Blog"> |
| <i class="none"></i> |
| Blog</a> |
| </li> |
| <li class="nav-header">Releases</li> |
| |
| <li> |
| |
| <a href="http://www.apache.org/dyn/closer.lua/falcon/0.11" class="externalLink" title="0.11"> |
| <i class="none"></i> |
| 0.11</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://www.apache.org/dyn/closer.lua/falcon/0.10" class="externalLink" title="0.10"> |
| <i class="none"></i> |
| 0.10</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://www.apache.org/dyn/closer.lua/falcon/0.9" class="externalLink" title="0.9"> |
| <i class="none"></i> |
| 0.9</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://www.apache.org/dyn/closer.lua/falcon/0.8" class="externalLink" title="0.8"> |
| <i class="none"></i> |
| 0.8</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://www.apache.org/dyn/closer.lua/falcon/0.7" class="externalLink" title="0.7"> |
| <i class="none"></i> |
| 0.7</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://archive.apache.org/dist/falcon/0.6.1" class="externalLink" title="0.6.1"> |
| <i class="none"></i> |
| 0.6.1</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://archive.apache.org/dist/incubator/falcon/0.6-incubating" class="externalLink" title="0.6-incubating"> |
| <i class="none"></i> |
| 0.6-incubating</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://archive.apache.org/dist/incubator/falcon/0.5-incubating" class="externalLink" title="0.5-incubating"> |
| <i class="none"></i> |
| 0.5-incubating</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://archive.apache.org/dist/incubator/falcon/0.4-incubating" class="externalLink" title="0.4-incubating"> |
| <i class="none"></i> |
| 0.4-incubating</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://archive.apache.org/dist/incubator/falcon/0.3-incubating" class="externalLink" title="0.3-incubating"> |
| <i class="none"></i> |
| 0.3-incubating</a> |
| </li> |
| |
| <li> |
| |
| <a href="https://cwiki.apache.org/confluence/display/FALCON/Roadmap" class="externalLink" title="Coming soon"> |
| <i class="none"></i> |
| Coming soon</a> |
| </li> |
| <li class="nav-header">Documentation</li> |
| |
| <li> |
| |
| <a href="0.11/index.html" title="0.11 (Current)"> |
| <i class="none"></i> |
| 0.11 (Current)</a> |
| </li> |
| |
| <li> |
| |
| <a href="0.10/index.html" title="0.10"> |
| <i class="none"></i> |
| 0.10</a> |
| </li> |
| |
| <li> |
| |
| <a href="0.9/index.html" title="0.9"> |
| <i class="none"></i> |
| 0.9</a> |
| </li> |
| |
| <li> |
| |
| <a href="0.8/index.html" title="0.8"> |
| <i class="none"></i> |
| 0.8</a> |
| </li> |
| |
| <li> |
| |
| <a href="0.7/index.html" title="0.7"> |
| <i class="none"></i> |
| 0.7</a> |
| </li> |
| |
| <li> |
| |
| <a href="0.6.1/index.html" title="0.6.1"> |
| <i class="none"></i> |
| 0.6.1</a> |
| </li> |
| |
| <li> |
| |
| <a href="0.6-incubating/index.html" title="0.6-incubating"> |
| <i class="none"></i> |
| 0.6-incubating</a> |
| </li> |
| |
| <li> |
| |
| <a href="0.5-incubating/index.html" title="0.5-incubating"> |
| <i class="none"></i> |
| 0.5-incubating</a> |
| </li> |
| |
| <li> |
| |
| <a href="0.4-incubating/index.html" title="0.4-incubating"> |
| <i class="none"></i> |
| 0.4-incubating</a> |
| </li> |
| |
| <li> |
| |
| <a href="0.3-incubating/index.html" title="0.3-incubating"> |
| <i class="none"></i> |
| 0.3-incubating</a> |
| </li> |
| <li class="nav-header">ASF</li> |
| |
| <li> |
| |
| <a href="http://www.apache.org/foundation/how-it-works.html" class="externalLink" title="How Apache Works"> |
| <i class="none"></i> |
| How Apache Works</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://www.apache.org/foundation/" class="externalLink" title="Foundation"> |
| <i class="none"></i> |
| Foundation</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://www.apache.org/foundation/sponsorship.html" class="externalLink" title="Sponsoring Apache"> |
| <i class="none"></i> |
| Sponsoring Apache</a> |
| </li> |
| |
| <li> |
| |
| <a href="http://www.apache.org/foundation/thanks.html" class="externalLink" title="Thanks"> |
| <i class="none"></i> |
| Thanks</a> |
| </li> |
| </ul> |
| |
| |
| |
| <hr class="divider" /> |
| |
| <div id="poweredBy"> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <div class="clear"></div> |
| <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"> |
| <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> |
| </a> |
| </div> |
| </div> |
| </div> |
| |
| |
| <div id="bodyColumn" class="span9" > |
| |
| <div class="section"> |
| <h2>Hive Disaster Recovery<a name="Hive_Disaster_Recovery"></a></h2></div> |
| <div class="section"> |
| <h3>Overview<a name="Overview"></a></h3> |
| <p>Falcon provides feature to replicate Hive metadata and data events from source cluster to destination cluster. This is supported for secure and unsecure cluster through Falcon Recipes.</p></div> |
| <div class="section"> |
| <h3>Prerequisites<a name="Prerequisites"></a></h3> |
| <p>Following is the prerequisites to use Hive DR</p> |
| <p></p> |
| <ul> |
| <li><b>Hive 1.2.0+</b></li> |
| <li><b>Oozie 4.2.0+</b></li></ul> |
| <p><b>Note:</b> Set following properties in hive-site.xml for replicating the Hive events on source and destination Hive cluster:</p> |
| <div class="source"> |
| <pre> |
| <property> |
| <name>hive.metastore.event.listeners</name> |
| <value>org.apache.hive.hcatalog.listener.DbNotificationListener</value> |
| <description>event listeners that are notified of any metastore changes</description> |
| </property> |
| |
| <property> |
| <name>hive.metastore.dml.events</name> |
| <value>true</value> |
| </property> |
| |
| </pre></div></div> |
| <div class="section"> |
| <h3>Usage<a name="Usage"></a></h3></div> |
| <div class="section"> |
| <h4>Bootstrap<a name="Bootstrap"></a></h4> |
| <p>Perform initial bootstrap of Table and Database from source cluster to destination cluster</p> |
| <ul> |
| <li><b>Database Bootstrap</b></li></ul>For bootstrapping DB replication, first destination DB should be created. This step is expected, since DB replication definitions can be set up by users only on pre-existing DB’s. Second, Export all tables in the source db and Import it in the destination db, as described in Table bootstrap. |
| <p></p> |
| <ul> |
| <li><b>Table Bootstrap</b></li></ul>For bootstrapping table replication, essentially after having turned on the DbNotificationListener on the source db, perform an Export of the table, distcp the Export over to the destination warehouse and do an Import over there. Check the following <a class="externalLink" href="https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport">Hive Export-Import</a> for syntax details and examples. This will set up the destination table so that the events on the source cluster that modify the table will then be replicated.</div> |
| <div class="section"> |
| <h4>Setup cluster definition<a name="Setup_cluster_definition"></a></h4> |
| <div class="source"> |
| <pre> |
| $FALCON_HOME/bin/falcon entity -submit -type cluster -file /cluster/definition.xml |
| |
| </pre></div></div> |
| <div class="section"> |
| <h4>Update recipes properties<a name="Update_recipes_properties"></a></h4> |
| <p>Copy Hive DR recipe properties, workflow and template file from $FALCON_HOME/data-mirroring/hive-disaster-recovery to the accessible directory path or to the recipe directory path (<b>falcon.recipe.path=<recipe directory path></b>). <b>"falcon.recipe.path"</b> must be specified in Falcon conf client.properties. Now update the copied recipe properties file with required attributes to replicate metadata and data from source cluster to destination cluster for Hive DR.</p></div> |
| <div class="section"> |
| <h4>Submit Hive DR recipe<a name="Submit_Hive_DR_recipe"></a></h4> |
| <p>After updating the recipe properties file with required attributes in directory path or in falcon.recipe.path, there are two ways of submitting the Hive DR recipe:</p> |
| <p></p> |
| <ul> |
| <li>1. Specify Falcon recipe properties file through recipe command line.</li></ul> |
| <div class="source"> |
| <pre> |
| $FALCON_HOME/bin/falcon recipe -name hive-disaster-recovery -operation HIVE_DISASTER_RECOVERY |
| -properties /cluster/hive-disaster-recovery.properties |
| |
| </pre></div> |
| <p></p> |
| <ul> |
| <li>2. Use Falcon recipe path specified in Falcon conf client.properties .</li></ul> |
| <div class="source"> |
| <pre> |
| $FALCON_HOME/bin/falcon recipe -name hive-disaster-recovery -operation HIVE_DISASTER_RECOVERY |
| |
| </pre></div> |
| <p><b>Note:</b></p> |
| <ul> |
| <li>Recipe properties file, workflow file and template file name must match to the recipe name, it must be unique and in the same directory.</li> |
| <li>If kerberos security is enabled on cluster, use the secure templates for Hive DR from $FALCON_HOME/data-mirroring/hive-disaster-recovery .</li></ul></div> |
| </div> |
| </div> |
| </div> |
| |
| <hr/> |
| |
| <footer> |
| <div class="container-fluid"> |
| <div class="row span12">Copyright © 2013-2018 |
| <a href="http://www.apache.org">Apache Software Foundation</a>. |
| All Rights Reserved. |
| |
| </div> |
| |
| |
| |
| </div> |
| </footer> |
| </body> |
| </html> |
