| <!DOCTYPE html> |
| <!-- |
| | Generated by Apache Maven Doxia at 2017-03-16 |
| | Rendered using Apache Maven Fluido Skin 1.3.0 |
| --> |
| <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> |
| <head> |
| <meta charset="UTF-8" /> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> |
| <meta name="Date-Revision-yyyymmdd" content="20170316" /> |
| <meta http-equiv="Content-Language" content="en" /> |
| <title>Apache Atlas – Sqoop Atlas Bridge</title> |
| <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> |
| <link rel="stylesheet" href="./css/site.css" /> |
| <link rel="stylesheet" href="./css/print.css" media="print" /> |
| |
| |
| <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> |
| |
| |
| |
| <script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> |
| |
| </head> |
| <body class="topBarEnabled"> |
| |
| |
| |
| |
| |
| <div id="topbar" class="navbar navbar-fixed-top "> |
| <div class="navbar-inner"> |
| <div class="container" style="width: 68%;"><div class="nav-collapse"> |
| |
| |
| <ul class="nav"> |
| <li class="dropdown"> |
| <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| |
| <li> <a href="index.html" title="About">About</a> |
| </li> |
| |
| <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS" title="Wiki">Wiki</a> |
| </li> |
| |
| <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS" title="News">News</a> |
| </li> |
| |
| <li> <a href="https://git-wip-us.apache.org/repos/asf/incubator-atlas.git" title="Git">Git</a> |
| </li> |
| |
| <li> <a href="https://issues.apache.org/jira/browse/ATLAS" title="Jira">Jira</a> |
| </li> |
| |
| <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy" title="Powered by">Powered by</a> |
| </li> |
| |
| <li> <a href="http://blogs.apache.org/atlas/" title="Blog">Blog</a> |
| </li> |
| </ul> |
| </li> |
| <li class="dropdown"> |
| <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| |
| <li> <a href="project-info.html" title="Summary">Summary</a> |
| </li> |
| |
| <li> <a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a> |
| </li> |
| |
| <li> <a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4" title="IRC">IRC</a> |
| </li> |
| |
| <li> <a href="team-list.html" title="Team">Team</a> |
| </li> |
| |
| <li> <a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a> |
| </li> |
| |
| <li> <a href="source-repository.html" title="Source Repository">Source Repository</a> |
| </li> |
| |
| <li> <a href="license.html" title="License">License</a> |
| </li> |
| </ul> |
| </li> |
| <li class="dropdown"> |
| <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| |
| <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.8.0-incubating/" title="0.8-incubating">0.8-incubating</a> |
| </li> |
| |
| <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/" title="0.7.1-incubating">0.7.1-incubating</a> |
| </li> |
| |
| <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/" title="0.7-incubating">0.7-incubating</a> |
| </li> |
| |
| <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/" title="0.6-incubating">0.6-incubating</a> |
| </li> |
| |
| <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/" title="0.5-incubating">0.5-incubating</a> |
| </li> |
| </ul> |
| </li> |
| <li class="dropdown"> |
| <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| <li><a href="../index.html" title="latest">latest</a></li> |
| <li><a href="../0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li> |
| <li><a href="../0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li> |
| <li><a href="../0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li> |
| <li><a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li> |
| <li><a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li> |
| </ul> |
| </li> |
| <li class="dropdown"> |
| <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| |
| <li> <a href="http://www.apache.org/foundation/how-it-works.html" title="How Apache Works">How Apache Works</a> |
| </li> |
| |
| <li> <a href="http://www.apache.org/foundation/" title="Foundation">Foundation</a> |
| </li> |
| |
| <li> <a href="http://www.apache.org/foundation/sponsorship.html" title="Sponsoring Apache">Sponsoring Apache</a> |
| </li> |
| |
| <li> <a href="http://www.apache.org/foundation/thanks.html" title="Thanks">Thanks</a> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| |
| <form id="search-form" action="http://www.google.com/search" method="get" class="navbar-search pull-right" > |
| |
| <input value="http://atlas.incubator.apache.org" name="sitesearch" type="hidden"/> |
| <input class="search-query" name="q" id="query" type="text" /> |
| </form> |
| <script type="text/javascript" src="http://www.google.com/coop/cse/brand?form=search-form"></script> |
| |
| |
| |
| |
| |
| <iframe src="http://www.facebook.com/plugins/like.php?href=http://atlas.incubator.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark" |
| scrolling="no" frameborder="0" |
| style="border:none; width:80px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> |
| |
| <script type="text/javascript" src="https://apis.google.com/js/plusone.js"></script> |
| |
| <ul class="nav pull-right"><li style="margin-top: 10px;"> |
| |
| <div class="g-plusone" data-href="http://atlas.incubator.apache.org/atlas-docs" data-size="medium" width="60px" align="right" ></div> |
| |
| </li></ul> |
| |
| |
| </div> |
| |
| </div> |
| </div> |
| </div> |
| |
| <div class="container"> |
| <div id="banner"> |
| <div class="pull-left"> |
| <a href=".." id="bannerLeft"> |
| <img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/> |
| </a> |
| </div> |
| <div class="pull-right"> <a href="http://incubator.apache.org" id="bannerRight"> |
| <img src="images/apache-incubator-logo.png" alt="Apache Incubator"/> |
| </a> |
| </div> |
| <div class="clear"><hr/></div> |
| </div> |
| |
| <div id="breadcrumbs"> |
| <ul class="breadcrumb"> |
| |
| |
| <li class=""> |
| <a href="http://www.apache.org" class="externalLink" title="Apache"> |
| Apache</a> |
| </li> |
| <li class="divider ">/</li> |
| <li class=""> |
| <a href="index.html" title="Atlas"> |
| Atlas</a> |
| </li> |
| <li class="divider ">/</li> |
| <li class="">Sqoop Atlas Bridge</li> |
| |
| |
| |
| <li id="publishDate" class="pull-right">Last Published: 2017-03-16</li> <li class="divider pull-right">|</li> |
| <li id="projectVersion" class="pull-right">Version: 0.8-incubating</li> |
| |
| </ul> |
| </div> |
| |
| |
| |
| <div id="bodyColumn" > |
| |
| <div class="section"> |
| <h2><a name="Sqoop_Atlas_Bridge"></a>Sqoop Atlas Bridge</h2></div> |
| <div class="section"> |
| <h3><a name="Sqoop_Model"></a>Sqoop Model</h3> |
| <p>The default Sqoop modelling is available in org.apache.atlas.sqoop.model.SqoopDataModelGenerator. It defines the following types:</p> |
| <div class="source"> |
| <pre> |
| sqoop_operation_type(EnumType) - values [IMPORT, EXPORT, EVAL] |
| sqoop_dbstore_usage(EnumType) - values [TABLE, QUERY, PROCEDURE, OTHER] |
| sqoop_process(ClassType) - super types [Process] - attributes [name, operation, dbStore, hiveTable, commandlineOpts, startTime, endTime, userName] |
| sqoop_dbdatastore(ClassType) - super types [DataSet] - attributes [name, dbStoreType, storeUse, storeUri, source, description, ownerName] |
| |
| </pre></div> |
| <p>The entities are created and de-duped using unique qualified name. They provide namespace and can be used for querying as well: sqoop_process - attribute name - sqoop-dbStoreType-storeUri-endTime sqoop_dbdatastore - attribute name - dbStoreType-connectorUrl-source</p></div> |
| <div class="section"> |
| <h3><a name="Sqoop_Hook"></a>Sqoop Hook</h3> |
| <p>Sqoop added a SqoopJobDataPublisher that publishes data to Atlas after completion of import Job. Today, only hiveImport is supported in sqoopHook. This is used to add entities in Atlas using the model defined in org.apache.atlas.sqoop.model.SqoopDataModelGenerator. Follow these instructions in your sqoop set-up to add sqoop hook for Atlas in <sqoop-conf>/sqoop-site.xml:</p> |
| <p></p> |
| <ul> |
| <li>Sqoop Job publisher class. Currently only one publishing class is supported</li></ul><property> <name>sqoop.job.data.publish.class</name> <value>org.apache.atlas.sqoop.hook.SqoopHook</value> </property> |
| <ul> |
| <li>Atlas cluster name</li></ul><property> <name>atlas.cluster.name</name> <value><clustername></value> </property> |
| <ul> |
| <li>Copy <atlas-conf>/atlas-application.properties to to the sqoop conf directory <sqoop-conf>/</li> |
| <li>Link <atlas-home>/hook/sqoop/*.jar in sqoop lib</li></ul> |
| <p>Refer <a href="./Configuration.html">Configuration</a> for notification related configurations</p></div> |
| <div class="section"> |
| <h3><a name="Limitations"></a>Limitations</h3> |
| <p></p> |
| <ul> |
| <li>Only the following sqoop operations are captured by sqoop hook currently - hiveImport</li></ul></div> |
| </div> |
| </div> |
| |
| <hr/> |
| |
| <footer> |
| <div class="container"> |
| <div class="row span12">Copyright © 2015-2017 |
| <a href="http://www.apache.org">Apache Software Foundation</a>. |
| All Rights Reserved. |
| |
| </div> |
| |
| |
| <p id="poweredBy" class="pull-right"> |
| <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"> |
| <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> |
| </a> |
| </p> |
| |
| </div> |
| </footer> |
| </body> |
| </html> |
