| <div role="region" aria-label="Skip to main content"><a class="skipToContent_fXgn" href="#docusaurus_skipToContent_fallback">Skip to main content</a></div><nav class="navbar navbar--fixed-top"><div class="navbar__inner"><div class="navbar__items"><button aria-label="Toggle navigation bar" aria-expanded="false" class="navbar__toggle clean-btn" type="button"><svg width="30" height="30" viewBox="0 0 30 30" aria-hidden="true"><path stroke="currentColor" stroke-linecap="round" stroke-miterlimit="10" stroke-width="2" d="M4 7h22M4 15h22M4 23h22"></path></svg></button><a class="navbar__brand" href="/zh-CN/"><div class="navbar__logo"><img src="/zh-CN/img/logo.png" alt="Apache Linkis Logo" class="themedImage_ToTc themedImage--light_HNdA"><img src="/zh-CN/img/logo.png" alt="Apache Linkis Logo" class="themedImage_ToTc themedImage--dark_i4oU"></div><b class="navbar__title text--truncate">Apache Linkis</b></a><a class="navbar__item navbar__link" href="/zh-CN/download/main">下载</a><a class="navbar__item navbar__link" href="/zh-CN/community/how-to-subscribe">社区</a><a class="navbar__item navbar__link" href="/zh-CN/blog">博客</a><a class="navbar__item navbar__link" href="/zh-CN/team">团队</a><a class="navbar__item navbar__link" href="/zh-CN/user">用户</a><a href="https://docs.qq.com/doc/DUkdTTGhVSlZ0VXVt" target="_blank" rel="noopener noreferrer" class="navbar__item navbar__link">FAQ</a><div class="navbar__item dropdown dropdown--hoverable"><a href="#" aria-haspopup="true" aria-expanded="false" role="button" class="navbar__link">Apache</a><ul class="dropdown__menu"><li><a href="https://www.apache.org/" target="_blank" rel="noopener noreferrer" class="dropdown__link">基金会</a></li><li><a href="https://www.apache.org/licenses/" target="_blank" rel="noopener noreferrer" class="dropdown__link">证书</a></li><li><a href="https://www.apache.org/events/current-event" target="_blank" rel="noopener noreferrer" class="dropdown__link">事件</a></li><li><a href="https://www.apache.org/security/" target="_blank" rel="noopener noreferrer" class="dropdown__link">安全</a></li><li><a href="https://www.apache.org/foundation/sponsorship.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">赞助</a></li><li><a href="https://privacy.apache.org/policies/privacy-policy-public.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">Privacy</a></li><li><a href="https://www.apache.org/foundation/thanks.html" target="_blank" rel="noopener noreferrer" class="dropdown__link">致谢</a></li></ul></div></div><div class="navbar__items navbar__items--right"><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a href="#" aria-haspopup="true" aria-expanded="false" role="button" class="navbar__link">文档</a><ul class="dropdown__menu"><li><a class="dropdown__link" href="/zh-CN/docs/latest/about/introduction">1.5.0</a></li><li><a class="dropdown__link" href="/zh-CN/docs/1.4.0/about/introduction">1.4.0</a></li><li><a class="dropdown__link" href="/zh-CN/docs/1.3.2/about/introduction">1.3.2</a></li><li><a class="dropdown__link" href="/zh-CN/docs/1.6.0/about/introduction">Next(1.6.0)</a></li><li><a class="dropdown__link" href="/zh-CN/versions">All Version</a></li></ul></div><a href="https://github.com/apache/linkis" target="_blank" rel="noopener noreferrer" class="navbar__item navbar__link header-github-link" aria-label="GitHub"></a><div class="navbar__item dropdown dropdown--hoverable dropdown--right"><a href="#" aria-haspopup="true" aria-expanded="false" role="button" class="navbar__link"><svg viewBox="0 0 24 24" width="20" height="20" aria-hidden="true" class="iconLanguage_nlXk"><path fill="currentColor" d="M12.87 15.07l-2.54-2.51.03-.03c1.74-1.94 2.98-4.17 3.71-6.53H17V4h-7V2H8v2H1v1.99h11.17C11.5 7.92 10.44 9.75 9 11.35 8.07 10.32 7.3 9.19 6.69 8h-2c.73 1.63 1.73 3.17 2.98 4.56l-5.09 5.02L4 19l5-5 3.11 3.11.76-2.04zM18.5 10h-2L12 22h2l1.12-3h4.75L21 22h2l-4.5-12zm-2.62 7l1.62-4.33L19.12 17h-3.24z"></path></svg>简体中文</a><ul class="dropdown__menu"><li><a href="/docs/1.2.0/engine-usage/sqoop" target="_self" rel="noopener noreferrer" class="dropdown__link" lang="en">English</a></li><li><a href="/zh-CN/docs/1.2.0/engine-usage/sqoop" target="_self" rel="noopener noreferrer" class="dropdown__link dropdown__link--active" lang="zh-CN">简体中文</a></li></ul></div><div class="searchBox_ZlJk"><button type="button" class="DocSearch DocSearch-Button" aria-label="搜索"><span class="DocSearch-Button-Container"><svg width="20" height="20" class="DocSearch-Search-Icon" viewBox="0 0 20 20"><path d="M14.386 14.386l4.0877 4.0877-4.0877-4.0877c-2.9418 2.9419-7.7115 2.9419-10.6533 0-2.9419-2.9418-2.9419-7.7115 0-10.6533 2.9418-2.9419 7.7115-2.9419 10.6533 0 2.9419 2.9418 2.9419 7.7115 0 10.6533z" stroke="currentColor" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"></path></svg><span class="DocSearch-Button-Placeholder">搜索</span></span><span class="DocSearch-Button-Keys"></span></button></div></div></div><div role="presentation" class="navbar-sidebar__backdrop"></div></nav><div id="docusaurus_skipToContent_fallback" class="main-wrapper mainWrapper_z2l0 docsWrapper_BCFX"><button aria-label="回到顶部" class="clean-btn theme-back-to-top-button backToTopButton_sjWU" type="button"></button><div class="docPage__5DB"><aside class="theme-doc-sidebar-container docSidebarContainer_b6E3"><div class="sidebar_njMd"><nav class="menu thin-scrollbar menu_SIkG"><ul class="theme-doc-sidebar-menu menu__list"><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-1 menu__list-item"><a class="menu__link" href="/zh-CN/docs/1.2.0/introduction">Linkis 简述</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-1 menu__list-item"><a class="menu__link" href="/zh-CN/docs/1.2.0/release">版本总览</a></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/zh-CN/docs/1.2.0/deployment/quick-deploy">安装部署</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/zh-CN/docs/1.2.0/user-guide/overview">用户手册</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret menu__link--active" aria-expanded="true" href="/zh-CN/docs/1.2.0/engine-usage/overview">引擎使用</a></div><ul style="display:block;overflow:visible;height:auto" class="menu__list"><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/zh-CN/docs/1.2.0/engine-usage/overview">总览</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/zh-CN/docs/1.2.0/engine-usage/spark">Spark 引擎</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/zh-CN/docs/1.2.0/engine-usage/hive">Hive 引擎</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/zh-CN/docs/1.2.0/engine-usage/python">Python 引擎</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/zh-CN/docs/1.2.0/engine-usage/shell">Shell 引擎</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/zh-CN/docs/1.2.0/engine-usage/jdbc">JDBC 引擎</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/zh-CN/docs/1.2.0/engine-usage/flink">Flink 引擎</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/zh-CN/docs/1.2.0/engine-usage/openlookeng">openLooKeng 引擎</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link menu__link--active" aria-current="page" tabindex="0" href="/zh-CN/docs/1.2.0/engine-usage/sqoop">Sqoop 引擎</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/zh-CN/docs/1.2.0/engine-usage/pipeline">Pipeline 引擎</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/zh-CN/docs/1.2.0/engine-usage/elasticsearch">ElasticSearch 引擎</a></li><li class="theme-doc-sidebar-item-link theme-doc-sidebar-item-link-level-2 menu__list-item"><a class="menu__link" tabindex="0" href="/zh-CN/docs/1.2.0/engine-usage/presto">Presto 引擎</a></li></ul></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/zh-CN/docs/1.2.0/api/overview">API文档</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/zh-CN/docs/1.2.0/table/udf-table">表结构</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/zh-CN/docs/1.2.0/architecture/overview">技术架构</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/zh-CN/docs/1.2.0/development/linkis-compile-and-package">开发文档</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/zh-CN/docs/1.2.0/upgrade/upgrade-from-0.X-to-1.0-guide">版本升级</a></div></li><li class="theme-doc-sidebar-item-category theme-doc-sidebar-item-category-level-1 menu__list-item menu__list-item--collapsed"><div class="menu__list-item-collapsible"><a class="menu__link menu__link--sublist menu__link--sublist-caret" aria-expanded="false" href="/zh-CN/docs/1.2.0/tuning-and-troubleshooting/overview">调优排障</a></div></li></ul></nav></div></aside><main class="docMainContainer_gTbr"><div class="container padding-top--md padding-bottom--lg"><div class="row"><div class="col docItemCol_VOVn"><div class="theme-doc-version-banner alert alert--warning margin-bottom--md" role="alert"><div>This is documentation for <!-- -->Apache Linkis<!-- --> <b>1.2.0</b>, 非最新版本。</div><div class="margin-top--md">有关最新文档，请参阅最新稳定版本<b><a href="/zh-CN/docs/latest/engine-usage/sqoop">latest version</a></b> (<!-- -->1.5.0<!-- -->).</div></div><div class="docItemContainer_Djhp"><article><nav class="theme-doc-breadcrumbs breadcrumbsContainer_Z_bl" aria-label="页面路径"><ul class="breadcrumbs" itemscope="" itemtype="https://schema.org/BreadcrumbList"><li class="breadcrumbs__item"><a aria-label="主页面" class="breadcrumbs__link" href="/zh-CN/"><svg viewBox="0 0 24 24" class="breadcrumbHomeIcon_OVgt"><path d="M10 19v-5h4v5c0 .55.45 1 1 1h3c.55 0 1-.45 1-1v-7h1.7c.46 0 .68-.57.33-.87L12.67 3.6c-.38-.34-.96-.34-1.34 0l-8.36 7.53c-.34.3-.13.87.33.87H5v7c0 .55.45 1 1 1h3c.55 0 1-.45 1-1z" fill="currentColor"></path></svg></a></li><li class="breadcrumbs__item"><span class="breadcrumbs__link">引擎使用</span><meta itemprop="position" content="1"></li><li itemscope="" itemprop="itemListElement" itemtype="https://schema.org/ListItem" class="breadcrumbs__item breadcrumbs__item--active"><span class="breadcrumbs__link" itemprop="name">Sqoop 引擎</span><meta itemprop="position" content="2"></li></ul></nav><span class="theme-doc-version-badge badge badge--secondary">版本：1.2.0</span><div class="tocCollapsible_ETCw theme-doc-toc-mobile tocMobile_ITEo"><button type="button" class="clean-btn tocCollapsibleButton_TO0P">On this page</button></div><div class="theme-doc-markdown markdown"><header><h1>Sqoop 引擎</h1></header><blockquote><p>本文主要介绍Sqoop(>=1.1.2 版本支持)引擎的配置、部署和使用。</p></blockquote><h2 class="anchor anchorWithStickyNavbar_LWe7" id="1-sqoop引擎linkis系统参数配置">1 Sqoop引擎Linkis系统参数配置<a class="hash-link" href="#1-sqoop引擎linkis系统参数配置" title="Direct link to heading"></a></h2><p>Sqoop引擎主要依赖Hadoop基础环境，如果该节点需要部署Sqoop引擎，需要部署Hadoop客户端环境。</p><p>强烈建议您在执行Sqoop任务之前，先在该节点使用原生的Sqoop执行测试任务，以检测该节点环境是否正常。</p><div class="language-shell codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-shell codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">#验证sqoop环境是否可用 参考示例：将hdfs的/user/hive/warehouse/hadoop/test_linkis_sqoop文件数据导入到mysql表 test_sqoop中</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">sqoop </span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">export</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--connect jdbc:mysql://127.0.0.1/test </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--username </span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">test</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--password test123</span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--table test_sqoop </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--columns user_id,user_code,user_name,email,status </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--export-dir /user/hive/warehouse/hadoop/test_linkis_sqoop </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--update-mode allowinsert </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--verbose </span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><table><thead><tr><th>环境变量名</th><th>环境变量内容</th><th>备注</th></tr></thead><tbody><tr><td>JAVA_HOME</td><td>JDK安装路径</td><td>必须</td></tr><tr><td>HADOOP_HOME</td><td>Hadoop安装路径</td><td>必须</td></tr><tr><td>HADOOP_CONF_DIR</td><td>Hadoop配置路径</td><td>必须</td></tr><tr><td>SQOOP_HOME</td><td>Sqoop安装路径</td><td>必须</td></tr><tr><td>SQOOP_CONF_DIR</td><td>Sqoop配置路径</td><td>非必须</td></tr><tr><td>HCAT_HOME</td><td>HCAT配置路径</td><td>非必须</td></tr><tr><td>HBASE_HOME</td><td>HBASE配置路径</td><td>非必须</td></tr></tbody></table><p>表1-1 环境配置清单</p><table><thead><tr><th>Linkis系统参数</th><th>参数</th><th>备注</th></tr></thead><tbody><tr><td>wds.linkis.hadoop.site.xml</td><td>设置sqoop加载hadoop参数文件位置</td><td>一般不需要单独配置，默认值"core-site.xml;hdfs-site.xml;yarn-site.xml;mapred-site.xml"</td></tr><tr><td>sqoop.fetch.status.interval</td><td>设置获取sqoop执行状态的间隔时间</td><td>一般不需要单独配置，默认值为5s</td></tr></tbody></table><h2 class="anchor anchorWithStickyNavbar_LWe7" id="2-sqoop引擎的配置和部署">2 Sqoop引擎的配置和部署<a class="hash-link" href="#2-sqoop引擎的配置和部署" title="Direct link to heading"></a></h2><h3 class="anchor anchorWithStickyNavbar_LWe7" id="21-sqoop版本的选择和编译">2.1 Sqoop版本的选择和编译<a class="hash-link" href="#21-sqoop版本的选择和编译" title="Direct link to heading"></a></h3><p>Linkis 1.1.2及以上支持的主流Sqoop版本1.4.6与1.4.7，更高版本可能需要修改部分代码重新编译。</p><h3 class="anchor anchorWithStickyNavbar_LWe7" id="22-sqoop-engineconn部署和加载">2.2 Sqoop engineConn部署和加载<a class="hash-link" href="#22-sqoop-engineconn部署和加载" title="Direct link to heading"></a></h3><p>注意: 编译sqoop引擎之前需要进行linkis项目全量编译 </p><div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">单独编译sqoop的方式</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">${linkis_code_dir}/linkis-engineconn-plugins/sqoop/</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">mvn clean install</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>安装方式是将编译出来的引擎包,位置在</p><div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token variable" style="color:rgb(189, 147, 249);font-style:italic">${linkis_code_dir}</span><span class="token plain">/linkis-enginepconn-plugins/engineconn-plugins/sqoop/target/sqoop-engineconn.zip</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>然后上传部署到linkis服务器</p><div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token variable" style="color:rgb(189, 147, 249);font-style:italic">${LINKIS_HOME}</span><span class="token plain">/lib/linkis-engineplugins</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>并重启linkis-engineplugin</p><div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token builtin class-name" style="color:rgb(189, 147, 249)">cd</span><span class="token plain"> </span><span class="token variable" style="color:rgb(189, 147, 249);font-style:italic">${LINKIS_HOME}</span><span class="token plain">/sbin</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token function" style="color:rgb(80, 250, 123)">sh</span><span class="token plain"> linkis-daemon.sh restart linkis-cg-linkismanager</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>engineplugin更详细的介绍可以参看下面的文章。</p><p><a href="https://linkis.apache.org/zh-CN/docs/latest/deployment/engine-conn-plugin-installation" target="_blank" rel="noopener noreferrer">https://linkis.apache.org/zh-CN/docs/latest/deployment/engine-conn-plugin-installation</a></p><h2 class="anchor anchorWithStickyNavbar_LWe7" id="3-sqoop引擎的使用">3 Sqoop引擎的使用<a class="hash-link" href="#3-sqoop引擎的使用" title="Direct link to heading"></a></h2><h3 class="anchor anchorWithStickyNavbar_LWe7" id="31-通过linkis-cli进行任务提交">3.1 通过Linkis-cli进行任务提交<a class="hash-link" href="#31-通过linkis-cli进行任务提交" title="Direct link to heading"></a></h3><p>hdfs文件导出到mysql</p><div class="language-shell codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-shell codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token function" style="color:rgb(80, 250, 123)">sh</span><span class="token plain"> linkis-cli-sqoop </span><span class="token builtin class-name" style="color:rgb(189, 147, 249)">export</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">-D mapreduce.job.queuename</span><span class="token operator">=</span><span class="token plain">ide </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--connect jdbc:mysql://127.0.0.1:9600/testdb </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--username password@123 </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--password password@123 </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--table test_sqoop_01_copy </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--columns user_id,user_code,user_name,email,status </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--export-dir /user/hive/warehouse/hadoop/test_linkis_sqoop_2 </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--update-mode allowinsert --verbose </span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><span class="token plain"> </span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p>mysql数据导入到hive库</p><div class="language-shell codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-shell codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain">mysql导入到hive 库linkis_test_ind.test_import_sqoop_1,表test_import_sqoop_1不存在 需要添加参数 --create-hive-table </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token function" style="color:rgb(80, 250, 123)">sh</span><span class="token plain"> linkis-cli-sqoop </span><span class="token function" style="color:rgb(80, 250, 123)">import</span><span class="token plain"> -D mapreduce.job.queuename</span><span class="token operator">=</span><span class="token plain">dws </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--connect jdbc:mysql://127.0.0.1:3306/casion_test </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--username hadoop </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--password password@123 </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--table test_sqoop_01 </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--columns user_id,user_code,user_name,email,status </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--fields-terminated-by </span><span class="token string" style="color:rgb(255, 121, 198)">','</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--hive-import --create-hive-table </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--hive-database casionxia_ind </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--hive-table test_import_sqoop_1 </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--hive-drop-import-delims </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--delete-target-dir </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--input-null-non-string </span><span class="token string" style="color:rgb(255, 121, 198)">'\\N'</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--input-null-string </span><span class="token string" style="color:rgb(255, 121, 198)">'\\N'</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--verbose </span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">mysql导入到hive 库linkis_test_ind.test_import_sqoop_1,表test_import_sqoop_1存在 移除参数--create-hive-table </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token function" style="color:rgb(80, 250, 123)">sh</span><span class="token plain"> linkis-cli-sqoop </span><span class="token function" style="color:rgb(80, 250, 123)">import</span><span class="token plain"> -D mapreduce.job.queuename</span><span class="token operator">=</span><span class="token plain">dws </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--connect jdbc:mysql://127.0.0.1:9600/testdb </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--username testdb </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--password password@123 </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--table test_sqoop_01 </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--columns user_id,user_code,user_name,email,status </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--fields-terminated-by </span><span class="token string" style="color:rgb(255, 121, 198)">','</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--hive-import </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--hive-database linkis_test_ind </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--hive-table test_import_sqoop_1 </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--hive-overwrite </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--hive-drop-import-delims </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--delete-target-dir </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--input-null-non-string </span><span class="token string" style="color:rgb(255, 121, 198)">'\\N'</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--input-null-string </span><span class="token string" style="color:rgb(255, 121, 198)">'\\N'</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(248, 248, 242)">\</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">--verbose </span><span class="token punctuation" style="color:rgb(248, 248, 242)">;</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h3 class="anchor anchorWithStickyNavbar_LWe7" id="31-onceengineconn方式">3.1 OnceEngineConn方式<a class="hash-link" href="#31-onceengineconn方式" title="Direct link to heading"></a></h3><p>OnceEngineConn的使用方式是通过LinkisManagerClient调用LinkisManager的createEngineConn的接口，并将代码发给创建的Sqoop引擎，然后Sqoop引擎就开始执行，此方式可以被其他系统进行调用，比如Exchangis。Client的使用方式也很简单，首先新建一个maven项目，或者在您的项目中引入以下的依赖</p><div class="language-xml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-xml codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token tag punctuation" style="color:rgb(248, 248, 242)"><</span><span class="token tag" style="color:rgb(255, 121, 198)">dependency</span><span class="token tag punctuation" style="color:rgb(248, 248, 242)">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token tag punctuation" style="color:rgb(248, 248, 242)"><</span><span class="token tag" style="color:rgb(255, 121, 198)">groupId</span><span class="token tag punctuation" style="color:rgb(248, 248, 242)">></span><span class="token plain">org.apache.linkis</span><span class="token tag punctuation" style="color:rgb(248, 248, 242)"></</span><span class="token tag" style="color:rgb(255, 121, 198)">groupId</span><span class="token tag punctuation" style="color:rgb(248, 248, 242)">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token tag punctuation" style="color:rgb(248, 248, 242)"><</span><span class="token tag" style="color:rgb(255, 121, 198)">artifactId</span><span class="token tag punctuation" style="color:rgb(248, 248, 242)">></span><span class="token plain">linkis-computation-client</span><span class="token tag punctuation" style="color:rgb(248, 248, 242)"></</span><span class="token tag" style="color:rgb(255, 121, 198)">artifactId</span><span class="token tag punctuation" style="color:rgb(248, 248, 242)">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> </span><span class="token tag punctuation" style="color:rgb(248, 248, 242)"><</span><span class="token tag" style="color:rgb(255, 121, 198)">version</span><span class="token tag punctuation" style="color:rgb(248, 248, 242)">></span><span class="token plain">${linkis.version}</span><span class="token tag punctuation" style="color:rgb(248, 248, 242)"></</span><span class="token tag" style="color:rgb(255, 121, 198)">version</span><span class="token tag punctuation" style="color:rgb(248, 248, 242)">></span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token tag punctuation" style="color:rgb(248, 248, 242)"></</span><span class="token tag" style="color:rgb(255, 121, 198)">dependency</span><span class="token tag punctuation" style="color:rgb(248, 248, 242)">></span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><p><strong>测试用例：</strong></p><div class="language-scala codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:#282A36"><div class="codeBlockContent_biex"><pre tabindex="0" class="prism-code language-scala codeBlock_bY9V thin-scrollbar"><code class="codeBlockLines_e6Vv"><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">import java.util.concurrent.TimeUnit</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">import java.util</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">import org.apache.linkis.computation.client.LinkisJobBuilder</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">import org.apache.linkis.computation.client.once.simple.{SimpleOnceJob, SimpleOnceJobBuilder, SubmittableSimpleOnceJob}</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">import org.apache.linkis.computation.client.operator.impl.{EngineConnLogOperator, EngineConnMetricsOperator, EngineConnProgressOperator}</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">import org.apache.linkis.computation.client.utils.LabelKeyUtils</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">import scala.collection.JavaConverters._</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">object SqoopOnceJobTest extends App {</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> LinkisJobBuilder.setDefaultServerUrl("http://127.0.0.1:9001")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> val logPath = "C:\\Users\\resources\\log4j.properties"</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> System.setProperty("log4j.configurationFile", logPath)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> val startUpMap = new util.HashMap[String, Any]</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> startUpMap.put("wds.linkis.engineconn.java.driver.memory", "1g")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> val builder = SimpleOnceJob.builder().setCreateService("Linkis-Client")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .addLabel(LabelKeyUtils.ENGINE_TYPE_LABEL_KEY, "sqoop-1.4.6")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .addLabel(LabelKeyUtils.USER_CREATOR_LABEL_KEY, "Client")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .addLabel(LabelKeyUtils.ENGINE_CONN_MODE_LABEL_KEY, "once")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .setStartupParams(startUpMap)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .setMaxSubmitTime(30000)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .addExecuteUser("freeuser")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> val onceJob = importJob(builder)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> val time = System.currentTimeMillis()</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> onceJob.submit()</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> println(onceJob.getId)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> val logOperator = onceJob.getOperator(EngineConnLogOperator.OPERATOR_NAME).asInstanceOf[EngineConnLogOperator]</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> println(onceJob.getECMServiceInstance)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> logOperator.setFromLine(0)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> logOperator.setECMServiceInstance(onceJob.getECMServiceInstance)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> logOperator.setEngineConnType("sqoop")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> logOperator.setIgnoreKeywords("[main],[SpringContextShutdownHook]")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> var progressOperator = onceJob.getOperator(EngineConnProgressOperator.OPERATOR_NAME).asInstanceOf[EngineConnProgressOperator]</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> var metricOperator = onceJob.getOperator(EngineConnMetricsOperator.OPERATOR_NAME).asInstanceOf[EngineConnMetricsOperator]</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> var end = false</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> var rowBefore = 1</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> while (!end || rowBefore > 0){</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> if(onceJob.isCompleted) {</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> end = true</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> metricOperator = null</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> }</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> logOperator.setPageSize(100)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> Utils.tryQuietly{</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> val logs = logOperator.apply()</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> logs.logs.asScala.foreach( log => {</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> println(log)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> })</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> rowBefore = logs.logs.size</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> }</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> Thread.sleep(3000)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> Option(metricOperator).foreach( operator => {</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> if (!onceJob.isCompleted){</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> println(s"Metric监控: ${operator.apply()}")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> println(s"进度: ${progressOperator.apply()}")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> }</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> })</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> }</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> onceJob.isCompleted</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> onceJob.waitForCompleted()</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> println(onceJob.getStatus)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> println(TimeUnit.SECONDS.convert(System.currentTimeMillis() - time, TimeUnit.MILLISECONDS) + "s")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> System.exit(0)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> def importJob(jobBuilder: SimpleOnceJobBuilder): SubmittableSimpleOnceJob = {</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> jobBuilder</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .addJobContent("sqoop.env.mapreduce.job.queuename", "queue_10")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .addJobContent("sqoop.mode", "import")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .addJobContent("sqoop.args.connect", "jdbc:mysql://127.0.0.1:3306/exchangis")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .addJobContent("sqoop.args.username", "free")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .addJobContent("sqoop.args.password", "testpwd")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .addJobContent("sqoop.args.query", "select id as order_number, sno as time from" +</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> " exchangis where sno =1 and $CONDITIONS")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .addJobContent("sqoop.args.hcatalog.database", "freedb")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .addJobContent("sqoop.args.hcatalog.table", "zy_test")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .addJobContent("sqoop.args.hcatalog.partition.keys", "month")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .addJobContent("sqoop.args.hcatalog.partition.values", "3")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .addJobContent("sqoop.args.num.mappers", "1")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .build()</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> }</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> def exportJob(jobBuilder: SimpleOnceJobBuilder): SubmittableSimpleOnceJob = {</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> jobBuilder</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .addJobContent("sqoop.env.mapreduce.job.queuename", "queue1")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .addJobContent("sqoop.mode", "import")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .addJobContent("sqoop.args.connect", "jdbc:mysql://127.0.0.1:3306/exchangis")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .addJobContent("sqoop.args.query", "select id as order, sno as great_time from" +</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> " exchangis_table where sno =1 and $CONDITIONS")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .addJobContent("sqoop.args.hcatalog.database", "hadoop")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .addJobContent("sqoop.args.hcatalog.table", "partition_33")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .addJobContent("sqoop.args.hcatalog.partition.keys", "month")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .addJobContent("sqoop.args.hcatalog.partition.values", "4")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .addJobContent("sqoop.args.num.mappers", "1")</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> .build()</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> }</span><br></span></code></pre><div class="buttonGroup__atx"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_eSgA" aria-hidden="true"><svg class="copyButtonIcon_y97N" viewBox="0 0 24 24"><path d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"></path></svg><svg class="copyButtonSuccessIcon_LjdS" viewBox="0 0 24 24"><path d="M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"></path></svg></span></button></div></div></div><h2 class="anchor anchorWithStickyNavbar_LWe7" id="4-参数">4 参数<a class="hash-link" href="#4-参数" title="Direct link to heading"></a></h2><h3 class="anchor anchorWithStickyNavbar_LWe7" id="41-common-arguments">4.1 Common arguments<a class="hash-link" href="#41-common-arguments" title="Direct link to heading"></a></h3><table><thead><tr><th>参数</th><th>key</th><th>说明</th></tr></thead><tbody><tr><td></td><td>sqoop.mode</td><td>import/export/…</td></tr><tr><td>-Dmapreduce.job.queuename</td><td>sqoop.env.mapreduce.job.queuename</td><td></td></tr><tr><td>-<!-- -->-connect <jdbc-uri<!-- -->></td><td>sqoop.args.connect</td><td>Specify JDBC connect string</td></tr><tr><td>-<!-- -->-connection-manager <class-name<!-- -->></td><td>sqoop.args.connection.manager</td><td>Specify connection manager class name</td></tr><tr><td>-<!-- -->-connection-param-file <properties-file<!-- -->></td><td>sqoop.args.connection.param.file</td><td>Specify connection parameters file</td></tr><tr><td>-<!-- -->-driver <class-name<!-- -->></td><td>sqoop.args.driver</td><td>Manually specify JDBC driver class to use</td></tr><tr><td>-<!-- -->-hadoop-home <hdir<!-- -->></td><td>sqoop.args.hadoop.home</td><td>Override $HADOOP<!-- -->_<!-- -->MAPRED<!-- -->_<!-- -->HOME<!-- -->_<!-- -->ARG</td></tr><tr><td>-<!-- -->-hadoop-mapred-home <dir<!-- -->></td><td>sqoop.args.hadoop.mapred.home</td><td>Override $HADOOP<!-- -->_<!-- -->MAPRED<!-- -->_<!-- -->HOME<!-- -->_<!-- -->ARG</td></tr><tr><td>-<!-- -->-help</td><td>sqoop.args.help</td><td>Print usage instructions</td></tr><tr><td>-<!-- -->P</td><td></td><td>Read password from console</td></tr><tr><td>-<!-- -->-password <password<!-- -->></td><td>sqoop.args.password</td><td>Set authentication password</td></tr><tr><td>-<!-- -->-password-alias <password-alias<!-- -->></td><td>sqoop.args.password.alias</td><td>Credential provider password alias</td></tr><tr><td>-<!-- -->-password-file <password-file<!-- -->></td><td>sqoop.args.password.file</td><td>Set authentication password file path</td></tr><tr><td>-<!-- -->-relaxed-isolation</td><td>sqoop.args.relaxed.isolation</td><td>Use read-uncommitted isolation for imports</td></tr><tr><td>-<!-- -->-skip-dist-cache</td><td>sqoop.args.skip.dist.cache</td><td>Skip copying jars to distributed cache</td></tr><tr><td>-<!-- -->-username <username<!-- -->></td><td>sqoop.args.username</td><td>Set authentication username</td></tr><tr><td>-<!-- -->-verbose</td><td>sqoop.args.verbose</td><td>Print more information while working</td></tr><tr><td></td><td></td><td></td></tr></tbody></table><h3 class="anchor anchorWithStickyNavbar_LWe7" id="42-export-import-arguments">4.2 Export import arguments<a class="hash-link" href="#42-export-import-arguments" title="Direct link to heading"></a></h3><table><thead><tr><th>参数</th><th>key</th><th>说明</th></tr></thead><tbody><tr><td>-<!-- -->-batch</td><td>sqoop.args.batch</td><td> Indicates underlying statements to be executed in batch mode</td></tr><tr><td>-<!-- -->-call <arg<!-- -->></td><td>sqoop.args.call</td><td>Populate the table using this stored procedure (one call per row)</td></tr><tr><td>-<!-- -->-clear-staging-table</td><td>sqoop.args.clear.staging.table</td><td>Indicates that any data in staging table can be deleted</td></tr><tr><td>-<!-- -->-columns <col,col,col...<!-- -->></td><td>sqoop.args.columns</td><td>Columns to export to table</td></tr><tr><td>-<!-- -->-direct</td><td>sqoop.args.direct</td><td>Use direct export fast path</td></tr><tr><td>-<!-- -->-export-dir <dir<!-- -->></td><td>sqoop.args.export.dir</td><td>HDFS source path for the export</td></tr><tr><td>-<!-- -->m,--num-mappers <n<!-- -->></td><td>sqoop.args.num.mappers</td><td>Use 'n' map tasks to export in parallel</td></tr><tr><td>-<!-- -->-mapreduce-job-name <name<!-- -->></td><td>sqoop.args.mapreduce.job.name</td><td>Set name for generated mapreduce job</td></tr><tr><td>-<!-- -->-staging-table <table-name<!-- -->></td><td>sqoop.args.staging.table</td><td>Intermediate staging table</td></tr><tr><td>-<!-- -->-table <table-name<!-- -->></td><td>sqoop.args.table</td><td>Table to populate</td></tr><tr><td>-<!-- -->-update-key <key<!-- -->></td><td>sqoop.args.update.key</td><td>Update records by specified key column</td></tr><tr><td>-<!-- -->-update-mode <mode<!-- -->></td><td>sqoop.args.update.mode</td><td>Specifies how updates are performed when new rows are found with non-matching keys in database</td></tr><tr><td>-<!-- -->-validate</td><td>sqoop.args.validate</td><td>Validate the copy using the configured validator</td></tr><tr><td>-<!-- -->-validation-failurehandler <validation-failurehandler<!-- -->></td><td>sqoop.args.validation.failurehandler</td><td>Validate the copy using the configured validator</td></tr><tr><td>-<!-- -->-validation-threshold <validation-threshold<!-- -->></td><td>sqoop.args.validation.threshold</td><td> Fully qualified class name for ValidationThreshold</td></tr><tr><td>-<!-- -->-validator <validator<!-- -->></td><td>sqoop.args.validator</td><td>Fully qualified class name for the Validator</td></tr><tr><td></td><td></td><td></td></tr></tbody></table><h3 class="anchor anchorWithStickyNavbar_LWe7" id="43-import-control-arguments">4.3 Import control arguments<a class="hash-link" href="#43-import-control-arguments" title="Direct link to heading"></a></h3><table><thead><tr><th>参数</th><th>key</th><th>说明</th></tr></thead><tbody><tr><td>-<!-- -->-append</td><td>sqoop.args.append</td><td>Imports data in append mode</td></tr><tr><td>-<!-- -->-as-avrodatafile</td><td>sqoop.args.as.avrodatafile</td><td>Imports data to Avro data files</td></tr><tr><td>-<!-- -->-as-parquetfile</td><td>sqoop.args.as.parquetfile</td><td>Imports data to Parquet files</td></tr><tr><td>-<!-- -->-as-sequencefile</td><td>sqoop.args.as.sequencefile</td><td>Imports data to SequenceFiles</td></tr><tr><td>-<!-- -->-as-textfile</td><td>sqoop.args.as.textfile</td><td>Imports data as plain text (default)</td></tr><tr><td>-<!-- -->-autoreset-to-one-mapper</td><td>sqoop.args.autoreset.to.one.mapper</td><td>Reset the number of mappers to one mapper if no split key available</td></tr><tr><td>-<!-- -->-boundary-query <statement<!-- -->></td><td>sqoop.args.boundary.query</td><td>Set boundary query for retrieving max and min value of the primary key</td></tr><tr><td>-<!-- -->-case-insensitive</td><td>sqoop.args.case.insensitive</td><td>Data Base is case insensitive, split where condition transfrom to lower case!</td></tr><tr><td>-<!-- -->-columns <col,col,col...<!-- -->></td><td>sqoop.args.columns</td><td>Columns to import from table</td></tr><tr><td>-<!-- -->-compression-codec <codec<!-- -->></td><td>sqoop.args.compression.codec</td><td>Compression codec to use for import</td></tr><tr><td>-<!-- -->-delete-target-dir</td><td>sqoop.args.delete.target.dir</td><td>Imports data in delete mode</td></tr><tr><td>-<!-- -->-direct</td><td>sqoop.args.direct</td><td>Use direct import fast path</td></tr><tr><td>-<!-- -->-direct-split-size <n<!-- -->></td><td>sqoop.args.direct.split.size</td><td>Split the input stream every 'n' bytes when importing in direct mode</td></tr><tr><td>-<!-- -->e,--query <statement<!-- -->></td><td>sqoop.args.query</td><td>Import results of SQL 'statement'</td></tr><tr><td>-<!-- -->-fetch-size <n<!-- -->></td><td>sqoop.args.fetch.size</td><td>Set number 'n' of rows to fetch from the database when more rows are needed</td></tr><tr><td>-<!-- -->-inline-lob-limit <n<!-- -->></td><td>sqoop.args.inline.lob.limit</td><td>Set the maximum size for an inline LOB</td></tr><tr><td>-<!-- -->m,--num-mappers <n<!-- -->></td><td>sqoop.args.num.mappers</td><td>Use 'n' map tasks to import in parallel</td></tr><tr><td>-<!-- -->-mapreduce-job-name <name<!-- -->></td><td>sqoop.args.mapreduce.job.name</td><td>Set name for generated mapreduce job</td></tr><tr><td>-<!-- -->-merge-key <column<!-- -->></td><td>sqoop.args.merge.key</td><td>Key column to use to join results</td></tr><tr><td>-<!-- -->-split-by <column-name<!-- -->></td><td>sqoop.args.split.by</td><td>Column of the table used to split work units</td></tr><tr><td>-<!-- -->-table <table-name<!-- -->></td><td>sqoop.args.table</td><td>Table to read</td></tr><tr><td>-<!-- -->-target-dir <dir<!-- -->></td><td>sqoop.args.target.dir</td><td>HDFS plain table destination</td></tr><tr><td>-<!-- -->-validate</td><td>sqoop.args.validate</td><td>Validate the copy using the configured validator</td></tr><tr><td>-<!-- -->-validation-failurehandler <validation-failurehandler<!-- -->></td><td>sqoop.args.validation.failurehandler</td><td>Fully qualified class name for ValidationFa ilureHandler</td></tr><tr><td>-<!-- -->-validation-threshold <validation-threshold<!-- -->></td><td>sqoop.args.validation.threshold</td><td>Fully qualified class name for ValidationTh reshold</td></tr><tr><td>-<!-- -->-validator <validator<!-- -->></td><td>sqoop.args.validator</td><td>Fully qualified class name for the Validator</td></tr><tr><td>-<!-- -->-warehouse-dir <dir<!-- -->></td><td>sqoop.args.warehouse.dir</td><td>HDFS parent for table destination</td></tr><tr><td>-<!-- -->-where <where clause<!-- -->></td><td>sqoop.args.where</td><td>WHERE clause to use during import</td></tr><tr><td>-<!-- -->z,--compress</td><td>sqoop.args.compress</td><td>Enable compression</td></tr><tr><td></td><td></td><td></td></tr></tbody></table><h3 class="anchor anchorWithStickyNavbar_LWe7" id="44-incremental-import-argument">4.4 Incremental import argument<a class="hash-link" href="#44-incremental-import-argument" title="Direct link to heading"></a></h3><table><thead><tr><th>参数</th><th>key</th><th>说明</th></tr></thead><tbody><tr><td>-<!-- -->-check-column <column<!-- -->></td><td>sqoop.args.check.column</td><td>Source column to check for incremental change</td></tr><tr><td>-<!-- -->-incremental <import-type<!-- -->></td><td>sqoop.args.incremental</td><td>Define an incremental import of type 'append' or 'lastmodified'</td></tr><tr><td>-<!-- -->-last-value <value<!-- -->></td><td>sqoop.args.last.value</td><td>Last imported value in the incremental check column</td></tr><tr><td></td><td></td><td></td></tr></tbody></table><h3 class="anchor anchorWithStickyNavbar_LWe7" id="45-output-line-formatting-arguments">4.5 Output line formatting arguments<a class="hash-link" href="#45-output-line-formatting-arguments" title="Direct link to heading"></a></h3><table><thead><tr><th>参数</th><th>key</th><th>说明</th></tr></thead><tbody><tr><td>-<!-- -->-enclosed-by <char<!-- -->></td><td>sqoop.args.enclosed.by</td><td>Sets a required field enclosing character</td></tr><tr><td>-<!-- -->-escaped-by <char<!-- -->></td><td>sqoop.args.escaped.by</td><td>Sets the escape character</td></tr><tr><td>-<!-- -->-fields-terminated-by <char<!-- -->></td><td>sqoop.args.fields.terminated.by</td><td>Sets the field separator character</td></tr><tr><td>-<!-- -->-lines-terminated-by <char<!-- -->></td><td>sqoop.args.lines.terminated.by</td><td>Sets the end-of-line character</td></tr><tr><td>-<!-- -->-mysql-delimiters</td><td>sqoop.args.mysql.delimiters</td><td>Uses MySQL's default delimiter set: fields: , lines: <!-- -->\<!-- -->n escaped-by: <!-- -->\<!-- --> optionally-enclosed-by: '</td></tr><tr><td>-<!-- -->-optionally-enclosed-by <char<!-- -->></td><td>sqoop.args.optionally.enclosed.by</td><td>Sets a field enclosing character</td></tr><tr><td></td><td></td><td></td></tr></tbody></table><h3 class="anchor anchorWithStickyNavbar_LWe7" id="46-input-parsing-arguments">4.6 Input parsing arguments<a class="hash-link" href="#46-input-parsing-arguments" title="Direct link to heading"></a></h3><table><thead><tr><th>参数</th><th>key</th><th>说明</th></tr></thead><tbody><tr><td>-<!-- -->-input-enclosed-by <char<!-- -->></td><td>sqoop.args.input.enclosed.by</td><td>Sets a required field encloser</td></tr><tr><td>-<!-- -->-input-escaped-by <char<!-- -->></td><td>sqoop.args.input.escaped.by</td><td>Sets the input escape character</td></tr><tr><td>-<!-- -->-input-fields-terminated-by <char<!-- -->></td><td>sqoop.args.input.fields.terminated.by</td><td>Sets the input field separator</td></tr><tr><td>-<!-- -->-input-lines-terminated-by <char<!-- -->></td><td>sqoop.args.input.lines.terminated.by</td><td>Sets the input end-of-line char</td></tr><tr><td>-<!-- -->-input-optionally-enclosed-by <char<!-- -->></td><td>sqoop.args.input.optionally.enclosed.by</td><td>Sets a field enclosing character</td></tr><tr><td></td><td></td><td></td></tr></tbody></table><h3 class="anchor anchorWithStickyNavbar_LWe7" id="47-hive-arguments">4.7 Hive arguments<a class="hash-link" href="#47-hive-arguments" title="Direct link to heading"></a></h3><table><thead><tr><th>参数</th><th>key</th><th>说明</th></tr></thead><tbody><tr><td>-<!-- -->-create-hive-table</td><td>sqoop.args.create.hive.table</td><td>Fail if the target hive table exists</td></tr><tr><td>-<!-- -->-hive-database <database-name<!-- -->></td><td>sqoop.args.hive.database</td><td>Sets the database name to use when importing to hive</td></tr><tr><td>-<!-- -->-hive-delims-replacement <arg<!-- -->></td><td>sqoop.args.hive.delims.replacement</td><td>Replace Hive record <!-- -->\<!-- -->0x01 and row delimiters (<!-- -->\<!-- -->n<!-- -->\<!-- -->r) from imported string fields with user-defined string</td></tr><tr><td>-<!-- -->-hive-drop-import-delims</td><td>sqoop.args.hive.drop.import.delims</td><td>Drop Hive record <!-- -->\<!-- -->0x01 and row delimiters (<!-- -->\<!-- -->n<!-- -->\<!-- -->r) from imported string fields</td></tr><tr><td>-<!-- -->-hive-home <dir<!-- -->></td><td>sqoop.args.hive.home</td><td>Override $HIVE<!-- -->_<!-- -->HOME</td></tr><tr><td>-<!-- -->-hive-import</td><td>sqoop.args.hive.import</td><td>Import tables into Hive (Uses Hive's default delimiters if none are set.)</td></tr><tr><td>-<!-- -->-hive-overwrite</td><td>sqoop.args.hive.overwrite</td><td>Overwrite existing data in the Hive table</td></tr><tr><td>-<!-- -->-hive-partition-key <partition-key<!-- -->></td><td>sqoop.args.hive.partition.key</td><td>Sets the partition key to use when importing to hive</td></tr><tr><td>-<!-- -->-hive-partition-value <partition-value<!-- -->></td><td>sqoop.args.hive.partition.value</td><td>Sets the partition value to use when importing to hive</td></tr><tr><td>-<!-- -->-hive-table <table-name<!-- -->></td><td>sqoop.args.hive.table</td><td>Sets the table name to use when importing to hive</td></tr><tr><td>-<!-- -->-map-column-hive <arg<!-- -->></td><td>sqoop.args.map.column.hive</td><td>Override mapping for specific column to hive types.</td></tr></tbody></table><h3 class="anchor anchorWithStickyNavbar_LWe7" id="48-hbase-arguments">4.8 HBase arguments<a class="hash-link" href="#48-hbase-arguments" title="Direct link to heading"></a></h3><table><thead><tr><th>参数</th><th>key</th><th>说明</th></tr></thead><tbody><tr><td>-<!-- -->-column-family <family<!-- -->></td><td>sqoop.args.column.family</td><td>Sets the target column family for the import</td></tr><tr><td>-<!-- -->-hbase-bulkload</td><td>sqoop.args.hbase.bulkload</td><td>Enables HBase bulk loading</td></tr><tr><td>-<!-- -->-hbase-create-table</td><td>sqoop.args.hbase.create.table</td><td>If specified, create missing HBase tables</td></tr><tr><td>-<!-- -->-hbase-row-key <col<!-- -->></td><td>sqoop.args.hbase.row.key</td><td>Specifies which input column to use as the row key</td></tr><tr><td>-<!-- -->-hbase-table <table<!-- -->></td><td>sqoop.args.hbase.table</td><td>Import to <table<!-- -->><!-- -->in HBase</td></tr><tr><td></td><td></td><td></td></tr></tbody></table><h3 class="anchor anchorWithStickyNavbar_LWe7" id="49-hcatalog-arguments">4.9 HCatalog arguments<a class="hash-link" href="#49-hcatalog-arguments" title="Direct link to heading"></a></h3><table><thead><tr><th>参数</th><th>key</th><th>说明</th></tr></thead><tbody><tr><td>-<!-- -->-hcatalog-database <arg<!-- -->></td><td>sqoop.args.hcatalog.database</td><td>HCatalog database name</td></tr><tr><td>-<!-- -->-hcatalog-home <hdir<!-- -->></td><td>sqoop.args.hcatalog.home</td><td>Override $HCAT<!-- -->_<!-- -->HOME</td></tr><tr><td>-<!-- -->-hcatalog-partition-keys <partition-key<!-- -->></td><td>sqoop.args.hcatalog.partition.keys</td><td>Sets the partition keys to use when importing to hive</td></tr><tr><td>-<!-- -->-hcatalog-partition-values <partition-value<!-- -->></td><td>sqoop.args.hcatalog.partition.values</td><td>Sets the partition values to use when importing to hive</td></tr><tr><td>-<!-- -->-hcatalog-table <arg<!-- -->></td><td>sqoop.args.hcatalog.table</td><td>HCatalog table name</td></tr><tr><td>-<!-- -->-hive-home <dir<!-- -->></td><td>sqoop.args.hive.home</td><td>Override $HIVE<!-- -->_<!-- -->HOME</td></tr><tr><td>-<!-- -->-hive-partition-key <partition-key<!-- -->></td><td>sqoop.args.hive.partition.key</td><td>Sets the partition key to use when importing to hive</td></tr><tr><td>-<!-- -->-hive-partition-value <partition-value<!-- -->></td><td>sqoop.args.hive.partition.value</td><td>Sets the partition value to use when importing to hive</td></tr><tr><td>-<!-- -->-map-column-hive <arg<!-- -->></td><td>sqoop.args.map.column.hive</td><td>Override mapping for specific column to hive types.</td></tr><tr><td></td><td></td><td></td></tr><tr><td>HCatalog import specific options:</td><td></td><td></td></tr><tr><td>-<!-- -->-create-hcatalog-table</td><td>sqoop.args.create.hcatalog.table</td><td>Create HCatalog before import</td></tr><tr><td>-<!-- -->-hcatalog-storage-stanza <arg<!-- -->></td><td>sqoop.args.hcatalog.storage.stanza</td><td>HCatalog storage stanza for table creation</td></tr><tr><td></td><td></td><td></td></tr></tbody></table><h3 class="anchor anchorWithStickyNavbar_LWe7" id="410-accumulo-arguments">4.10 Accumulo arguments<a class="hash-link" href="#410-accumulo-arguments" title="Direct link to heading"></a></h3><table><thead><tr><th>参数</th><th>key</th><th>说明</th></tr></thead><tbody><tr><td>-<!-- -->-accumulo-batch-size <size<!-- -->></td><td>sqoop.args.accumulo.batch.size</td><td>Batch size in bytes</td></tr><tr><td>-<!-- -->-accumulo-column-family <family<!-- -->></td><td>sqoop.args.accumulo.column.family</td><td>Sets the target column family for the import</td></tr><tr><td>-<!-- -->-accumulo-create-table</td><td>sqoop.args.accumulo.create.table</td><td>If specified, create missing Accumulo tables</td></tr><tr><td>-<!-- -->-accumulo-instance <instance<!-- -->></td><td>sqoop.args.accumulo.instance</td><td>Accumulo instance name.</td></tr><tr><td>-<!-- -->-accumulo-max-latency <latency<!-- -->></td><td>sqoop.args.accumulo.max.latency</td><td>Max write latency in milliseconds</td></tr><tr><td>-<!-- -->-accumulo-password <password<!-- -->></td><td>sqoop.args.accumulo.password</td><td>Accumulo password.</td></tr><tr><td>-<!-- -->-accumulo-row-key <col<!-- -->></td><td>sqoop.args.accumulo.row.key</td><td>Specifies which input column to use as the row key</td></tr><tr><td>-<!-- -->-accumulo-table <table<!-- -->></td><td>sqoop.args.accumulo.table</td><td>Import to <table<!-- -->><!-- -->in Accumulo</td></tr><tr><td>-<!-- -->-accumulo-user <user<!-- -->></td><td>sqoop.args.accumulo.user</td><td>Accumulo user name.</td></tr><tr><td>-<!-- -->-accumulo-visibility <vis<!-- -->></td><td>sqoop.args.accumulo.visibility</td><td>Visibility token to be applied to all rows imported</td></tr><tr><td>-<!-- -->-accumulo-zookeepers <zookeepers<!-- -->></td><td>sqoop.args.accumulo.zookeepers</td><td>Comma-separated list of zookeepers (host:port)</td></tr><tr><td></td><td></td><td></td></tr></tbody></table><h3 class="anchor anchorWithStickyNavbar_LWe7" id="411-code-generation-arguments">4.11 Code generation arguments<a class="hash-link" href="#411-code-generation-arguments" title="Direct link to heading"></a></h3><table><thead><tr><th>参数</th><th>key</th><th>说明</th></tr></thead><tbody><tr><td>-<!-- -->-bindir <dir<!-- -->></td><td>sqoop.args.bindir</td><td>Output directory for compiled objects</td></tr><tr><td>-<!-- -->-class-name <name<!-- -->></td><td>sqoop.args.class.name</td><td>Sets the generated class name. This overrides --package-name. When combined with --jar-file, sets the input class.</td></tr><tr><td>-<!-- -->-input-null-non-string <null-str<!-- -->></td><td>sqoop.args.input.null.non.string</td><td>Input null non-string representation</td></tr><tr><td>-<!-- -->-input-null-string <null-str<!-- -->></td><td>sqoop.args.input.null.string</td><td>Input null string representation</td></tr><tr><td>-<!-- -->-jar-file <file<!-- -->></td><td>sqoop.args.jar.file</td><td>Disable code generation; use specified jar</td></tr><tr><td>-<!-- -->-map-column-java <arg<!-- -->></td><td>sqoop.args.map.column.java</td><td>Override mapping for specific columns to java types</td></tr><tr><td>-<!-- -->-null-non-string <null-str<!-- -->></td><td>sqoop.args.null.non.string</td><td>Null non-string representation</td></tr><tr><td>-<!-- -->-null-string <null-str<!-- -->></td><td>sqoop.args.null.string</td><td>Null string representation</td></tr><tr><td>-<!-- -->-outdir <dir<!-- -->></td><td>sqoop.args.outdir</td><td>Output directory for generated code</td></tr><tr><td>-<!-- -->-package-name <name<!-- -->></td><td>sqoop.args.package.name</td><td>Put auto-generated classes in this package</td></tr><tr><td></td><td></td><td></td></tr></tbody></table><h3 class="anchor anchorWithStickyNavbar_LWe7" id="412-generic-hadoop-command-line-arguments">4.12 Generic Hadoop command-line arguments<a class="hash-link" href="#412-generic-hadoop-command-line-arguments" title="Direct link to heading"></a></h3><blockquote><p>must preceed any tool-specific arguments,Generic options supported are </p></blockquote><table><thead><tr><th>参数</th><th>key</th><th>说明</th></tr></thead><tbody><tr><td>-<!-- -->conf <configuration file<!-- -->></td><td>sqoop.args.conf</td><td>specify an application configuration file</td></tr><tr><td>-<!-- -->D <property=value<!-- -->></td><td>sqoop.args.D</td><td>use value for given property</td></tr><tr><td>-<!-- -->fs <local</td><td>namenode:port<!-- -->></td><td>sqoop.args.fs</td></tr><tr><td>-<!-- -->jt <local</td><td>resourcemanager:port<!-- -->></td><td>sqoop.args.jt</td></tr><tr><td>-<!-- -->files <comma separated list of files<!-- -->></td><td>sqoop.args.files</td><td>specify comma separated files to be copied to the map reduce cluster</td></tr><tr><td>-<!-- -->libjars <comma separated list of jars<!-- -->></td><td>sqoop.args.libjars</td><td>specify comma separated jar files to include in the classpath.</td></tr><tr><td>-<!-- -->archives <comma separated list of archives<!-- -->></td><td>sqoop.args.archives</td><td>specify comma separated archives to be unarchived on the compute machines.</td></tr></tbody></table></div><footer class="theme-doc-footer docusaurus-mt-lg"><div class="theme-doc-footer-edit-meta-row row"><div class="col"><a href="https://github.com/apache/linkis-website/edit/dev/i18n/zh-CN/docusaurus-plugin-content-docs/version-1.2.0/engine-usage/sqoop.md" target="_blank" rel="noreferrer noopener" class="theme-edit-this-page"><svg fill="currentColor" height="20" width="20" viewBox="0 0 40 40" class="iconEdit_Z9Sw" aria-hidden="true"><g><path d="m34.5 11.7l-3 3.1-6.3-6.3 3.1-3q0.5-0.5 1.2-0.5t1.1 0.5l3.9 3.9q0.5 0.4 0.5 1.1t-0.5 1.2z m-29.5 17.1l18.4-18.5 6.3 6.3-18.4 18.4h-6.3v-6.2z"></path></g></svg>编辑</a></div><div class="col lastUpdated_vwxv"></div></div></footer></article><nav class="pagination-nav docusaurus-mt-lg" aria-label="Docs pages navigation"><a class="pagination-nav__link pagination-nav__link--prev" href="/zh-CN/docs/1.2.0/engine-usage/openlookeng"><div class="pagination-nav__sublabel">上一篇</div><div class="pagination-nav__label">openLooKeng 引擎</div></a><a class="pagination-nav__link pagination-nav__link--next" href="/zh-CN/docs/1.2.0/engine-usage/pipeline"><div class="pagination-nav__sublabel">下一篇</div><div class="pagination-nav__label">Pipeline 引擎</div></a></nav></div></div><div class="col col--3"><div class="tableOfContents_bqdL thin-scrollbar theme-doc-toc-desktop"><ul class="table-of-contents table-of-contents__left-border"><li><a href="#1-sqoop引擎linkis系统参数配置" class="table-of-contents__link toc-highlight">1 Sqoop引擎Linkis系统参数配置</a></li><li><a href="#2-sqoop引擎的配置和部署" class="table-of-contents__link toc-highlight">2 Sqoop引擎的配置和部署</a><ul><li><a href="#21-sqoop版本的选择和编译" class="table-of-contents__link toc-highlight">2.1 Sqoop版本的选择和编译</a></li><li><a href="#22-sqoop-engineconn部署和加载" class="table-of-contents__link toc-highlight">2.2 Sqoop engineConn部署和加载</a></li></ul></li><li><a href="#3-sqoop引擎的使用" class="table-of-contents__link toc-highlight">3 Sqoop引擎的使用</a><ul><li><a href="#31-通过linkis-cli进行任务提交" class="table-of-contents__link toc-highlight">3.1 通过Linkis-cli进行任务提交</a></li><li><a href="#31-onceengineconn方式" class="table-of-contents__link toc-highlight">3.1 OnceEngineConn方式</a></li></ul></li><li><a href="#4-参数" class="table-of-contents__link toc-highlight">4 参数</a><ul><li><a href="#41-common-arguments" class="table-of-contents__link toc-highlight">4.1 Common arguments</a></li><li><a href="#42-export-import-arguments" class="table-of-contents__link toc-highlight">4.2 Export import arguments</a></li><li><a href="#43-import-control-arguments" class="table-of-contents__link toc-highlight">4.3 Import control arguments</a></li><li><a href="#44-incremental-import-argument" class="table-of-contents__link toc-highlight">4.4 Incremental import argument</a></li><li><a href="#45-output-line-formatting-arguments" class="table-of-contents__link toc-highlight">4.5 Output line formatting arguments</a></li><li><a href="#46-input-parsing-arguments" class="table-of-contents__link toc-highlight">4.6 Input parsing arguments</a></li><li><a href="#47-hive-arguments" class="table-of-contents__link toc-highlight">4.7 Hive arguments</a></li><li><a href="#48-hbase-arguments" class="table-of-contents__link toc-highlight">4.8 HBase arguments</a></li><li><a href="#49-hcatalog-arguments" class="table-of-contents__link toc-highlight">4.9 HCatalog arguments</a></li><li><a href="#410-accumulo-arguments" class="table-of-contents__link toc-highlight">4.10 Accumulo arguments</a></li><li><a href="#411-code-generation-arguments" class="table-of-contents__link toc-highlight">4.11 Code generation arguments</a></li><li><a href="#412-generic-hadoop-command-line-arguments" class="table-of-contents__link toc-highlight">4.12 Generic Hadoop command-line arguments</a></li></ul></li></ul></div></div></div></div></main></div></div><footer class="footer"><div class="container container-fluid"><div class="row footer__links"><div class="col footer__col"><div class="footer__title">Linkis</div><ul class="footer__items clean-list"><li class="footer__item"><a class="footer__link-item" href="/zh-CN/docs/latest/about/introduction">文档</a></li><li class="footer__item"><a class="footer__link-item" href="/zh-CN/faq/main">FAQ</a></li><li class="footer__item"><a href="https://github.com/apache/linkis/releases" target="_blank" rel="noopener noreferrer" class="footer__link-item">版本</a></li></ul></div><div class="col footer__col"><div class="footer__title">社区</div><ul class="footer__items clean-list"><li class="footer__item"><a href="https://github.com/apache/linkis" target="_blank" rel="noopener noreferrer" class="footer__link-item">GitHub</a></li><li class="footer__item"><a href="https://github.com/apache/linkis/issues" target="_blank" rel="noopener noreferrer" class="footer__link-item">Issue Tracker</a></li><li class="footer__item"><a href="https://github.com/apache/linkis/pulls" target="_blank" rel="noopener noreferrer" class="footer__link-item">Pull Requests</a></li></ul></div><div class="col footer__col"><div class="footer__title">订阅邮件组</div><ul class="footer__items clean-list"><li class="footer__item"><a class="footer__link-item" href="/zh-CN/community/how-to-subscribe">如何订阅</a></li><li class="footer__item"><a href="mailto:dev-subscribe@linkis.apache.org" target="_blank" rel="noopener noreferrer" class="footer__link-item">订阅邮件</a></li><li class="footer__item"><a href="https://lists.apache.org/list.html?dev@linkis.apache.org" target="_blank" rel="noopener noreferrer" class="footer__link-item">邮件归档</a></li></ul></div></div><div class="footer__bottom text--center"><div class="footer__copyright"><div style="text-align: left;"> |
