| <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
| <!-- |
| | Generated by Apache Maven Doxia at 2021-06-15 |
| | Rendered using Apache Maven Stylus Skin 1.5 |
| --> |
| <html xmlns="http://www.w3.org/1999/xhtml"> |
| <head> |
| <title>Apache Hadoop 3.3.1 – Quick Start</title> |
| <style type="text/css" media="all"> |
| @import url("../css/maven-base.css"); |
| @import url("../css/maven-theme.css"); |
| @import url("../css/site.css"); |
| </style> |
| <link rel="stylesheet" href="../css/print.css" type="text/css" media="print" /> |
| <meta name="Date-Revision-yyyymmdd" content="20210615" /> |
| <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> |
| </head> |
| <body class="composite"> |
| <div id="banner"> |
| <a href="http://hadoop.apache.org/" id="bannerLeft"> |
| <img src="http://hadoop.apache.org/images/hadoop-logo.jpg" alt="" /> |
| </a> |
| <a href="http://www.apache.org/" id="bannerRight"> |
| <img src="http://www.apache.org/images/asf_logo_wide.png" alt="" /> |
| </a> |
| <div class="clear"> |
| <hr/> |
| </div> |
| </div> |
| <div id="breadcrumbs"> |
| |
| <div class="xleft"> |
| <a href="http://www.apache.org/" class="externalLink">Apache</a> |
| > |
| <a href="http://hadoop.apache.org/" class="externalLink">Hadoop</a> |
| > |
| <a href="../../index.html">Apache Hadoop YARN</a> |
| > |
| <a href="../index.html">Apache Hadoop 3.3.1</a> |
| > |
| Quick Start |
| </div> |
| <div class="xright"> <a href="http://wiki.apache.org/hadoop" class="externalLink">Wiki</a> |
| | |
| <a href="https://gitbox.apache.org/repos/asf/hadoop.git" class="externalLink">git</a> |
| | |
| <a href="http://hadoop.apache.org/" class="externalLink">Apache Hadoop</a> |
| |
| | Last Published: 2021-06-15 |
| | Version: 3.3.1 |
| </div> |
| <div class="clear"> |
| <hr/> |
| </div> |
| </div> |
| <div id="leftColumn"> |
| <div id="navcolumn"> |
| |
| <h5>General</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../index.html">Overview</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/SingleCluster.html">Single Node Setup</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/ClusterSetup.html">Cluster Setup</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/CommandsManual.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/FileSystemShell.html">FileSystem Shell</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/Compatibility.html">Compatibility Specification</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/DownstreamDev.html">Downstream Developer's Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/AdminCompatibilityGuide.html">Admin Compatibility Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/InterfaceClassification.html">Interface Classification</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/filesystem/index.html">FileSystem Specification</a> |
| </li> |
| </ul> |
| <h5>Common</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/CLIMiniCluster.html">CLI Mini Cluster</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/FairCallQueue.html">Fair Call Queue</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/NativeLibraries.html">Native Libraries</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/Superusers.html">Proxy User</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/RackAwareness.html">Rack Awareness</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/SecureMode.html">Secure Mode</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/ServiceLevelAuth.html">Service Level Authorization</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/HttpAuthentication.html">HTTP Authentication</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/CredentialProviderAPI.html">Credential Provider API</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-kms/index.html">Hadoop KMS</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/Tracing.html">Tracing</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/UnixShellGuide.html">Unix Shell Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/registry/index.html">Registry</a> |
| </li> |
| </ul> |
| <h5>HDFS</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsDesign.html">Architecture</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html">User Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HDFSCommands.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html">NameNode HA With QJM</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html">NameNode HA With NFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html">Observer NameNode</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/Federation.html">Federation</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/ViewFs.html">ViewFs</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/ViewFsOverloadScheme.html">ViewFsOverloadScheme</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html">Snapshots</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsEditsViewer.html">Edits Viewer</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html">Image Viewer</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html">Permissions and HDFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html">Quotas and HDFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/LibHdfs.html">libhdfs (C API)</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/WebHDFS.html">WebHDFS (REST API)</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-hdfs-httpfs/index.html">HttpFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html">Short Circuit Local Reads</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html">Centralized Cache Management</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html">NFS Gateway</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html">Rolling Upgrade</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/ExtendedAttributes.html">Extended Attributes</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html">Transparent Encryption</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html">Multihoming</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html">Storage Policies</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/MemoryStorage.html">Memory Storage Support</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/SLGUserGuide.html">Synthetic Load Generator</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html">Erasure Coding</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HDFSDiskbalancer.html">Disk Balancer</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsUpgradeDomain.html">Upgrade Domain</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsDataNodeAdminGuide.html">DataNode Admin</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html">Router Federation</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsProvidedStorage.html">Provided Storage</a> |
| </li> |
| </ul> |
| <h5>MapReduce</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html">Tutorial</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html">Compatibility with 1.x</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html">Encrypted Shuffle</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html">Pluggable Shuffle/Sort</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html">Distributed Cache Deploy</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/SharedCacheSupport.html">Support for YARN Shared Cache</a> |
| </li> |
| </ul> |
| <h5>MapReduce REST APIs</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredAppMasterRest.html">MR Application Master</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-hs/HistoryServerRest.html">MR History Server</a> |
| </li> |
| </ul> |
| <h5>YARN</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/YARN.html">Architecture</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/YarnCommands.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html">Capacity Scheduler</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/FairScheduler.html">Fair Scheduler</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html">ResourceManager Restart</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html">ResourceManager HA</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/ResourceModel.html">Resource Model</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/NodeLabel.html">Node Labels</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/NodeAttributes.html">Node Attributes</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html">Web Application Proxy</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html">Timeline Server</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html">Timeline Service V.2</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html">Writing YARN Applications</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html">YARN Application Security</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/NodeManager.html">NodeManager</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/DockerContainers.html">Running Applications in Docker Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/RuncContainers.html">Running Applications in runC Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/NodeManagerCgroups.html">Using CGroups</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/SecureContainer.html">Secure Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/ReservationSystem.html">Reservation System</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html">Graceful Decommission</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html">Opportunistic Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/Federation.html">YARN Federation</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/SharedCache.html">Shared Cache</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/UsingGpus.html">Using GPU</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/UsingFPGA.html">Using FPGA</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html">Placement Constraints</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/YarnUI2.html">YARN UI2</a> |
| </li> |
| </ul> |
| <h5>YARN REST APIs</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html">Introduction</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html">Resource Manager</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/NodeManagerRest.html">Node Manager</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html#Timeline_Server_REST_API_v1">Timeline Server</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html#Timeline_Service_v.2_REST_API">Timeline Service V.2</a> |
| </li> |
| </ul> |
| <h5>YARN Service</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/yarn-service/Overview.html">Overview</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/yarn-service/QuickStart.html">QuickStart</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/yarn-service/Concepts.html">Concepts</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/yarn-service/YarnServiceAPI.html">Yarn Service API</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/yarn-service/ServiceDiscovery.html">Service Discovery</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/yarn-service/SystemServices.html">System Services</a> |
| </li> |
| </ul> |
| <h5>Hadoop Compatible File Systems</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-aliyun/tools/hadoop-aliyun/index.html">Aliyun OSS</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-aws/tools/hadoop-aws/index.html">Amazon S3</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-azure/index.html">Azure Blob Storage</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-azure-datalake/index.html">Azure Data Lake Storage</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-openstack/index.html">OpenStack Swift</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-cos/cloud-storage/index.html">Tencent COS</a> |
| </li> |
| </ul> |
| <h5>Auth</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-auth/index.html">Overview</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-auth/Examples.html">Examples</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-auth/Configuration.html">Configuration</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-auth/BuildingIt.html">Building</a> |
| </li> |
| </ul> |
| <h5>Tools</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-streaming/HadoopStreaming.html">Hadoop Streaming</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-archives/HadoopArchives.html">Hadoop Archives</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-archive-logs/HadoopArchiveLogs.html">Hadoop Archive Logs</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-distcp/DistCp.html">DistCp</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-gridmix/GridMix.html">GridMix</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-rumen/Rumen.html">Rumen</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-resourceestimator/ResourceEstimator.html">Resource Estimator Service</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-sls/SchedulerLoadSimulator.html">Scheduler Load Simulator</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/Benchmarking.html">Hadoop Benchmarking</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-dynamometer/Dynamometer.html">Dynamometer</a> |
| </li> |
| </ul> |
| <h5>Reference</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/release/">Changelog and Release Notes</a> |
| </li> |
| <li class="none"> |
| <a href="../../../api/index.html">Java API docs</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/UnixShellAPI.html">Unix Shell API</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/Metrics.html">Metrics</a> |
| </li> |
| </ul> |
| <h5>Configuration</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/core-default.xml">core-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/hdfs-default.xml">hdfs-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs-rbf/hdfs-rbf-default.xml">hdfs-rbf-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml">mapred-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-common/yarn-default.xml">yarn-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-kms/kms-default.html">kms-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-hdfs-httpfs/httpfs-default.html">httpfs-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/DeprecatedProperties.html">Deprecated Properties</a> |
| </li> |
| </ul> |
| <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"> |
| <img alt="Built by Maven" src="../images/logos/maven-feather.png"/> |
| </a> |
| |
| </div> |
| </div> |
| <div id="bodyColumn"> |
| <div id="contentBox"> |
| <!--- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. See accompanying LICENSE file. |
| --> |
| <h1>Quick Start</h1> |
| <p>This document describes how to deploy services on YARN using the YARN Service framework.</p> |
| <ul> |
| <li><a href="#Configure_and_start_HDFS_and_YARN_components">Configure and start HDFS and YARN components</a></li> |
| <li><a href="#Example_service">Example service</a></li> |
| <li><a href="#Manage_services_on_YARN_via_CLI">Manage services on YARN via CLI</a> |
| <ul> |
| <li><a href="#Deploy_a_service">Deploy a service</a></li> |
| <li><a href="#Flex_a_component_of_a_service">Flex a component of a service</a></li> |
| <li><a href="#Stop_a_service">Stop a service</a></li> |
| <li><a href="#Restart_a_stopped_service">Restart a stopped service</a></li> |
| <li><a href="#Destroy_a_service">Destroy a service</a></li></ul></li> |
| <li><a href="#Manage_services_on_YARN_via_REST_API">Manage services on YARN via REST API</a> |
| <ul> |
| <li><a href="#Deploy_a_service">Deploy a service</a></li> |
| <li><a href="#Get_a_service_status">Get a service status</a></li> |
| <li><a href="#Flex_a_component_of_a_service">Flex a component of a service</a></li> |
| <li><a href="#Stop_a_service">Stop a service</a></li> |
| <li><a href="#Restart_a_stopped_service">Restart a stopped service</a></li> |
| <li><a href="#Destroy_a_service">Destroy a service</a></li></ul></li> |
| <li><a href="#Services_UI_with_YARN_UI2_and_Timeline_Service_v2">Services UI with YARN UI2 and Timeline Service v2</a> |
| <ul> |
| <li><a href="#Enable_Timeline_Service_v2">Enable Timeline Service v2</a></li> |
| <li><a href="#Enable_new_YARN_UI">Enable new YARN UI</a></li></ul></li></ul> |
| |
| <div class="section"> |
| <h2><a name="Configure_and_start_HDFS_and_YARN_components"></a>Configure and start HDFS and YARN components</h2> |
| <p>Start all the hadoop components for HDFS and YARN as usual. To enable the YARN Service framework, add this property to <tt>yarn-site.xml</tt> and restart the ResourceManager or set the property before the ResourceManager is started. This property is required for using the YARN Service framework through the CLI or the REST API.</p> |
| |
| <div> |
| <div> |
| <pre class="source"> <property> |
| <description> |
| Enable services rest api on ResourceManager. |
| </description> |
| <name>yarn.webapp.api-service.enable</name> |
| <value>true</value> |
| </property> |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h2><a name="Example_service"></a>Example service</h2> |
| <p>Below is a simple service definition that launches sleep containers on YARN by writing a simple spec file and without writing any code.</p> |
| |
| <div> |
| <div> |
| <pre class="source">{ |
| "name": "sleeper-service", |
| "version": "1.0", |
| "components" : |
| [ |
| { |
| "name": "sleeper", |
| "number_of_containers": 1, |
| "launch_command": "sleep 900000", |
| "resource": { |
| "cpus": 1, |
| "memory": "256" |
| } |
| } |
| ] |
| } |
| </pre></div></div> |
| |
| <p>User can simply run a pre-built example service on YARN using below command:</p> |
| |
| <div> |
| <div> |
| <pre class="source">yarn app -launch <service-name> <example-name> |
| </pre></div></div> |
| |
| <p>e.g. Below command launches a <tt>sleeper</tt> service named as <tt>my-sleeper</tt> on YARN.</p> |
| |
| <div> |
| <div> |
| <pre class="source">yarn app -launch my-sleeper sleeper |
| </pre></div></div> |
| |
| <p>For launching docker based services using YARN Service framework, please refer to <a href="YarnServiceAPI.html">API doc</a>.</p></div> |
| <div class="section"> |
| <h2><a name="Manage_services_on_YARN_via_CLI"></a>Manage services on YARN via CLI</h2> |
| <p>Below steps walk you through deploying a services on YARN using CLI. Refer to <a href="../YarnCommands.html">Yarn Commands</a> for the full list of commands and options.</p> |
| <div class="section"> |
| <h3><a name="Deploy_a_service"></a>Deploy a service</h3> |
| |
| <div> |
| <div> |
| <pre class="source">yarn app -launch ${SERVICE_NAME} ${PATH_TO_SERVICE_DEF_FILE} |
| </pre></div></div> |
| |
| <p>Params:</p> |
| <ul> |
| |
| <li>SERVICE_NAME: The name of the service. Note that this needs to be unique across running services for the current user.</li> |
| <li>PATH_TO_SERVICE_DEF: The path to the service definition file in JSON format.</li> |
| </ul> |
| <p>For example:</p> |
| |
| <div> |
| <div> |
| <pre class="source">yarn app -launch sleeper-service /path/to/local/sleeper.json |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h3><a name="Flex_a_component_of_a_service"></a>Flex a component of a service</h3> |
| <p>Increase or decrease the number of containers for a component.</p> |
| |
| <div> |
| <div> |
| <pre class="source">yarn app -flex ${SERVICE_NAME} -component ${COMPONENT_NAME} ${NUMBER_OF_CONTAINERS} |
| </pre></div></div> |
| |
| <p>For example, for a service named <tt>sleeper-service</tt>:</p> |
| <p>Set the <tt>sleeper</tt> component to <tt>2</tt> containers (absolute number).</p> |
| |
| <div> |
| <div> |
| <pre class="source">yarn app -flex sleeper-service -component sleeper 2 |
| </pre></div></div> |
| |
| <p>Relative changes are also supported for the ${NUMBER_OF_CONTAINERS} in the flex command, such as +2 or -2.</p></div> |
| <div class="section"> |
| <h3><a name="Stop_a_service"></a>Stop a service</h3> |
| <p>Stopping a service will stop all containers of the service and the ApplicationMaster, but does not delete the state of a service, such as the service root folder on hdfs.</p> |
| |
| <div> |
| <div> |
| <pre class="source">yarn app -stop ${SERVICE_NAME} |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h3><a name="Restart_a_stopped_service"></a>Restart a stopped service</h3> |
| <p>Restarting a stopped service is easy - just call start!</p> |
| |
| <div> |
| <div> |
| <pre class="source">yarn app -start ${SERVICE_NAME} |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h3><a name="Destroy_a_service"></a>Destroy a service</h3> |
| <p>In addition to stopping a service, it also deletes the service root folder on hdfs and the records in YARN Service Registry.</p> |
| |
| <div> |
| <div> |
| <pre class="source">yarn app -destroy ${SERVICE_NAME} |
| </pre></div></div> |
| </div></div> |
| <div class="section"> |
| <h2><a name="Manage_services_on_YARN_via_REST_API"></a>Manage services on YARN via REST API</h2> |
| <p>The YARN API Server REST API is activated as part of the ResourceManager when <tt>yarn.webapp.api-service.enable</tt> is set to true.</p> |
| <p>Services can be deployed on YARN through the ResourceManager web endpoint.</p> |
| <p>Refer to <a href="YarnServiceAPI.html">API doc</a> for the detailed API specificatiosn.</p> |
| <div class="section"> |
| <h3><a name="Deploy_a_service"></a>Deploy a service</h3> |
| <p>POST the aforementioned example service definition to the ResourceManager api-server endpoint:</p> |
| |
| <div> |
| <div> |
| <pre class="source">POST http://localhost:8088/app/v1/services |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h3><a name="Get_a_service_status"></a>Get a service status</h3> |
| |
| <div> |
| <div> |
| <pre class="source">GET http://localhost:8088/app/v1/services/${SERVICE_NAME} |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h3><a name="Flex_a_component_of_a_service"></a>Flex a component of a service</h3> |
| |
| <div> |
| <div> |
| <pre class="source">PUT http://localhost:8088/app/v1/services/${SERVICE_NAME}/components/${COMPONENT_NAME} |
| </pre></div></div> |
| |
| <p><tt>PUT</tt> Request body:</p> |
| |
| <div> |
| <div> |
| <pre class="source">{ |
| "name": "${COMPONENT_NAME}", |
| "number_of_containers": ${COUNT} |
| } |
| </pre></div></div> |
| |
| <p>For example:</p> |
| |
| <div> |
| <div> |
| <pre class="source">{ |
| "name": "sleeper", |
| "number_of_containers": 2 |
| } |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h3><a name="Stop_a_service"></a>Stop a service</h3> |
| <p>Stopping a service will stop all containers of the service and the ApplicationMaster, but does not delete the state of a service, such as the service root folder on hdfs.</p> |
| |
| <div> |
| <div> |
| <pre class="source">PUT http://localhost:8088/app/v1/services/${SERVICE_NAME} |
| </pre></div></div> |
| |
| <p><tt>PUT</tt> Request body:</p> |
| |
| <div> |
| <div> |
| <pre class="source">{ |
| "name": "${SERVICE_NAME}", |
| "state": "STOPPED" |
| } |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h3><a name="Restart_a_stopped_service"></a>Restart a stopped service</h3> |
| <p>Restarting a stopped service is easy.</p> |
| |
| <div> |
| <div> |
| <pre class="source">PUT http://localhost:8088/app/v1/services/${SERVICE_NAME} |
| </pre></div></div> |
| |
| <p><tt>PUT</tt> Request body:</p> |
| |
| <div> |
| <div> |
| <pre class="source">{ |
| "name": "${SERVICE_NAME}", |
| "state": "STARTED" |
| } |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h3><a name="Destroy_a_service"></a>Destroy a service</h3> |
| <p>In addition to stopping a service, it also deletes the service root folder on hdfs and the records in YARN Service Registry.</p> |
| |
| <div> |
| <div> |
| <pre class="source">DELETE http://localhost:8088/app/v1/services/${SERVICE_NAME} |
| </pre></div></div> |
| </div></div> |
| <div class="section"> |
| <h2><a name="Services_UI_with_YARN_UI2_and_Timeline_Service_v2"></a>Services UI with YARN UI2 and Timeline Service v2</h2> |
| <p>A new <tt>service</tt> tab is added in the YARN UI2 specially to show YARN Services in a first class manner. The services framework posts the data into TimelineService and the <tt>service</tt> UI reads data from TimelineService to render its content.</p> |
| <div class="section"> |
| <h3><a name="Enable_Timeline_Service_v2"></a>Enable Timeline Service v2</h3> |
| <p>Please refer to <a href="../TimelineServiceV2.html">TimeLineService v2 doc</a> for how to enable Timeline Service v2.</p></div> |
| <div class="section"> |
| <h3><a name="Enable_new_YARN_UI"></a>Enable new YARN UI</h3> |
| <p>Set below config in <tt>yarn-site.xml</tt> and start ResourceManager. If you are building from source code, make sure you use <tt>-Pyarn-ui</tt> in the <tt>mvn</tt> command - this will generate the war file for the new YARN UI.</p> |
| |
| <div> |
| <div> |
| <pre class="source"> <property> |
| <description>To enable RM web ui2 application.</description> |
| <name>yarn.webapp.ui2.enable</name> |
| <value>true</value> |
| </property> |
| </pre></div></div> |
| |
| <h1>Run with security</h1> |
| <p>YARN service framework supports running in a secure (kerberized) environment. User needs to specify the kerberos principal name and keytab when they launch the service. E.g. A typical configuration looks like below:</p> |
| |
| <div> |
| <div> |
| <pre class="source">{ |
| "name": "sample-service", |
| ... |
| ... |
| "kerberos_principal" : { |
| "principal_name" : "hdfs-demo/_HOST@EXAMPLE.COM", |
| "keytab" : "file:///etc/security/keytabs/hdfs.headless.keytab" |
| } |
| } |
| </pre></div></div> |
| |
| <p>Note that <tt>_HOST</tt> is required in the <tt>principal_name</tt> field because Hadoop client validates that the server’s (in this case, the AM’s) principal has hostname present when communicating to the server. * principal_name : the principal name of the user who launches the service * keytab : URI of the keytab. Currently supports only files present on the bare host. * URI starts with <tt>file://</tt> - A path on the local host where the keytab is stored. It is assumed that admin pre-installs the keytabs on the local host before AM launches.</p> |
| <h1>Run with Docker</h1> |
| <p>The above example is only for a non-docker container based service. YARN Service Framework also provides first-class support for managing docker based services. Most of the steps for managing docker based services are the same except that in docker the <tt>Artifact</tt> type for a component is <tt>DOCKER</tt> and the Artifact <tt>id</tt> is the name of the docker image. For details in how to setup docker on YARN, please check <a href="../DockerContainers.html">Docker on YARN</a>.</p> |
| <p>With docker support, it also opens up a set of new possibilities to implement features such as discovering service containers on YARN with DNS. Check <a href="ServiceDiscovery.html">ServiceDiscovery</a> for more details.</p></div></div> |
| </div> |
| </div> |
| <div class="clear"> |
| <hr/> |
| </div> |
| <div id="footer"> |
| <div class="xright"> |
| © 2008-2021 |
| Apache Software Foundation |
| |
| - <a href="http://maven.apache.org/privacy-policy.html">Privacy Policy</a>. |
| Apache Maven, Maven, Apache, the Apache feather logo, and the Apache Maven project logos are trademarks of The Apache Software Foundation. |
| </div> |
| <div class="clear"> |
| <hr/> |
| </div> |
| </div> |
| </body> |
| </html> |
