| <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
| <!-- |
| | Generated by Apache Maven Doxia at 2021-06-15 |
| | Rendered using Apache Maven Stylus Skin 1.5 |
| --> |
| <html xmlns="http://www.w3.org/1999/xhtml"> |
| <head> |
| <title>Apache Hadoop 3.3.1 – YARN Service Configurations</title> |
| <style type="text/css" media="all"> |
| @import url("../css/maven-base.css"); |
| @import url("../css/maven-theme.css"); |
| @import url("../css/site.css"); |
| </style> |
| <link rel="stylesheet" href="../css/print.css" type="text/css" media="print" /> |
| <meta name="Date-Revision-yyyymmdd" content="20210615" /> |
| <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> |
| </head> |
| <body class="composite"> |
| <div id="banner"> |
| <a href="http://hadoop.apache.org/" id="bannerLeft"> |
| <img src="http://hadoop.apache.org/images/hadoop-logo.jpg" alt="" /> |
| </a> |
| <a href="http://www.apache.org/" id="bannerRight"> |
| <img src="http://www.apache.org/images/asf_logo_wide.png" alt="" /> |
| </a> |
| <div class="clear"> |
| <hr/> |
| </div> |
| </div> |
| <div id="breadcrumbs"> |
| |
| <div class="xleft"> |
| <a href="http://www.apache.org/" class="externalLink">Apache</a> |
| > |
| <a href="http://hadoop.apache.org/" class="externalLink">Hadoop</a> |
| > |
| <a href="../../index.html">Apache Hadoop YARN</a> |
| > |
| <a href="../index.html">Apache Hadoop 3.3.1</a> |
| > |
| YARN Service Configurations |
| </div> |
| <div class="xright"> <a href="http://wiki.apache.org/hadoop" class="externalLink">Wiki</a> |
| | |
| <a href="https://gitbox.apache.org/repos/asf/hadoop.git" class="externalLink">git</a> |
| | |
| <a href="http://hadoop.apache.org/" class="externalLink">Apache Hadoop</a> |
| |
| | Last Published: 2021-06-15 |
| | Version: 3.3.1 |
| </div> |
| <div class="clear"> |
| <hr/> |
| </div> |
| </div> |
| <div id="leftColumn"> |
| <div id="navcolumn"> |
| |
| <h5>General</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../index.html">Overview</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/SingleCluster.html">Single Node Setup</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/ClusterSetup.html">Cluster Setup</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/CommandsManual.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/FileSystemShell.html">FileSystem Shell</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/Compatibility.html">Compatibility Specification</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/DownstreamDev.html">Downstream Developer's Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/AdminCompatibilityGuide.html">Admin Compatibility Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/InterfaceClassification.html">Interface Classification</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/filesystem/index.html">FileSystem Specification</a> |
| </li> |
| </ul> |
| <h5>Common</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/CLIMiniCluster.html">CLI Mini Cluster</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/FairCallQueue.html">Fair Call Queue</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/NativeLibraries.html">Native Libraries</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/Superusers.html">Proxy User</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/RackAwareness.html">Rack Awareness</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/SecureMode.html">Secure Mode</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/ServiceLevelAuth.html">Service Level Authorization</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/HttpAuthentication.html">HTTP Authentication</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/CredentialProviderAPI.html">Credential Provider API</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-kms/index.html">Hadoop KMS</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/Tracing.html">Tracing</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/UnixShellGuide.html">Unix Shell Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/registry/index.html">Registry</a> |
| </li> |
| </ul> |
| <h5>HDFS</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsDesign.html">Architecture</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html">User Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HDFSCommands.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html">NameNode HA With QJM</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html">NameNode HA With NFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html">Observer NameNode</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/Federation.html">Federation</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/ViewFs.html">ViewFs</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/ViewFsOverloadScheme.html">ViewFsOverloadScheme</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html">Snapshots</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsEditsViewer.html">Edits Viewer</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html">Image Viewer</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html">Permissions and HDFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html">Quotas and HDFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/LibHdfs.html">libhdfs (C API)</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/WebHDFS.html">WebHDFS (REST API)</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-hdfs-httpfs/index.html">HttpFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html">Short Circuit Local Reads</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html">Centralized Cache Management</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html">NFS Gateway</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html">Rolling Upgrade</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/ExtendedAttributes.html">Extended Attributes</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html">Transparent Encryption</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html">Multihoming</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html">Storage Policies</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/MemoryStorage.html">Memory Storage Support</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/SLGUserGuide.html">Synthetic Load Generator</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html">Erasure Coding</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HDFSDiskbalancer.html">Disk Balancer</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsUpgradeDomain.html">Upgrade Domain</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsDataNodeAdminGuide.html">DataNode Admin</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html">Router Federation</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/HdfsProvidedStorage.html">Provided Storage</a> |
| </li> |
| </ul> |
| <h5>MapReduce</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html">Tutorial</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html">Compatibility with 1.x</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html">Encrypted Shuffle</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html">Pluggable Shuffle/Sort</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html">Distributed Cache Deploy</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/SharedCacheSupport.html">Support for YARN Shared Cache</a> |
| </li> |
| </ul> |
| <h5>MapReduce REST APIs</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredAppMasterRest.html">MR Application Master</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-hs/HistoryServerRest.html">MR History Server</a> |
| </li> |
| </ul> |
| <h5>YARN</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/YARN.html">Architecture</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/YarnCommands.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html">Capacity Scheduler</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/FairScheduler.html">Fair Scheduler</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html">ResourceManager Restart</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html">ResourceManager HA</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/ResourceModel.html">Resource Model</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/NodeLabel.html">Node Labels</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/NodeAttributes.html">Node Attributes</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html">Web Application Proxy</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html">Timeline Server</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html">Timeline Service V.2</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html">Writing YARN Applications</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html">YARN Application Security</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/NodeManager.html">NodeManager</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/DockerContainers.html">Running Applications in Docker Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/RuncContainers.html">Running Applications in runC Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/NodeManagerCgroups.html">Using CGroups</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/SecureContainer.html">Secure Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/ReservationSystem.html">Reservation System</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html">Graceful Decommission</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html">Opportunistic Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/Federation.html">YARN Federation</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/SharedCache.html">Shared Cache</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/UsingGpus.html">Using GPU</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/UsingFPGA.html">Using FPGA</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html">Placement Constraints</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/YarnUI2.html">YARN UI2</a> |
| </li> |
| </ul> |
| <h5>YARN REST APIs</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html">Introduction</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html">Resource Manager</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/NodeManagerRest.html">Node Manager</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html#Timeline_Server_REST_API_v1">Timeline Server</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html#Timeline_Service_v.2_REST_API">Timeline Service V.2</a> |
| </li> |
| </ul> |
| <h5>YARN Service</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/yarn-service/Overview.html">Overview</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/yarn-service/QuickStart.html">QuickStart</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/yarn-service/Concepts.html">Concepts</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/yarn-service/YarnServiceAPI.html">Yarn Service API</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/yarn-service/ServiceDiscovery.html">Service Discovery</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-site/yarn-service/SystemServices.html">System Services</a> |
| </li> |
| </ul> |
| <h5>Hadoop Compatible File Systems</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-aliyun/tools/hadoop-aliyun/index.html">Aliyun OSS</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-aws/tools/hadoop-aws/index.html">Amazon S3</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-azure/index.html">Azure Blob Storage</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-azure-datalake/index.html">Azure Data Lake Storage</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-openstack/index.html">OpenStack Swift</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-cos/cloud-storage/index.html">Tencent COS</a> |
| </li> |
| </ul> |
| <h5>Auth</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-auth/index.html">Overview</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-auth/Examples.html">Examples</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-auth/Configuration.html">Configuration</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-auth/BuildingIt.html">Building</a> |
| </li> |
| </ul> |
| <h5>Tools</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-streaming/HadoopStreaming.html">Hadoop Streaming</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-archives/HadoopArchives.html">Hadoop Archives</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-archive-logs/HadoopArchiveLogs.html">Hadoop Archive Logs</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-distcp/DistCp.html">DistCp</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-gridmix/GridMix.html">GridMix</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-rumen/Rumen.html">Rumen</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-resourceestimator/ResourceEstimator.html">Resource Estimator Service</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-sls/SchedulerLoadSimulator.html">Scheduler Load Simulator</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/Benchmarking.html">Hadoop Benchmarking</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-dynamometer/Dynamometer.html">Dynamometer</a> |
| </li> |
| </ul> |
| <h5>Reference</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/release/">Changelog and Release Notes</a> |
| </li> |
| <li class="none"> |
| <a href="../../../api/index.html">Java API docs</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/UnixShellAPI.html">Unix Shell API</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/Metrics.html">Metrics</a> |
| </li> |
| </ul> |
| <h5>Configuration</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/core-default.xml">core-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs/hdfs-default.xml">hdfs-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-hdfs-rbf/hdfs-rbf-default.xml">hdfs-rbf-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml">mapred-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-yarn/hadoop-yarn-common/yarn-default.xml">yarn-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-kms/kms-default.html">kms-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-hdfs-httpfs/httpfs-default.html">httpfs-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../../hadoop-project-dist/hadoop-common/DeprecatedProperties.html">Deprecated Properties</a> |
| </li> |
| </ul> |
| <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"> |
| <img alt="Built by Maven" src="../images/logos/maven-feather.png"/> |
| </a> |
| |
| </div> |
| </div> |
| <div id="bodyColumn"> |
| <div id="contentBox"> |
| <!--- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. See accompanying LICENSE file. |
| --> |
| <h1>YARN Service Configurations</h1> |
| <p>This document describes how to configure the services to be deployed on YARN.</p> |
| <p>There are mainly three types of configurations:</p> |
| <ul> |
| |
| <li>The configurations specific to the custom service running on YARN . E.g. the hbase-site.xml for a Hbase service running on YARN. |
| <ul> |
| |
| <li>It can be specified at both global service level(<tt>Service#configuration</tt>) or component level(<tt>Component#configuration</tt>).</li> |
| <li>Service-level configs are considered as the default configs for all components and component-level config can override service level config.</li> |
| <li>All config properties that uses constant variables as described below are subject to substitutions.</li> |
| </ul> |
| </li> |
| <li>The configurations specific to YARN service AM. (<tt>Configuration#properties</tt>). |
| <ul> |
| |
| <li>E.g. The <tt>yarn.service.am-restart.max-attempts</tt> which controls how many times the framework AM can be retried if it fails. These configs are mainly to control the behavior of the framework AM , rather than the custom services launched by the framework.</li> |
| </ul> |
| </li> |
| <li>Some constants such as <tt>SERVICE_NAME</tt> for referring some system properties. |
| <ul> |
| |
| <li>They are substituted by the service AM before writing the config files.</li> |
| </ul> |
| </li> |
| </ul> |
| <p>Below describes the details for each type of configurations.</p> |
| <div class="section"> |
| <h2><a name="Configuration_for_custom_service"></a>Configuration for custom service</h2> |
| <p>Below is how a configuration object typically looks like:</p> |
| |
| <div> |
| <div> |
| <pre class="source">"configuration" : { |
| "properties" : { |
| "yarn.service.am-restart.max-attempts" : 10 // config for the service AM |
| }, |
| "env" : { // The environment variables to be exported when container gets launched |
| "env1" : "val1" |
| }, |
| "files" : [ // The list of configuration files to be mounted for the container |
| { |
| "type": "HADOOP_XML", // The format of the config file into which the "properties" are dumped |
| "dest_file": "/etc/hadoop/conf/core-site.xml", // The location where the config file is mounted inside the container |
| "properties" : { // The list of key/value pairs to be dumped into the config file |
| "fs.defaultFS" : "hdfs://myhdfs" // This property will be written into core-site.xml |
| } |
| }, |
| { |
| "type": "HADOOP_XML", // The format of the config file. |
| "src_file" : ""hdfs://mycluster/tmp/conf/yarn-site.xml"" // The location of the source config file to be downloaded |
| "dest_file": "/etc/hadoop/conf/yarn-site.xml", // The location where the config file will be mounted inside the container/ |
| "properties" : { |
| "yarn.resourcemanager.webapp.address" : "${COMPONENT_INSTANCE_NAME}.${SERVICE_NAME}.${USER}.${DOMAIN}" // Merge into (or override existing property in) yarn-site.xml |
| } |
| } |
| ] |
| } |
| </pre></div></div> |
| |
| <ul> |
| |
| <li>properties: the configurations for service AM. Details below.</li> |
| <li>env : the environment variables to be exported when container gets launched.</li> |
| <li>files : The list of configuration files to be mounted inside the container. |
| <ul> |
| |
| <li>type: The format of the config file(<tt>dest_file</tt>) to be mounted inside the container. If <tt>src_file</tt> is specified, it is also the format for both <tt>src_file</tt> and <tt>dest_file</tt>.</li> |
| <li>HADOOP_XML : the hadoop xml format. If <tt>src_file</tt> is specified, the file content will be read as parsed in hadoop xml format.</li> |
| <li>XML : the standard xml format</li> |
| <li>JSON : the standard JSON format</li> |
| <li>YAML : the YAML format</li> |
| <li>PROPERTIES : the java PROPERTIES format</li> |
| <li>TEMPLATE : the plain text format. If <tt>src_file</tt> is specified, the content of the <tt>src_file</tt> will be written into <tt>dest_file</tt> post constant substitution. If <tt>src_file</tt> is not specified, use <tt>content</tt> as the key in <tt>properties</tt> field, and the value will be the actual content to be written in the <tt>dest_file</tt> post constant substitution. E.g |
| |
| <div> |
| <div> |
| <pre class="source">{ |
| "type": "TEMPLATE" |
| "dest_file": "/etc/conf/hello" |
| "properties" : { |
| "content" : "Hello world" |
| } |
| } |
| </pre></div></div> |
| |
| <p>The string <tt>Hello world</tt> will be written into a file located at <tt>/etc/conf/hello</tt> inside the container.</p></li> |
| <li>src_file : [optional], the source location of the config file at a network accessible location such as hdfs.</li> |
| <li>The format of both <tt>src_file</tt> and <tt>dest_file</tt> are defined by <tt>type</tt>.</li> |
| <li>Currently, It only works with <tt>HADOOP_XML</tt> and <tt>TEMPLATE</tt> type.</li> |
| <li>The <tt>src_file</tt> will be downloaded by YARN NodeManager and be mounted inside the container as in the location specified by <tt>dest_file</tt>.</li> |
| <li>If any properties specified in the <tt>properties</tt> field, they are added into (or overwriting existing properties in) the <tt>src_file</tt>.</li> |
| <li>If <tt>src_file</tt> is not specified, only the properties in the <tt>properties</tt> field will be written into the <tt>dest_file</tt> file.</li> |
| <li>dest_file : the location where the config file is mounted inside the container. The file format is defined by <tt>type</tt>. dest_file can be an absolute path or a relative path. If it’s a relative path, the file will be located in the <tt>$PWD/conf</tt> directory (where <tt>$PWD</tt> is the container local directory which is mounted to all docker containers launched by YARN)</li> |
| <li>properties : The list of key/value pair configs to be written into the <tt>dest_file</tt> in the format as defined in <tt>type</tt>. If <tt>src_file</tt> is specified, these properties will be added into (or overwriting existing properties in) the <tt>src_file</tt>.</li> |
| </ul> |
| </li> |
| </ul></div> |
| <div class="section"> |
| <h2><a name="Configuration_for_YARN_service_AM"></a>Configuration for YARN service AM</h2> |
| <p>This section describes the configurations for configuring the YARN service AM.</p> |
| <div class="section"> |
| <h3><a name="System-wide_configuration_properties"></a>System-wide configuration properties</h3> |
| <p>System-wide service AM properties can only be configured in the cluster <tt>yarn-site.xml</tt> file.</p> |
| <table border="0" class="bodyTable"> |
| <thead> |
| |
| <tr class="a"> |
| <th> System-Level Config Name </th> |
| <th> Description </th></tr> |
| </thead><tbody> |
| |
| <tr class="b"> |
| <td>yarn.service.framework.path </td> |
| <td> HDFS path of the service AM dependency tarball. When no file exists at this location, AM dependencies will be uploaded by the RM the first time a service is started or launched. If the RM user does not have permission to upload the file to this location or the location is not world readable, the AM dependency jars will be uploaded each time a service is started or launched. If unspecified, value will be assumed to be /yarn-services/${hadoop.version}/service-dep.tar.gz.</td></tr> |
| <tr class="a"> |
| <td>yarn.service.base.path </td> |
| <td> HDFS parent directory where service artifacts will be stored (default ${user_home_dir}/.yarn/).</td></tr> |
| <tr class="b"> |
| <td>yarn.service.client-am.retry.max-wait-ms </td> |
| <td> Max retry time in milliseconds for the service client to talk to the service AM (default 900000, i.e. 15 minutes).</td></tr> |
| <tr class="a"> |
| <td>yarn.service.client-am.retry-interval-ms </td> |
| <td> Retry interval in milliseconds for the service client to talk to the service AM (default 2000, i.e. 2 seconds).</td></tr> |
| <tr class="b"> |
| <td>yarn.service.queue </td> |
| <td> Default queue to which the service will be submitted (default submits to the <tt>default</tt> queue). Note that queue can be specified per-service through the queue field, rather than through the service-level configuration properties.</td></tr> |
| </tbody> |
| </table></div> |
| <div class="section"> |
| <h3><a name="Service-level_configuration_properties"></a>Service-level configuration properties</h3> |
| <p>Service-level service AM configuration properties can be specified either in the cluster <tt>yarn-site.xml</tt> at the global level (effectively overriding the default values system-wide) or specified per service in the <tt>properties</tt> field of the <tt>Configuration</tt> object as in the example below:</p> |
| |
| <div> |
| <div> |
| <pre class="source">{ |
| "configuration" : { |
| "properties" : { |
| "yarn.service.am-restart.max-attempts" : 10 |
| } |
| } |
| } |
| </pre></div></div> |
| |
| <p>The above config allows the service AM to be retried a maximum of 10 times.</p> |
| <table border="0" class="bodyTable"> |
| <thead> |
| |
| <tr class="a"> |
| <th> Service-Level Config Name </th> |
| <th> Description </th></tr> |
| </thead><tbody> |
| |
| <tr class="b"> |
| <td>yarn.service.am-restart.max-attempts </td> |
| <td> Max number of times to start the service AM, after which the service will be killed (default 20).</td></tr> |
| <tr class="a"> |
| <td>yarn.service.am-resource.memory </td> |
| <td> Memory size in MB for the service AM (default 1024).</td></tr> |
| <tr class="b"> |
| <td>yarn.service.am.java.opts </td> |
| <td> Additional JVM options for the service AM (default " -Xmx768m" will be appended to any JVM opts that do not specify -Xmx).</td></tr> |
| <tr class="a"> |
| <td>yarn.service.container-recovery.timeout.ms </td> |
| <td> Timeout in milliseconds after which a newly started service AM releases all the containers of previous AM attempts which are not yet recovered from the RM (default 120000, i.e. 2 minutes).</td></tr> |
| <tr class="b"> |
| <td>yarn.service.failure-count-reset.window </td> |
| <td> Interval in seconds after which the container failure counts that will be evaluated for the per-component <tt>yarn.service.container-failure-per-component.threshold</tt> and <tt>yarn.service.node-blacklist.threshold</tt> are reset (default 21600, i.e. 6 hours).</td></tr> |
| <tr class="a"> |
| <td>yarn.service.readiness-check-interval.seconds </td> |
| <td> Interval in seconds between readiness checks (default 30 seconds).</td></tr> |
| <tr class="b"> |
| <td>yarn.service.log.include-pattern </td> |
| <td> Regex expression for including log files by name when aggregating the logs after the application completes (default includes all files).</td></tr> |
| <tr class="a"> |
| <td>yarn.service.log.exclude-pattern </td> |
| <td> Regex expression for excluding log files by name when aggregating the logs after the application completes. If the log file name matches both include and exclude pattern, this file will be excluded (default does not exclude any files).</td></tr> |
| <tr class="b"> |
| <td>yarn.service.rolling-log.include-pattern </td> |
| <td> Regex expression for including log files by name when aggregating the logs while app is running.</td></tr> |
| <tr class="a"> |
| <td>yarn.service.rolling-log.exclude-pattern </td> |
| <td> Regex expression for excluding log files by name when aggregating the logs while app is running. If the log file name matches both include and exclude pattern, this file will be excluded.</td></tr> |
| <tr class="b"> |
| <td>yarn.service.classpath </td> |
| <td> Comma separated extra class path parameters for yarn services AM. These path elements will be appended to the end of the YARN service AM classpath. </td></tr> |
| <tr class="a"> |
| <td>yarn.service.am.client.port-range </td> |
| <td> Range of ports that the Yarn Service AM can use when binding. Leave blank if you want all possible ports. For example 50000-50050,50100-50200. </td></tr> |
| </tbody> |
| </table></div> |
| <div class="section"> |
| <h3><a name="Component-level_configuration_properties"></a>Component-level configuration properties</h3> |
| <p>Component-level service AM configuration properties can be specified either in the cluster <tt>yarn-site.xml</tt> at the global level (effectively overriding the default values system-wide), specified per service in the <tt>properties</tt> field of the <tt>Configuration</tt> object, or specified per component in the <tt>properties</tt> field of the component’s <tt>Configuration</tt> object.</p> |
| <table border="0" class="bodyTable"> |
| <thead> |
| |
| <tr class="a"> |
| <th> Component-Level Config Name </th> |
| <th> Description </th></tr> |
| </thead><tbody> |
| |
| <tr class="b"> |
| <td>yarn.service.container-failure.retry.max </td> |
| <td> Max number of retries for the container to be auto restarted if it fails (default -1, which means forever).</td></tr> |
| <tr class="a"> |
| <td>yarn.service.container-failure.retry-interval-ms </td> |
| <td> Retry interval in milliseconds for the container to be restarted (default 30000, i.e. 30 seconds).</td></tr> |
| <tr class="b"> |
| <td>yarn.service.container-failure.validity-interval-ms </td> |
| <td> Failure validity interval in milliseconds. When set to a value greater than 0, the container retry policy will not take the failures that happened outside of this interval into the failure count (default -1, which means that all the failures so far will be included in the failure count).</td></tr> |
| <tr class="a"> |
| <td>yarn.service.container-failure-per-component.threshold </td> |
| <td> Max absolute number of container failures (not including retries) for a given component before the AM stops the service (default 10).</td></tr> |
| <tr class="b"> |
| <td>yarn.service.node-blacklist.threshold </td> |
| <td> Maximum number of container failures on a node (not including retries) before the node is blacklisted by the AM (default 3).</td></tr> |
| <tr class="a"> |
| <td>yarn.service.default-readiness-check.enabled </td> |
| <td> Whether or not the default readiness check is enabled (default true).</td></tr> |
| <tr class="b"> |
| <td>yarn.service.container-health-threshold.percent </td> |
| <td> The container health threshold percent when explicitly set for a specific component or globally for all components, will schedule a health check monitor to periodically check for the percentage of healthy containers. A container is healthy if it is in READY state. It runs the check at a specified/default poll frequency. It allows a component to be below the health threshold for a specified/default window after which it considers the service to be unhealthy and triggers a service stop. When health threshold percent is enabled, yarn.service.container-failure-per-component.threshold is ignored.</td></tr> |
| <tr class="a"> |
| <td>yarn.service.container-health-threshold.poll-frequency-secs </td> |
| <td> Health check monitor poll frequency. It is an advanced setting and does not need to be set unless the service owner understands the implication and does not want the default. The default is 10 secs.</td></tr> |
| <tr class="b"> |
| <td>yarn.service.container-health-threshold.window-secs </td> |
| <td> The amount of time the health check monitor allows a specific component to be below the health threshold after which it considers the service to be unhealthy. The default is 600 secs (10 mins).</td></tr> |
| <tr class="a"> |
| <td>yarn.service.container-health-threshold.init-delay-secs </td> |
| <td> The amount of initial time the health check monitor waits before the first check kicks in. It gives a lead time for the service containers to come up for the first time. The default is 600 secs (10 mins).</td></tr> |
| <tr class="b"> |
| <td>yarn.service.container-state-report-as-service-state </td> |
| <td> The boolean flag indicates that if this component is finished, the service is also terminated. The default is false.</td></tr> |
| </tbody> |
| </table> |
| <p>There is one component-level configuration property that is set differently in the <tt>yarn-site.xml</tt> file than it is in the service specification. To select the docker network type that will be used for docker containers, <tt>docker.network</tt> may be set in the service <tt>Configuration</tt> <tt>properties</tt> or the component <tt>Configuration</tt> <tt>properties</tt>. The system-wide default for the docker network type (for both YARN service containers and all other application containers) is set via the <tt>yarn.nodemanager.runtime.linux.docker.default-container-network</tt> property in the <tt>yarn-site.xml</tt> file.</p></div> |
| <div class="section"> |
| <h3><a name="Component-level_readiness_check_properties"></a>Component-level readiness check properties</h3> |
| <p>The AM can be configured to perform readiness checks for containers through the <tt>Component</tt> field <tt>readiness_check</tt>. A container will not reach the <tt>READY</tt> state until its readiness check succeeds. If no readiness check is specified, the default readiness check is performed unless it is disabled through the <tt>yarn.service.default-readiness-check.enabled</tt> component-level configuration property.</p> |
| <p>The default readiness check succeeds when an IP becomes available for a container. There are also optional properties that configure a DNS check in addition to the IP check. DNS checking ensures that a DNS lookup succeeds for the container hostname before the container is considered ready. For example, DNS checking can be enabled for the default readiness check as follows:</p> |
| |
| <div> |
| <div> |
| <pre class="source"> "readiness_check": { |
| "type": "DEFAULT", |
| "properties": { |
| "dns.check.enabled": "true" |
| } |
| }, |
| </pre></div></div> |
| |
| <p>Here is a full list of configurable properties for readiness checks that can be performed by the AM.</p> |
| <table border="0" class="bodyTable"> |
| <thead> |
| |
| <tr class="a"> |
| <th> Readiness Check </th> |
| <th> Configurable Property </th> |
| <th> Description </th></tr> |
| </thead><tbody> |
| |
| <tr class="b"> |
| <td>DEFAULT, HTTP, PORT</td> |
| <td> dns.check.enabled </td> |
| <td> true if DNS check should be performed (default false)</td></tr> |
| <tr class="a"> |
| <td>DEFAULT, HTTP, PORT</td> |
| <td> dns.address </td> |
| <td> optional IP:port address of DNS server to use for DNS check</td></tr> |
| <tr class="b"> |
| <td>HTTP</td> |
| <td> url </td> |
| <td> required URL for HTTP response check, e.g. <a class="externalLink" href="http://${THIS_HOST}:8080">http://${THIS_HOST}:8080</a></td></tr> |
| <tr class="a"> |
| <td>HTTP</td> |
| <td> timeout </td> |
| <td> connection timeout (default 1000)</td></tr> |
| <tr class="b"> |
| <td>HTTP</td> |
| <td> min.success </td> |
| <td> minimum response code considered successful (default 200)</td></tr> |
| <tr class="a"> |
| <td>HTTP</td> |
| <td> max.success </td> |
| <td> maximum response code considered successful (default 299)</td></tr> |
| <tr class="b"> |
| <td>PORT</td> |
| <td> port </td> |
| <td> required port for socket connection</td></tr> |
| <tr class="a"> |
| <td>PORT</td> |
| <td> timeout </td> |
| <td> socket connection timeout (default 1000)</td></tr> |
| </tbody> |
| </table> |
| <p>HTTP readiness check example:</p> |
| |
| <div> |
| <div> |
| <pre class="source"> "readiness_check": { |
| "type": "HTTP", |
| "properties": { |
| "url": "http://${THIS_HOST}:8080" |
| } |
| }, |
| </pre></div></div> |
| |
| <p>PORT readiness check example:</p> |
| |
| <div> |
| <div> |
| <pre class="source"> "readiness_check": { |
| "type": "PORT", |
| "properties": { |
| "port": "8080" |
| } |
| }, |
| </pre></div></div> |
| |
| <div class="section"> |
| <h4><a name="Warning_on_configuring_readiness_checks_with_host_network_for_docker_containers"></a>Warning on configuring readiness checks with <tt>host</tt> network for docker containers</h4> |
| <p>When the <tt>host</tt> docker network is configured for a component that has more than one container and the containers are binding to a specific port, there will be a port collision if the containers happen to be allocated on the same host. HTTP and PORT readiness checks will not be valid in this situation. In particular, both containers (the one that successfully binds to the port and the one that does not) may have their HTTP or PORT readiness check succeed since the checks are being performed against the same IP (the host’s IP). A valid configuration for such a service could use the anti-affinity placement policy, ensuring that containers will be assigned on different hosts so that port collisions will not occur.</p></div></div></div> |
| <div class="section"> |
| <h2><a name="Constant_variables_for_custom_service"></a>Constant variables for custom service</h2> |
| <p>The service framework provides some constant variables for user to configure their services. These variables are either dynamically generated by the system or are static ones such as service name defined by the user. User can use these constants in their configurations to be dynamically substituted by the service AM. E.g.</p> |
| |
| <div> |
| <div> |
| <pre class="source">{ |
| "type" : "HADOOP_XML", |
| "dest_file" : "/etc/hadoop/hbase-site.xml", |
| "properties" : { |
| "hbase.regionserver.hostname": "${COMPONENT_INSTANCE_NAME}.${SERVICE_NAME}.${USER}.${DOMAIN}" |
| } |
| } |
| </pre></div></div> |
| |
| <p>Here, <tt>COMPONENT_INSTANCE_NAME</tt> and <tt>SERVICE_NAME</tt> are the constants to be substituted by the system.</p> |
| <p>Suppose the <tt>COMPONENT_INSTANCE_NAME</tt> is <tt>regionserver-0</tt> and <tt>SERVICE_NAME</tt> is defined by user as <tt>hbase</tt>, user name is <tt>devuser</tt> and domain name is <tt>dev.test</tt>. Then, the config will be substituted by the service AM and written in the config file <tt>/etc/hadoop/hbase-site.xml</tt> inside the container as below:</p> |
| |
| <div> |
| <div> |
| <pre class="source"><property> |
| <name>hbase.regionserver.hostname</name> |
| <value>regionserver-0.hbase.devuser.dev.test</value> |
| </property> |
| </pre></div></div> |
| |
| <p>where <tt>regionserver-0</tt> is the actual component instance name assigned by the system for this container.</p> |
| <div class="section"> |
| <div class="section"> |
| <h4><a name="Available_constants:"></a>Available constants:</h4> |
| <table border="0" class="bodyTable"> |
| <thead> |
| |
| <tr class="a"> |
| <th> Name </th> |
| <th> Description </th></tr> |
| </thead><tbody> |
| |
| <tr class="b"> |
| <td> SERVICE_NAME </td> |
| <td> name of the service defined by the user</td></tr> |
| <tr class="a"> |
| <td> USER </td> |
| <td> user who submits the service. Note that user name which has “_” will be converted to use “-”, to conform with DNS hostname RFC format which doesn’t allow “_”, and all characters will be lowercased E.g. “Bob_dev” will be converted to “bob-dev” </td></tr> |
| <tr class="b"> |
| <td> DOMAIN </td> |
| <td> the domain name for the cluster </td></tr> |
| <tr class="a"> |
| <td> COMPONENT_NAME </td> |
| <td> the name for a given component </td></tr> |
| <tr class="b"> |
| <td> COMPONENT_INSTANCE_NAME </td> |
| <td> the name for a given component instance (i.e. container) </td></tr> |
| <tr class="a"> |
| <td> COMPONENT_ID </td> |
| <td> the monotonically increasing integer for a given component</td></tr> |
| <tr class="b"> |
| <td> CONTAINER_ID </td> |
| <td> the YARN container Id for a given container </td></tr> |
| <tr class="a"> |
| <td> ${COMPONENT_INSTANCE_NAME}_HOST </td> |
| <td> the hostname for a component instance (i.e. container), e.g. REGIONSERVER-0_HOST will be substituted by the actual hostname of the component instance. Note all characters must be uppercase. </td></tr> |
| <tr class="b"> |
| <td> ${COMPONENT_INSTANCE_NAME}_IP </td> |
| <td> the ip for a component instance (i.e. container), e.g. REGIONSERVER-0_IP will be substituted by the actual IP address of the component instance. Note all characters must be uppercase. </td></tr> |
| <tr class="a"> |
| <td> CLUSTER_FS_URI </td> |
| <td> the URI of the cluster hdfs </td></tr> |
| </tbody> |
| </table></div></div></div> |
| </div> |
| </div> |
| <div class="clear"> |
| <hr/> |
| </div> |
| <div id="footer"> |
| <div class="xright"> |
| © 2008-2021 |
| Apache Software Foundation |
| |
| - <a href="http://maven.apache.org/privacy-policy.html">Privacy Policy</a>. |
| Apache Maven, Maven, Apache, the Apache feather logo, and the Apache Maven project logos are trademarks of The Apache Software Foundation. |
| </div> |
| <div class="clear"> |
| <hr/> |
| </div> |
| </div> |
| </body> |
| </html> |
