| <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
| <!-- |
| | Generated by Apache Maven Doxia at 2021-06-15 |
| | Rendered using Apache Maven Stylus Skin 1.5 |
| --> |
| <html xmlns="http://www.w3.org/1999/xhtml"> |
| <head> |
| <title>Apache Hadoop 3.3.1 – YARN Node Labels</title> |
| <style type="text/css" media="all"> |
| @import url("./css/maven-base.css"); |
| @import url("./css/maven-theme.css"); |
| @import url("./css/site.css"); |
| </style> |
| <link rel="stylesheet" href="./css/print.css" type="text/css" media="print" /> |
| <meta name="Date-Revision-yyyymmdd" content="20210615" /> |
| <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> |
| </head> |
| <body class="composite"> |
| <div id="banner"> |
| <a href="http://hadoop.apache.org/" id="bannerLeft"> |
| <img src="http://hadoop.apache.org/images/hadoop-logo.jpg" alt="" /> |
| </a> |
| <a href="http://www.apache.org/" id="bannerRight"> |
| <img src="http://www.apache.org/images/asf_logo_wide.png" alt="" /> |
| </a> |
| <div class="clear"> |
| <hr/> |
| </div> |
| </div> |
| <div id="breadcrumbs"> |
| |
| <div class="xleft"> |
| <a href="http://www.apache.org/" class="externalLink">Apache</a> |
| > |
| <a href="http://hadoop.apache.org/" class="externalLink">Hadoop</a> |
| > |
| <a href="../index.html">Apache Hadoop YARN</a> |
| > |
| <a href="index.html">Apache Hadoop 3.3.1</a> |
| > |
| YARN Node Labels |
| </div> |
| <div class="xright"> <a href="http://wiki.apache.org/hadoop" class="externalLink">Wiki</a> |
| | |
| <a href="https://gitbox.apache.org/repos/asf/hadoop.git" class="externalLink">git</a> |
| | |
| <a href="http://hadoop.apache.org/" class="externalLink">Apache Hadoop</a> |
| |
| | Last Published: 2021-06-15 |
| | Version: 3.3.1 |
| </div> |
| <div class="clear"> |
| <hr/> |
| </div> |
| </div> |
| <div id="leftColumn"> |
| <div id="navcolumn"> |
| |
| <h5>General</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../index.html">Overview</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/SingleCluster.html">Single Node Setup</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/ClusterSetup.html">Cluster Setup</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/CommandsManual.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/FileSystemShell.html">FileSystem Shell</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/Compatibility.html">Compatibility Specification</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/DownstreamDev.html">Downstream Developer's Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/AdminCompatibilityGuide.html">Admin Compatibility Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/InterfaceClassification.html">Interface Classification</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/filesystem/index.html">FileSystem Specification</a> |
| </li> |
| </ul> |
| <h5>Common</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/CLIMiniCluster.html">CLI Mini Cluster</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/FairCallQueue.html">Fair Call Queue</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/NativeLibraries.html">Native Libraries</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/Superusers.html">Proxy User</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/RackAwareness.html">Rack Awareness</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/SecureMode.html">Secure Mode</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/ServiceLevelAuth.html">Service Level Authorization</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/HttpAuthentication.html">HTTP Authentication</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/CredentialProviderAPI.html">Credential Provider API</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-kms/index.html">Hadoop KMS</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/Tracing.html">Tracing</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/UnixShellGuide.html">Unix Shell Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/registry/index.html">Registry</a> |
| </li> |
| </ul> |
| <h5>HDFS</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsDesign.html">Architecture</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html">User Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HDFSCommands.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html">NameNode HA With QJM</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html">NameNode HA With NFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html">Observer NameNode</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/Federation.html">Federation</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/ViewFs.html">ViewFs</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/ViewFsOverloadScheme.html">ViewFsOverloadScheme</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html">Snapshots</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsEditsViewer.html">Edits Viewer</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html">Image Viewer</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html">Permissions and HDFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html">Quotas and HDFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/LibHdfs.html">libhdfs (C API)</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/WebHDFS.html">WebHDFS (REST API)</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-hdfs-httpfs/index.html">HttpFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html">Short Circuit Local Reads</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html">Centralized Cache Management</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html">NFS Gateway</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html">Rolling Upgrade</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/ExtendedAttributes.html">Extended Attributes</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html">Transparent Encryption</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html">Multihoming</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html">Storage Policies</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/MemoryStorage.html">Memory Storage Support</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/SLGUserGuide.html">Synthetic Load Generator</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html">Erasure Coding</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HDFSDiskbalancer.html">Disk Balancer</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsUpgradeDomain.html">Upgrade Domain</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsDataNodeAdminGuide.html">DataNode Admin</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html">Router Federation</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsProvidedStorage.html">Provided Storage</a> |
| </li> |
| </ul> |
| <h5>MapReduce</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html">Tutorial</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html">Compatibility with 1.x</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html">Encrypted Shuffle</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html">Pluggable Shuffle/Sort</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html">Distributed Cache Deploy</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/SharedCacheSupport.html">Support for YARN Shared Cache</a> |
| </li> |
| </ul> |
| <h5>MapReduce REST APIs</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredAppMasterRest.html">MR Application Master</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-hs/HistoryServerRest.html">MR History Server</a> |
| </li> |
| </ul> |
| <h5>YARN</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/YARN.html">Architecture</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/YarnCommands.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html">Capacity Scheduler</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/FairScheduler.html">Fair Scheduler</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html">ResourceManager Restart</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html">ResourceManager HA</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/ResourceModel.html">Resource Model</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/NodeLabel.html">Node Labels</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/NodeAttributes.html">Node Attributes</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html">Web Application Proxy</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html">Timeline Server</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html">Timeline Service V.2</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html">Writing YARN Applications</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html">YARN Application Security</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/NodeManager.html">NodeManager</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/DockerContainers.html">Running Applications in Docker Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/RuncContainers.html">Running Applications in runC Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/NodeManagerCgroups.html">Using CGroups</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/SecureContainer.html">Secure Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/ReservationSystem.html">Reservation System</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html">Graceful Decommission</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html">Opportunistic Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/Federation.html">YARN Federation</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/SharedCache.html">Shared Cache</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/UsingGpus.html">Using GPU</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/UsingFPGA.html">Using FPGA</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html">Placement Constraints</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/YarnUI2.html">YARN UI2</a> |
| </li> |
| </ul> |
| <h5>YARN REST APIs</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html">Introduction</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html">Resource Manager</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/NodeManagerRest.html">Node Manager</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html#Timeline_Server_REST_API_v1">Timeline Server</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html#Timeline_Service_v.2_REST_API">Timeline Service V.2</a> |
| </li> |
| </ul> |
| <h5>YARN Service</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/Overview.html">Overview</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/QuickStart.html">QuickStart</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/Concepts.html">Concepts</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/YarnServiceAPI.html">Yarn Service API</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/ServiceDiscovery.html">Service Discovery</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/SystemServices.html">System Services</a> |
| </li> |
| </ul> |
| <h5>Hadoop Compatible File Systems</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-aliyun/tools/hadoop-aliyun/index.html">Aliyun OSS</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-aws/tools/hadoop-aws/index.html">Amazon S3</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-azure/index.html">Azure Blob Storage</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-azure-datalake/index.html">Azure Data Lake Storage</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-openstack/index.html">OpenStack Swift</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-cos/cloud-storage/index.html">Tencent COS</a> |
| </li> |
| </ul> |
| <h5>Auth</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-auth/index.html">Overview</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-auth/Examples.html">Examples</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-auth/Configuration.html">Configuration</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-auth/BuildingIt.html">Building</a> |
| </li> |
| </ul> |
| <h5>Tools</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-streaming/HadoopStreaming.html">Hadoop Streaming</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-archives/HadoopArchives.html">Hadoop Archives</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-archive-logs/HadoopArchiveLogs.html">Hadoop Archive Logs</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-distcp/DistCp.html">DistCp</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-gridmix/GridMix.html">GridMix</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-rumen/Rumen.html">Rumen</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-resourceestimator/ResourceEstimator.html">Resource Estimator Service</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-sls/SchedulerLoadSimulator.html">Scheduler Load Simulator</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/Benchmarking.html">Hadoop Benchmarking</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-dynamometer/Dynamometer.html">Dynamometer</a> |
| </li> |
| </ul> |
| <h5>Reference</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/release/">Changelog and Release Notes</a> |
| </li> |
| <li class="none"> |
| <a href="../../api/index.html">Java API docs</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/UnixShellAPI.html">Unix Shell API</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/Metrics.html">Metrics</a> |
| </li> |
| </ul> |
| <h5>Configuration</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/core-default.xml">core-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/hdfs-default.xml">hdfs-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs-rbf/hdfs-rbf-default.xml">hdfs-rbf-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml">mapred-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-common/yarn-default.xml">yarn-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-kms/kms-default.html">kms-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-hdfs-httpfs/httpfs-default.html">httpfs-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/DeprecatedProperties.html">Deprecated Properties</a> |
| </li> |
| </ul> |
| <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"> |
| <img alt="Built by Maven" src="./images/logos/maven-feather.png"/> |
| </a> |
| |
| </div> |
| </div> |
| <div id="bodyColumn"> |
| <div id="contentBox"> |
| <!--- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. See accompanying LICENSE file. |
| --> |
| <h1>YARN Node Labels</h1> |
| <ul> |
| <li><a href="#Overview">Overview</a></li> |
| <li><a href="#Features">Features</a></li> |
| <li><a href="#Configuration">Configuration</a> |
| <ul> |
| <li><a href="#Setting_up_ResourceManager_to_enable_Node_Labels">Setting up ResourceManager to enable Node Labels</a></li> |
| <li><a href="#Add.2Fmodify_node_labels_list_to_YARN">Add/modify node labels list to YARN</a></li> |
| <li><a href="#Remove_node_labels_from_YARN">Remove node labels from YARN</a></li> |
| <li><a href="#Add.2Fmodify_node-to-labels_mapping_to_YARN">Add/modify node-to-labels mapping to YARN</a></li> |
| <li><a href="#Configuration_of_Schedulers_for_node_labels">Configuration of Schedulers for node labels</a></li></ul></li> |
| <li><a href="#Specifying_node_label_for_application">Specifying node label for application</a></li> |
| <li><a href="#Monitoring">Monitoring</a> |
| <ul> |
| <li><a href="#Monitoring_through_web_UI">Monitoring through web UI</a></li> |
| <li><a href="#Monitoring_through_commandline">Monitoring through commandline</a></li></ul></li> |
| <li><a href="#Useful_links">Useful links</a></li></ul> |
| |
| <div class="section"> |
| <h2><a name="Overview"></a>Overview</h2> |
| <p>Node label is a way to group nodes with similar characteristics and applications can specify where to run.</p> |
| <p>Now we only support node partition, which is:</p> |
| <ul> |
| |
| <li>One node can have only one node partition, so a cluster is partitioned to several disjoint sub-clusters by node partitions. By default, nodes belong to DEFAULT partition (partition="")</li> |
| <li>User need to configure how much resources of each partition can be used by different queues. For more detail, please refer next section.</li> |
| <li>There are two kinds of node partitions: |
| <ul> |
| |
| <li>Exclusive: containers will be allocated to nodes with exactly match node partition. (e.g. asking partition=“x” will be allocated to node with partition=“x”, asking DEFAULT partition will be allocated to DEFAULT partition nodes).</li> |
| <li>Non-exclusive: if a partition is non-exclusive, it shares idle resource to container requesting DEFAULT partition.</li> |
| </ul> |
| </li> |
| </ul> |
| <p>User can specify set of node labels which can be accessed by each queue, one application can only use subset of node labels that can be accessed by the queue which contains the application.</p></div> |
| <div class="section"> |
| <h2><a name="Features"></a>Features</h2> |
| <p>The <tt>Node Labels</tt> supports the following features for now:</p> |
| <ul> |
| |
| <li>Partition cluster - each node can be assigned one label, so the cluster will be divided to several smaller disjoint partitions.</li> |
| <li>ACL of node-labels on queues - user can set accessible node labels on each queue so only some nodes can only be accessed by specific queues.</li> |
| <li>Specify percentage of resource of a partition which can be accessed by a queue - user can set percentage like: queue A can access 30% of resources on nodes with label=hbase. Such percentage setting will be consistent with existing resource manager</li> |
| <li>Specify required node label in resource request, it will only be allocated when node has the same label. If no node label requirement specified, such Resource Request will only be allocated on nodes belong to DEFAULT partition.</li> |
| <li>Operability |
| <ul> |
| |
| <li>Node labels and node labels mapping can be recovered across RM restart</li> |
| <li>Update node labels - admin can update labels on nodes and labels on queues when RM is running</li> |
| </ul> |
| </li> |
| <li>Mapping of NM to node labels can be done in three ways, but in all of the approaches Partition Label should be one among the valid node labels list configured in the RM. |
| <ul> |
| |
| <li><b>Centralized :</b> Node to labels mapping can be done through RM exposed CLI, REST or RPC.</li> |
| <li><b>Distributed :</b> Node to labels mapping will be set by a configured Node Labels Provider in NM. We have two different providers in YARN: <i>Script</i> based provider and <i>Configuration</i> based provider. In case of script, NM can be configured with a script path and the script can emit the labels of the node. In case of config, node Labels can be directly configured in the NM’s yarn-site.xml. In both of these options dynamic refresh of the label mapping is supported.</li> |
| <li><b>Delegated-Centralized :</b> Node to labels mapping will be set by a configured Node Labels Provider in RM. This would be helpful when label mapping cannot be provided by each node due to security concerns and to avoid interaction through RM Interfaces for each node in a large cluster. Labels will be fetched from this interface during NM registration and periodical refresh is also supported.</li> |
| </ul> |
| </li> |
| </ul></div> |
| <div class="section"> |
| <h2><a name="Configuration"></a>Configuration</h2> |
| <div class="section"> |
| <h3><a name="Setting_up_ResourceManager_to_enable_Node_Labels"></a>Setting up ResourceManager to enable Node Labels</h3> |
| <p>Setup following properties in <tt>yarn-site.xml</tt></p> |
| <table border="0" class="bodyTable"> |
| <thead> |
| |
| <tr class="a"> |
| <th>Property </th> |
| <th> Value</th></tr> |
| </thead><tbody> |
| |
| <tr class="b"> |
| <td>yarn.node-labels.fs-store.root-dir </td> |
| <td> <a class="externalLink" href="hdfs://namenode:port/path/to/store/node-labels/">hdfs://namenode:port/path/to/store/node-labels/</a></td></tr> |
| <tr class="a"> |
| <td>yarn.node-labels.enabled </td> |
| <td> true</td></tr> |
| <tr class="b"> |
| <td>yarn.node-labels.configuration-type </td> |
| <td> Set configuration type for node labels. Administrators can specify “centralized”, “delegated-centralized” or “distributed”. Default value is “centralized”.</td></tr> |
| </tbody> |
| </table> |
| <p>Notes:</p> |
| <ul> |
| |
| <li>Make sure <tt>yarn.node-labels.fs-store.root-dir</tt> is created and <tt>ResourceManager</tt> has permission to access it. (Typically from “yarn” user)</li> |
| <li>If user want to store node label to local file system of RM (instead of HDFS), paths like <tt>file:///home/yarn/node-label</tt> can be used</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="Add.2Fmodify_node_labels_list_to_YARN"></a>Add/modify node labels list to YARN</h3> |
| <ul> |
| |
| <li>Add cluster node labels list: |
| <ul> |
| |
| <li>Executing <tt>yarn rmadmin -addToClusterNodeLabels "label_1(exclusive=true/false),label_2(exclusive=true/false)"</tt> to add node label.</li> |
| <li>If user don’t specify “(exclusive=…)”, exclusive will be <tt>true</tt> by default.</li> |
| <li>Run <tt>yarn cluster --list-node-labels</tt> to check added node labels are visible in the cluster.</li> |
| </ul> |
| </li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="Remove_node_labels_from_YARN"></a>Remove node labels from YARN</h3> |
| <ul> |
| |
| <li>Remove cluster node labels: |
| <ul> |
| |
| <li>To remove one or more node labels, execute the following command: <tt>yarn rmadmin -removeFromClusterNodeLabels "<label>[,<label>,...]"</tt>. The command argument should be a comma-separated list of node labels to remove.</li> |
| <li>It is not allowed to remove a label which has been associated with queues, i.e., one or more queues have access to this label.</li> |
| <li>To verify if specified node labels have been successfully removed, run <tt>yarn cluster --list-node-labels</tt>.</li> |
| </ul> |
| </li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="Add.2Fmodify_node-to-labels_mapping_to_YARN"></a>Add/modify node-to-labels mapping to YARN</h3> |
| <ul> |
| |
| <li> |
| |
| <p>Configuring nodes to labels mapping in <b>Centralized</b> NodeLabel setup</p> |
| <ul> |
| |
| <li>Executing <tt>yarn rmadmin -replaceLabelsOnNode “node1[:port]=label1 node2=label2” [-failOnUnknownNodes]</tt>. Added label1 to node1, label2 to node2. If user don’t specify port, it adds the label to all <tt>NodeManagers</tt> running on the node. If option <tt>-failOnUnknownNodes</tt> is set, this command will fail if specified nodes are unknown.</li> |
| </ul> |
| </li> |
| <li> |
| |
| <p>Configuring nodes to labels mapping in <b>Distributed</b> NodeLabel setup</p> |
| </li> |
| </ul> |
| <table border="0" class="bodyTable"> |
| <thead> |
| |
| <tr class="a"> |
| <th>Property </th> |
| <th> Value</th></tr> |
| </thead><tbody> |
| |
| <tr class="b"> |
| <td>yarn.node-labels.configuration-type </td> |
| <td> Needs to be set as <i>“distributed”</i> in RM, to fetch node to labels mapping from a configured Node Labels Provider in NM.</td></tr> |
| <tr class="a"> |
| <td>yarn.nodemanager.node-labels.provider </td> |
| <td> When <i>“yarn.node-labels.configuration-type”</i> is configured with <i>“distributed”</i> in RM, Administrators can configure the provider for the node labels by configuring this parameter in NM. Administrators can configure <i>“config”</i>, <i>“script”</i> or the <i>class name</i> of the provider. Configured class needs to extend <i>org.apache.hadoop.yarn.server.nodemanager.nodelabels.NodeLabelsProvider</i>. If <i>“config”</i> is configured, then <i>“ConfigurationNodeLabelsProvider”</i> and if <i>“script”</i> is configured, then <i>“ScriptNodeLabelsProvider”</i> will be used.</td></tr> |
| <tr class="b"> |
| <td>yarn.nodemanager.node-labels.resync-interval-ms </td> |
| <td> Interval at which NM syncs its node labels with RM. NM will send its loaded labels every x intervals configured, along with heartbeat to RM. This resync is required even when the labels are not modified because admin might have removed the cluster label which was provided by NM. Default is 2 mins.</td></tr> |
| <tr class="a"> |
| <td>yarn.nodemanager.node-labels.provider.fetch-interval-ms </td> |
| <td> When <i>“yarn.nodemanager.node-labels.provider”</i> is configured with <i>“config”</i>, <i>“script”</i> or the <i>configured class</i> extends AbstractNodeLabelsProvider, then periodically node labels are retrieved from the node labels provider. This configuration is to define the interval period. If -1 is configured, then node labels are retrieved from provider only during initialization. Defaults to 10 mins.</td></tr> |
| <tr class="b"> |
| <td>yarn.nodemanager.node-labels.provider.fetch-timeout-ms </td> |
| <td> When <i>“yarn.nodemanager.node-labels.provider”</i> is configured with <i>“script”</i>, then this configuration provides the timeout period after which it will interrupt the script which queries the node labels. Defaults to 20 mins.</td></tr> |
| <tr class="a"> |
| <td>yarn.nodemanager.node-labels.provider.script.path </td> |
| <td> The node label script to run. Script output Line starting with <i>“NODE_PARTITION:”</i> will be considered as node label Partition. In case multiple lines of script output have this pattern, then the last one will be considered.</td></tr> |
| <tr class="b"> |
| <td>yarn.nodemanager.node-labels.provider.script.opts </td> |
| <td> The arguments to pass to the node label script.</td></tr> |
| <tr class="a"> |
| <td>yarn.nodemanager.node-labels.provider.configured-node-partition </td> |
| <td> When <i>“yarn.nodemanager.node-labels.provider”</i> is configured with <i>“config”</i>, then ConfigurationNodeLabelsProvider fetches the partition label from this parameter.</td></tr> |
| </tbody> |
| </table> |
| <ul> |
| |
| <li>Configuring nodes to labels mapping in <b>Delegated-Centralized</b> NodeLabel setup</li> |
| </ul> |
| <table border="0" class="bodyTable"> |
| <thead> |
| |
| <tr class="a"> |
| <th>Property </th> |
| <th> Value</th></tr> |
| </thead><tbody> |
| |
| <tr class="b"> |
| <td>yarn.node-labels.configuration-type </td> |
| <td> Needs to be set as <i>“delegated-centralized”</i> to fetch node to labels mapping from a configured Node Labels Provider in RM.</td></tr> |
| <tr class="a"> |
| <td>yarn.resourcemanager.node-labels.provider </td> |
| <td> When <i>“yarn.node-labels.configuration-type”</i> is configured with <i>“delegated-centralized”</i>, then administrators should configure the class for fetching node labels by ResourceManager. Configured class needs to extend <i>org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsMappingProvider</i>.</td></tr> |
| <tr class="b"> |
| <td>yarn.resourcemanager.node-labels.provider.fetch-interval-ms </td> |
| <td> When <i>“yarn.node-labels.configuration-type”</i> is configured with <i>“delegated-centralized”</i>, then periodically node labels are retrieved from the node labels provider. This configuration is to define the interval. If -1 is configured, then node labels are retrieved from provider only once for each node after it registers. Defaults to 30 mins.</td></tr> |
| </tbody> |
| </table></div> |
| <div class="section"> |
| <h3><a name="Configuration_of_Schedulers_for_node_labels"></a>Configuration of Schedulers for node labels</h3> |
| <ul> |
| |
| <li>Capacity Scheduler Configuration</li> |
| </ul> |
| <table border="0" class="bodyTable"> |
| <thead> |
| |
| <tr class="a"> |
| <th>Property </th> |
| <th> Value</th></tr> |
| </thead><tbody> |
| |
| <tr class="b"> |
| <td>yarn.scheduler.capacity.<tt><queue-path></tt>.capacity </td> |
| <td> Set the percentage of the queue can access to nodes belong to DEFAULT partition. The sum of DEFAULT capacities for direct children under each parent, must be equal to 100.</td></tr> |
| <tr class="a"> |
| <td>yarn.scheduler.capacity.<tt><queue-path></tt>.accessible-node-labels </td> |
| <td> Admin need specify labels can be accessible by each queue, split by comma, like “hbase,storm” means queue can access label hbase and storm. All queues can access to nodes without label, user don’t have to specify that. If user don’t specify this field, it will inherit from its parent. If user want to explicitly specify a queue can only access nodes without labels, just put a space as the value.</td></tr> |
| <tr class="b"> |
| <td>yarn.scheduler.capacity.<tt><queue-path></tt>.accessible-node-labels.<tt><label></tt>.capacity </td> |
| <td> Set the percentage of the queue can access to nodes belong to <tt><label></tt> partition . The sum of <tt><label></tt> capacities for direct children under each parent, must be equal to 100. By default, it’s 0.</td></tr> |
| <tr class="a"> |
| <td>yarn.scheduler.capacity.<tt><queue-path></tt>.accessible-node-labels.<tt><label></tt>.maximum-capacity </td> |
| <td> Similar to yarn.scheduler.capacity.<tt><queue-path></tt>.maximum-capacity, it is for maximum-capacity for labels of each queue. By default, it’s 100.</td></tr> |
| <tr class="b"> |
| <td>yarn.scheduler.capacity.<tt><queue-path></tt>.default-node-label-expression </td> |
| <td> Value like “hbase”, which means: if applications submitted to the queue without specifying node label in their resource requests, it will use “hbase” as default-node-label-expression. By default, this is empty, so application will get containers from nodes without label.</td></tr> |
| </tbody> |
| </table> |
| <p><b>An example of node label configuration</b>:</p> |
| <p>Assume we have a queue structure</p> |
| |
| <div> |
| <div> |
| <pre class="source"> root |
| / | \ |
| engineer sales marketing |
| </pre></div></div> |
| |
| <p>We have 5 nodes (hostname=h1..h5) in the cluster, each of them has 24G memory, 24 vcores. 1 among the 5 nodes has GPU (assume it’s h5). So admin added GPU label to h5.</p> |
| <p>Assume user have a Capacity Scheduler configuration like: (key=value is used here for readability)</p> |
| |
| <div> |
| <div> |
| <pre class="source">yarn.scheduler.capacity.root.queues=engineering,marketing,sales |
| yarn.scheduler.capacity.root.engineering.capacity=33 |
| yarn.scheduler.capacity.root.marketing.capacity=34 |
| yarn.scheduler.capacity.root.sales.capacity=33 |
| |
| yarn.scheduler.capacity.root.engineering.accessible-node-labels=GPU |
| yarn.scheduler.capacity.root.marketing.accessible-node-labels=GPU |
| |
| yarn.scheduler.capacity.root.engineering.accessible-node-labels.GPU.capacity=50 |
| yarn.scheduler.capacity.root.marketing.accessible-node-labels.GPU.capacity=50 |
| |
| yarn.scheduler.capacity.root.engineering.default-node-label-expression=GPU |
| </pre></div></div> |
| |
| <p>You can see root.engineering/marketing/sales.capacity=33, so each of them can has guaranteed resource equals to 1/3 of resource <b>without partition</b>. So each of them can use 1/3 resource of h1..h4, which is 24 * 4 * (1/3) = (32G mem, 32 v-cores).</p> |
| <p>And only engineering/marketing queue has permission to access GPU partition (see root.<tt><queue-name></tt>.accessible-node-labels).</p> |
| <p>Each of engineering/marketing queue has guaranteed resource equals to 1/2 of resource <b>with partition=GPU</b>. So each of them can use 1/2 resource of h5, which is 24 * 0.5 = (12G mem, 12 v-cores).</p> |
| <p>Notes:</p> |
| <ul> |
| |
| <li>After finishing configuration of CapacityScheduler, execute <tt>yarn rmadmin -refreshQueues</tt> to apply changes</li> |
| <li>Go to scheduler page of RM Web UI to check if you have successfully set configuration.</li> |
| </ul></div></div> |
| <div class="section"> |
| <h2><a name="Specifying_node_label_for_application"></a>Specifying node label for application</h2> |
| <p>Applications can use following Java APIs to specify node label to request</p> |
| <ul> |
| |
| <li><tt>ApplicationSubmissionContext.setNodeLabelExpression(..)</tt> to set node label expression for all containers of the application.</li> |
| <li><tt>ResourceRequest.setNodeLabelExpression(..)</tt> to set node label expression for individual resource requests. This can overwrite node label expression set in ApplicationSubmissionContext</li> |
| <li>Specify <tt>setAMContainerResourceRequest.setNodeLabelExpression</tt> in <tt>ApplicationSubmissionContext</tt> to indicate expected node label for application master container.</li> |
| </ul></div> |
| <div class="section"> |
| <h2><a name="Monitoring"></a>Monitoring</h2> |
| <div class="section"> |
| <h3><a name="Monitoring_through_web_UI"></a>Monitoring through web UI</h3> |
| <p>Following label-related fields can be seen on web UI:</p> |
| <ul> |
| |
| <li>Nodes page: <a class="externalLink" href="http://RM-Address:port/cluster/nodes">http://RM-Address:port/cluster/nodes</a>, you can get labels on each node</li> |
| <li>Node labels page: <a class="externalLink" href="http://RM-Address:port/cluster/nodelabels">http://RM-Address:port/cluster/nodelabels</a>, you can get type (exclusive/non-exclusive), number of active node managers, total resource of each partition</li> |
| <li>Scheduler page: <a class="externalLink" href="http://RM-Address:port/cluster/scheduler">http://RM-Address:port/cluster/scheduler</a>, you can get label-related settings of each queue, and resource usage of queue partitions.</li> |
| </ul></div> |
| <div class="section"> |
| <h3><a name="Monitoring_through_commandline"></a>Monitoring through commandline</h3> |
| <ul> |
| |
| <li>Use <tt>yarn cluster --list-node-labels</tt> to get labels in the cluster</li> |
| <li>Use <tt>yarn node -status <NodeId></tt> to get node status including labels on a given node</li> |
| </ul></div></div> |
| <div class="section"> |
| <h2><a name="Useful_links"></a>Useful links</h2> |
| <ul> |
| |
| <li><a href="./CapacityScheduler.html">YARN Capacity Scheduler</a>, if you need more understanding about how to configure Capacity Scheduler</li> |
| <li>Write YARN application using node labels, you can see following two links as examples: <a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-2502">YARN distributed shell</a>, <a class="externalLink" href="https://issues.apache.org/jira/browse/MAPREDUCE-6304">Hadoop MapReduce</a></li> |
| </ul></div> |
| </div> |
| </div> |
| <div class="clear"> |
| <hr/> |
| </div> |
| <div id="footer"> |
| <div class="xright"> |
| © 2008-2021 |
| Apache Software Foundation |
| |
| - <a href="http://maven.apache.org/privacy-policy.html">Privacy Policy</a>. |
| Apache Maven, Maven, Apache, the Apache feather logo, and the Apache Maven project logos are trademarks of The Apache Software Foundation. |
| </div> |
| <div class="clear"> |
| <hr/> |
| </div> |
| </div> |
| </body> |
| </html> |
