| <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
| <!-- |
| | Generated by Apache Maven Doxia at 2021-06-15 |
| | Rendered using Apache Maven Stylus Skin 1.5 |
| --> |
| <html xmlns="http://www.w3.org/1999/xhtml"> |
| <head> |
| <title>Apache Hadoop 3.3.1 – YARN Application Security</title> |
| <style type="text/css" media="all"> |
| @import url("./css/maven-base.css"); |
| @import url("./css/maven-theme.css"); |
| @import url("./css/site.css"); |
| </style> |
| <link rel="stylesheet" href="./css/print.css" type="text/css" media="print" /> |
| <meta name="Date-Revision-yyyymmdd" content="20210615" /> |
| <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> |
| </head> |
| <body class="composite"> |
| <div id="banner"> |
| <a href="http://hadoop.apache.org/" id="bannerLeft"> |
| <img src="http://hadoop.apache.org/images/hadoop-logo.jpg" alt="" /> |
| </a> |
| <a href="http://www.apache.org/" id="bannerRight"> |
| <img src="http://www.apache.org/images/asf_logo_wide.png" alt="" /> |
| </a> |
| <div class="clear"> |
| <hr/> |
| </div> |
| </div> |
| <div id="breadcrumbs"> |
| |
| <div class="xleft"> |
| <a href="http://www.apache.org/" class="externalLink">Apache</a> |
| > |
| <a href="http://hadoop.apache.org/" class="externalLink">Hadoop</a> |
| > |
| <a href="../index.html">Apache Hadoop YARN</a> |
| > |
| <a href="index.html">Apache Hadoop 3.3.1</a> |
| > |
| YARN Application Security |
| </div> |
| <div class="xright"> <a href="http://wiki.apache.org/hadoop" class="externalLink">Wiki</a> |
| | |
| <a href="https://gitbox.apache.org/repos/asf/hadoop.git" class="externalLink">git</a> |
| | |
| <a href="http://hadoop.apache.org/" class="externalLink">Apache Hadoop</a> |
| |
| | Last Published: 2021-06-15 |
| | Version: 3.3.1 |
| </div> |
| <div class="clear"> |
| <hr/> |
| </div> |
| </div> |
| <div id="leftColumn"> |
| <div id="navcolumn"> |
| |
| <h5>General</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../index.html">Overview</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/SingleCluster.html">Single Node Setup</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/ClusterSetup.html">Cluster Setup</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/CommandsManual.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/FileSystemShell.html">FileSystem Shell</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/Compatibility.html">Compatibility Specification</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/DownstreamDev.html">Downstream Developer's Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/AdminCompatibilityGuide.html">Admin Compatibility Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/InterfaceClassification.html">Interface Classification</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/filesystem/index.html">FileSystem Specification</a> |
| </li> |
| </ul> |
| <h5>Common</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/CLIMiniCluster.html">CLI Mini Cluster</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/FairCallQueue.html">Fair Call Queue</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/NativeLibraries.html">Native Libraries</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/Superusers.html">Proxy User</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/RackAwareness.html">Rack Awareness</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/SecureMode.html">Secure Mode</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/ServiceLevelAuth.html">Service Level Authorization</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/HttpAuthentication.html">HTTP Authentication</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/CredentialProviderAPI.html">Credential Provider API</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-kms/index.html">Hadoop KMS</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/Tracing.html">Tracing</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/UnixShellGuide.html">Unix Shell Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/registry/index.html">Registry</a> |
| </li> |
| </ul> |
| <h5>HDFS</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsDesign.html">Architecture</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html">User Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HDFSCommands.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html">NameNode HA With QJM</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html">NameNode HA With NFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html">Observer NameNode</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/Federation.html">Federation</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/ViewFs.html">ViewFs</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/ViewFsOverloadScheme.html">ViewFsOverloadScheme</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html">Snapshots</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsEditsViewer.html">Edits Viewer</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html">Image Viewer</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html">Permissions and HDFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html">Quotas and HDFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/LibHdfs.html">libhdfs (C API)</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/WebHDFS.html">WebHDFS (REST API)</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-hdfs-httpfs/index.html">HttpFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html">Short Circuit Local Reads</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html">Centralized Cache Management</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html">NFS Gateway</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html">Rolling Upgrade</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/ExtendedAttributes.html">Extended Attributes</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html">Transparent Encryption</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html">Multihoming</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html">Storage Policies</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/MemoryStorage.html">Memory Storage Support</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/SLGUserGuide.html">Synthetic Load Generator</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html">Erasure Coding</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HDFSDiskbalancer.html">Disk Balancer</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsUpgradeDomain.html">Upgrade Domain</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsDataNodeAdminGuide.html">DataNode Admin</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html">Router Federation</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsProvidedStorage.html">Provided Storage</a> |
| </li> |
| </ul> |
| <h5>MapReduce</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html">Tutorial</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html">Compatibility with 1.x</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html">Encrypted Shuffle</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html">Pluggable Shuffle/Sort</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html">Distributed Cache Deploy</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/SharedCacheSupport.html">Support for YARN Shared Cache</a> |
| </li> |
| </ul> |
| <h5>MapReduce REST APIs</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredAppMasterRest.html">MR Application Master</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-hs/HistoryServerRest.html">MR History Server</a> |
| </li> |
| </ul> |
| <h5>YARN</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/YARN.html">Architecture</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/YarnCommands.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html">Capacity Scheduler</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/FairScheduler.html">Fair Scheduler</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html">ResourceManager Restart</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html">ResourceManager HA</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/ResourceModel.html">Resource Model</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/NodeLabel.html">Node Labels</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/NodeAttributes.html">Node Attributes</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html">Web Application Proxy</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html">Timeline Server</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html">Timeline Service V.2</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html">Writing YARN Applications</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html">YARN Application Security</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/NodeManager.html">NodeManager</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/DockerContainers.html">Running Applications in Docker Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/RuncContainers.html">Running Applications in runC Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/NodeManagerCgroups.html">Using CGroups</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/SecureContainer.html">Secure Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/ReservationSystem.html">Reservation System</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html">Graceful Decommission</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html">Opportunistic Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/Federation.html">YARN Federation</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/SharedCache.html">Shared Cache</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/UsingGpus.html">Using GPU</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/UsingFPGA.html">Using FPGA</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html">Placement Constraints</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/YarnUI2.html">YARN UI2</a> |
| </li> |
| </ul> |
| <h5>YARN REST APIs</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html">Introduction</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html">Resource Manager</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/NodeManagerRest.html">Node Manager</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html#Timeline_Server_REST_API_v1">Timeline Server</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html#Timeline_Service_v.2_REST_API">Timeline Service V.2</a> |
| </li> |
| </ul> |
| <h5>YARN Service</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/Overview.html">Overview</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/QuickStart.html">QuickStart</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/Concepts.html">Concepts</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/YarnServiceAPI.html">Yarn Service API</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/ServiceDiscovery.html">Service Discovery</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/SystemServices.html">System Services</a> |
| </li> |
| </ul> |
| <h5>Hadoop Compatible File Systems</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-aliyun/tools/hadoop-aliyun/index.html">Aliyun OSS</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-aws/tools/hadoop-aws/index.html">Amazon S3</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-azure/index.html">Azure Blob Storage</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-azure-datalake/index.html">Azure Data Lake Storage</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-openstack/index.html">OpenStack Swift</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-cos/cloud-storage/index.html">Tencent COS</a> |
| </li> |
| </ul> |
| <h5>Auth</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-auth/index.html">Overview</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-auth/Examples.html">Examples</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-auth/Configuration.html">Configuration</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-auth/BuildingIt.html">Building</a> |
| </li> |
| </ul> |
| <h5>Tools</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-streaming/HadoopStreaming.html">Hadoop Streaming</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-archives/HadoopArchives.html">Hadoop Archives</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-archive-logs/HadoopArchiveLogs.html">Hadoop Archive Logs</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-distcp/DistCp.html">DistCp</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-gridmix/GridMix.html">GridMix</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-rumen/Rumen.html">Rumen</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-resourceestimator/ResourceEstimator.html">Resource Estimator Service</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-sls/SchedulerLoadSimulator.html">Scheduler Load Simulator</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/Benchmarking.html">Hadoop Benchmarking</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-dynamometer/Dynamometer.html">Dynamometer</a> |
| </li> |
| </ul> |
| <h5>Reference</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/release/">Changelog and Release Notes</a> |
| </li> |
| <li class="none"> |
| <a href="../../api/index.html">Java API docs</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/UnixShellAPI.html">Unix Shell API</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/Metrics.html">Metrics</a> |
| </li> |
| </ul> |
| <h5>Configuration</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/core-default.xml">core-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/hdfs-default.xml">hdfs-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs-rbf/hdfs-rbf-default.xml">hdfs-rbf-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml">mapred-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-common/yarn-default.xml">yarn-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-kms/kms-default.html">kms-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-hdfs-httpfs/httpfs-default.html">httpfs-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/DeprecatedProperties.html">Deprecated Properties</a> |
| </li> |
| </ul> |
| <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"> |
| <img alt="Built by Maven" src="./images/logos/maven-feather.png"/> |
| </a> |
| |
| </div> |
| </div> |
| <div id="bodyColumn"> |
| <div id="contentBox"> |
| <!--- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. See accompanying LICENSE file. |
| --> |
| <h1>YARN Application Security</h1> |
| <ul> |
| <li><a href="#How_YARN_Security_works">How YARN Security works</a></li> |
| <li><a href="#Other_Aspects_of_YARN_Security">Other Aspects of YARN Security</a></li> |
| <li><a href="#Securing_Long-lived_YARN_Services">Securing Long-lived YARN Services</a></li> |
| <li><a href="#Securing_YARN_Application_Web_UIs_and_REST_APIs">Securing YARN Application Web UIs and REST APIs</a></li> |
| <li><a href="#Securing_YARN_Application_REST_APIs">Securing YARN Application REST APIs</a></li> |
| <li><a href="#Checklist_for_YARN_Applications">Checklist for YARN Applications</a></li> |
| <li><a href="#Testing_YARN_applications_in_a_secure_cluster.">Testing YARN applications in a secure cluster.</a></li> |
| <li><a href="#Important">Important</a></li></ul> |
| |
| <p>Anyone writing a YARN application needs to understand the process, in order to write short-lived applications or long-lived services. They also need to start testing on secure clusters during early development stages, in order to write code that actually works.</p> |
| <div class="section"> |
| <h2><a name="How_YARN_Security_works"></a>How YARN Security works</h2> |
| <p>YARN Resource Managers (RMs) and Node Managers (NMs) co-operate to execute the user’s application with the identity and hence access rights of that user.</p> |
| <p>The (active) Resource Manager:</p> |
| <ol style="list-style-type: decimal"> |
| |
| <li> |
| |
| <p>Finds space in a cluster to deploy the core of the application, the Application Master (AM).</p> |
| </li> |
| <li> |
| |
| <p>Requests that the NM on that node allocate a container and start the AM in it.</p> |
| </li> |
| <li> |
| |
| <p>Communicates with the AM, so that the AM can request new containers and manipulate/release current ones, and to provide notifications about allocated and running containers.</p> |
| </li> |
| </ol> |
| <p>The Node Managers:</p> |
| <ol style="list-style-type: decimal"> |
| |
| <li> |
| |
| <p><i>Localize</i> resources: Download from HDFS or other filesystem into a local directory. This is done using the delegation tokens attached to the container launch context. (For non-HDFS resources, using other credentials such as object store login details in cluster configuration files)</p> |
| </li> |
| <li> |
| |
| <p>Start the application as the user.</p> |
| </li> |
| <li> |
| |
| <p>Monitor the application and report failure to the RM.</p> |
| </li> |
| </ol> |
| <p>To execute code in the cluster, a YARN application must:</p> |
| <ol style="list-style-type: decimal"> |
| |
| <li> |
| |
| <p>Have a client-side application which sets up the <tt>ApplicationSubmissionContext</tt> detailing what is to be launched. This includes:</p> |
| <ul> |
| |
| <li>A list of files in the cluster’s filesystem to be “localized”.</li> |
| <li>The environment variables to set in the container.</li> |
| <li>The commands to execute in the container to start the application.</li> |
| <li>Any security credentials needed by YARN to launch the application.</li> |
| <li>Any security credentials needed by the application to interact with any Hadoop cluster services and applications.</li> |
| </ul> |
| </li> |
| <li> |
| |
| <p>Have an Application Master which, when launched, registers with the YARN RM and listens for events. Any AM which wishes to execute work in other containers must request them off the RM, and, when allocated, create a <tt>ContainerLaunchContext</tt> containing the command to execute, the environment to execute the command, binaries to localize and all relevant security credentials.</p> |
| </li> |
| <li> |
| |
| <p>Even with the NM handling the localization process, the AM must itself be able to retrieve the security credentials supplied at launch time so that it itself may work with HDFS and any other services, and to pass some or all of these credentials down to the launched containers.</p> |
| </li> |
| </ol> |
| <div class="section"> |
| <h3><a name="Acquiring_and_Adding_tokens_to_a_YARN_Application"></a>Acquiring and Adding tokens to a YARN Application</h3> |
| <p>The delegation tokens which a YARN application needs must be acquired from a program executing as an authenticated user. For a YARN application, this means the user launching the application. It is the client-side part of the YARN application which must do this:</p> |
| <ol style="list-style-type: decimal"> |
| |
| <li>Log in via <tt>UserGroupInformation</tt>.</li> |
| <li>Identify all tokens which must be acquired.</li> |
| <li>Request these tokens from the specific Hadoop services.</li> |
| <li>Marshall all tokens into a byte buffer.</li> |
| <li>Add them to the <tt>ContainerLaunchContext</tt> within the <tt>ApplicationSubmissionContext</tt>.</li> |
| </ol> |
| <p>Which tokens are required? Normally, at least a token to access HDFS.</p> |
| <p>An application must request a delegation token from every filesystem with which it intends to interact —including the cluster’s main FS. <tt>FileSystem.addDelegationTokens(renewer, credentials)</tt> can be used to collect these; it is a no-op on those filesystems which do not issue tokens (including non-kerberized HDFS clusters).</p> |
| <p>Applications talking to other services, such as Apache HBase and Apache Hive, must request tokens from these services, using the libraries of these services to acquire delegation tokens. All tokens can be added to the same set of credentials, then saved to a byte buffer for submission.</p> |
| <p>The Application Timeline Server also needs a delegation token. This is handled automatically on AM launch.</p></div> |
| <div class="section"> |
| <h3><a name="Extracting_tokens_within_the_AM"></a>Extracting tokens within the AM</h3> |
| <p>When the Application Master is launched and any of the UGI/Hadoop operations which trigger a user login invoked, the UGI class will automatically load in all tokens saved in the file named by the environment variable <tt>HADOOP_TOKEN_FILE_LOCATION</tt>.</p> |
| <p>This happens on an insecure cluster along with a secure one, and on a secure cluster even if a keytab is used by the application. Why? Because the AM/RM token needed to authenticate the application with the YARN RM is always supplied this way.</p> |
| <p>This means you have a relative similar workflow across secure and insecure clusters.</p> |
| <ol style="list-style-type: decimal"> |
| |
| <li> |
| |
| <p>During AM startup, log in to Kerberos. A call to <tt>UserGroupInformation.isSecurityEnabled()</tt> will trigger this operation.</p> |
| </li> |
| <li> |
| |
| <p>Enumerate the current user’s credentials, through a call of <tt>UserGroupInformation.getCurrentUser().getCredentials()</tt>.</p> |
| </li> |
| <li> |
| |
| <p>Filter out the AMRM token, resulting in a new set of credentials. In an insecure cluster, the list of credentials will now be empty; in a secure cluster they will contain</p> |
| </li> |
| <li> |
| |
| <p>Set the credentials of all containers to be launched to this (possibly empty) list of credentials.</p> |
| </li> |
| <li> |
| |
| <p>If the filtered list of tokens to renew, is non-empty start up a thread to renew them.</p> |
| </li> |
| </ol></div> |
| <div class="section"> |
| <h3><a name="Token_Renewal"></a>Token Renewal</h3> |
| <p>Tokens <i>expire</i>: they have a limited lifespan. An application wishing to use a token past this expiry date must <i>renew</i> the token before the token expires.</p> |
| <p>Hadoop automatically sets up a delegation token renewal thread when needed, the <tt>DelegationTokenRenewer</tt>.</p> |
| <p>It is the responsibility of the application to renew all tokens other than the AMRM and timeline tokens.</p> |
| <p>Here are the different strategies</p> |
| <ol style="list-style-type: decimal"> |
| |
| <li> |
| |
| <p>Don’t rely on the lifespan of the application being so short that token renewal is not needed. For applications whose life can always be measured in minutes or tens of minutes, this is a viable strategy.</p> |
| </li> |
| <li> |
| |
| <p>Start a background thread/Executor to renew the tokens at a regular interval. This what most YARN applications do.</p> |
| </li> |
| </ol></div></div> |
| <div class="section"> |
| <h2><a name="Other_Aspects_of_YARN_Security"></a>Other Aspects of YARN Security</h2> |
| <div class="section"> |
| <h3><a name="AM.2FRM_Token_Refresh"></a>AM/RM Token Refresh</h3> |
| <p>The AM/RM token is renewed automatically; the RM sends out a new token to the AM within an <tt>allocate</tt> message. Consult the <tt>AMRMClientImpl</tt> class to see the process. <i>Your AM code does not need to worry about this process</i></p></div> |
| <div class="section"> |
| <h3><a name="Token_Renewal_on_AM_Restart"></a>Token Renewal on AM Restart</h3> |
| <p>Even if an application is renewing tokens regularly, if an AM fails and is restarted, it gets restarted from that original <tt>ApplicationSubmissionContext</tt>. The tokens there may have expired, so localization may fail, even before the issue of credentials to talk to other services.</p> |
| <p>How is this problem addressed? The YARN Resource Manager gets a new token for the node managers, if needed.</p> |
| <p>More precisely</p> |
| <ol style="list-style-type: decimal"> |
| |
| <li>The token passed by the RM to the NM for localization is refreshed/updated as needed.</li> |
| <li>Tokens in the app launch context for use by the application are <i>not</i> refreshed. That is, if it has an out of date HDFS token —that token is not renewed. This also holds for tokens for for Hive, HBase, etc.</li> |
| <li>Therefore, to survive AM restart after token expiry, your AM has to get the NMs to localize the keytab or make no HDFS accesses until (somehow) a new token has been passed to them from a client.</li> |
| </ol> |
| <p>This is primarily an issue for long-lived services (see below).</p></div> |
| <div class="section"> |
| <h3><a name="Unmanaged_Application_Masters"></a>Unmanaged Application Masters</h3> |
| <p>Unmanaged application masters are not launched in a container set up by the RM and NM, so cannot automatically pick up an AM/RM token at launch time. The <tt>YarnClient.getAMRMToken()</tt> API permits an Unmanaged AM to request an AM/RM token. Consult <tt>UnmanagedAMLauncher</tt> for the specifics.</p></div> |
| <div class="section"> |
| <h3><a name="Identity_on_an_insecure_cluster:_HADOOP_USER_NAME"></a>Identity on an insecure cluster: <tt>HADOOP_USER_NAME</tt></h3> |
| <p>In an insecure cluster, the application will run as the identity of the account of the node manager, such as <tt>yarn</tt> or <tt>mapred</tt>. By default, the application will access HDFS as that user, with a different home directory, and with a different user identified in audit logs and on file system owner attributes.</p> |
| <p>This can be avoided by having the client identify the identify of the HDFS/Hadoop user under which the application is expected to run. <i>This does not affect the OS-level user or the application’s access rights to the local machine</i>.</p> |
| <p>When Kerberos is disabled, the identity of a user is picked up by Hadoop first from the environment variable <tt>HADOOP_USER_NAME</tt>, then from the OS-level username (e.g. the system property <tt>user.name</tt>).</p> |
| <p>YARN applications should propagate the user name of the user launching an application by setting this environment variable.</p> |
| |
| <div> |
| <div> |
| <pre class="source">Map<String, String> env = new HashMap<>(); |
| String userName = UserGroupInformation.getCurrentUser().getUserName(); |
| env.put(UserGroupInformation.HADOOP_USER_NAME, userName); |
| containerLaunchContext.setEnvironment(env); |
| </pre></div></div> |
| |
| <p>Note that this environment variable is picked up in all applications which talk to HDFS via the hadoop libraries. That is, if set, it is the identity picked up by HBase and other applications executed within the environment of a YARN container within which this environment variable is set.</p></div> |
| <div class="section"> |
| <h3><a name="Oozie_integration_and_HADOOP_TOKEN_FILE_LOCATION"></a>Oozie integration and <tt>HADOOP_TOKEN_FILE_LOCATION</tt></h3> |
| <p>Apache Oozie can launch an application in a secure cluster either by acquiring all relevant credentials, saving them to a file in the local filesystem, then setting the path to this file in the environment variable <tt>HADOOP_TOKEN_FILE_LOCATION</tt>. This is of course the same environment variable passed down by YARN in launched containers, as is similar content: a byte array with credentials.</p> |
| <p>Here, however, the environment variable is set in the environment executing the YARN client. This client must use the token information saved in the named file <i>instead of acquiring any tokens of its own</i>.</p> |
| <p>Loading in the token file is automatic: UGI does it during user login.</p> |
| <p>The client is then responsible for passing the same credentials into the AM launch context. This can be done simply by passing down the current credentials.</p> |
| |
| <div> |
| <div> |
| <pre class="source">credentials = new Credentials( |
| UserGroupInformation.getCurrentUser().getCredentials()); |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h3><a name="Timeline_Server_integration"></a>Timeline Server integration</h3> |
| <p>The <a href="TimelineServer.html">Application Timeline Server</a> can be deployed as a secure service —in which case the application will need the relevant token to authenticate with it. This process is handled automatically in <tt>YarnClientImpl</tt> if ATS is enabled in a secure cluster. Similarly, the AM-side <tt>TimelineClient</tt> YARN service class manages token renewal automatically via the ATS’s SPNEGO-authenticated REST API.</p> |
| <p>If you need to prepare a set of delegation tokens for a YARN application launch via Oozie, this can be done via the timeline client API.</p> |
| |
| <div> |
| <div> |
| <pre class="source">try(TimelineClient timelineClient = TimelineClient.createTimelineClient()) { |
| timelineClient.init(conf); |
| timelineClient.start(); |
| Token<TimelineDelegationTokenIdentifier> token = |
| timelineClient.getDelegationToken(rmprincipal)); |
| credentials.addToken(token.getService(), token); |
| } |
| </pre></div></div> |
| </div> |
| <div class="section"> |
| <h3><a name="Cancelling_Tokens"></a>Cancelling Tokens</h3> |
| <p>Applications <i>may</i> wish to cancel tokens they hold when terminating their AM. This ensures that the tokens are no-longer valid.</p> |
| <p>This is not mandatory, and as a clean shutdown of a YARN application cannot be guaranteed, it is not possible to guarantee that the tokens will always be during application termination. However, it does reduce the window of vulnerability to stolen tokens.</p></div></div> |
| <div class="section"> |
| <h2><a name="Securing_Long-lived_YARN_Services"></a>Securing Long-lived YARN Services</h2> |
| <p>There is a time limit on all token renewals, after which tokens won’t renew, causing the application to stop working. This is somewhere between seventy-two hours and seven days.</p> |
| <p>Any YARN service intended to run for an extended period of time <i>must</i> have a strategy for renewing credentials.</p> |
| <p>Here are the strategies:</p> |
| <div class="section"> |
| <h3><a name="Pre-installed_Keytabs_for_AM_and_containers"></a>Pre-installed Keytabs for AM and containers</h3> |
| <p>A keytab is provided for the application’s use on every node.</p> |
| <p>This is done by:</p> |
| <ol style="list-style-type: decimal"> |
| |
| <li>Installing it in every cluster node’s local filesystem.</li> |
| <li>Providing the path to this in a configuration option.</li> |
| <li>The application loading the credentials via <tt>UserGroupInformation.loginUserFromKeytab()</tt>.</li> |
| </ol> |
| <p>The keytab must be in a secure directory path, where only the service (and other trusted accounts) can read it. Distribution becomes a responsibility of the cluster operations team.</p> |
| <p>This is effectively how all static Hadoop applications get their security credentials.</p></div> |
| <div class="section"> |
| <h3><a name="Keytabs_for_AM_and_containers_distributed_via_YARN"></a>Keytabs for AM and containers distributed via YARN</h3> |
| <ol style="list-style-type: decimal"> |
| |
| <li> |
| |
| <p>A keytab is uploaded to HDFS.</p> |
| </li> |
| <li> |
| |
| <p>When launching the AM, the keytab is listed as a resource to localize to the AM’s container.</p> |
| </li> |
| <li> |
| |
| <p>The Application Master is configured with the relative path to the keytab, and logs in with <tt>UserGroupInformation.loginUserFromKeytab()</tt>.</p> |
| </li> |
| <li> |
| |
| <p>When the AM launches the container, it lists the HDFS path to the keytab as a resource to localize.</p> |
| </li> |
| <li> |
| |
| <p>It adds the HDFS delegation token to the container launch context, so that the keytab and other application files can be localized.</p> |
| </li> |
| <li> |
| |
| <p>Launched containers must themselves log in via <tt>UserGroupInformation.loginUserFromKeytab()</tt>. UGI handles the login, and schedules a background thread to relogin the user periodically.</p> |
| </li> |
| <li> |
| |
| <p>Token creation is handled automatically in the Hadoop IPC and REST APIs, the containers stay logged in via kerberos for their entire duration.</p> |
| </li> |
| </ol> |
| <p>This avoids the administration task of installing keytabs for specific services across the entire cluster.</p> |
| <p>It does require the client to have access to the keytab and, as it is uploaded to the distributed filesystem, must be secured through the appropriate path permissions/ACLs.</p> |
| <p>As all containers have access to the keytab, all code executing in the containers has to be trusted. Malicious code (or code escaping some form of sandbox) could read the keytab, and hence have access to the cluster until the keys expire or are revoked.</p> |
| <p>This is the strategy implemented by Apache Slider (incubating).</p></div> |
| <div class="section"> |
| <h3><a name="AM_keytab_distributed_via_YARN.3B_AM_regenerates_delegation_tokens_for_containers."></a>AM keytab distributed via YARN; AM regenerates delegation tokens for containers.</h3> |
| <ol style="list-style-type: decimal"> |
| |
| <li> |
| |
| <p>A keytab is uploaded to HDFS by the client.</p> |
| </li> |
| <li> |
| |
| <p>When launching the AM, the keytab is listed as a resource to localize to the AM’s container.</p> |
| </li> |
| <li> |
| |
| <p>The Application Master is configured with the relative path to the keytab, and logs in with <tt>UserGroupInformation.loginUserFromKeytab()</tt>. The UGI codepath will still automatically load the file references by <tt>$HADOOP_TOKEN_FILE_LOCATION</tt>, which is how the AMRM token is picked up.</p> |
| </li> |
| <li> |
| |
| <p>When the AM launches a container, it acquires all the delegation tokens needed by that container, and adds them to the container’s container launch context.</p> |
| </li> |
| <li> |
| |
| <p>Launched containers must load the delegation tokens from <tt>$HADOOP_TOKEN_FILE_LOCATION</tt>, and use them (including renewals) until they can no longer be renewed.</p> |
| </li> |
| <li> |
| |
| <p>The AM must implement an IPC interface which permits containers to request a new set of delegation tokens; this interface must itself use authentication and ideally wire encryption.</p> |
| </li> |
| <li> |
| |
| <p>Before a delegation token is due to expire, the processes running in the containers must request new tokens from the Application Master over the IPC channel.</p> |
| </li> |
| <li> |
| |
| <p>When the containers need the new tokens, the AM, logged in with a keytab, asks the various cluster services for new tokens.</p> |
| </li> |
| </ol> |
| <p>(Note there is an alternative direction for refresh operations: from AM to the containers, again over whatever IPC channel is implemented between AM and containers). The rest of the algorithm: AM regenerated tokens passed to containers over IPC.</p> |
| <p>This is the strategy used by Apache Spark 1.5+, with a netty-based protocol between containers and the AM for token updates.</p> |
| <p>Because only the AM has direct access to the keytab, it is less exposed. Code running in the containers only has access to the delegation tokens.</p> |
| <p>However, those containers will have access to HDFS from the tokens passed in at container launch, so will have access to the copy of the keytab used for launching the AM. While the AM could delete that keytab on launch, doing so would stop YARN being able to successfully relaunch the AM after any failure.</p></div> |
| <div class="section"> |
| <h3><a name="Client-side_Token_Push"></a>Client-side Token Push</h3> |
| <p>This strategy may be the sole one acceptable to a strict operations team: a client process running on an account holding a Kerberos TGT negotiates with all needed cluster services for new delegation tokens, tokens which are then pushed out to the Application Master via some RPC interface.</p> |
| <p>This does require the client process to be re-executed on a regular basis; a cron or Oozie job can do this. The AM will need to implement an IPC API over which renewed tokens can be provided. (Note that as Oozie can collect the tokens itself, all the updater application needs to do whenever executed is set up an IPC connection with the AM and pass up the current user’s credentials).</p></div></div> |
| <div class="section"> |
| <h2><a name="Securing_YARN_Application_Web_UIs_and_REST_APIs"></a>Securing YARN Application Web UIs and REST APIs</h2> |
| <p>YARN provides a straightforward way of giving every YARN Application SPNEGO authenticated web pages: the RM implements SPNEGO authentication in the Resource Manager Proxy and restricts access to the Yarn Application’s Web UI to only the RM Proxy. There are two ways to do this:</p> |
| <div class="section"> |
| <div class="section"> |
| <h4><a name="Option_1:_AM_IP_Proxy_Filter"></a>Option 1: AM IP Proxy Filter</h4> |
| <p>A YARN Application’s Web Server should load the AM proxy filter (see the <tt>AmFilterInitializer</tt> class) when setting up its web UI; this filter will redirect all HTTP Requests coming from any host other than the RM Proxy hosts to an RM proxy, to which the client app/browser must re-issue the request. The client will authenticate against the principal of the RM Proxy (usually <tt>yarn</tt>), and, once authenticated, have its request forwarded.</p> |
| <p>Known weaknesses in this option are:</p> |
| <ol style="list-style-type: decimal"> |
| |
| <li> |
| |
| <p>The AM proxy filter only checks for the IP/hosts of the RM Proxy so any Application running on those hosts has unrestricted access to the YARN Application’s Web UI. This is why in a secure cluster the proxy hosts <i>must</i> run on cluster nodes which do not run end user code (i.e. not running YARN NodeManagers, and hence not schedule YARN containers; nor support logins by end users).</p> |
| </li> |
| <li> |
| |
| <p>The HTTP requests between RM proxy and the Yarn Application are not currently encrypted. That is: HTTPS is not supported.</p> |
| </li> |
| </ol></div> |
| <div class="section"> |
| <h4><a name="Option_2:_HTTPS_Mutual_Authentication"></a>Option 2: HTTPS Mutual Authentication</h4> |
| <p>By default, YARN Application Web UIs are not encrypted (i.e. HTTPS). It is up to the Application to provide support for HTTPS. This can either be done entirely independently with a valid HTTPS Certificate from a public CA or source that the RM or JVM is configured to trust. Or, alternatively, the RM can act as a limited CA and provide the Application with a Certificate it can use, which is only accepted by the RM proxy, and no other clients (e.g. web browsers). This is important because the Application cannot necessarily be trusted to not steal any issued Certificates or perform other malicious behavior. The Certificates the RM issues will be (a) expired, (b) have a Subject containing <tt>CN=<application-id></tt> instead of the typical <tt>CN=<hostname|domain></tt>, and (c) be issued by a self-signed CA Certificate generated by the RM.</p> |
| <p>For an Application to take advantage of this ability, it simply needs to load the provided Keystore into its Web Server of choice. The location of the Keystore can be found in the <tt>KEYSTORE_FILE_LOCATION</tt> environment variable, and its password in the <tt>KEYSTORE_PASSWORD</tt> environment variable. This will be available as long as <tt>yarn.resourcemanager.application-https.policy</tt> is <i>not</i> set to <tt>NONE</tt> (see table below), and it’s provided an HTTPS Tracking URL.</p> |
| <p>Additionally, the Application can verify that the RM proxy is in fact the RM via HTTPS Mutual Authentication. Besides the provided Keystore, there is also a provided Truststore with the RM proxy’s client Certificate. By loading this Truststore and enabling <tt>needsClientAuth</tt> (or equivalent) in its Web Server of choice, the AM’s Web Server should automatically require that the client (i.e. the RM proxy) provide a trusted Certificate, or it will fail the connection. This ensures that only the RM Proxy, which the client authenticated against, can access it.</p> |
| <table border="0" class="bodyTable"> |
| <thead> |
| |
| <tr class="a"> |
| <th align="left"> <tt>yarn.resourcemanager.application-https.policy</tt> </th> |
| <th align="left"> Behavior </th></tr> |
| </thead><tbody> |
| |
| <tr class="b"> |
| <td align="left"> <tt>NONE</tt> </td> |
| <td align="left"> The RM will do nothing special.</td></tr> |
| <tr class="a"> |
| <td align="left"> <tt>LENIENT</tt> </td> |
| <td align="left"> The RM will generate and provide a keystore and truststore to the AM, which it is free to use for HTTPS in its tracking URL web server. The RM proxy will still allow HTTP connections to AMs that opt not to use HTTPS.</td></tr> |
| <tr class="b"> |
| <td align="left"> <tt>STRICT</tt> </td> |
| <td align="left"> this is the same as LENIENT, except that the RM proxy will only allow HTTPS connections to AMs; HTTP connections will be blocked and result in a warning page to the user.</td></tr> |
| </tbody> |
| </table> |
| <p>The default value is <tt>OFF</tt>.</p></div></div></div> |
| <div class="section"> |
| <h2><a name="Securing_YARN_Application_REST_APIs"></a>Securing YARN Application REST APIs</h2> |
| <p>YARN REST APIs running on the same port as the registered web UI of a YARN application are automatically authenticated via SPNEGO authentication in the RM proxy.</p> |
| <p>Any REST endpoint (and equally, any web UI) brought up on a different port does not support SPNEGO authentication unless implemented in the YARN application itself.</p></div> |
| <div class="section"> |
| <h2><a name="Checklist_for_YARN_Applications"></a>Checklist for YARN Applications</h2> |
| <p>Here is the checklist of core actions which a YARN application must do to successfully launch in a YARN cluster.</p> |
| <div class="section"> |
| <h3><a name="Client"></a>Client</h3> |
| <p><tt>[ ]</tt> Client checks for security being enabled via <tt>UserGroupInformation.isSecurityEnabled()</tt></p> |
| <p>In a secure cluster:</p> |
| <p><tt>[ ]</tt> If <tt>HADOOP_TOKEN_FILE_LOCATION</tt> is unset, client acquires delegation tokens for the local filesystems, with the RM principal set as the renewer.</p> |
| <p><tt>[ ]</tt> If <tt>HADOOP_TOKEN_FILE_LOCATION</tt> is unset, client acquires delegation tokens for all other services to be used in the YARN application.</p> |
| <p><tt>[ ]</tt> If <tt>HADOOP_TOKEN_FILE_LOCATION</tt> is set, client uses the current user’s credentials as the source of all tokens to be added to the container launch context.</p> |
| <p><tt>[ ]</tt> Client sets all tokens on AM <tt>ContainerLaunchContext.setTokens()</tt>.</p> |
| <p><tt>[ ]</tt> Recommended: if it is set in the client’s environment, client sets the environment variable <tt>HADOOP_JAAS_DEBUG=true</tt> in the Container launch context of the AM.</p> |
| <p>In an insecure cluster:</p> |
| <p><tt>[ ]</tt> Propagate local username to YARN AM, hence HDFS identity via the <tt>HADOOP_USER_NAME</tt> environment variable.</p></div> |
| <div class="section"> |
| <h3><a name="App_Master"></a>App Master</h3> |
| <p><tt>[ ]</tt> In a secure cluster, AM retrieves security tokens from <tt>HADOOP_TOKEN_FILE_LOCATION</tt> environment variable (automatically done by UGI).</p> |
| <p><tt>[ ]</tt> A copy the token set is filtered to remove the AM/RM token and any timeline token.</p> |
| <p><tt>[ ]</tt> A thread or executor is started to renew threads on a regular basis.</p> |
| <p><tt>[ ]</tt> Recommended: AM cancels tokens when application completes.</p></div> |
| <div class="section"> |
| <h3><a name="Container_Launch_by_AM"></a>Container Launch by AM</h3> |
| <p><tt>[ ]</tt> Tokens to be passed to containers are passed via <tt>ContainerLaunchContext.setTokens()</tt>.</p> |
| <p><tt>[ ]</tt> In an insecure cluster, propagate the <tt>HADOOP_USER_NAME</tt> environment variable.</p> |
| <p><tt>[ ]</tt> Recommended: AM sets the environment variable <tt>HADOOP_JAAS_DEBUG=true</tt> in the Container launch context if it is set in the AM’s environment.</p></div> |
| <div class="section"> |
| <h3><a name="Launched_Containers"></a>Launched Containers</h3> |
| <p><tt>[ ]</tt> Call <tt>UserGroupInformation.isSecurityEnabled()</tt> to trigger security setup.</p> |
| <p><tt>[ ]</tt> A thread or executor is started to renew threads on a regular basis.</p></div> |
| <div class="section"> |
| <h3><a name="YARN_service"></a>YARN service</h3> |
| <p><tt>[ ]</tt> Application developers have chosen and implemented a token renewal strategy: shared keytab, AM keytab or client-side token refresh.</p> |
| <p><tt>[ ]</tt> In a secure cluster, the keytab is either already in HDFS (and checked for), or it is in the local FS of the client, in which case it must be uploaded and added to the list of resources to localize.</p> |
| <p><tt>[ ]</tt> If stored in HDFS, keytab permissions should be checked. If the keytab is readable by principals other than the current user, warn, and consider actually failing the launch (similar to the normal <tt>ssh</tt> application.)</p> |
| <p><tt>[ ]</tt> Client acquires HDFS delegation token and and attaches to the AM Container Launch Context,</p> |
| <p><tt>[ ]</tt> AM logs in as principal in keytab via <tt>loginUserFromKeytab()</tt>.</p> |
| <p><tt>[ ]</tt> (AM extracts AM/RM token from the <tt>HADOOP_TOKEN_FILE_LOCATION</tt> environment variable).</p> |
| <p><tt>[ ]</tt> For launched containers, either the keytab is propagated, or the AM acquires/attaches all required delegation tokens to the Container Launch context alongside the HDFS delegation token needed by the NMs.</p></div></div> |
| <div class="section"> |
| <h2><a name="Testing_YARN_applications_in_a_secure_cluster."></a>Testing YARN applications in a secure cluster.</h2> |
| <p>It is straightforward to be confident that a YARN application works in secure cluster. The process to do so is: test on a secure cluster.</p> |
| <p>Even a single VM-cluster can be set up with security enabled. If doing so, we recommend turning security up to its strictest, with SPNEGO-authenticated Web UIs (and hence RM Proxy), as well as IPC wire encryption. Setting the kerberos token expiry to under an hour will find kerberos expiry problems early —so is also recommended.</p> |
| <p><tt>[ ]</tt> Application launched in secure cluster.</p> |
| <p><tt>[ ]</tt> Launched application runs as user submitting job (tip: log <tt>user.name</tt> system property in AM).</p> |
| <p><tt>[ ]</tt> Web browser interaction verified in secure cluster.</p> |
| <p><tt>[ ]</tt> REST client interation (GET operations) tested.</p> |
| <p><tt>[ ]</tt> Application continues to run after Kerberos Token expiry.</p> |
| <p><tt>[ ]</tt> Application does not launch if user lacks Kerberos credentials.</p> |
| <p><tt>[ ]</tt> If the application supports the timeline server, verify that it publishes events in a secure cluster.</p> |
| <p><tt>[ ]</tt> If the application integrates with other applications, such as HBase or Hive, verify that the interaction works in a secure cluster.</p> |
| <p><tt>[ ]</tt> If the application communicates with remote HDFS clusters, verify that it can do so in a secure cluster (i.e. that the client extracted any delegation tokens for this at launch time)</p></div> |
| <div class="section"> |
| <h2><a name="Important"></a>Important</h2> |
| <p><i>If you don’t test your YARN application in a secure Hadoop cluster, it won’t work.</i></p> |
| <p>And without those tests: <i>your users will be the ones to find out that your application doesn’t work in a secure cluster.</i></p> |
| <p>Bear that in mind when considering how much development effort to put into Kerberos support.</p></div> |
| </div> |
| </div> |
| <div class="clear"> |
| <hr/> |
| </div> |
| <div id="footer"> |
| <div class="xright"> |
| © 2008-2021 |
| Apache Software Foundation |
| |
| - <a href="http://maven.apache.org/privacy-policy.html">Privacy Policy</a>. |
| Apache Maven, Maven, Apache, the Apache feather logo, and the Apache Maven project logos are trademarks of The Apache Software Foundation. |
| </div> |
| <div class="clear"> |
| <hr/> |
| </div> |
| </div> |
| </body> |
| </html> |
