| <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
| <!-- |
| | Generated by Apache Maven Doxia at 2021-06-15 |
| | Rendered using Apache Maven Stylus Skin 1.5 |
| --> |
| <html xmlns="http://www.w3.org/1999/xhtml"> |
| <head> |
| <title>Apache Hadoop 3.3.1 – MapReduce Commands Guide</title> |
| <style type="text/css" media="all"> |
| @import url("./css/maven-base.css"); |
| @import url("./css/maven-theme.css"); |
| @import url("./css/site.css"); |
| </style> |
| <link rel="stylesheet" href="./css/print.css" type="text/css" media="print" /> |
| <meta name="Date-Revision-yyyymmdd" content="20210615" /> |
| <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> |
| </head> |
| <body class="composite"> |
| <div id="banner"> |
| <a href="http://hadoop.apache.org/" id="bannerLeft"> |
| <img src="http://hadoop.apache.org/images/hadoop-logo.jpg" alt="" /> |
| </a> |
| <a href="http://www.apache.org/" id="bannerRight"> |
| <img src="http://www.apache.org/images/asf_logo_wide.png" alt="" /> |
| </a> |
| <div class="clear"> |
| <hr/> |
| </div> |
| </div> |
| <div id="breadcrumbs"> |
| |
| <div class="xleft"> |
| <a href="http://www.apache.org/" class="externalLink">Apache</a> |
| > |
| <a href="http://hadoop.apache.org/" class="externalLink">Hadoop</a> |
| > |
| <a href="../index.html">Apache Hadoop MapReduce Client</a> |
| > |
| <a href="index.html">Apache Hadoop 3.3.1</a> |
| > |
| MapReduce Commands Guide |
| </div> |
| <div class="xright"> <a href="http://wiki.apache.org/hadoop" class="externalLink">Wiki</a> |
| | |
| <a href="https://gitbox.apache.org/repos/asf/hadoop.git" class="externalLink">git</a> |
| | |
| <a href="http://hadoop.apache.org/" class="externalLink">Apache Hadoop</a> |
| |
| | Last Published: 2021-06-15 |
| | Version: 3.3.1 |
| </div> |
| <div class="clear"> |
| <hr/> |
| </div> |
| </div> |
| <div id="leftColumn"> |
| <div id="navcolumn"> |
| |
| <h5>General</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../index.html">Overview</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/SingleCluster.html">Single Node Setup</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/ClusterSetup.html">Cluster Setup</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/CommandsManual.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/FileSystemShell.html">FileSystem Shell</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/Compatibility.html">Compatibility Specification</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/DownstreamDev.html">Downstream Developer's Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/AdminCompatibilityGuide.html">Admin Compatibility Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/InterfaceClassification.html">Interface Classification</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/filesystem/index.html">FileSystem Specification</a> |
| </li> |
| </ul> |
| <h5>Common</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/CLIMiniCluster.html">CLI Mini Cluster</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/FairCallQueue.html">Fair Call Queue</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/NativeLibraries.html">Native Libraries</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/Superusers.html">Proxy User</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/RackAwareness.html">Rack Awareness</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/SecureMode.html">Secure Mode</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/ServiceLevelAuth.html">Service Level Authorization</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/HttpAuthentication.html">HTTP Authentication</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/CredentialProviderAPI.html">Credential Provider API</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-kms/index.html">Hadoop KMS</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/Tracing.html">Tracing</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/UnixShellGuide.html">Unix Shell Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/registry/index.html">Registry</a> |
| </li> |
| </ul> |
| <h5>HDFS</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsDesign.html">Architecture</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html">User Guide</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HDFSCommands.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html">NameNode HA With QJM</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html">NameNode HA With NFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html">Observer NameNode</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/Federation.html">Federation</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/ViewFs.html">ViewFs</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/ViewFsOverloadScheme.html">ViewFsOverloadScheme</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html">Snapshots</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsEditsViewer.html">Edits Viewer</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html">Image Viewer</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html">Permissions and HDFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html">Quotas and HDFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/LibHdfs.html">libhdfs (C API)</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/WebHDFS.html">WebHDFS (REST API)</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-hdfs-httpfs/index.html">HttpFS</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html">Short Circuit Local Reads</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html">Centralized Cache Management</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html">NFS Gateway</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html">Rolling Upgrade</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/ExtendedAttributes.html">Extended Attributes</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html">Transparent Encryption</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html">Multihoming</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html">Storage Policies</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/MemoryStorage.html">Memory Storage Support</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/SLGUserGuide.html">Synthetic Load Generator</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html">Erasure Coding</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HDFSDiskbalancer.html">Disk Balancer</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsUpgradeDomain.html">Upgrade Domain</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsDataNodeAdminGuide.html">DataNode Admin</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html">Router Federation</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsProvidedStorage.html">Provided Storage</a> |
| </li> |
| </ul> |
| <h5>MapReduce</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html">Tutorial</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html">Compatibility with 1.x</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html">Encrypted Shuffle</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html">Pluggable Shuffle/Sort</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html">Distributed Cache Deploy</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/SharedCacheSupport.html">Support for YARN Shared Cache</a> |
| </li> |
| </ul> |
| <h5>MapReduce REST APIs</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredAppMasterRest.html">MR Application Master</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-hs/HistoryServerRest.html">MR History Server</a> |
| </li> |
| </ul> |
| <h5>YARN</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/YARN.html">Architecture</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/YarnCommands.html">Commands Reference</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html">Capacity Scheduler</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/FairScheduler.html">Fair Scheduler</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html">ResourceManager Restart</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html">ResourceManager HA</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/ResourceModel.html">Resource Model</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/NodeLabel.html">Node Labels</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/NodeAttributes.html">Node Attributes</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html">Web Application Proxy</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html">Timeline Server</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html">Timeline Service V.2</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html">Writing YARN Applications</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html">YARN Application Security</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/NodeManager.html">NodeManager</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/DockerContainers.html">Running Applications in Docker Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/RuncContainers.html">Running Applications in runC Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/NodeManagerCgroups.html">Using CGroups</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/SecureContainer.html">Secure Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/ReservationSystem.html">Reservation System</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html">Graceful Decommission</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html">Opportunistic Containers</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/Federation.html">YARN Federation</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/SharedCache.html">Shared Cache</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/UsingGpus.html">Using GPU</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/UsingFPGA.html">Using FPGA</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html">Placement Constraints</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/YarnUI2.html">YARN UI2</a> |
| </li> |
| </ul> |
| <h5>YARN REST APIs</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html">Introduction</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html">Resource Manager</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/NodeManagerRest.html">Node Manager</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html#Timeline_Server_REST_API_v1">Timeline Server</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html#Timeline_Service_v.2_REST_API">Timeline Service V.2</a> |
| </li> |
| </ul> |
| <h5>YARN Service</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/Overview.html">Overview</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/QuickStart.html">QuickStart</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/Concepts.html">Concepts</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/YarnServiceAPI.html">Yarn Service API</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/ServiceDiscovery.html">Service Discovery</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/SystemServices.html">System Services</a> |
| </li> |
| </ul> |
| <h5>Hadoop Compatible File Systems</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-aliyun/tools/hadoop-aliyun/index.html">Aliyun OSS</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-aws/tools/hadoop-aws/index.html">Amazon S3</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-azure/index.html">Azure Blob Storage</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-azure-datalake/index.html">Azure Data Lake Storage</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-openstack/index.html">OpenStack Swift</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-cos/cloud-storage/index.html">Tencent COS</a> |
| </li> |
| </ul> |
| <h5>Auth</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-auth/index.html">Overview</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-auth/Examples.html">Examples</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-auth/Configuration.html">Configuration</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-auth/BuildingIt.html">Building</a> |
| </li> |
| </ul> |
| <h5>Tools</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-streaming/HadoopStreaming.html">Hadoop Streaming</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-archives/HadoopArchives.html">Hadoop Archives</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-archive-logs/HadoopArchiveLogs.html">Hadoop Archive Logs</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-distcp/DistCp.html">DistCp</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-gridmix/GridMix.html">GridMix</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-rumen/Rumen.html">Rumen</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-resourceestimator/ResourceEstimator.html">Resource Estimator Service</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-sls/SchedulerLoadSimulator.html">Scheduler Load Simulator</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/Benchmarking.html">Hadoop Benchmarking</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-dynamometer/Dynamometer.html">Dynamometer</a> |
| </li> |
| </ul> |
| <h5>Reference</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/release/">Changelog and Release Notes</a> |
| </li> |
| <li class="none"> |
| <a href="../../api/index.html">Java API docs</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/UnixShellAPI.html">Unix Shell API</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/Metrics.html">Metrics</a> |
| </li> |
| </ul> |
| <h5>Configuration</h5> |
| <ul> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/core-default.xml">core-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs/hdfs-default.xml">hdfs-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-hdfs-rbf/hdfs-rbf-default.xml">hdfs-rbf-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml">mapred-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-yarn/hadoop-yarn-common/yarn-default.xml">yarn-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-kms/kms-default.html">kms-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-hdfs-httpfs/httpfs-default.html">httpfs-default.xml</a> |
| </li> |
| <li class="none"> |
| <a href="../../hadoop-project-dist/hadoop-common/DeprecatedProperties.html">Deprecated Properties</a> |
| </li> |
| </ul> |
| <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"> |
| <img alt="Built by Maven" src="./images/logos/maven-feather.png"/> |
| </a> |
| |
| </div> |
| </div> |
| <div id="bodyColumn"> |
| <div id="contentBox"> |
| <!--- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. See accompanying LICENSE file. |
| --> |
| <h1>MapReduce Commands Guide</h1> |
| <ul> |
| <li><a href="#Overview">Overview</a></li> |
| <li><a href="#User_Commands">User Commands</a> |
| <ul> |
| <li><a href="#archive">archive</a></li> |
| <li><a href="#archive-logs">archive-logs</a></li> |
| <li><a href="#classpath">classpath</a></li> |
| <li><a href="#distcp">distcp</a></li> |
| <li><a href="#job">job</a></li> |
| <li><a href="#pipes">pipes</a></li> |
| <li><a href="#queue">queue</a></li> |
| <li><a href="#version">version</a></li> |
| <li><a href="#envvars">envvars</a></li></ul></li> |
| <li><a href="#Administration_Commands">Administration Commands</a> |
| <ul> |
| <li><a href="#historyserver">historyserver</a></li> |
| <li><a href="#hsadmin">hsadmin</a></li> |
| <li><a href="#frameworkuploader">frameworkuploader</a></li></ul></li></ul> |
| |
| <div class="section"> |
| <h2><a name="Overview"></a>Overview</h2> |
| <p>All mapreduce commands are invoked by the <tt>bin/mapred</tt> script. Running the mapred script without any arguments prints the description for all commands.</p> |
| <p>Usage: <tt>mapred [SHELL_OPTIONS] COMMAND [GENERIC_OPTIONS] [COMMAND_OPTIONS]</tt></p> |
| <p>Hadoop has an option parsing framework that employs parsing generic options as well as running classes.</p> |
| <table border="0" class="bodyTable"> |
| <thead> |
| |
| <tr class="a"> |
| <th align="left"> COMMAND_OPTIONS </th> |
| <th align="left"> Description </th></tr> |
| </thead><tbody> |
| |
| <tr class="b"> |
| <td align="left"> SHELL_OPTIONS </td> |
| <td align="left"> The common set of shell options. These are documented on the <a href="../../hadoop-project-dist/hadoop-common/CommandsManual.html#Shell_Options">Hadoop Commands Reference</a> page. </td></tr> |
| <tr class="a"> |
| <td align="left"> GENERIC_OPTIONS </td> |
| <td align="left"> The common set of options supported by multiple commands. See the <a href="../../hadoop-project-dist/hadoop-common/CommandsManual.html#Generic_Options">Hadoop Commands Reference</a> for more information. </td></tr> |
| <tr class="b"> |
| <td align="left"> COMMAND COMMAND_OPTIONS </td> |
| <td align="left"> Various commands with their options are described in the following sections. The commands have been grouped into <a href="#User_Commands">User Commands</a> and <a href="#Administration_Commands">Administration Commands</a>. </td></tr> |
| </tbody> |
| </table></div> |
| <div class="section"> |
| <h2><a name="User_Commands"></a>User Commands</h2> |
| <p>Commands useful for users of a hadoop cluster.</p> |
| <div class="section"> |
| <h3><a name="archive"></a><tt>archive</tt></h3> |
| <p>Creates a hadoop archive. More information can be found at <a href="../../hadoop-archives/HadoopArchives.html">Hadoop Archives Guide</a>.</p></div> |
| <div class="section"> |
| <h3><a name="archive-logs"></a><tt>archive-logs</tt></h3> |
| <p>A tool to combine YARN aggregated logs into Hadoop archives to reduce the number of files in HDFS. More information can be found at <a href="../../hadoop-archive-logs/HadoopArchiveLogs.html">Hadoop Archive Logs Guide</a>.</p></div> |
| <div class="section"> |
| <h3><a name="classpath"></a><tt>classpath</tt></h3> |
| <p>Usage: <tt>yarn classpath [--glob |--jar <path> |-h |--help]</tt></p> |
| <table border="0" class="bodyTable"> |
| <thead> |
| |
| <tr class="a"> |
| <th align="left"> COMMAND_OPTION </th> |
| <th align="left"> Description </th></tr> |
| </thead><tbody> |
| |
| <tr class="b"> |
| <td align="left"> <tt>--glob</tt> </td> |
| <td align="left"> expand wildcards </td></tr> |
| <tr class="a"> |
| <td align="left"> <tt>--jar</tt> <i>path</i> </td> |
| <td align="left"> write classpath as manifest in jar named <i>path</i> </td></tr> |
| <tr class="b"> |
| <td align="left"> <tt>-h</tt>, <tt>--help</tt> </td> |
| <td align="left"> print help </td></tr> |
| </tbody> |
| </table> |
| <p>Prints the class path needed to get the Hadoop jar and the required libraries. If called without arguments, then prints the classpath set up by the command scripts, which is likely to contain wildcards in the classpath entries. Additional options print the classpath after wildcard expansion or write the classpath into the manifest of a jar file. The latter is useful in environments where wildcards cannot be used and the expanded classpath exceeds the maximum supported command line length.</p></div> |
| <div class="section"> |
| <h3><a name="distcp"></a><tt>distcp</tt></h3> |
| <p>Copy file or directories recursively. More information can be found at <a href="../../hadoop-distcp/DistCp.html">Hadoop DistCp Guide</a>.</p></div> |
| <div class="section"> |
| <h3><a name="job"></a><tt>job</tt></h3> |
| <p>Command to interact with Map Reduce Jobs.</p> |
| <p>Usage: <tt>mapred job | [GENERIC_OPTIONS] | [-submit <job-file>] | [-status <job-id>] | [-counter <job-id> <group-name> <counter-name>] | [-kill <job-id>] | [-events <job-id> <from-event-#> <#-of-events>] | [-history [all] <jobHistoryFile|jobId> [-outfile <file>] [-format <human|json>]] | [-list [all]] | [-kill-task <task-id>] | [-fail-task <task-id>] | [-set-priority <job-id> <priority>] | [-list-active-trackers] | [-list-blacklisted-trackers] | [-list-attempt-ids <job-id> <task-type> <task-state>] [-logs <job-id> <task-attempt-id>] [-config <job-id> <file>]</tt></p> |
| <table border="0" class="bodyTable"> |
| <thead> |
| |
| <tr class="a"> |
| <th align="left"> COMMAND_OPTION </th> |
| <th align="left"> Description </th></tr> |
| </thead><tbody> |
| |
| <tr class="b"> |
| <td align="left"> -submit <i>job-file</i> </td> |
| <td align="left"> Submits the job. </td></tr> |
| <tr class="a"> |
| <td align="left"> -status <i>job-id</i> </td> |
| <td align="left"> Prints the map and reduce completion percentage and all job counters. </td></tr> |
| <tr class="b"> |
| <td align="left"> -counter <i>job-id</i> <i>group-name</i> <i>counter-name</i> </td> |
| <td align="left"> Prints the counter value. </td></tr> |
| <tr class="a"> |
| <td align="left"> -kill <i>job-id</i> </td> |
| <td align="left"> Kills the job. </td></tr> |
| <tr class="b"> |
| <td align="left"> -events <i>job-id</i> <i>from-event-#</i> <i>#-of-events</i> </td> |
| <td align="left"> Prints the events’ details received by jobtracker for the given range. </td></tr> |
| <tr class="a"> |
| <td align="left"> -history [all] <i>jobHistoryFilejobId</i> [-outfile <i>file</i>] [-format <i>humanjson</i>] </td> |
| <td align="left"> Prints job details, failed and killed task details. More details about the job such as successful tasks, task attempts made for each task, task counters, etc can be viewed by specifying the [all] option. An optional file output path (instead of stdout) can be specified. The format defaults to human-readable but can also be changed to JSON with the [-format] option. </td></tr> |
| <tr class="b"> |
| <td align="left"> -list [all] </td> |
| <td align="left"> Displays jobs which are yet to complete. <tt>-list all</tt> displays all jobs. </td></tr> |
| <tr class="a"> |
| <td align="left"> -kill-task <i>task-id</i> </td> |
| <td align="left"> Kills the task. Killed tasks are NOT counted against failed attempts. </td></tr> |
| <tr class="b"> |
| <td align="left"> -fail-task <i>task-id</i> </td> |
| <td align="left"> Fails the task. Failed tasks are counted against failed attempts. </td></tr> |
| <tr class="a"> |
| <td align="left"> -set-priority <i>job-id</i> <i>priority</i> </td> |
| <td align="left"> Changes the priority of the job. Allowed priority values are VERY_HIGH, HIGH, NORMAL, LOW, VERY_LOW </td></tr> |
| <tr class="b"> |
| <td align="left"> -list-active-trackers </td> |
| <td align="left"> List all the active NodeManagers in the cluster. </td></tr> |
| <tr class="a"> |
| <td align="left"> -list-blacklisted-trackers </td> |
| <td align="left"> List the black listed task trackers in the cluster. This command is not supported in MRv2 based cluster. </td></tr> |
| <tr class="b"> |
| <td align="left"> -list-attempt-ids <i>job-id</i> <i>task-type</i> <i>task-state</i> </td> |
| <td align="left"> List the attempt-ids based on the task type and the status given. Valid values for task-type are REDUCE, MAP. Valid values for task-state are running, pending, completed, failed, killed. </td></tr> |
| <tr class="a"> |
| <td align="left"> -logs <i>job-id</i> <i>task-attempt-id</i> </td> |
| <td align="left"> Dump the container log for a job if taskAttemptId is not specified, otherwise dump the log for the task with the specified taskAttemptId. The logs will be dumped in system out. </td></tr> |
| <tr class="b"> |
| <td align="left"> -config <i>job-id</i> <i>file</i> </td> |
| <td align="left"> Download the job configuration file. </td></tr> |
| </tbody> |
| </table></div> |
| <div class="section"> |
| <h3><a name="pipes"></a><tt>pipes</tt></h3> |
| <p>Runs a pipes job.</p> |
| <p>Usage: <tt>mapred pipes [-conf <path>] [-jobconf <key=value>, <key=value>, ...] [-input <path>] [-output <path>] [-jar <jar file>] [-inputformat <class>] [-map <class>] [-partitioner <class>] [-reduce <class>] [-writer <class>] [-program <executable>] [-reduces <num>]</tt></p> |
| <table border="0" class="bodyTable"> |
| <thead> |
| |
| <tr class="a"> |
| <th align="left"> COMMAND_OPTION </th> |
| <th align="left"> Description </th></tr> |
| </thead><tbody> |
| |
| <tr class="b"> |
| <td align="left"> -conf <i>path</i> </td> |
| <td align="left"> Configuration for job </td></tr> |
| <tr class="a"> |
| <td align="left"> -jobconf <i>key=value</i>, <i>key=value</i>, … </td> |
| <td align="left"> Add/override configuration for job </td></tr> |
| <tr class="b"> |
| <td align="left"> -input <i>path</i> </td> |
| <td align="left"> Input directory </td></tr> |
| <tr class="a"> |
| <td align="left"> -output <i>path</i> </td> |
| <td align="left"> Output directory </td></tr> |
| <tr class="b"> |
| <td align="left"> -jar <i>jar file</i> </td> |
| <td align="left"> Jar filename </td></tr> |
| <tr class="a"> |
| <td align="left"> -inputformat <i>class</i> </td> |
| <td align="left"> InputFormat class </td></tr> |
| <tr class="b"> |
| <td align="left"> -map <i>class</i> </td> |
| <td align="left"> Java Map class </td></tr> |
| <tr class="a"> |
| <td align="left"> -partitioner <i>class</i> </td> |
| <td align="left"> Java Partitioner </td></tr> |
| <tr class="b"> |
| <td align="left"> -reduce <i>class</i> </td> |
| <td align="left"> Java Reduce class </td></tr> |
| <tr class="a"> |
| <td align="left"> -writer <i>class</i> </td> |
| <td align="left"> Java RecordWriter </td></tr> |
| <tr class="b"> |
| <td align="left"> -program <i>executable</i> </td> |
| <td align="left"> Executable URI </td></tr> |
| <tr class="a"> |
| <td align="left"> -reduces <i>num</i> </td> |
| <td align="left"> Number of reduces </td></tr> |
| </tbody> |
| </table></div> |
| <div class="section"> |
| <h3><a name="queue"></a><tt>queue</tt></h3> |
| <p>command to interact and view Job Queue information</p> |
| <p>Usage: <tt>mapred queue [-list] | [-info <job-queue-name> [-showJobs]] | [-showacls]</tt></p> |
| <table border="0" class="bodyTable"> |
| <thead> |
| |
| <tr class="a"> |
| <th align="left"> COMMAND_OPTION </th> |
| <th align="left"> Description </th></tr> |
| </thead><tbody> |
| |
| <tr class="b"> |
| <td align="left"> -list </td> |
| <td align="left"> Gets list of Job Queues configured in the system. Along with scheduling information associated with the job queues. </td></tr> |
| <tr class="a"> |
| <td align="left"> -info <i>job-queue-name</i> [-showJobs] </td> |
| <td align="left"> Displays the job queue information and associated scheduling information of particular job queue. If <tt>-showJobs</tt> options is present a list of jobs submitted to the particular job queue is displayed. </td></tr> |
| <tr class="b"> |
| <td align="left"> -showacls </td> |
| <td align="left"> Displays the queue name and associated queue operations allowed for the current user. The list consists of only those queues to which the user has access. </td></tr> |
| </tbody> |
| </table></div> |
| <div class="section"> |
| <h3><a name="version"></a><tt>version</tt></h3> |
| <p>Prints the version.</p> |
| <p>Usage: <tt>mapred version</tt></p></div> |
| <div class="section"> |
| <h3><a name="envvars"></a><tt>envvars</tt></h3> |
| <p>Usage: <tt>mapred envvars</tt></p> |
| <p>Display computed Hadoop environment variables.</p></div></div> |
| <div class="section"> |
| <h2><a name="Administration_Commands"></a>Administration Commands</h2> |
| <p>Commands useful for administrators of a hadoop cluster.</p> |
| <div class="section"> |
| <h3><a name="historyserver"></a><tt>historyserver</tt></h3> |
| <p>Start JobHistoryServer.</p> |
| <p>Usage: <tt>mapred historyserver</tt></p></div> |
| <div class="section"> |
| <h3><a name="hsadmin"></a><tt>hsadmin</tt></h3> |
| <p>Runs a MapReduce hsadmin client for execute JobHistoryServer administrative commands.</p> |
| <p>Usage: <tt>mapred hsadmin [-refreshUserToGroupsMappings] | [-refreshSuperUserGroupsConfiguration] | [-refreshAdminAcls] | [-refreshLoadedJobCache] | [-refreshLogRetentionSettings] | [-refreshJobRetentionSettings] | [-getGroups [username]] | [-help [cmd]]</tt></p> |
| <table border="0" class="bodyTable"> |
| <thead> |
| |
| <tr class="a"> |
| <th align="left"> COMMAND_OPTION </th> |
| <th align="left"> Description </th></tr> |
| </thead><tbody> |
| |
| <tr class="b"> |
| <td align="left"> -refreshUserToGroupsMappings </td> |
| <td align="left"> Refresh user-to-groups mappings </td></tr> |
| <tr class="a"> |
| <td align="left"> -refreshSuperUserGroupsConfiguration </td> |
| <td align="left"> Refresh superuser proxy groups mappings </td></tr> |
| <tr class="b"> |
| <td align="left"> -refreshAdminAcls </td> |
| <td align="left"> Refresh acls for administration of Job history server </td></tr> |
| <tr class="a"> |
| <td align="left"> -refreshLoadedJobCache </td> |
| <td align="left"> Refresh loaded job cache of Job history server </td></tr> |
| <tr class="b"> |
| <td align="left"> -refreshJobRetentionSettings </td> |
| <td align="left"> Refresh job history period, job cleaner settings </td></tr> |
| <tr class="a"> |
| <td align="left"> -refreshLogRetentionSettings </td> |
| <td align="left"> Refresh log retention period and log retention check interval </td></tr> |
| <tr class="b"> |
| <td align="left"> -getGroups [username] </td> |
| <td align="left"> Get the groups which given user belongs to </td></tr> |
| <tr class="a"> |
| <td align="left"> -help [cmd] </td> |
| <td align="left"> Displays help for the given command or all commands if none is specified. </td></tr> |
| </tbody> |
| </table></div> |
| <div class="section"> |
| <h3><a name="frameworkuploader"></a><tt>frameworkuploader</tt></h3> |
| <p>Collects framework jars and uploads them to HDFS as a tarball.</p> |
| <p>Usage: <tt>mapred frameworkuploader -target <target> [-fs <filesystem>] [-input <classpath>] [-blacklist <list>] [-whitelist <list>] [-initialReplication <num>] [-acceptableReplication <num>] [-finalReplication <num>] [-timeout <seconds>] [-nosymlink]</tt></p> |
| <table border="0" class="bodyTable"> |
| <thead> |
| |
| <tr class="a"> |
| <th align="left"> COMMAND_OPTION </th> |
| <th align="left"> Description </th></tr> |
| </thead><tbody> |
| |
| <tr class="b"> |
| <td align="left"> -input <i>classpath</i> </td> |
| <td align="left"> This is the input classpath that is searched for jar files to be included in the tarball. </td></tr> |
| <tr class="a"> |
| <td align="left"> -fs <i>filesystem</i> </td> |
| <td align="left"> The target file system. Defaults to the default filesystem set by fs.defaultFS. </td></tr> |
| <tr class="b"> |
| <td align="left"> -target <i>target</i> </td> |
| <td align="left"> This is the target location of the framework tarball, optionally followed by a # with the localized alias. An example would be /usr/lib/framework.tar#framework. Make sure the target directory is readable by all users but it is not writable by others than administrators to protect cluster security.</td></tr> |
| <tr class="a"> |
| <td align="left"> -blacklist <i>list</i> </td> |
| <td align="left"> This is a comma separated regex array to filter the jar file names to exclude from the class path. It can be used for example to exclude test jars or Hadoop services that are not necessary to localize. </td></tr> |
| <tr class="b"> |
| <td align="left"> -whitelist <i>list</i> </td> |
| <td align="left"> This is a comma separated regex array to include certain jar files. This can be used to provide additional security, so that no external source can include malicious code in the classpath when the tool runs. </td></tr> |
| <tr class="a"> |
| <td align="left"> -nosymlink </td> |
| <td align="left"> This flag can be used to exclude symlinks that point to the same directory. This is not widely used. For example, <tt>/a/foo.jar</tt> and a symlink <tt>/a/bar.jar</tt> that points to <tt>/a/foo.jar</tt> would normally add <tt>foo.jar</tt> and <tt>bar.jar</tt> to the tarball as separate files despite them actually being the same file. This flag would make the tool exclude <tt>/a/bar.jar</tt> so only one copy of the file is added. </td></tr> |
| <tr class="b"> |
| <td align="left"> -initialReplication <i>num</i> </td> |
| <td align="left"> This is the replication count that the framework tarball is created with. It is safe to leave this value at the default 3. This is the tested scenario. </td></tr> |
| <tr class="a"> |
| <td align="left"> -finalReplication <i>num</i> </td> |
| <td align="left"> The uploader tool sets the replication once all blocks are collected and uploaded. If quick initial startup is required, then it is advised to set this to the commissioned node count divided by two but not more than 512. </td></tr> |
| <tr class="b"> |
| <td align="left"> -acceptableReplication <i>num</i> </td> |
| <td align="left"> The tool will wait until the tarball has been replicated this number of times before exiting. This should be a replication count less than or equal to the value in <tt>finalReplication</tt>. This is typically a 90% of the value in <tt>finalReplication</tt> to accomodate failing nodes. </td></tr> |
| <tr class="a"> |
| <td align="left"> -timeout <i>seconds</i> </td> |
| <td align="left"> A timeout in seconds to wait to reach <tt>acceptableReplication</tt> before the tool exits. The tool logs an error otherwise and returns.</td></tr> |
| </tbody> |
| </table></div></div> |
| </div> |
| </div> |
| <div class="clear"> |
| <hr/> |
| </div> |
| <div id="footer"> |
| <div class="xright"> |
| © 2008-2021 |
| Apache Software Foundation |
| |
| - <a href="http://maven.apache.org/privacy-policy.html">Privacy Policy</a>. |
| Apache Maven, Maven, Apache, the Apache feather logo, and the Apache Maven project logos are trademarks of The Apache Software Foundation. |
| </div> |
| <div class="clear"> |
| <hr/> |
| </div> |
| </div> |
| </body> |
| </html> |
