Google Git
Sign in
apache / hadoop-site / 952d547359cafef280f0a0324055741b2ed55be1 / . / content / docs / r3.3.1 / hadoop-streaming / HadoopStreaming.html
blob: 577b5c412ded5b362fbfd62a1d98f334ae680377 [file] [log] [blame]
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<!--
| Generated by Apache Maven Doxia at 2021-06-15
| Rendered using Apache Maven Stylus Skin 1.5
-->
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Apache Hadoop MapReduce Streaming &#x2013; Hadoop Streaming</title>
<style type="text/css" media="all">
@import url("./css/maven-base.css");
@import url("./css/maven-theme.css");
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
<meta name="Date-Revision-yyyymmdd" content="20210615" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body class="composite">
<div id="banner">
<a href="http://hadoop.apache.org/" id="bannerLeft">
<img src="http://hadoop.apache.org/images/hadoop-logo.jpg" alt="" />
</a>
<a href="http://www.apache.org/" id="bannerRight">
<img src="http://www.apache.org/images/asf_logo_wide.png" alt="" />
</a>
<div class="clear">
<hr/>
</div>
</div>
<div id="breadcrumbs">
<div class="xleft">
<a href="http://www.apache.org/" class="externalLink">Apache</a>
&gt;
<a href="http://hadoop.apache.org/" class="externalLink">Hadoop</a>
&gt;
<a href="index.html">Apache Hadoop MapReduce Streaming</a>
&gt;
Hadoop Streaming
</div>
<div class="xright"> <a href="http://wiki.apache.org/hadoop" class="externalLink">Wiki</a>
|
<a href="https://gitbox.apache.org/repos/asf/hadoop.git" class="externalLink">git</a>
&nbsp;| Last Published: 2021-06-15
&nbsp;| Version: 3.3.1
</div>
<div class="clear">
<hr/>
</div>
</div>
<div id="leftColumn">
<div id="navcolumn">
<h5>General</h5>
<ul>
<li class="none">
<a href="../index.html">Overview</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/SingleCluster.html">Single Node Setup</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/ClusterSetup.html">Cluster Setup</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/CommandsManual.html">Commands Reference</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/FileSystemShell.html">FileSystem Shell</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/Compatibility.html">Compatibility Specification</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/DownstreamDev.html">Downstream Developer's Guide</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/AdminCompatibilityGuide.html">Admin Compatibility Guide</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/InterfaceClassification.html">Interface Classification</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/filesystem/index.html">FileSystem Specification</a>
</li>
</ul>
<h5>Common</h5>
<ul>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/CLIMiniCluster.html">CLI Mini Cluster</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/FairCallQueue.html">Fair Call Queue</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/NativeLibraries.html">Native Libraries</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/Superusers.html">Proxy User</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/RackAwareness.html">Rack Awareness</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/SecureMode.html">Secure Mode</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/ServiceLevelAuth.html">Service Level Authorization</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/HttpAuthentication.html">HTTP Authentication</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/CredentialProviderAPI.html">Credential Provider API</a>
</li>
<li class="none">
<a href="../hadoop-kms/index.html">Hadoop KMS</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/Tracing.html">Tracing</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/UnixShellGuide.html">Unix Shell Guide</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/registry/index.html">Registry</a>
</li>
</ul>
<h5>HDFS</h5>
<ul>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/HdfsDesign.html">Architecture</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html">User Guide</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/HDFSCommands.html">Commands Reference</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html">NameNode HA With QJM</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html">NameNode HA With NFS</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html">Observer NameNode</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/Federation.html">Federation</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/ViewFs.html">ViewFs</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/ViewFsOverloadScheme.html">ViewFsOverloadScheme</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html">Snapshots</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/HdfsEditsViewer.html">Edits Viewer</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html">Image Viewer</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html">Permissions and HDFS</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html">Quotas and HDFS</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/LibHdfs.html">libhdfs (C API)</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/WebHDFS.html">WebHDFS (REST API)</a>
</li>
<li class="none">
<a href="../hadoop-hdfs-httpfs/index.html">HttpFS</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html">Short Circuit Local Reads</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html">Centralized Cache Management</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html">NFS Gateway</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html">Rolling Upgrade</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/ExtendedAttributes.html">Extended Attributes</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html">Transparent Encryption</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html">Multihoming</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html">Storage Policies</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/MemoryStorage.html">Memory Storage Support</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/SLGUserGuide.html">Synthetic Load Generator</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html">Erasure Coding</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/HDFSDiskbalancer.html">Disk Balancer</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/HdfsUpgradeDomain.html">Upgrade Domain</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/HdfsDataNodeAdminGuide.html">DataNode Admin</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html">Router Federation</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/HdfsProvidedStorage.html">Provided Storage</a>
</li>
</ul>
<h5>MapReduce</h5>
<ul>
<li class="none">
<a href="../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html">Tutorial</a>
</li>
<li class="none">
<a href="../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html">Commands Reference</a>
</li>
<li class="none">
<a href="../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html">Compatibility with 1.x</a>
</li>
<li class="none">
<a href="../hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html">Encrypted Shuffle</a>
</li>
<li class="none">
<a href="../hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html">Pluggable Shuffle/Sort</a>
</li>
<li class="none">
<a href="../hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html">Distributed Cache Deploy</a>
</li>
<li class="none">
<a href="../hadoop-mapreduce-client/hadoop-mapreduce-client-core/SharedCacheSupport.html">Support for YARN Shared Cache</a>
</li>
</ul>
<h5>MapReduce REST APIs</h5>
<ul>
<li class="none">
<a href="../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredAppMasterRest.html">MR Application Master</a>
</li>
<li class="none">
<a href="../hadoop-mapreduce-client/hadoop-mapreduce-client-hs/HistoryServerRest.html">MR History Server</a>
</li>
</ul>
<h5>YARN</h5>
<ul>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/YARN.html">Architecture</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/YarnCommands.html">Commands Reference</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html">Capacity Scheduler</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/FairScheduler.html">Fair Scheduler</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html">ResourceManager Restart</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html">ResourceManager HA</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/ResourceModel.html">Resource Model</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/NodeLabel.html">Node Labels</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/NodeAttributes.html">Node Attributes</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html">Web Application Proxy</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/TimelineServer.html">Timeline Server</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html">Timeline Service V.2</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html">Writing YARN Applications</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html">YARN Application Security</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/NodeManager.html">NodeManager</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/DockerContainers.html">Running Applications in Docker Containers</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/RuncContainers.html">Running Applications in runC Containers</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/NodeManagerCgroups.html">Using CGroups</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/SecureContainer.html">Secure Containers</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/ReservationSystem.html">Reservation System</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html">Graceful Decommission</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html">Opportunistic Containers</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/Federation.html">YARN Federation</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/SharedCache.html">Shared Cache</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/UsingGpus.html">Using GPU</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/UsingFPGA.html">Using FPGA</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html">Placement Constraints</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/YarnUI2.html">YARN UI2</a>
</li>
</ul>
<h5>YARN REST APIs</h5>
<ul>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html">Introduction</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html">Resource Manager</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/NodeManagerRest.html">Node Manager</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/TimelineServer.html#Timeline_Server_REST_API_v1">Timeline Server</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html#Timeline_Service_v.2_REST_API">Timeline Service V.2</a>
</li>
</ul>
<h5>YARN Service</h5>
<ul>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/yarn-service/Overview.html">Overview</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/yarn-service/QuickStart.html">QuickStart</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/yarn-service/Concepts.html">Concepts</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/yarn-service/YarnServiceAPI.html">Yarn Service API</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/yarn-service/ServiceDiscovery.html">Service Discovery</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-site/yarn-service/SystemServices.html">System Services</a>
</li>
</ul>
<h5>Hadoop Compatible File Systems</h5>
<ul>
<li class="none">
<a href="../hadoop-aliyun/tools/hadoop-aliyun/index.html">Aliyun OSS</a>
</li>
<li class="none">
<a href="../hadoop-aws/tools/hadoop-aws/index.html">Amazon S3</a>
</li>
<li class="none">
<a href="../hadoop-azure/index.html">Azure Blob Storage</a>
</li>
<li class="none">
<a href="../hadoop-azure-datalake/index.html">Azure Data Lake Storage</a>
</li>
<li class="none">
<a href="../hadoop-openstack/index.html">OpenStack Swift</a>
</li>
<li class="none">
<a href="../hadoop-cos/cloud-storage/index.html">Tencent COS</a>
</li>
</ul>
<h5>Auth</h5>
<ul>
<li class="none">
<a href="../hadoop-auth/index.html">Overview</a>
</li>
<li class="none">
<a href="../hadoop-auth/Examples.html">Examples</a>
</li>
<li class="none">
<a href="../hadoop-auth/Configuration.html">Configuration</a>
</li>
<li class="none">
<a href="../hadoop-auth/BuildingIt.html">Building</a>
</li>
</ul>
<h5>Tools</h5>
<ul>
<li class="none">
<a href="../hadoop-streaming/HadoopStreaming.html">Hadoop Streaming</a>
</li>
<li class="none">
<a href="../hadoop-archives/HadoopArchives.html">Hadoop Archives</a>
</li>
<li class="none">
<a href="../hadoop-archive-logs/HadoopArchiveLogs.html">Hadoop Archive Logs</a>
</li>
<li class="none">
<a href="../hadoop-distcp/DistCp.html">DistCp</a>
</li>
<li class="none">
<a href="../hadoop-gridmix/GridMix.html">GridMix</a>
</li>
<li class="none">
<a href="../hadoop-rumen/Rumen.html">Rumen</a>
</li>
<li class="none">
<a href="../hadoop-resourceestimator/ResourceEstimator.html">Resource Estimator Service</a>
</li>
<li class="none">
<a href="../hadoop-sls/SchedulerLoadSimulator.html">Scheduler Load Simulator</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/Benchmarking.html">Hadoop Benchmarking</a>
</li>
<li class="none">
<a href="../hadoop-dynamometer/Dynamometer.html">Dynamometer</a>
</li>
</ul>
<h5>Reference</h5>
<ul>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/release/">Changelog and Release Notes</a>
</li>
<li class="none">
<a href="../api/index.html">Java API docs</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/UnixShellAPI.html">Unix Shell API</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/Metrics.html">Metrics</a>
</li>
</ul>
<h5>Configuration</h5>
<ul>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/core-default.xml">core-default.xml</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs/hdfs-default.xml">hdfs-default.xml</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-hdfs-rbf/hdfs-rbf-default.xml">hdfs-rbf-default.xml</a>
</li>
<li class="none">
<a href="../hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml">mapred-default.xml</a>
</li>
<li class="none">
<a href="../hadoop-yarn/hadoop-yarn-common/yarn-default.xml">yarn-default.xml</a>
</li>
<li class="none">
<a href="../hadoop-kms/kms-default.html">kms-default.xml</a>
</li>
<li class="none">
<a href="../hadoop-hdfs-httpfs/httpfs-default.html">httpfs-default.xml</a>
</li>
<li class="none">
<a href="../hadoop-project-dist/hadoop-common/DeprecatedProperties.html">Deprecated Properties</a>
</li>
</ul>
<a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
<img alt="Built by Maven" src="./images/logos/maven-feather.png"/>
</a>
</div>
</div>
<div id="bodyColumn">
<div id="contentBox">
<!---
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<h1>Hadoop Streaming</h1>
<ul>
<li><a href="#Hadoop_Streaming">Hadoop Streaming</a>
<ul>
<li><a href="#Hadoop_Streaming">Hadoop Streaming</a></li>
<li><a href="#How_Streaming_Works">How Streaming Works</a></li>
<li><a href="#Streaming_Command_Options">Streaming Command Options</a>
<ul>
<li><a href="#Specifying_a_Java_Class_as_the_MapperReducer">Specifying a Java Class as the Mapper/Reducer</a></li>
<li><a href="#Packaging_Files_With_Job_Submissions">Packaging Files With Job Submissions</a></li>
<li><a href="#Specifying_Other_Plugins_for_Jobs">Specifying Other Plugins for Jobs</a></li>
<li><a href="#Setting_Environment_Variables">Setting Environment Variables</a></li>
</ul>
</li>
<li><a href="#Generic_Command_Options">Generic Command Options</a>
<ul>
<li><a href="#Specifying_Configuration_Variables_with_the_-D_Option">Specifying Configuration Variables with the -D Option</a>
<ul>
<li><a href="#Specifying_Directories">Specifying Directories</a></li>
<li><a href="#Specifying_Map-Only_Jobs">Specifying Map-Only Jobs</a></li>
<li><a href="#Specifying_the_Number_of_Reducers">Specifying the Number of Reducers</a></li>
<li><a href="#Customizing_How_Lines_are_Split_into_KeyValue_Pairs">Customizing How Lines are Split into Key/Value Pairs</a></li>
</ul>
</li>
<li><a href="#Working_with_Large_Files_and_Archives">Working with Large Files and Archives</a>
<ul>
<li><a href="#Making_Files_Available_to_Tasks">Making Files Available to Tasks</a></li>
<li><a href="#Making_Archives_Available_to_Tasks">Making Archives Available to Tasks</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#More_Usage_Examples">More Usage Examples</a>
<ul>
<li><a href="#Hadoop_Partitioner_Class">Hadoop Partitioner Class</a></li>
<li><a href="#Hadoop_Comparator_Class">Hadoop Comparator Class</a></li>
<li><a href="#Hadoop_Aggregate_Package">Hadoop Aggregate Package</a></li>
<li><a href="#Hadoop_Field_Selection_Class">Hadoop Field Selection Class</a></li>
</ul>
</li>
<li><a href="#Frequently_Asked_Questions">Frequently Asked Questions</a>
<ul>
<li><a href="#How_do_I_use_Hadoop_Streaming_to_run_an_arbitrary_set_of_semi_independent_tasks">How do I use Hadoop Streaming to run an arbitrary set of (semi) independent tasks?</a></li>
<li><a href="#How_do_I_process_files_one_per_map">How do I process files, one per map?</a></li>
<li><a href="#How_many_reducers_should_I_use">How many reducers should I use?</a></li>
<li><a href="#If_I_set_up_an_alias_in_my_shell_script_will_that_work_after_-mapper">If I set up an alias in my shell script, will that work after -mapper?</a></li>
<li><a href="#Can_I_use_UNIX_pipes">Can I use UNIX pipes?</a></li>
<li><a href="#What_do_I_do_if_I_get_the_No_space_left_on_device_error">What do I do if I get the &#x201c;No space left on device&#x201d; error?</a></li>
<li><a href="#How_do_I_specify_multiple_input_directories">How do I specify multiple input directories?</a></li>
<li><a href="#How_do_I_generate_output_files_with_gzip_format">How do I generate output files with gzip format?</a></li>
<li><a href="#How_do_I_provide_my_own_inputoutput_format_with_streaming">How do I provide my own input/output format with streaming?</a></li>
<li><a href="#How_do_I_parse_XML_documents_using_streaming">How do I parse XML documents using streaming?</a></li>
<li><a href="#How_do_I_update_counters_in_streaming_applications">How do I update counters in streaming applications?</a></li>
<li><a href="#How_do_I_update_status_in_streaming_applications">How do I update status in streaming applications?</a></li>
<li><a href="#How_do_I_get_the_Job_variables_in_a_streaming_jobs_mapperreducer">How do I get the Job variables in a streaming job&#x2019;s mapper/reducer?</a></li>
<li><a href="#What_do_I_do_if_I_get_a_error_Argument_list_too_long">What do I do if I get a &#x201c;error=7, Argument list too long&#x201d;</a></li>
</ul>
</li>
</ul>
</li>
</ul>
<div class="section">
<h2><a name="Hadoop_Streaming"></a>Hadoop Streaming</h2>
<p>Hadoop streaming is a utility that comes with the Hadoop distribution. The utility allows you to create and run Map/Reduce jobs with any executable or script as the mapper and/or the reducer. For example:</p>
<div>
<div>
<pre class="source">mapred streaming \
-input myInputDirs \
-output myOutputDir \
-mapper /bin/cat \
-reducer /usr/bin/wc
</pre></div></div>
</div>
<div class="section">
<h2><a name="How_Streaming_Works"></a>How Streaming Works</h2>
<p>In the above example, both the mapper and the reducer are executables that read the input from stdin (line by line) and emit the output to stdout. The utility will create a Map/Reduce job, submit the job to an appropriate cluster, and monitor the progress of the job until it completes.</p>
<p>When an executable is specified for mappers, each mapper task will launch the executable as a separate process when the mapper is initialized. As the mapper task runs, it converts its inputs into lines and feed the lines to the stdin of the process. In the meantime, the mapper collects the line oriented outputs from the stdout of the process and converts each line into a key/value pair, which is collected as the output of the mapper. By default, the <i>prefix of a line up to the first tab character</i> is the <tt>key</tt> and the rest of the line (excluding the tab character) will be the <tt>value</tt>. If there is no tab character in the line, then entire line is considered as key and the value is null. However, this can be customized by setting <tt>-inputformat</tt> command option, as discussed later.</p>
<p>When an executable is specified for reducers, each reducer task will launch the executable as a separate process then the reducer is initialized. As the reducer task runs, it converts its input key/values pairs into lines and feeds the lines to the stdin of the process. In the meantime, the reducer collects the line oriented outputs from the stdout of the process, converts each line into a key/value pair, which is collected as the output of the reducer. By default, the prefix of a line up to the first tab character is the key and the rest of the line (excluding the tab character) is the value. However, this can be customized by setting <tt>-outputformat</tt> command option, as discussed later.</p>
<p>This is the basis for the communication protocol between the Map/Reduce framework and the streaming mapper/reducer.</p>
<p>User can specify <tt>stream.non.zero.exit.is.failure</tt> as <tt>true</tt> or <tt>false</tt> to make a streaming task that exits with a non-zero status to be <tt>Failure</tt> or <tt>Success</tt> respectively. By default, streaming tasks exiting with non-zero status are considered to be failed tasks.</p></div>
<div class="section">
<h2><a name="Streaming_Command_Options"></a>Streaming Command Options</h2>
<p>Streaming supports streaming command options as well as <a href="#Generic_Command_Options">generic command options</a>. The general command line syntax is shown below.</p>
<p><b>Note:</b> Be sure to place the generic options before the streaming options, otherwise the command will fail. For an example, see <a href="#Making_Archives_Available_to_Tasks">Making Archives Available to Tasks</a>.</p>
<div>
<div>
<pre class="source">mapred streaming [genericOptions] [streamingOptions]
</pre></div></div>
<p>The Hadoop streaming command options are listed here:</p>
<table border="0" class="bodyTable">
<thead>
<tr class="a">
<th align="left"> Parameter </th>
<th align="left"> Optional/Required </th>
<th align="left"> Description </th></tr>
</thead><tbody>
<tr class="b">
<td align="left"> -input directoryname or filename </td>
<td align="left"> Required </td>
<td align="left"> Input location for mapper </td></tr>
<tr class="a">
<td align="left"> -output directoryname </td>
<td align="left"> Required </td>
<td align="left"> Output location for reducer </td></tr>
<tr class="b">
<td align="left"> -mapper executable or JavaClassName </td>
<td align="left"> Optional </td>
<td align="left"> Mapper executable. If not specified, IdentityMapper is used as the default </td></tr>
<tr class="a">
<td align="left"> -reducer executable or JavaClassName </td>
<td align="left"> Optional </td>
<td align="left"> Reducer executable. If not specified, IdentityReducer is used as the default </td></tr>
<tr class="b">
<td align="left"> -file filename </td>
<td align="left"> Optional </td>
<td align="left"> Make the mapper, reducer, or combiner executable available locally on the compute nodes </td></tr>
<tr class="a">
<td align="left"> -inputformat JavaClassName </td>
<td align="left"> Optional </td>
<td align="left"> Class you supply should return key/value pairs of Text class. If not specified, TextInputFormat is used as the default </td></tr>
<tr class="b">
<td align="left"> -outputformat JavaClassName </td>
<td align="left"> Optional </td>
<td align="left"> Class you supply should take key/value pairs of Text class. If not specified, TextOutputformat is used as the default </td></tr>
<tr class="a">
<td align="left"> -partitioner JavaClassName </td>
<td align="left"> Optional </td>
<td align="left"> Class that determines which reduce a key is sent to </td></tr>
<tr class="b">
<td align="left"> -combiner streamingCommand or JavaClassName </td>
<td align="left"> Optional </td>
<td align="left"> Combiner executable for map output </td></tr>
<tr class="a">
<td align="left"> -cmdenv name=value </td>
<td align="left"> Optional </td>
<td align="left"> Pass environment variable to streaming commands </td></tr>
<tr class="b">
<td align="left"> -inputreader </td>
<td align="left"> Optional </td>
<td align="left"> For backwards-compatibility: specifies a record reader class (instead of an input format class) </td></tr>
<tr class="a">
<td align="left"> -verbose </td>
<td align="left"> Optional </td>
<td align="left"> Verbose output </td></tr>
<tr class="b">
<td align="left"> -lazyOutput </td>
<td align="left"> Optional </td>
<td align="left"> Create output lazily. For example, if the output format is based on FileOutputFormat, the output file is created only on the first call to Context.write </td></tr>
<tr class="a">
<td align="left"> -numReduceTasks </td>
<td align="left"> Optional </td>
<td align="left"> Specify the number of reducers </td></tr>
<tr class="b">
<td align="left"> -mapdebug </td>
<td align="left"> Optional </td>
<td align="left"> Script to call when map task fails </td></tr>
<tr class="a">
<td align="left"> -reducedebug </td>
<td align="left"> Optional </td>
<td align="left"> Script to call when reduce task fails </td></tr>
</tbody>
</table>
<div class="section">
<h3><a name="Specifying_a_Java_Class_as_the_Mapper.2FReducer"></a>Specifying a Java Class as the Mapper/Reducer</h3>
<p>You can supply a Java class as the mapper and/or the reducer.</p>
<div>
<div>
<pre class="source">mapred streaming \
-input myInputDirs \
-output myOutputDir \
-inputformat org.apache.hadoop.mapred.KeyValueTextInputFormat \
-mapper org.apache.hadoop.mapred.lib.IdentityMapper \
-reducer /usr/bin/wc
</pre></div></div>
<p>You can specify <tt>stream.non.zero.exit.is.failure</tt> as <tt>true</tt> or <tt>false</tt> to make a streaming task that exits with a non-zero status to be <tt>Failure</tt> or <tt>Success</tt> respectively. By default, streaming tasks exiting with non-zero status are considered to be failed tasks.</p></div>
<div class="section">
<h3><a name="Packaging_Files_With_Job_Submissions"></a>Packaging Files With Job Submissions</h3>
<p>You can specify any executable as the mapper and/or the reducer. The executables do not need to pre-exist on the machines in the cluster; however, if they don&#x2019;t, you will need to use &#x201c;-file&#x201d; option to tell the framework to pack your executable files as a part of job submission. For example:</p>
<div>
<div>
<pre class="source">mapred streaming \
-input myInputDirs \
-output myOutputDir \
-mapper myPythonScript.py \
-reducer /usr/bin/wc \
-file myPythonScript.py
</pre></div></div>
<p>The above example specifies a user defined Python executable as the mapper. The option &#x201c;-file myPythonScript.py&#x201d; causes the python executable shipped to the cluster machines as a part of job submission.</p>
<p>In addition to executable files, you can also package other auxiliary files (such as dictionaries, configuration files, etc) that may be used by the mapper and/or the reducer. For example:</p>
<div>
<div>
<pre class="source">mapred streaming \
-input myInputDirs \
-output myOutputDir \
-mapper myPythonScript.py \
-reducer /usr/bin/wc \
-file myPythonScript.py \
-file myDictionary.txt
</pre></div></div>
</div>
<div class="section">
<h3><a name="Specifying_Other_Plugins_for_Jobs"></a>Specifying Other Plugins for Jobs</h3>
<p>Just as with a normal Map/Reduce job, you can specify other plugins for a streaming job:</p>
<div>
<div>
<pre class="source"> -inputformat JavaClassName
-outputformat JavaClassName
-partitioner JavaClassName
-combiner streamingCommand or JavaClassName
</pre></div></div>
<p>The class you supply for the input format should return key/value pairs of Text class. If you do not specify an input format class, the TextInputFormat is used as the default. Since the TextInputFormat returns keys of LongWritable class, which are actually not part of the input data, the keys will be discarded; only the values will be piped to the streaming mapper.</p>
<p>The class you supply for the output format is expected to take key/value pairs of Text class. If you do not specify an output format class, the TextOutputFormat is used as the default.</p></div>
<div class="section">
<h3><a name="Setting_Environment_Variables"></a>Setting Environment Variables</h3>
<p>To set an environment variable in a streaming command use:</p>
<div>
<div>
<pre class="source"> -cmdenv EXAMPLE_DIR=/home/example/dictionaries/
</pre></div></div>
</div></div>
<div class="section">
<h2><a name="Generic_Command_Options"></a>Generic Command Options</h2>
<p>Streaming supports <a href="#Streaming_Command_Options">streaming command options</a> as well as generic command options. The general command line syntax is shown below.</p>
<p><b>Note:</b> Be sure to place the generic options before the streaming options, otherwise the command will fail. For an example, see <a href="#Making_Archives_Available_to_Tasks">Making Archives Available to Tasks</a>.</p>
<div>
<div>
<pre class="source">hadoop command [genericOptions] [streamingOptions]
</pre></div></div>
<p>The Hadoop generic command options you can use with streaming are listed here:</p>
<table border="0" class="bodyTable">
<thead>
<tr class="a">
<th align="left"> Parameter </th>
<th align="left"> Optional/Required </th>
<th align="left"> Description </th></tr>
</thead><tbody>
<tr class="b">
<td align="left"> -conf configuration_file </td>
<td align="left"> Optional </td>
<td align="left"> Specify an application configuration file </td></tr>
<tr class="a">
<td align="left"> -D property=value </td>
<td align="left"> Optional </td>
<td align="left"> Use value for given property </td></tr>
<tr class="b">
<td align="left"> -fs host:port or local </td>
<td align="left"> Optional </td>
<td align="left"> Specify a namenode </td></tr>
<tr class="a">
<td align="left"> -files </td>
<td align="left"> Optional </td>
<td align="left"> Specify comma-separated files to be copied to the Map/Reduce cluster </td></tr>
<tr class="b">
<td align="left"> -libjars </td>
<td align="left"> Optional </td>
<td align="left"> Specify comma-separated jar files to include in the classpath </td></tr>
<tr class="a">
<td align="left"> -archives </td>
<td align="left"> Optional </td>
<td align="left"> Specify comma-separated archives to be unarchived on the compute machines </td></tr>
</tbody>
</table>
<div class="section">
<h3><a name="Specifying_Configuration_Variables_with_the_-D_Option"></a>Specifying Configuration Variables with the -D Option</h3>
<p>You can specify additional configuration variables by using &#x201c;-D &lt;property&gt;=&lt;value&gt;&#x201d;.</p>
<div class="section">
<h4><a name="Specifying_Directories"></a>Specifying Directories</h4>
<p>To change the local temp directory use:</p>
<div>
<div>
<pre class="source"> -D dfs.data.dir=/tmp
</pre></div></div>
<p>To specify additional local temp directories use:</p>
<div>
<div>
<pre class="source"> -D mapred.local.dir=/tmp/local
-D mapred.system.dir=/tmp/system
-D mapred.temp.dir=/tmp/temp
</pre></div></div>
<p><b>Note:</b> For more details on job configuration parameters see: <a href="../hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml">mapred-default.xml</a></p></div>
<div class="section">
<h4><a name="Specifying_Map-Only_Jobs"></a>Specifying Map-Only Jobs</h4>
<p>Often, you may want to process input data using a map function only. To do this, simply set <tt>mapreduce.job.reduces</tt> to zero. The Map/Reduce framework will not create any reducer tasks. Rather, the outputs of the mapper tasks will be the final output of the job.</p>
<div>
<div>
<pre class="source"> -D mapreduce.job.reduces=0
</pre></div></div>
<p>To be backward compatible, Hadoop Streaming also supports the &#x201c;-reducer NONE&#x201d; option, which is equivalent to &#x201c;-D mapreduce.job.reduces=0&#x201d;.</p></div>
<div class="section">
<h4><a name="Specifying_the_Number_of_Reducers"></a>Specifying the Number of Reducers</h4>
<p>To specify the number of reducers, for example two, use:</p>
<div>
<div>
<pre class="source">mapred streaming \
-D mapreduce.job.reduces=2 \
-input myInputDirs \
-output myOutputDir \
-mapper /bin/cat \
-reducer /usr/bin/wc
</pre></div></div>
</div>
<div class="section">
<h4><a name="Customizing_How_Lines_are_Split_into_Key.2FValue_Pairs"></a>Customizing How Lines are Split into Key/Value Pairs</h4>
<p>As noted earlier, when the Map/Reduce framework reads a line from the stdout of the mapper, it splits the line into a key/value pair. By default, the prefix of the line up to the first tab character is the key and the rest of the line (excluding the tab character) is the value.</p>
<p>However, you can customize this default. You can specify a field separator other than the tab character (the default), and you can specify the nth (n &gt;= 1) character rather than the first character in a line (the default) as the separator between the key and value. For example:</p>
<div>
<div>
<pre class="source">mapred streaming \
-D stream.map.output.field.separator=. \
-D stream.num.map.output.key.fields=4 \
-input myInputDirs \
-output myOutputDir \
-mapper /bin/cat \
-reducer /bin/cat
</pre></div></div>
<p>In the above example, &#x201c;-D stream.map.output.field.separator=.&#x201d; specifies &#x201c;.&#x201d; as the field separator for the map outputs, and the prefix up to the fourth &#x201c;.&#x201d; in a line will be the key and the rest of the line (excluding the fourth &#x201c;.&#x201d;) will be the value. If a line has less than four &#x201c;.&#x201c;s, then the whole line will be the key and the value will be an empty Text object (like the one created by new Text(&#x201d;&#x201d;)).</p>
<p>Similarly, you can use &#x201c;-D stream.reduce.output.field.separator=SEP&#x201d; and &#x201c;-D stream.num.reduce.output.fields=NUM&#x201d; to specify the nth field separator in a line of the reduce outputs as the separator between the key and the value.</p>
<p>Similarly, you can specify &#x201c;stream.map.input.field.separator&#x201d; and &#x201c;stream.reduce.input.field.separator&#x201d; as the input separator for Map/Reduce inputs. By default the separator is the tab character.</p></div></div>
<div class="section">
<h3><a name="Working_with_Large_Files_and_Archives"></a>Working with Large Files and Archives</h3>
<p>The -files and -archives options allow you to make files and archives available to the tasks. The argument is a URI to the file or archive that you have already uploaded to HDFS. These files and archives are cached across jobs. You can retrieve the host and fs_port values from the fs.default.name config variable.</p>
<p><b>Note:</b> The -files and -archives options are generic options. Be sure to place the generic options before the command options, otherwise the command will fail.</p>
<div class="section">
<h4><a name="Making_Files_Available_to_Tasks"></a>Making Files Available to Tasks</h4>
<p>The -files option creates a symlink in the current working directory of the tasks that points to the local copy of the file.</p>
<p>In this example, Hadoop automatically creates a symlink named testfile.txt in the current working directory of the tasks. This symlink points to the local copy of testfile.txt.</p>
<div>
<div>
<pre class="source">-files hdfs://host:fs_port/user/testfile.txt
</pre></div></div>
<p>User can specify a different symlink name for -files using #.</p>
<div>
<div>
<pre class="source">-files hdfs://host:fs_port/user/testfile.txt#testfile
</pre></div></div>
<p>Multiple entries can be specified like this:</p>
<div>
<div>
<pre class="source">-files hdfs://host:fs_port/user/testfile1.txt,hdfs://host:fs_port/user/testfile2.txt
</pre></div></div>
</div>
<div class="section">
<h4><a name="Making_Archives_Available_to_Tasks"></a>Making Archives Available to Tasks</h4>
<p>The -archives option allows you to copy jars locally to the current working directory of tasks and automatically unjar the files.</p>
<p>In this example, Hadoop automatically creates a symlink named testfile.jar in the current working directory of tasks. This symlink points to the directory that stores the unjarred contents of the uploaded jar file.</p>
<div>
<div>
<pre class="source">-archives hdfs://host:fs_port/user/testfile.jar
</pre></div></div>
<p>User can specify a different symlink name for -archives using #.</p>
<div>
<div>
<pre class="source">-archives hdfs://host:fs_port/user/testfile.tgz#tgzdir
</pre></div></div>
<p>In this example, the input.txt file has two lines specifying the names of the two files: cachedir.jar/cache.txt and cachedir.jar/cache2.txt. &#x201c;cachedir.jar&#x201d; is a symlink to the archived directory, which has the files &#x201c;cache.txt&#x201d; and &#x201c;cache2.txt&#x201d;.</p>
<div>
<div>
<pre class="source">mapred streaming \
-archives 'hdfs://hadoop-nn1.example.com/user/me/samples/cachefile/cachedir.jar' \
-D mapreduce.job.maps=1 \
-D mapreduce.job.reduces=1 \
-D mapreduce.job.name=&quot;Experiment&quot; \
-input &quot;/user/me/samples/cachefile/input.txt&quot; \
-output &quot;/user/me/samples/cachefile/out&quot; \
-mapper &quot;xargs cat&quot; \
-reducer &quot;cat&quot;
$ ls test_jar/
cache.txt cache2.txt
$ jar cvf cachedir.jar -C test_jar/ .
added manifest
adding: cache.txt(in = 30) (out= 29)(deflated 3%)
adding: cache2.txt(in = 37) (out= 35)(deflated 5%)
$ hdfs dfs -put cachedir.jar samples/cachefile
$ hdfs dfs -cat /user/me/samples/cachefile/input.txt
cachedir.jar/cache.txt
cachedir.jar/cache2.txt
$ cat test_jar/cache.txt
This is just the cache string
$ cat test_jar/cache2.txt
This is just the second cache string
$ hdfs dfs -ls /user/me/samples/cachefile/out
Found 2 items
-rw-r--r-* 1 me supergroup 0 2013-11-14 17:00 /user/me/samples/cachefile/out/_SUCCESS
-rw-r--r-* 1 me supergroup 69 2013-11-14 17:00 /user/me/samples/cachefile/out/part-00000
$ hdfs dfs -cat /user/me/samples/cachefile/out/part-00000
This is just the cache string
This is just the second cache string
</pre></div></div>
</div></div></div>
<div class="section">
<h2><a name="More_Usage_Examples"></a>More Usage Examples</h2>
<div class="section">
<h3><a name="Hadoop_Partitioner_Class"></a>Hadoop Partitioner Class</h3>
<p>Hadoop has a library class, <a href="../api/org/apache/hadoop/mapred/lib/KeyFieldBasedPartitioner.html">KeyFieldBasedPartitioner</a>, that is useful for many applications. This class allows the Map/Reduce framework to partition the map outputs based on certain key fields, not the whole keys. For example:</p>
<div>
<div>
<pre class="source">mapred streaming \
-D stream.map.output.field.separator=. \
-D stream.num.map.output.key.fields=4 \
-D map.output.key.field.separator=. \
-D mapreduce.partition.keypartitioner.options=-k1,2 \
-D mapreduce.job.reduces=12 \
-input myInputDirs \
-output myOutputDir \
-mapper /bin/cat \
-reducer /bin/cat \
-partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner
</pre></div></div>
<p>Here, <i>-D stream.map.output.field.separator=.</i> and <i>-D stream.num.map.output.key.fields=4</i> are as explained in previous example. The two variables are used by streaming to identify the key/value pair of mapper.</p>
<p>The map output keys of the above Map/Reduce job normally have four fields separated by &#x201c;.&#x201d;. However, the Map/Reduce framework will partition the map outputs by the first two fields of the keys using the <i>-D mapred.text.key.partitioner.options=-k1,2</i> option. Here, <i>-D map.output.key.field.separator=.</i> specifies the separator for the partition. This guarantees that all the key/value pairs with the same first two fields in the keys will be partitioned into the same reducer.</p>
<p><i>This is effectively equivalent to specifying the first two fields as the primary key and the next two fields as the secondary. The primary key is used for partitioning, and the combination of the primary and secondary keys is used for sorting.</i> A simple illustration is shown here:</p>
<p>Output of map (the keys)</p>
<div>
<div>
<pre class="source">11.12.1.2
11.14.2.3
11.11.4.1
11.12.1.1
11.14.2.2
</pre></div></div>
<p>Partition into 3 reducers (the first 2 fields are used as keys for partition)</p>
<div>
<div>
<pre class="source">11.11.4.1
-----------
11.12.1.2
11.12.1.1
-----------
11.14.2.3
11.14.2.2
</pre></div></div>
<p>Sorting within each partition for the reducer(all 4 fields used for sorting)</p>
<div>
<div>
<pre class="source">11.11.4.1
-----------
11.12.1.1
11.12.1.2
-----------
11.14.2.2
11.14.2.3
</pre></div></div>
</div>
<div class="section">
<h3><a name="Hadoop_Comparator_Class"></a>Hadoop Comparator Class</h3>
<p>Hadoop has a library class, <a href="../api/org/apache/hadoop/mapreduce/lib/partition/KeyFieldBasedComparator.html">KeyFieldBasedComparator</a>, that is useful for many applications. This class provides a subset of features provided by the Unix/GNU Sort. For example:</p>
<div>
<div>
<pre class="source">mapred streaming \
-D mapreduce.job.output.key.comparator.class=org.apache.hadoop.mapreduce.lib.partition.KeyFieldBasedComparator \
-D stream.map.output.field.separator=. \
-D stream.num.map.output.key.fields=4 \
-D mapreduce.map.output.key.field.separator=. \
-D mapreduce.partition.keycomparator.options=-k2,2nr \
-D mapreduce.job.reduces=1 \
-input myInputDirs \
-output myOutputDir \
-mapper /bin/cat \
-reducer /bin/cat
</pre></div></div>
<p>The map output keys of the above Map/Reduce job normally have four fields separated by &#x201c;.&#x201d;. However, the Map/Reduce framework will sort the outputs by the second field of the keys using the <i>-D mapreduce.partition.keycomparator.options=-k2,2nr</i> option. Here, <i>-n</i> specifies that the sorting is numerical sorting and <i>-r</i> specifies that the result should be reversed. A simple illustration is shown below:</p>
<p>Output of map (the keys)</p>
<div>
<div>
<pre class="source">11.12.1.2
11.14.2.3
11.11.4.1
11.12.1.1
11.14.2.2
</pre></div></div>
<p>Sorting output for the reducer (where second field used for sorting)</p>
<div>
<div>
<pre class="source">11.14.2.3
11.14.2.2
11.12.1.2
11.12.1.1
11.11.4.1
</pre></div></div>
</div>
<div class="section">
<h3><a name="Hadoop_Aggregate_Package"></a>Hadoop Aggregate Package</h3>
<p>Hadoop has a library package called <a href="../api/org/apache/hadoop/mapred/lib/aggregate/package-summary.html">Aggregate</a>. Aggregate provides a special reducer class and a special combiner class, and a list of simple aggregators that perform aggregations such as &#x201c;sum&#x201d;, &#x201c;max&#x201d;, &#x201c;min&#x201d; and so on over a sequence of values. Aggregate allows you to define a mapper plugin class that is expected to generate &#x201c;aggregatable items&#x201d; for each input key/value pair of the mappers. The combiner/reducer will aggregate those aggregatable items by invoking the appropriate aggregators.</p>
<p>To use Aggregate, simply specify &#x201c;-reducer aggregate&#x201d;:</p>
<div>
<div>
<pre class="source">mapred streaming \
-input myInputDirs \
-output myOutputDir \
-mapper myAggregatorForKeyCount.py \
-reducer aggregate \
-file myAggregatorForKeyCount.py
</pre></div></div>
<p>The python program myAggregatorForKeyCount.py looks like:</p>
<div>
<div>
<pre class="source">#!/usr/bin/python3
import sys
def generateLongCountToken(id):
return &quot;LongValueSum:&quot; + id + &quot;\t&quot; + &quot;1&quot;
def main(argv):
line = sys.stdin.readline()
try:
while line:
line = line[:-1]
fields = line.split(&quot;\t&quot;)
print(generateLongCountToken(fields[0]))
line = sys.stdin.readline()
except &quot;end of file&quot;:
return None
if __name__ == &quot;__main__&quot;:
main(sys.argv)
</pre></div></div>
</div>
<div class="section">
<h3><a name="Hadoop_Field_Selection_Class"></a>Hadoop Field Selection Class</h3>
<p>Hadoop has a library class, <a href="../api/org/apache/hadoop/mapred/lib/FieldSelectionMapReduce.html">FieldSelectionMapReduce</a>, that effectively allows you to process text data like the unix &#x201c;cut&#x201d; utility. The map function defined in the class treats each input key/value pair as a list of fields. You can specify the field separator (the default is the tab character). You can select an arbitrary list of fields as the map output key, and an arbitrary list of fields as the map output value. Similarly, the reduce function defined in the class treats each input key/value pair as a list of fields. You can select an arbitrary list of fields as the reduce output key, and an arbitrary list of fields as the reduce output value. For example:</p>
<div>
<div>
<pre class="source">mapred streaming \
-D mapreduce.map.output.key.field.separator=. \
-D mapreduce.partition.keypartitioner.options=-k1,2 \
-D mapreduce.fieldsel.data.field.separator=. \
-D mapreduce.fieldsel.map.output.key.value.fields.spec=6,5,1-3:0- \
-D mapreduce.fieldsel.reduce.output.key.value.fields.spec=0-2:5- \
-D mapreduce.map.output.key.class=org.apache.hadoop.io.Text \
-D mapreduce.job.reduces=12 \
-input myInputDirs \
-output myOutputDir \
-mapper org.apache.hadoop.mapred.lib.FieldSelectionMapReduce \
-reducer org.apache.hadoop.mapred.lib.FieldSelectionMapReduce \
-partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner
</pre></div></div>
<p>The option &#x201c;-D mapreduce.fieldsel.map.output.key.value.fields.spec=6,5,1-3:0-&#x201d; specifies key/value selection for the map outputs. Key selection spec and value selection spec are separated by &#x201c;:&#x201d;. In this case, the map output key will consist of fields 6, 5, 1, 2, and 3. The map output value will consist of all fields (0- means field 0 and all the subsequent fields).</p>
<p>The option &#x201c;-D mapreduce.fieldsel.reduce.output.key.value.fields.spec=0-2:5-&#x201d; specifies key/value selection for the reduce outputs. In this case, the reduce output key will consist of fields 0, 1, 2 (corresponding to the original fields 6, 5, 1). The reduce output value will consist of all fields starting from field 5 (corresponding to all the original fields).</p></div></div>
<div class="section">
<h2><a name="Frequently_Asked_Questions"></a>Frequently Asked Questions</h2>
<div class="section">
<h3><a name="How_do_I_use_Hadoop_Streaming_to_run_an_arbitrary_set_of_.28semi.29_independent_tasks.3F"></a>How do I use Hadoop Streaming to run an arbitrary set of (semi) independent tasks?</h3>
<p>Often you do not need the full power of Map Reduce, but only need to run multiple instances of the same program - either on different parts of the data, or on the same data, but with different parameters. You can use Hadoop Streaming to do this.</p></div>
<div class="section">
<h3><a name="How_do_I_process_files.2C_one_per_map.3F"></a>How do I process files, one per map?</h3>
<p>As an example, consider the problem of zipping (compressing) a set of files across the hadoop cluster. You can achieve this by using Hadoop Streaming and custom mapper script:</p>
<ul>
<li>
<p>Generate a file containing the full HDFS path of the input files. Each map task would get one file name as input.</p>
</li>
<li>
<p>Create a mapper script which, given a filename, will get the file to local disk, gzip the file and put it back in the desired output directory.</p>
</li>
</ul></div>
<div class="section">
<h3><a name="How_many_reducers_should_I_use.3F"></a>How many reducers should I use?</h3>
<p>See MapReduce Tutorial for details: <a href="../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Reducer">Reducer</a></p></div>
<div class="section">
<h3><a name="If_I_set_up_an_alias_in_my_shell_script.2C_will_that_work_after_-mapper.3F"></a>If I set up an alias in my shell script, will that work after -mapper?</h3>
<p>For example, say I do: alias c1=&#x2018;cut -f1&#x2019;. Will -mapper &#x201c;c1&#x201d; work?</p>
<p>Using an alias will not work, but variable substitution is allowed as shown in this example:</p>
<div>
<div>
<pre class="source">$ hdfs dfs -cat /user/me/samples/student_marks
alice 50
bruce 70
charlie 80
dan 75
$ c2='cut -f2'; mapred streaming \
-D mapreduce.job.name='Experiment' \
-input /user/me/samples/student_marks \
-output /user/me/samples/student_out \
-mapper &quot;$c2&quot; -reducer 'cat'
$ hdfs dfs -cat /user/me/samples/student_out/part-00000
50
70
75
80
</pre></div></div>
</div>
<div class="section">
<h3><a name="Can_I_use_UNIX_pipes.3F"></a>Can I use UNIX pipes?</h3>
<p>For example, will -mapper &#x201c;cut -f1 | sed s/foo/bar/g&#x201d; work?</p>
<p>Currently this does not work and gives an &#x201c;java.io.IOException: Broken pipe&#x201d; error. This is probably a bug that needs to be investigated.</p></div>
<div class="section">
<h3><a name="What_do_I_do_if_I_get_the_.E2.80.9CNo_space_left_on_device.E2.80.9D_error.3F"></a>What do I do if I get the &#x201c;No space left on device&#x201d; error?</h3>
<p>For example, when I run a streaming job by distributing large executables (for example, 3.6G) through the -file option, I get a &#x201c;No space left on device&#x201d; error.</p>
<p>The jar packaging happens in a directory pointed to by the configuration variable stream.tmpdir. The default value of stream.tmpdir is /tmp. Set the value to a directory with more space:</p>
<p>-D stream.tmpdir=/export/bigspace/&#x2026;</p></div>
<div class="section">
<h3><a name="How_do_I_specify_multiple_input_directories.3F"></a>How do I specify multiple input directories?</h3>
<p>You can specify multiple input directories with multiple &#x2018;-input&#x2019; options:</p>
<div>
<div>
<pre class="source">mapred streaming \
-input '/user/foo/dir1' -input '/user/foo/dir2' \
(rest of the command)
</pre></div></div>
</div>
<div class="section">
<h3><a name="How_do_I_generate_output_files_with_gzip_format.3F"></a>How do I generate output files with gzip format?</h3>
<p>Instead of plain text files, you can generate gzip files as your generated output. Pass &#x2018;-D mapreduce.output.fileoutputformat.compress=true -D mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.GzipCodec&#x2019; as option to your streaming job.</p></div>
<div class="section">
<h3><a name="How_do_I_provide_my_own_input.2Foutput_format_with_streaming.3F"></a>How do I provide my own input/output format with streaming?</h3>
<p>You can specify your own custom class by packing them and putting the custom jar to <tt>$HADOOP_CLASSPATH</tt>.</p></div>
<div class="section">
<h3><a name="How_do_I_parse_XML_documents_using_streaming.3F"></a>How do I parse XML documents using streaming?</h3>
<p>You can use the record reader StreamXmlRecordReader to process XML documents.</p>
<div>
<div>
<pre class="source">mapred streaming \
-inputreader &quot;StreamXmlRecord,begin=BEGIN_STRING,end=END_STRING&quot; \
(rest of the command)
</pre></div></div>
<p>Anything found between BEGIN_STRING and END_STRING would be treated as one record for map tasks.</p>
<p>The name-value properties that StreamXmlRecordReader understands are:</p>
<ul>
<li>(strings) &#x2018;begin&#x2019; - Characters marking beginning of record, and &#x2018;end&#x2019; - Characters marking end of record.</li>
<li>(boolean) &#x2018;slowmatch&#x2019; - Toggle to look for begin and end characters, but within CDATA instead of regular tags. Defaults to false.</li>
<li>(integer) &#x2018;lookahead&#x2019; - Maximum lookahead bytes to sync CDATA when using &#x2018;slowmatch&#x2019;, should be larger than &#x2018;maxrec&#x2019;. Defaults to 2*&#x2018;maxrec&#x2019;.</li>
<li>(integer) &#x2018;maxrec&#x2019; - Maximum record size to read between each match during &#x2018;slowmatch&#x2019;. Defaults to 50000 bytes.</li>
</ul></div>
<div class="section">
<h3><a name="How_do_I_update_counters_in_streaming_applications.3F"></a>How do I update counters in streaming applications?</h3>
<p>A streaming process can use the stderr to emit counter information. <tt>reporter:counter:&lt;group&gt;,&lt;counter&gt;,&lt;amount&gt;</tt> should be sent to stderr to update the counter.</p></div>
<div class="section">
<h3><a name="How_do_I_update_status_in_streaming_applications.3F"></a>How do I update status in streaming applications?</h3>
<p>A streaming process can use the stderr to emit status information. To set a status, <tt>reporter:status:&lt;message&gt;</tt> should be sent to stderr.</p></div>
<div class="section">
<h3><a name="How_do_I_get_the_Job_variables_in_a_streaming_job.E2.80.99s_mapper.2Freducer.3F"></a>How do I get the Job variables in a streaming job&#x2019;s mapper/reducer?</h3>
<p>See <a href="../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Configured_Parameters">Configured Parameters</a>. During the execution of a streaming job, the names of the &#x201c;mapred&#x201d; parameters are transformed. The dots ( . ) become underscores ( _ ). For example, mapreduce.job.id becomes mapreduce_job_id and mapreduce.job.jar becomes mapreduce_job_jar. In your code, use the parameter names with the underscores.</p></div>
<div class="section">
<h3><a name="What_do_I_do_if_I_get_a_.E2.80.9Cerror.3D7.2C_Argument_list_too_long.E2.80.9D"></a>What do I do if I get a &#x201c;error=7, Argument list too long&#x201d;</h3>
<p>The job copies the whole configuration to the environment. If the job is processing a large number of input files adding the job configuration to the environment could cause an overrun of the environment. The job configuration copy in the environment is not essential for running the job and can be truncated by setting:</p>
<p>-D stream.jobconf.truncate.limit=20000</p>
<p>By default the values are not truncated (-1). Zero (0) will only copy the names and not values. For almost all cases 20000 is a safe value that will prevent the overrun of the environment.</p></div></div>
</div>
</div>
<div class="clear">
<hr/>
</div>
<div id="footer">
<div class="xright">
&#169; 2008-2021
Apache Software Foundation
- <a href="http://maven.apache.org/privacy-policy.html">Privacy Policy</a>.
Apache Maven, Maven, Apache, the Apache feather logo, and the Apache Maven project logos are trademarks of The Apache Software Foundation.
</div>
<div class="clear">
<hr/>
</div>
</div>
</body>
</html>