Google Git
Sign in
apache / hadoop-site / 952d547359cafef280f0a0324055741b2ed55be1 / . / content / docs / r3.3.1 / hadoop-project-dist / hadoop-common / release / 0.17.0 / RELEASENOTES.0.17.0.html
blob: 357dc9e9c7b6c6dd0cebef258fcc69ea0124c811 [file] [log] [blame]
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<!--
| Generated by Apache Maven Doxia at 2021-06-15
| Rendered using Apache Maven Stylus Skin 1.5
-->
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Apache Hadoop 3.3.1 &#x2013; Apache Hadoop 0.17.0 Release Notes</title>
<style type="text/css" media="all">
@import url("../../css/maven-base.css");
@import url("../../css/maven-theme.css");
@import url("../../css/site.css");
</style>
<link rel="stylesheet" href="../../css/print.css" type="text/css" media="print" />
<meta name="Date-Revision-yyyymmdd" content="20210615" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body class="composite">
<div id="banner">
<a href="http://hadoop.apache.org/" id="bannerLeft">
<img src="http://hadoop.apache.org/images/hadoop-logo.jpg" alt="" />
</a>
<a href="http://www.apache.org/" id="bannerRight">
<img src="http://www.apache.org/images/asf_logo_wide.png" alt="" />
</a>
<div class="clear">
<hr/>
</div>
</div>
<div id="breadcrumbs">
<div class="xleft">
<a href="http://www.apache.org/" class="externalLink">Apache</a>
&gt;
<a href="http://hadoop.apache.org/" class="externalLink">Hadoop</a>
&gt;
<a href="../../../index.html">Apache Hadoop Project Dist POM</a>
&gt;
<a href="../../index.html">Apache Hadoop 3.3.1</a>
&gt;
Apache Hadoop 0.17.0 Release Notes
</div>
<div class="xright"> <a href="http://wiki.apache.org/hadoop" class="externalLink">Wiki</a>
|
<a href="https://gitbox.apache.org/repos/asf/hadoop.git" class="externalLink">git</a>
|
<a href="http://hadoop.apache.org/" class="externalLink">Apache Hadoop</a>
&nbsp;| Last Published: 2021-06-15
&nbsp;| Version: 3.3.1
</div>
<div class="clear">
<hr/>
</div>
</div>
<div id="leftColumn">
<div id="navcolumn">
<h5>General</h5>
<ul>
<li class="none">
<a href="../../../../index.html">Overview</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/SingleCluster.html">Single Node Setup</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/ClusterSetup.html">Cluster Setup</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/CommandsManual.html">Commands Reference</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/FileSystemShell.html">FileSystem Shell</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/Compatibility.html">Compatibility Specification</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/DownstreamDev.html">Downstream Developer's Guide</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/AdminCompatibilityGuide.html">Admin Compatibility Guide</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/InterfaceClassification.html">Interface Classification</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/filesystem/index.html">FileSystem Specification</a>
</li>
</ul>
<h5>Common</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/CLIMiniCluster.html">CLI Mini Cluster</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/FairCallQueue.html">Fair Call Queue</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/NativeLibraries.html">Native Libraries</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/Superusers.html">Proxy User</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/RackAwareness.html">Rack Awareness</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/SecureMode.html">Secure Mode</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/ServiceLevelAuth.html">Service Level Authorization</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/HttpAuthentication.html">HTTP Authentication</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/CredentialProviderAPI.html">Credential Provider API</a>
</li>
<li class="none">
<a href="../../../../hadoop-kms/index.html">Hadoop KMS</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/Tracing.html">Tracing</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/UnixShellGuide.html">Unix Shell Guide</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/registry/index.html">Registry</a>
</li>
</ul>
<h5>HDFS</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsDesign.html">Architecture</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html">User Guide</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HDFSCommands.html">Commands Reference</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html">NameNode HA With QJM</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html">NameNode HA With NFS</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html">Observer NameNode</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/Federation.html">Federation</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/ViewFs.html">ViewFs</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/ViewFsOverloadScheme.html">ViewFsOverloadScheme</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html">Snapshots</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsEditsViewer.html">Edits Viewer</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html">Image Viewer</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html">Permissions and HDFS</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html">Quotas and HDFS</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/LibHdfs.html">libhdfs (C API)</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/WebHDFS.html">WebHDFS (REST API)</a>
</li>
<li class="none">
<a href="../../../../hadoop-hdfs-httpfs/index.html">HttpFS</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html">Short Circuit Local Reads</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html">Centralized Cache Management</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html">NFS Gateway</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html">Rolling Upgrade</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/ExtendedAttributes.html">Extended Attributes</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html">Transparent Encryption</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html">Multihoming</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html">Storage Policies</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/MemoryStorage.html">Memory Storage Support</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/SLGUserGuide.html">Synthetic Load Generator</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html">Erasure Coding</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HDFSDiskbalancer.html">Disk Balancer</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsUpgradeDomain.html">Upgrade Domain</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsDataNodeAdminGuide.html">DataNode Admin</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html">Router Federation</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsProvidedStorage.html">Provided Storage</a>
</li>
</ul>
<h5>MapReduce</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html">Tutorial</a>
</li>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html">Commands Reference</a>
</li>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html">Compatibility with 1.x</a>
</li>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html">Encrypted Shuffle</a>
</li>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html">Pluggable Shuffle/Sort</a>
</li>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html">Distributed Cache Deploy</a>
</li>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/SharedCacheSupport.html">Support for YARN Shared Cache</a>
</li>
</ul>
<h5>MapReduce REST APIs</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredAppMasterRest.html">MR Application Master</a>
</li>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-hs/HistoryServerRest.html">MR History Server</a>
</li>
</ul>
<h5>YARN</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/YARN.html">Architecture</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/YarnCommands.html">Commands Reference</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html">Capacity Scheduler</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/FairScheduler.html">Fair Scheduler</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html">ResourceManager Restart</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html">ResourceManager HA</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/ResourceModel.html">Resource Model</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/NodeLabel.html">Node Labels</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/NodeAttributes.html">Node Attributes</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html">Web Application Proxy</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html">Timeline Server</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html">Timeline Service V.2</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html">Writing YARN Applications</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html">YARN Application Security</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/NodeManager.html">NodeManager</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/DockerContainers.html">Running Applications in Docker Containers</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/RuncContainers.html">Running Applications in runC Containers</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/NodeManagerCgroups.html">Using CGroups</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/SecureContainer.html">Secure Containers</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/ReservationSystem.html">Reservation System</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html">Graceful Decommission</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html">Opportunistic Containers</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/Federation.html">YARN Federation</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/SharedCache.html">Shared Cache</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/UsingGpus.html">Using GPU</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/UsingFPGA.html">Using FPGA</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html">Placement Constraints</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/YarnUI2.html">YARN UI2</a>
</li>
</ul>
<h5>YARN REST APIs</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html">Introduction</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html">Resource Manager</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/NodeManagerRest.html">Node Manager</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html#Timeline_Server_REST_API_v1">Timeline Server</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html#Timeline_Service_v.2_REST_API">Timeline Service V.2</a>
</li>
</ul>
<h5>YARN Service</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/yarn-service/Overview.html">Overview</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/yarn-service/QuickStart.html">QuickStart</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/yarn-service/Concepts.html">Concepts</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/yarn-service/YarnServiceAPI.html">Yarn Service API</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/yarn-service/ServiceDiscovery.html">Service Discovery</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/yarn-service/SystemServices.html">System Services</a>
</li>
</ul>
<h5>Hadoop Compatible File Systems</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-aliyun/tools/hadoop-aliyun/index.html">Aliyun OSS</a>
</li>
<li class="none">
<a href="../../../../hadoop-aws/tools/hadoop-aws/index.html">Amazon S3</a>
</li>
<li class="none">
<a href="../../../../hadoop-azure/index.html">Azure Blob Storage</a>
</li>
<li class="none">
<a href="../../../../hadoop-azure-datalake/index.html">Azure Data Lake Storage</a>
</li>
<li class="none">
<a href="../../../../hadoop-openstack/index.html">OpenStack Swift</a>
</li>
<li class="none">
<a href="../../../../hadoop-cos/cloud-storage/index.html">Tencent COS</a>
</li>
</ul>
<h5>Auth</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-auth/index.html">Overview</a>
</li>
<li class="none">
<a href="../../../../hadoop-auth/Examples.html">Examples</a>
</li>
<li class="none">
<a href="../../../../hadoop-auth/Configuration.html">Configuration</a>
</li>
<li class="none">
<a href="../../../../hadoop-auth/BuildingIt.html">Building</a>
</li>
</ul>
<h5>Tools</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-streaming/HadoopStreaming.html">Hadoop Streaming</a>
</li>
<li class="none">
<a href="../../../../hadoop-archives/HadoopArchives.html">Hadoop Archives</a>
</li>
<li class="none">
<a href="../../../../hadoop-archive-logs/HadoopArchiveLogs.html">Hadoop Archive Logs</a>
</li>
<li class="none">
<a href="../../../../hadoop-distcp/DistCp.html">DistCp</a>
</li>
<li class="none">
<a href="../../../../hadoop-gridmix/GridMix.html">GridMix</a>
</li>
<li class="none">
<a href="../../../../hadoop-rumen/Rumen.html">Rumen</a>
</li>
<li class="none">
<a href="../../../../hadoop-resourceestimator/ResourceEstimator.html">Resource Estimator Service</a>
</li>
<li class="none">
<a href="../../../../hadoop-sls/SchedulerLoadSimulator.html">Scheduler Load Simulator</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/Benchmarking.html">Hadoop Benchmarking</a>
</li>
<li class="none">
<a href="../../../../hadoop-dynamometer/Dynamometer.html">Dynamometer</a>
</li>
</ul>
<h5>Reference</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/release/">Changelog and Release Notes</a>
</li>
<li class="none">
<a href="../../../../api/index.html">Java API docs</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/UnixShellAPI.html">Unix Shell API</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/Metrics.html">Metrics</a>
</li>
</ul>
<h5>Configuration</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/core-default.xml">core-default.xml</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/hdfs-default.xml">hdfs-default.xml</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs-rbf/hdfs-rbf-default.xml">hdfs-rbf-default.xml</a>
</li>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml">mapred-default.xml</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-common/yarn-default.xml">yarn-default.xml</a>
</li>
<li class="none">
<a href="../../../../hadoop-kms/kms-default.html">kms-default.xml</a>
</li>
<li class="none">
<a href="../../../../hadoop-hdfs-httpfs/httpfs-default.html">httpfs-default.xml</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/DeprecatedProperties.html">Deprecated Properties</a>
</li>
</ul>
<a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
<img alt="Built by Maven" src="../../images/logos/maven-feather.png"/>
</a>
</div>
</div>
<div id="bodyColumn">
<div id="contentBox">
<!---
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-->
<h1>Apache Hadoop 0.17.0 Release Notes</h1>
<p>These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-1593">HADOOP-1593</a> | <i>Major</i> | <b>FsShell should work with paths in non-default FileSystem</b></li>
</ul>
<p>This bug allows non default path to specifeid in fsshell commands.</p>
<p>So, you can now specify hadoop dfs -ls <a class="externalLink" href="hdfs://remotehost1:port/path">hdfs://remotehost1:port/path</a> and hadoop dfs -ls <a class="externalLink" href="hdfs://remotehost2:port/path">hdfs://remotehost2:port/path</a> without changing the config.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2345">HADOOP-2345</a> | <i>Major</i> | <b>new transactions to support HDFS Appends</b></li>
</ul>
<p>Introduce new namenode transactions to support appending to HDFS files.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2178">HADOOP-2178</a> | <i>Major</i> | <b>Job history on HDFS</b></li>
</ul>
<p>This feature provides facility to store job history on DFS. Now cluster admin can provide either localFS location or DFS location using configuration property &#x201c;mapred.job.history.location&#x201d; to store job histroy. History will be logged in user specified location also. User can specify history location using configuration property &#x201c;mapred.job.history.user.location&#x201d; . The classes org.apache.hadoop.mapred.DefaultJobHistoryParser.MasterIndex and org.apache.hadoop.mapred.DefaultJobHistoryParser.MasterIndexParseListener, and public method org.apache.hadoop.mapred.DefaultJobHistoryParser.parseMasterIndex are not available. The signature of public method org.apache.hadoop.mapred.DefaultJobHistoryParser.parseJobTasks(File jobHistoryFile, JobHistory.JobInfo job) is changed to DefaultJobHistoryParser.parseJobTasks(String jobHistoryFile, JobHistory.JobInfo job, FileSystem fs). The signature of public method org.apache.hadoop.mapred.JobHistory.parseHistory(File path, Listener l) is changed to JobHistory.parseHistoryFromFS(String path, Listener l, FileSystem fs)</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2192">HADOOP-2192</a> | <i>Major</i> | <b>dfs mv command differs from POSIX standards</b></li>
</ul>
<p>this patch makes dfs -mv more like linux mv command getting rid of unnecessary output in dfs -mv and returns an error message when moving non existent files/directories &#x2014; mv: cannot stat &#x201c;filename&#x201d;: No such file or directory.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2873">HADOOP-2873</a> | <i>Major</i> | <b>Namenode fails to re-start after cluster shutdown - DFSClient: Could not obtain blocks even all datanodes were up &amp; live</b></li>
</ul>
<p><b>WARNING: No release note provided for this change.</b></p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2063">HADOOP-2063</a> | <i>Blocker</i> | <b>Command to pull corrupted files</b></li>
</ul>
<p>Added a new option -ignoreCrc to fs -get, or equivalently, fs -copyToLocal, such that crc checksum will be ignored for the command. The use of this option is to download the corrupted files.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-1985">HADOOP-1985</a> | <i>Major</i> | <b>Abstract node to switch mapping into a topology service class used by namenode and jobtracker</b></li>
</ul>
<p>This issue introduces rack awareness for map tasks. It also moves the rack resolution logic to the central servers - NameNode &amp; JobTracker. The administrator can specify a loadable class given by topology.node.switch.mapping.impl to specify the class implementing the logic for rack resolution. The class must implement a method - resolve(List&lt;String&gt; names), where names is the list of DNS-names/IP-addresses that we want resolved. The return value is a list of resolved network paths of the form /foo/rack, where rack is the rackID where the node belongs to and foo is the switch where multiple racks are connected, and so on. The default implementation of this class is packaged along with hadoop and points to org.apache.hadoop.net.ScriptBasedMapping and this class loads a script that can be used for rack resolution. The script location is configurable. It is specified by topology.script.file.name and defaults to an empty script. In the case where the script name is empty, /default-rack is returned for all dns-names/IP-addresses. The loadable topology.node.switch.mapping.impl provides administrators fleixibilty to define how their site&#x2019;s node resolution should happen. For mapred, one can also specify the level of the cache w.r.t the number of levels in the resolved network path - defaults to two. This means that the JobTracker will cache tasks at the host level and at the rack level. Known issue: the task caching will not work with levels greater than 2 (beyond racks). This bug is tracked in HADOOP-3296.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-1986">HADOOP-1986</a> | <i>Major</i> | <b>Add support for a general serialization mechanism for Map Reduce</b></li>
</ul>
<p>Programs that implement the raw Mapper or Reducer interfaces will need modification to compile with this release. For example,</p>
<p>class MyMapper implements Mapper { public void map(WritableComparable key, Writable val, OutputCollector out, Reporter reporter) throws IOException { // &#x2026; } // &#x2026; }</p>
<p>will need to be changed to refer to the parameterized type. For example:</p>
<p>class MyMapper implements Mapper&lt;WritableComparable, Writable, WritableComparable, Writable&gt; { public void map(WritableComparable key, Writable val, OutputCollector&lt;WritableComparable, Writable&gt; out, Reporter reporter) throws IOException { // &#x2026; } // &#x2026; }</p>
<p>Similarly implementations of the following raw interfaces will need modification: InputFormat, OutputCollector, OutputFormat, Partitioner, RecordReader, RecordWriter</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-910">HADOOP-910</a> | <i>Major</i> | <b>Reduces can do merges for the on-disk map output files in parallel with their copying</b></li>
</ul>
<p>Reducers now perform merges of shuffle data (both in-memory and on disk) while fetching map outputs. Earlier, during shuffle they used to merge only the in-memory outputs.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2219">HADOOP-2219</a> | <i>Major</i> | <b>du like command to count number of files under a given directory</b></li>
</ul>
<p>Added a new fs command fs -count for counting the number of bytes, files and directories under a given path.</p>
<p>Added a new RPC getContentSummary(String path) to ClientProtocol.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2820">HADOOP-2820</a> | <i>Major</i> | <b>Remove deprecated classes in streaming</b></li>
</ul>
<p>The deprecated classes org.apache.hadoop.streaming.StreamLineRecordReader, org.apache.hadoop.streaming.StreamOutputFormat and org.apache.hadoop.streaming.StreamSequenceRecordReader are removed</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2819">HADOOP-2819</a> | <i>Major</i> | <b>Remove deprecated methods in JobConf()</b></li>
</ul>
<p>The following deprecated methods are removed from org.apache.hadoop.JobConf : public Class getInputKeyClass() public void setInputKeyClass(Class theClass) public Class getInputValueClass() public void setInputValueClass(Class theClass)</p>
<p>The methods, public boolean org.apache.hadoop.JobConf.getSpeculativeExecution() and public void org.apache.hadoop.JobConf.setSpeculativeExecution(boolean speculativeExecution) are undeprecated.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2817">HADOOP-2817</a> | <i>Major</i> | <b>Remove deprecated mapred.tasktracker.tasks.maximum and clusterStatus.getMaxTasks()</b></li>
</ul>
<p>The deprecated method public int org.apache.hadoop.mapred.ClusterStatus.getMaxTasks() is removed. The deprecated configuration property &#x201c;mapred.tasktracker.tasks.maximum&#x201d; is removed.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2821">HADOOP-2821</a> | <i>Major</i> | <b>Remove deprecated classes in util</b></li>
</ul>
<p>The deprecated classes org.apache.hadoop.util.ShellUtil and org.apache.hadoop.util.ToolBase are removed.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2758">HADOOP-2758</a> | <i>Major</i> | <b>Reduce memory copies when data is read from DFS</b></li>
</ul>
<p>DataNode takes 50% less CPU while serving data to clients.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-771">HADOOP-771</a> | <i>Major</i> | <b>Namenode should return error when trying to delete non-empty directory</b></li>
</ul>
<p>This patch adds a new api to file system i.e delete(path, boolean), deprecating the previous delete(path). the new api recursively deletes files only if boolean is set to true. If path is a file, the boolean value does not matter, if path is a directory and the directory is non empty delete(path, false) will throw an exception and delete(path, true) will delete all files recursively.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2765">HADOOP-2765</a> | <i>Major</i> | <b>setting memory limits for tasks</b></li>
</ul>
<p>This feature enables specifying ulimits for streaming/pipes tasks. Now pipes and streaming tasks have same virtual memory available as the java process which invokes them. Ulimit value will be the same as -Xmx value for java processes provided using mapred.child.java.opts.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2657">HADOOP-2657</a> | <i>Major</i> | <b>Enhancements to DFSClient to support flushing data at any point in time</b></li>
</ul>
<p>A new API DFSOututStream.flush() flushes all outstanding data to the pipeline of datanodes.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2399">HADOOP-2399</a> | <i>Major</i> | <b>Input key and value to combiner and reducer should be reused</b></li>
</ul>
<p>The key and value objects that are given to the Combiner and Reducer are now reused between calls. This is much more efficient, but the user can not assume the objects are constant.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2423">HADOOP-2423</a> | <i>Major</i> | <b>The codes in FSDirectory.mkdirs(&#x2026;) is inefficient.</b></li>
</ul>
<p>Improved FSDirectory.mkdirs(&#x2026;) performance. In NNThroughputBenchmark-create, the ops per sec in was improved ~54%.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2470">HADOOP-2470</a> | <i>Major</i> | <b>Open and isDir should be removed from ClientProtocol</b></li>
</ul>
<p>Open and isDir were removed from ClientProtocol.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2775">HADOOP-2775</a> | <i>Major</i> | <b>[HOD] Put in place unit test framework for HOD</b></li>
</ul>
<p>A unit testing framework based on pyunit is added to HOD. Developers contributing patches to HOD should now contribute unit tests along with the patches where possible.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2825">HADOOP-2825</a> | <i>Major</i> | <b>MapOutputLocation.getFile() needs to be removed</b></li>
</ul>
<p>The deprecated method, public long org.apache.hadoop.mapred.MapOutputLocation.getFile(FileSystem fileSys, Path localFilename, int reduce, Progressable pingee, int timeout) is removed.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2822">HADOOP-2822</a> | <i>Major</i> | <b>Remove deprecated classes in mapred</b></li>
</ul>
<p>The deprecated classes org.apache.hadoop.mapred.InputFormatBase and org.apache.hadoop.mapred.PhasedFileSystem are removed.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2559">HADOOP-2559</a> | <i>Major</i> | <b>DFS should place one replica per rack</b></li>
</ul>
<p>Change DFS block placement to allocate the first replica locally, the second off-rack, and the third intra-rack from the second.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2239">HADOOP-2239</a> | <i>Major</i> | <b>Security: Need to be able to encrypt Hadoop socket connections</b></li>
</ul>
<p>This patch adds a new FileSystem, HftpsFileSystem, that allows access to HDFS data over HTTPS.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2027">HADOOP-2027</a> | <i>Major</i> | <b>FileSystem should provide byte ranges for file locations</b></li>
</ul>
<p>New FileSystem API getFileBlockLocations to return the number of bytes in each block in a file via a single rpc to the namenode to speed up job planning. Deprecates getFileCacheHints.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2899">HADOOP-2899</a> | <i>Major</i> | <b>[HOD] <a class="externalLink" href="hdfs:///mapredsystem">hdfs:///mapredsystem</a> directory not cleaned up after deallocation</b></li>
</ul>
<p>The mapred system directory generated by HOD is cleaned up at cluster deallocation time.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2116">HADOOP-2116</a> | <i>Major</i> | <b>Job.local.dir to be exposed to tasks</b></li>
</ul>
<p>This issue restructures local job directory on the tasktracker. Users are provided with a job-specific shared directory (mapred-local/taskTracker/jobcache/$jobid/ work) for using it as scratch space, through configuration property and system property &#x201c;job.local.dir&#x201d;. Now, the directory &#x201c;../work&#x201d; is not available from the task&#x2019;s cwd.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2796">HADOOP-2796</a> | <i>Major</i> | <b>For script option hod should exit with distinguishable exit codes for script code and hod exit code.</b></li>
</ul>
<p>A provision to reliably detect a failing script&#x2019;s exit code was added. In case the hod script option returned a non-zero exit code, users can now look for a &#x2018;script.exitcode&#x2019; file written to the HOD cluster directory. If this file is present, it means the script failed with the returned exit code.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2828">HADOOP-2828</a> | <i>Major</i> | <b>Remove deprecated methods in Configuration.java</b></li>
</ul>
<p>The following deprecated methods in org.apache.hadoop.conf.Configuration are removed. public Object getObject(String name) public void setObject(String name, Object value) public Object get(String name, Object defaultValue) public void set(String name, Object value) and public Iterator entries()</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2824">HADOOP-2824</a> | <i>Major</i> | <b>One of MiniMRCluster constructors needs tobe removed</b></li>
</ul>
<p>The deprecated constructor org.apache.hadoop.mapred.MiniMRCluster.MiniMRCluster(int jobTrackerPort, int taskTrackerPort, int numTaskTrackers, String namenode, boolean taskTrackerFirst) is removed.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2823">HADOOP-2823</a> | <i>Major</i> | <b>SimpleCharStream.getColumn(), getLine() methods to be removed.</b></li>
</ul>
<p>The deprecated methods in org.apache.hadoop.record.compiler.generated.SimpleCharStream : public int getColumn() and public int getLine() are removed</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-3040">HADOOP-3040</a> | <i>Major</i> | <b>Streaming should assume an empty key if the first character on a line is the seperator (stream.map.output.field.separator, by default, tab)</b></li>
</ul>
<p>If the first character on a line is the separator, empty key is assumed, and the whole line is the value (due to a bug this was not the case).</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-1622">HADOOP-1622</a> | <i>Major</i> | <b>Hadoop should provide a way to allow the user to specify jar file(s) the user job depends on</b></li>
</ul>
<p>This patch allows new command line options for</p>
<p>hadoop jar which are</p>
<p>hadoop jar -files &lt;comma seperated list of files&gt; -libjars &lt;comma seperated list of jars&gt; -archives &lt;comma seperated list of archives&gt;</p>
<p>-files options allows you to speficy comma seperated list of path which would be present in your current working directory of your task -libjars option allows you to add jars to the classpaths of the maps and reduces. -archives allows you to pass archives as arguments that are unzipped/unjarred and a link with name of the jar/zip are created in the current working directory if tasks.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2119">HADOOP-2119</a> | <i>Critical</i> | <b>JobTracker becomes non-responsive if the task trackers finish task too fast</b></li>
</ul>
<p>This removes many inefficiencies in task placement and scheduling logic. The JobTracker would perform linear scans of the list of submitted tasks in cases where it did not find an obvious candidate task for a node. With better data structures for managing job state, all task placement operations now run in constant time (in most cases). Also, the task output promotions are batched.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-3091">HADOOP-3091</a> | <i>Major</i> | <b>hadoop dfs -put should support multiple src</b></li>
</ul>
<p>hadoop dfs -put accepts multiple sources when destination is a directory.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-3073">HADOOP-3073</a> | <i>Blocker</i> | <b>SocketOutputStream.close() should close the channel.</b></li>
</ul>
<p>SocketOutputStream.close() closes the underlying channel. Increase compatibility with java.net.Socket.getOutputStream. User Impact : none.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2982">HADOOP-2982</a> | <i>Blocker</i> | <b>[HOD] checknodes should look for free nodes without the jobs attribute</b></li>
</ul>
<p>The number of free nodes in the cluster is computed using a better algorithm that filters out inconsistencies in node status as reported by Torque.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-3060">HADOOP-3060</a> | <i>Major</i> | <b>MiniMRCluster is ignoring parameter taskTrackerFirst</b></li>
</ul>
<p>The parameter boolean taskTrackerFirst is removed from org.apache.hadoop.mapred.MiniMRCluster constructors. Thus signature of following APIs public MiniMRCluster(int jobTrackerPort, int taskTrackerPort, int numTaskTrackers, String namenode, boolean taskTrackerFirst, int numDir) public MiniMRCluster(int jobTrackerPort, int taskTrackerPort, int numTaskTrackers, String namenode, boolean taskTrackerFirst, int numDir, String[] racks) public MiniMRCluster(int jobTrackerPort, int taskTrackerPort, int numTaskTrackers, String namenode, boolean taskTrackerFirst, int numDir, String[] racks, String[] hosts) public MiniMRCluster(int jobTrackerPort, int taskTrackerPort, int numTaskTrackers, String namenode, boolean taskTrackerFirst, int numDir, String[] racks, String[] hosts, UnixUserGroupInformation ugi ) is changed to public MiniMRCluster(int jobTrackerPort, int taskTrackerPort, int numTaskTrackers, String namenode, int numDir) public MiniMRCluster(int jobTrackerPort, int taskTrackerPort, int numTaskTrackers, String namenode, int numDir, String[] racks) public MiniMRCluster(int jobTrackerPort, int taskTrackerPort, int numTaskTrackers, String namenode, int numDir, String[] racks, String[] hosts) public MiniMRCluster(int jobTrackerPort, int taskTrackerPort, int numTaskTrackers, String namenode, int numDir, String[] racks, String[] hosts, UnixUserGroupInformation ugi ) respectively. Since the old signatures were not deprecated, any code using the old constructors must be changed to use the new constructors.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2055">HADOOP-2055</a> | <i>Minor</i> | <b>JobConf should have a setInputPathFilter method</b></li>
</ul>
<p>This issue provides users the ability to specify what paths to ignore for processing in the job input directory (apart from the filenames that start with &#x201c;_&#x201d; and &#x201c;.&#x201d;). Defines two new APIs - FileInputFormat.setInputPathFilter(JobConf, PathFilter), and, FileInputFormat.getInputPathFilter(JobConf).</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2854">HADOOP-2854</a> | <i>Blocker</i> | <b>Remove the deprecated ipc.Server.getUserInfo()</b></li>
</ul>
<p>Removes deprecated method Server.getUserInfo()</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2563">HADOOP-2563</a> | <i>Blocker</i> | <b>Remove deprecated FileSystem#listPaths()</b></li>
</ul>
<p>Removes deprecated method FileSystem#listPaths()</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2855">HADOOP-2855</a> | <i>Blocker</i> | <b>[HOD] HOD fails to allocate a cluster if the tarball specified is a relative path</b></li>
</ul>
<p>Changes were made to handle relative paths correctly for important HOD options such as the cluster directory, tarball option, and script file.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2818">HADOOP-2818</a> | <i>Major</i> | <b>Remove deprecated Counters.getDisplayName(), getCounterNames(), getCounter(String counterName)</b></li>
</ul>
<p>The deprecated methods public String org.apache.hadoop.mapred.Counters.getDisplayName(String counter) and public synchronized Collection&lt;String&gt; org.apache.hadoop.mapred.Counters.getCounterNames() are removed. The deprecated method public synchronized long org.apache.hadoop.mapred.Counters.getCounter(String counterName) is undeprecated.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2831">HADOOP-2831</a> | <i>Blocker</i> | <b>Remove the deprecated INode.getAbsoluteName()</b></li>
</ul>
<p>Removes deprecated method INode#getAbsoluteName()</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2947">HADOOP-2947</a> | <i>Blocker</i> | <b>[HOD] Hod should redirect stderr and stdout of Hadoop daemons to assist debugging</b></li>
</ul>
<p>The stdout and stderr streams of daemons are redirected to files that are created under the hadoop log directory. Users can now send kill 3 signals to the daemons to get stack traces and thread dumps for debugging.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-3137">HADOOP-3137</a> | <i>Major</i> | <b>[HOD] Update hod version number</b></li>
</ul>
<p>Build script was changed to make HOD versions follow Hadoop version numbers. As a result of this change, the next version of HOD would not be 0.5, but would be synchronized to the Hadoop version number. Users who rely on the version number of HOD should note the unexpected jump in version numbers.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-3093">HADOOP-3093</a> | <i>Major</i> | <b>ma/reduce throws the following exception if &#x201c;io.serializations&#x201d; is not set:</b></li>
</ul>
<p>The following public APIs are added in org.apache.hadoop.conf.Configuration String[] Configuration.getStrings(String name, String&#x2026; defaultValue) and void Configuration.setStrings(String name, String&#x2026; values)</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2839">HADOOP-2839</a> | <i>Blocker</i> | <b>Remove deprecated methods in FileSystem</b></li>
</ul>
<p>Removes deprecated API FileSystem#globPaths()</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2551">HADOOP-2551</a> | <i>Blocker</i> | <b>hadoop-env.sh needs finer granularity</b></li>
</ul>
<p>New environment variables were introduced to allow finer grained control of Java options passed to server and client JVMs. See the new *_OPTS variables in conf/hadoop-env.sh.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2634">HADOOP-2634</a> | <i>Blocker</i> | <b>Deprecate exists() and isDir() to simplify ClientProtocol.</b></li>
</ul>
<p>Deprecates exists() from ClientProtocol</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-3099">HADOOP-3099</a> | <i>Blocker</i> | <b>Need new options in distcp for preserving ower, group and permission</b></li>
</ul>
<p>Added a new option -p to distcp for preserving file/directory status. -p[rbugp] Preserve status r: replication number b: block size u: user g: group p: permission -p alone is equivalent to -prbugp</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-3001">HADOOP-3001</a> | <i>Blocker</i> | <b>FileSystems should track how many bytes are read and written</b></li>
</ul>
<p>Adds new framework map/reduce counters that track the number of bytes read and written to HDFS, local, KFS, and S3 file systems.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-3048">HADOOP-3048</a> | <i>Blocker</i> | <b>Stringifier</b></li>
</ul>
<p>A new Interface and a default implementation to convert and restore serializations of objects to strings.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2410">HADOOP-2410</a> | <i>Major</i> | <b>Make EC2 cluster nodes more independent of each other</b></li>
</ul>
<p>The command &#x201c;hadoop-ec2 run&#x201d; has been replaced by &#x201c;hadoop-ec2 launch-cluster &lt;group&gt; &lt;number of instances&gt;&#x201d;, and &#x201c;hadoop-ec2 start-hadoop&#x201d; has been removed since Hadoop is started on instance start up. See <a class="externalLink" href="http://wiki.apache.org/hadoop/AmazonEC2">http://wiki.apache.org/hadoop/AmazonEC2</a> for details.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-2826">HADOOP-2826</a> | <i>Major</i> | <b>FileSplit.getFile(), LineRecordReader. readLine() need to be removed</b></li>
</ul>
<p>The deprecated methods, public File org.apache.hadoop.mapred.FileSplit.getFile() and public static long org.apache.hadoop.mapred.LineRecordReader.readLine(InputStream in, OutputStream out) are removed. The constructor org.apache.hadoop.mapred.LineRecordReader.LineReader(InputStream in, Configuration conf) &#x2019;s visibility is made public. The signature of the public org.apache.hadoop.streaming.UTF8ByteArrayUtils.readLIne(InputStream) method is changed to UTF8ByteArrayUtils.readLIne(LineReader, Text). Since the old signature is not deprecated, any code using the old method must be changed to use the new method.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-3140">HADOOP-3140</a> | <i>Major</i> | <b>JobTracker should not try to promote a (map) task if it does not write to DFS at all</b></li>
</ul>
<p>Tasks that don&#x2019;t generate any output are not inserted in the commit queue of the JobTracker. They are marked as SUCCESSFUL by the TaskTracker and the JobTracker updates their state short-circuiting the commit queue.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-3041">HADOOP-3041</a> | <i>Blocker</i> | <b>Within a task, the value ofJobConf.getOutputPath() method is modified</b></li>
</ul>
<ol style="list-style-type: decimal">
<li>Deprecates JobConf.setOutputPath and JobConf.getOutputPath JobConf.getOutputPath() still returns the same value that it used to return.</li>
<li>Deprecates OutputFormatBase. Adds FileOutputFormat. Existing output formats extending OutputFormatBase, now extend FileOutputFormat.</li>
<li>Adds the following APIs in FileOutputFormat : public static void setOutputPath(JobConf conf, Path outputDir); // sets mapred.output.dir public static Path getOutputPath(JobConf conf) ; // gets mapred.output.dir public static Path getWorkOutputPath(JobConf conf); // gets mapred.work.output.dir</li>
<li>static void setWorkOutputPath(JobConf conf, Path outputDir) is also added to FileOutputFormat. This is used by the framework to set mapred.work.output.dir as task&#x2019;s temporary output dir .</li>
</ol><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-3168">HADOOP-3168</a> | <i>Major</i> | <b>reduce amount of logging in hadoop streaming</b></li>
</ul>
<p>Decreases the frequency of logging from streaming from every 100 records to every 10,000 records.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-3152">HADOOP-3152</a> | <i>Minor</i> | <b>Make index interval configuable when using MapFileOutputFormat for map-reduce job</b></li>
</ul>
<p>Add a static method MapFile#setIndexInterval(Configuration, int interval) so that MapReduce jobs that use MapFileOutputFormat can set the index interval.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-3223">HADOOP-3223</a> | <i>Blocker</i> | <b>Hadoop dfs -help for permissions contains a typo</b></li>
</ul>
<p>Minor typo fix in help message for chmod. impact : none.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-3204">HADOOP-3204</a> | <i>Blocker</i> | <b>LocalFSMerger needs to catch throwable</b></li>
</ul>
<p>Fixes LocalFSMerger in ReduceTask.java to handle errors/exceptions better. Prior to this all exceptions except IOException would be silently ignored.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-3239">HADOOP-3239</a> | <i>Major</i> | <b>exists() calls logs FileNotFoundException in namenode log</b></li>
</ul>
<p>getFileInfo returns null for File not found instead of throwing FileNotFoundException</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-3162">HADOOP-3162</a> | <i>Blocker</i> | <b>Map/reduce stops working with comma separated input paths</b></li>
</ul>
<p>The public methods org.apache.hadoop.mapred.JobConf.setInputPath(Path) and org.apache.hadoop.mapred.JobConf.addInputPath(Path) are deprecated. And the methods have the semantics of branch 0.16. The following public APIs are added in org.apache.hadoop.mapred.FileInputFormat : public static void setInputPaths(JobConf job, Path&#x2026; paths); public static void setInputPaths(JobConf job, String commaSeparatedPaths); public static void addInputPath(JobConf job, Path path); public static void addInputPaths(JobConf job, String commaSeparatedPaths); Earlier code calling JobConf.setInputPath(Path), JobConf.addInputPath(Path) should now call FileInputFormat.setInputPaths(JobConf, Path&#x2026;) and FileInputFormat.addInputPath(Path) respectively</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-3124">HADOOP-3124</a> | <i>Major</i> | <b>DFS data node should not use hard coded 10 minutes as write timeout.</b></li>
</ul>
<p>Makes DataNode socket write timeout configurable. User impact : none.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-3266">HADOOP-3266</a> | <i>Major</i> | <b>Remove HOD changes from CHANGES.txt, as they are now inside src/contrib/hod</b></li>
</ul>
<p>Moved HOD change items from CHANGES.txt to a new file src/contrib/hod/CHANGES.txt.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-3280">HADOOP-3280</a> | <i>Blocker</i> | <b>virtual address space limits break streaming apps</b></li>
</ul>
<p>This patch adds the mapred.child.ulimit to limit the virtual memory for children processes to the given value.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-3382">HADOOP-3382</a> | <i>Blocker</i> | <b>Memory leak when files are not cleanly closed</b></li>
</ul>
<p>Fixed a memory leak associated with &#x2018;abandoned&#x2019; files (i.e. not cleanly closed). This held up significant amounts of memory depending on activity and how long NameNode has been running.</p>
</div>
</div>
<div class="clear">
<hr/>
</div>
<div id="footer">
<div class="xright">
&#169; 2008-2021
Apache Software Foundation
- <a href="http://maven.apache.org/privacy-policy.html">Privacy Policy</a>.
Apache Maven, Maven, Apache, the Apache feather logo, and the Apache Maven project logos are trademarks of The Apache Software Foundation.
</div>
<div class="clear">
<hr/>
</div>
</div>
</body>
</html>