Google Git
Sign in
apache / hadoop-site / 952d547359cafef280f0a0324055741b2ed55be1 / . / content / docs / r3.3.1 / hadoop-project-dist / hadoop-common / release / 3.0.0-beta1 / RELEASENOTES.3.0.0-beta1.html
blob: f7d8126a83d5dd2503434016dfa7ae71ce8e9592 [file] [log] [blame]
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<!--
| Generated by Apache Maven Doxia at 2021-06-15
| Rendered using Apache Maven Stylus Skin 1.5
-->
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Apache Hadoop 3.3.1 &#x2013; Apache Hadoop 3.0.0-beta1 Release Notes</title>
<style type="text/css" media="all">
@import url("../../css/maven-base.css");
@import url("../../css/maven-theme.css");
@import url("../../css/site.css");
</style>
<link rel="stylesheet" href="../../css/print.css" type="text/css" media="print" />
<meta name="Date-Revision-yyyymmdd" content="20210615" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body class="composite">
<div id="banner">
<a href="http://hadoop.apache.org/" id="bannerLeft">
<img src="http://hadoop.apache.org/images/hadoop-logo.jpg" alt="" />
</a>
<a href="http://www.apache.org/" id="bannerRight">
<img src="http://www.apache.org/images/asf_logo_wide.png" alt="" />
</a>
<div class="clear">
<hr/>
</div>
</div>
<div id="breadcrumbs">
<div class="xleft">
<a href="http://www.apache.org/" class="externalLink">Apache</a>
&gt;
<a href="http://hadoop.apache.org/" class="externalLink">Hadoop</a>
&gt;
<a href="../../../index.html">Apache Hadoop Project Dist POM</a>
&gt;
<a href="../../index.html">Apache Hadoop 3.3.1</a>
&gt;
Apache Hadoop 3.0.0-beta1 Release Notes
</div>
<div class="xright"> <a href="http://wiki.apache.org/hadoop" class="externalLink">Wiki</a>
|
<a href="https://gitbox.apache.org/repos/asf/hadoop.git" class="externalLink">git</a>
|
<a href="http://hadoop.apache.org/" class="externalLink">Apache Hadoop</a>
&nbsp;| Last Published: 2021-06-15
&nbsp;| Version: 3.3.1
</div>
<div class="clear">
<hr/>
</div>
</div>
<div id="leftColumn">
<div id="navcolumn">
<h5>General</h5>
<ul>
<li class="none">
<a href="../../../../index.html">Overview</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/SingleCluster.html">Single Node Setup</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/ClusterSetup.html">Cluster Setup</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/CommandsManual.html">Commands Reference</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/FileSystemShell.html">FileSystem Shell</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/Compatibility.html">Compatibility Specification</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/DownstreamDev.html">Downstream Developer's Guide</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/AdminCompatibilityGuide.html">Admin Compatibility Guide</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/InterfaceClassification.html">Interface Classification</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/filesystem/index.html">FileSystem Specification</a>
</li>
</ul>
<h5>Common</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/CLIMiniCluster.html">CLI Mini Cluster</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/FairCallQueue.html">Fair Call Queue</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/NativeLibraries.html">Native Libraries</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/Superusers.html">Proxy User</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/RackAwareness.html">Rack Awareness</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/SecureMode.html">Secure Mode</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/ServiceLevelAuth.html">Service Level Authorization</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/HttpAuthentication.html">HTTP Authentication</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/CredentialProviderAPI.html">Credential Provider API</a>
</li>
<li class="none">
<a href="../../../../hadoop-kms/index.html">Hadoop KMS</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/Tracing.html">Tracing</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/UnixShellGuide.html">Unix Shell Guide</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/registry/index.html">Registry</a>
</li>
</ul>
<h5>HDFS</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsDesign.html">Architecture</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html">User Guide</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HDFSCommands.html">Commands Reference</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html">NameNode HA With QJM</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html">NameNode HA With NFS</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html">Observer NameNode</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/Federation.html">Federation</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/ViewFs.html">ViewFs</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/ViewFsOverloadScheme.html">ViewFsOverloadScheme</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html">Snapshots</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsEditsViewer.html">Edits Viewer</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html">Image Viewer</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html">Permissions and HDFS</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html">Quotas and HDFS</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/LibHdfs.html">libhdfs (C API)</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/WebHDFS.html">WebHDFS (REST API)</a>
</li>
<li class="none">
<a href="../../../../hadoop-hdfs-httpfs/index.html">HttpFS</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html">Short Circuit Local Reads</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html">Centralized Cache Management</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html">NFS Gateway</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html">Rolling Upgrade</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/ExtendedAttributes.html">Extended Attributes</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html">Transparent Encryption</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html">Multihoming</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html">Storage Policies</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/MemoryStorage.html">Memory Storage Support</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/SLGUserGuide.html">Synthetic Load Generator</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html">Erasure Coding</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HDFSDiskbalancer.html">Disk Balancer</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsUpgradeDomain.html">Upgrade Domain</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsDataNodeAdminGuide.html">DataNode Admin</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html">Router Federation</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsProvidedStorage.html">Provided Storage</a>
</li>
</ul>
<h5>MapReduce</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html">Tutorial</a>
</li>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html">Commands Reference</a>
</li>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html">Compatibility with 1.x</a>
</li>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html">Encrypted Shuffle</a>
</li>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html">Pluggable Shuffle/Sort</a>
</li>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html">Distributed Cache Deploy</a>
</li>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/SharedCacheSupport.html">Support for YARN Shared Cache</a>
</li>
</ul>
<h5>MapReduce REST APIs</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredAppMasterRest.html">MR Application Master</a>
</li>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-hs/HistoryServerRest.html">MR History Server</a>
</li>
</ul>
<h5>YARN</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/YARN.html">Architecture</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/YarnCommands.html">Commands Reference</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html">Capacity Scheduler</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/FairScheduler.html">Fair Scheduler</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html">ResourceManager Restart</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html">ResourceManager HA</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/ResourceModel.html">Resource Model</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/NodeLabel.html">Node Labels</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/NodeAttributes.html">Node Attributes</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html">Web Application Proxy</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html">Timeline Server</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html">Timeline Service V.2</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html">Writing YARN Applications</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html">YARN Application Security</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/NodeManager.html">NodeManager</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/DockerContainers.html">Running Applications in Docker Containers</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/RuncContainers.html">Running Applications in runC Containers</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/NodeManagerCgroups.html">Using CGroups</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/SecureContainer.html">Secure Containers</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/ReservationSystem.html">Reservation System</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html">Graceful Decommission</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html">Opportunistic Containers</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/Federation.html">YARN Federation</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/SharedCache.html">Shared Cache</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/UsingGpus.html">Using GPU</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/UsingFPGA.html">Using FPGA</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html">Placement Constraints</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/YarnUI2.html">YARN UI2</a>
</li>
</ul>
<h5>YARN REST APIs</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html">Introduction</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html">Resource Manager</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/NodeManagerRest.html">Node Manager</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html#Timeline_Server_REST_API_v1">Timeline Server</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html#Timeline_Service_v.2_REST_API">Timeline Service V.2</a>
</li>
</ul>
<h5>YARN Service</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/yarn-service/Overview.html">Overview</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/yarn-service/QuickStart.html">QuickStart</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/yarn-service/Concepts.html">Concepts</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/yarn-service/YarnServiceAPI.html">Yarn Service API</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/yarn-service/ServiceDiscovery.html">Service Discovery</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/yarn-service/SystemServices.html">System Services</a>
</li>
</ul>
<h5>Hadoop Compatible File Systems</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-aliyun/tools/hadoop-aliyun/index.html">Aliyun OSS</a>
</li>
<li class="none">
<a href="../../../../hadoop-aws/tools/hadoop-aws/index.html">Amazon S3</a>
</li>
<li class="none">
<a href="../../../../hadoop-azure/index.html">Azure Blob Storage</a>
</li>
<li class="none">
<a href="../../../../hadoop-azure-datalake/index.html">Azure Data Lake Storage</a>
</li>
<li class="none">
<a href="../../../../hadoop-openstack/index.html">OpenStack Swift</a>
</li>
<li class="none">
<a href="../../../../hadoop-cos/cloud-storage/index.html">Tencent COS</a>
</li>
</ul>
<h5>Auth</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-auth/index.html">Overview</a>
</li>
<li class="none">
<a href="../../../../hadoop-auth/Examples.html">Examples</a>
</li>
<li class="none">
<a href="../../../../hadoop-auth/Configuration.html">Configuration</a>
</li>
<li class="none">
<a href="../../../../hadoop-auth/BuildingIt.html">Building</a>
</li>
</ul>
<h5>Tools</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-streaming/HadoopStreaming.html">Hadoop Streaming</a>
</li>
<li class="none">
<a href="../../../../hadoop-archives/HadoopArchives.html">Hadoop Archives</a>
</li>
<li class="none">
<a href="../../../../hadoop-archive-logs/HadoopArchiveLogs.html">Hadoop Archive Logs</a>
</li>
<li class="none">
<a href="../../../../hadoop-distcp/DistCp.html">DistCp</a>
</li>
<li class="none">
<a href="../../../../hadoop-gridmix/GridMix.html">GridMix</a>
</li>
<li class="none">
<a href="../../../../hadoop-rumen/Rumen.html">Rumen</a>
</li>
<li class="none">
<a href="../../../../hadoop-resourceestimator/ResourceEstimator.html">Resource Estimator Service</a>
</li>
<li class="none">
<a href="../../../../hadoop-sls/SchedulerLoadSimulator.html">Scheduler Load Simulator</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/Benchmarking.html">Hadoop Benchmarking</a>
</li>
<li class="none">
<a href="../../../../hadoop-dynamometer/Dynamometer.html">Dynamometer</a>
</li>
</ul>
<h5>Reference</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/release/">Changelog and Release Notes</a>
</li>
<li class="none">
<a href="../../../../api/index.html">Java API docs</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/UnixShellAPI.html">Unix Shell API</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/Metrics.html">Metrics</a>
</li>
</ul>
<h5>Configuration</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/core-default.xml">core-default.xml</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/hdfs-default.xml">hdfs-default.xml</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs-rbf/hdfs-rbf-default.xml">hdfs-rbf-default.xml</a>
</li>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml">mapred-default.xml</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-common/yarn-default.xml">yarn-default.xml</a>
</li>
<li class="none">
<a href="../../../../hadoop-kms/kms-default.html">kms-default.xml</a>
</li>
<li class="none">
<a href="../../../../hadoop-hdfs-httpfs/httpfs-default.html">httpfs-default.xml</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/DeprecatedProperties.html">Deprecated Properties</a>
</li>
</ul>
<a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
<img alt="Built by Maven" src="../../images/logos/maven-feather.png"/>
</a>
</div>
</div>
<div id="bodyColumn">
<div id="contentBox">
<!---
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-->
<h1>Apache Hadoop 3.0.0-beta1 Release Notes</h1>
<p>These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14535">HADOOP-14535</a> | <i>Major</i> | <b>wasb: implement high-performance random access and seek of block blobs</b></li>
</ul>
<p>Random access and seek improvements for the <a class="externalLink" href="wasb://">wasb://</a> (Azure) file system.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-6798">YARN-6798</a> | <i>Major</i> | <b>Fix NM startup failure with old state store due to version mismatch</b></li>
</ul><!-- markdown -->
<p>This fixes the LevelDB state store for the NodeManager. As of this patch, the state store versions now correspond to the following table.</p>
<ul>
<li>Previous Patch: YARN-5049</li>
<li>LevelDB Key: queued</li>
<li>Hadoop Versions: 2.9.0, 3.0.0-alpha1</li>
<li>Corresponding LevelDB Version: 1.2</li>
<li>Previous Patch: YARN-6127</li>
<li>LevelDB Key: AMRMProxy/NextMasterKey</li>
<li>Hadoop Versions: 2.9.0, 3.0.0-alpha4</li>
<li>Corresponding LevelDB Version: 1.1</li>
</ul><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14539">HADOOP-14539</a> | <i>Major</i> | <b>Move commons logging APIs over to slf4j in hadoop-common</b></li>
</ul>
<p>In Hadoop common, fatal log level is changed to error because slf4j API does not support fatal log level.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14518">HADOOP-14518</a> | <i>Minor</i> | <b>Customize User-Agent header sent in HTTP/HTTPS requests by WASB.</b></li>
</ul>
<p>WASB now includes the current Apache Hadoop version in the User-Agent string passed to Azure Blob service. Users also may include optional additional information to identify their application. See the documentation of configuration property fs.wasb.user.agent.id for further details.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-11875">HADOOP-11875</a> | <i>Major</i> | <b>[JDK9] Add a second copy of Hamlet without _ as a one-character identifier</b></li>
</ul>
<p>Added org.apache.hadoop.yarn.webapp.hamlet2 package without _ as a one-character identifier. Please use this package instead of org.apache.hadoop.yarn.webapp.hamlet.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-12206">HDFS-12206</a> | <i>Major</i> | <b>Rename the split EC / replicated block metrics</b></li>
</ul>
<p>The metrics and MBeans introduced in HDFS-10999 have been renamed for brevity and clarity.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13595">HADOOP-13595</a> | <i>Blocker</i> | <b>Rework hadoop_usage to be broken up by clients/daemons/etc.</b></li>
</ul>
<p>This patch changes how usage output is generated to now require a sub-command type. This allows users to see who the intended audience for a command is or it is a daemon.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-6984">HDFS-6984</a> | <i>Major</i> | <b>Serialize FileStatus via protobuf</b></li>
</ul>
<p>FileStatus and FsPermission Writable serialization is deprecated and its implementation (incompatibly) replaced with protocol buffers. The FsPermissionProto record moved from hdfs.proto to acl.proto. HdfsFileStatus is now a subtype of FileStatus. FsPermissionExtension with its associated flags for ACLs, encryption, and erasure coding has been deprecated; users should query these attributes on the FileStatus object directly. The FsPermission instance in AclStatus no longer retains or reports these extended attributes (likely unused).</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14722">HADOOP-14722</a> | <i>Major</i> | <b>Azure: BlockBlobInputStream position incorrect after seek</b></li>
</ul>
<p>Bug fix to Azure Filesystem related to HADOOP-14535.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-6961">YARN-6961</a> | <i>Minor</i> | <b>Remove commons-logging dependency from hadoop-yarn-server-applicationhistoryservice module</b></li>
</ul>
<p>commons-logging dependency was removed from hadoop-yarn-server-applicationhistoryservice. If you rely on the transitive commons-logging dependency, please define the dependency explicitly.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14680">HADOOP-14680</a> | <i>Minor</i> | <b>Azure: IndexOutOfBoundsException in BlockBlobInputStream</b></li>
</ul>
<p>Bug fix to Azure Filesystem related to HADOOP-14535</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-10326">HDFS-10326</a> | <i>Major</i> | <b>Disable setting tcp socket send/receive buffers for write pipelines</b></li>
</ul>
<p>The size of the TCP socket buffers are no longer hardcoded by default. Instead the OS now will automatically tune the size for the buffer.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-11957">HDFS-11957</a> | <i>Major</i> | <b>Enable POSIX ACL inheritance by default</b></li>
</ul><!-- markdown -->
<p>HDFS-6962 introduced POSIX ACL inheritance feature but it is disable by default. Now enable the feature by default. Please be aware any code expecting the old ACL inheritance behavior will have to be updated. Please see the HDFS Permissions Guide for further details.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/MAPREDUCE-6870">MAPREDUCE-6870</a> | <i>Major</i> | <b>Add configuration for MR job to finish when all reducers are complete (even with unfinished mappers)</b></li>
</ul>
<p>Enables mapreduce.job.finish-when-all-reducers-done by default. With this enabled, a MapReduce job will complete as soon as all of its reducers are complete, even if some mappers are still running. This can occur if a mapper was relaunched after node failure but the relaunched task&#x2019;s output is not actually needed. Previously the job would wait for all mappers to complete.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14260">HADOOP-14260</a> | <i>Major</i> | <b>Configuration.dumpConfiguration should redact sensitive information</b></li>
</ul><!-- markdown -->
<p>Configuration.dumpConfiguration no longer prints out the clear text values for the sensitive keys listed in <tt>hadoop.security.sensitive-config-keys</tt>. Callers can override the default list of sensitive keys either to redact more keys or print the clear text values for a few extra keys for debugging purpose.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-12221">HDFS-12221</a> | <i>Major</i> | <b>Replace xerces in XmlEditsVisitor</b></li>
</ul>
<p>New patch with changes in maven dependency. (removed apache xcerces as dependency)</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14726">HADOOP-14726</a> | <i>Minor</i> | <b>Mark FileStatus::isDir as final</b></li>
</ul>
<p>The deprecated FileStatus::isDir method has been marked as final. FileSystems should override FileStatus::isDirectory</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14660">HADOOP-14660</a> | <i>Major</i> | <b>wasb: improve throughput by 34% when account limit exceeded</b></li>
</ul>
<p>Up to 34% throughput improvement for the <a class="externalLink" href="wasb://">wasb://</a> (Azure) file system when fs.azure.selfthrottling.enable is false fs.azure.autothrottling.enable is true.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14769">HADOOP-14769</a> | <i>Major</i> | <b>WASB: delete recursive should not fail if a file is deleted</b></li>
</ul>
<p>Recursive directory delete improvement for the wasb filesystem.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-6959">YARN-6959</a> | <i>Major</i> | <b>RM may allocate wrong AM Container for new attempt</b></li>
</ul>
<p>ResourceManager will now record ResourceRequests from different attempts into different objects.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-12303">HDFS-12303</a> | <i>Blocker</i> | <b>Change default EC cell size to 1MB for better performance</b></li>
</ul>
<p>The cell size of the provided HDFS erasure coding policies has been changed from 64k to 1024k for better performance. The policy names have all been changed accordingly, i.e. RS-6-3.1024k.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-12258">HDFS-12258</a> | <i>Major</i> | <b>ec -listPolicies should list all policies in system, no matter it&#x2019;s enabled or disabled</b></li>
</ul><!-- markdown -->
<p><tt>hdfs ec -listPolicies</tt> now lists enabled, disabled, and removed policies, rather than just enabled policies.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/MAPREDUCE-6892">MAPREDUCE-6892</a> | <i>Major</i> | <b>Issues with the count of failed/killed tasks in the jhist file</b></li>
</ul>
<p>This adds some new job counters, the number of failed MAP/REDUCE tasks and the number of killed MAP/REDUCE tasks.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-5355">YARN-5355</a> | <i>Critical</i> | <b>YARN Timeline Service v.2: alpha 2</b></li>
</ul>
<p>We are releasing the alpha2 version of a major revision of YARN Timeline Service: v.2. YARN Timeline Service v.2 addresses two major challenges: improving scalability and reliability of Timeline Service, and enhancing usability by introducing flows and aggregation.</p>
<p>YARN Timeline Service v.2 alpha1 was introduced in 3.0.0-alpha1 via YARN-2928.</p>
<p>YARN Timeline Service v.2 alpha2 is now being provided so that users and developers can test it and provide feedback and suggestions for making it a ready replacement for Timeline Service v.1.x. Security is provided via Kerberos Authentication and delegation tokens. There is also a simple read level authorization provided via whitelists.</p>
<p>Some of the notable improvements since alpha-1 are: - Security via Kerberos Authentication and delegation tokens - Read side simple authorization via whitelist - Client configurable entity sort ordering - New REST APIs for apps, app attempts, containers, fetching metrics by timerange, pagination, sub-app entities - Support for storing sub-application entities (entities that exist outside the scope of an application) - Configurable TTLs (time-to-live) for tables, configurable table prefixes, configurable hbase cluster - Flow level aggregations done as dynamic (table level) coprocessors - Uses latest stable HBase release 1.2.6</p>
<p>More details are available in the <a href="./hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html">YARN Timeline Service v.2</a> documentation.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14364">HADOOP-14364</a> | <i>Major</i> | <b>refresh changelog/release notes with newer Apache Yetus build</b></li>
</ul>
<p>Additionally, this patch updates maven-site-plugin to 3.6 and doxia-module-markdown to 1.8-SNAPSHOT to work around problems with unknown schemas in URLs in markdown formatted content.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13345">HADOOP-13345</a> | <i>Major</i> | <b>S3Guard: Improved Consistency for S3A</b></li>
</ul>
<p>S3Guard (pronounced see-guard) is a new feature for the S3A connector to Amazon S3, which uses DynamoDB for a high performance and consistent metadata repository. Essentially: S3Guard caches directory information, so your S3A clients get faster lookups and resilience to inconsistency between S3 list operations and the status of objects. When files are created, with S3Guard, they&#x2019;ll always be found.</p>
<p>S3Guard does not address update consistency: if a file is updated, while the directory information will be updated, calling open() on the path may still return the old data. Similarly, deleted objects may also potentially be opened.</p>
<p>Please consult the S3Guard documentation in the Amazon S3 section of our documentation.</p>
<p>Note: part of this update includes moving to a new version of the AWS SDK 1.11, one which includes the Dynamo DB client and its a shaded version of Jackson 2. The large aws-sdk-bundle JAR is needed to use the S3A client with or without S3Guard enabled. The good news: because Jackson is shaded, there will be no conflict between any Jackson version used in your application and that which the AWS SDK needs.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14414">HADOOP-14414</a> | <i>Minor</i> | <b>Calling maven-site-plugin directly for docs profile is unnecessary</b></li>
</ul>
<p>maven-site-plugin is no longer called directly at package phase.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-12300">HDFS-12300</a> | <i>Major</i> | <b>Audit-log delegation token related operations</b></li>
</ul><!-- markdown -->
<p>NameNode now audit-logs <tt>getDelegationToken</tt>, <tt>renewDelegationToken</tt>, <tt>cancelDelegationToken</tt>.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-12218">HDFS-12218</a> | <i>Blocker</i> | <b>Rename split EC / replicated block metrics in BlockManager</b></li>
</ul>
<p>This renames ClientProtocol#getECBlockGroupsStats to ClientProtocol#getEcBlockGroupStats and ClientProtocol#getBlockStats to ClientProtocol#getReplicatedBlockStats. The return-type classes have also been similarly renamed. Their fields have also been renamed to drop the &#x201c;stats&#x201d; suffix.</p>
<p>Additionally, ECBlockGroupStats#pendingDeletionBlockGroups has been renamed to ECBlockGroupStats#pendingDeletionBlocks, to clarify that this is the number of blocks, not block groups, pending deletion. The corresponding NameNode metric has also been renamed to PendingDeletionECBlocks.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13421">HADOOP-13421</a> | <i>Minor</i> | <b>Switch to v2 of the S3 List Objects API in S3A</b></li>
</ul>
<p>S3A now defaults to using the &#x201c;v2&#x201d; S3 list API, which speeds up large-scale path listings. Non-AWS S3 implementations may not support this API: consult the S3A documentation on how to revert to the v1 API.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14847">HADOOP-14847</a> | <i>Blocker</i> | <b>Remove Guava Supplier and change to java Supplier in AMRMClient and AMRMClientAysnc</b></li>
</ul>
<p>AMRMClient#waitFor and AMRMClientAsync#waitFor now take java.util.function.Supplier as an argument, rather than com.google.common.base.Supplier.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14520">HADOOP-14520</a> | <i>Major</i> | <b>WASB: Block compaction for Azure Block Blobs</b></li>
</ul>
<p>Block Compaction for Azure Block Blobs. When the number of blocks in a block blob is above 32000, the process of compaction replaces a sequence of small blocks with with one big block.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-12412">HDFS-12412</a> | <i>Major</i> | <b>Change ErasureCodingWorker.stripedReadPool to cached thread pool</b></li>
</ul>
<p>Changed {{stripedReadPool}} to unbounded cachedThreadPool. User should combine {{dfs.datanode.ec.reconstruction.stripedblock.threads}} and {{dfs.namenode.replication.max-streams}} to tune recovery performance.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-12414">HDFS-12414</a> | <i>Major</i> | <b>Ensure to use CLI command to enable/disable erasure coding policy</b></li>
</ul>
<p>dfs.namenode.ec.policies.enabled was removed in order to ensure there is only one approach to enable/disable erasure coding policies to avoid sync up.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-12438">HDFS-12438</a> | <i>Major</i> | <b>Rename dfs.datanode.ec.reconstruction.stripedblock.threads.size to dfs.datanode.ec.reconstruction.threads</b></li>
</ul><!-- markdown -->
<p>Config key <tt>dfs.datanode.ec.reconstruction.stripedblock.threads.size</tt> has been renamed to <tt>dfs.datanode.ec.reconstruction.threads</tt>.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14738">HADOOP-14738</a> | <i>Blocker</i> | <b>Remove S3N and obsolete bits of S3A; rework docs</b></li>
</ul>
<p>* The <a class="externalLink" href="s3n://">s3n://</a> client has been removed. Please upgrade to the <a class="externalLink" href="s3a://">s3a://</a> client. * The s3a&#x2019;s original output stream has been removed, the &#x201c;fast&#x201d; output stream is the sole option available. There is no need to explicitly enable this, and trying to disable it (fs.s3a.fast.upload=false) will have no effect.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-7859">HDFS-7859</a> | <i>Major</i> | <b>Erasure Coding: Persist erasure coding policies in NameNode</b></li>
</ul>
<p>Persist all built-in erasure coding policies and user defined erasure coding policies into NameNode fsImage and editlog reliably, so that all erasure coding policies remain consistent after NameNode restart.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-12395">HDFS-12395</a> | <i>Major</i> | <b>Support erasure coding policy operations in namenode edit log</b></li>
</ul>
<p>See RN in HDFS-7859 as part of the work.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14670">HADOOP-14670</a> | <i>Major</i> | <b>Increase minimum cmake version for all platforms</b></li>
</ul>
<p>CMake v3.1.0 is now the minimum version required to build Apache Hadoop&#x2019;s native components.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-11799">HDFS-11799</a> | <i>Major</i> | <b>Introduce a config to allow setting up write pipeline with fewer nodes than replication factor</b></li>
</ul>
<p>Added new configuration &#x201c;dfs.client.block.write.replace-datanode-on-failure.min-replication&#x201d;.</p>
<div>
<div>
<pre class="source">The minimum number of replications that are needed to not to fail
the write pipeline if new datanodes can not be found to replace
failed datanodes (could be due to network failure) in the write pipeline.
If the number of the remaining datanodes in the write pipeline is greater
than or equal to this property value, continue writing to the remaining nodes.
Otherwise throw exception.
If this is set to 0, an exception will be thrown, when a replacement
can not be found.
</pre></div></div>
<hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-12447">HDFS-12447</a> | <i>Major</i> | <b>Rename AddECPolicyResponse to AddErasureCodingPolicyResponse</b></li>
</ul>
<p>HdfsAdmin#addErasureCodingPolicies now returns an AddErasureCodingPolicyResponse[] rather than AddECPolicyResponse[]. The corresponding RPC definition and PB message have also been renamed.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-7337">HDFS-7337</a> | <i>Critical</i> | <b>Configurable and pluggable erasure codec and policy</b></li>
</ul>
<p>This allows users to: * develop and plugin their own erasure codec and coders. The plugin will be loaded automatically from hadoop jars, the corresponding codec and coder will be registered for runtime use. * define their own erasure coding policies thru an xml file and CLI command. The added policies will be persisted into fsimage.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-2915">YARN-2915</a> | <i>Major</i> | <b>Enable YARN RM scale out via federation using multiple RM&#x2019;s</b></li>
</ul>
<p>A federation-based approach to transparently scale a single YARN cluster to tens of thousands of nodes, by federating multiple YARN standalone clusters (sub-clusters). The applications running in this federated environment will see a single massive YARN cluster and will be able to schedule tasks on any node of the federated cluster. Under the hood, the federation system will negotiate with sub-clusters ResourceManagers and provide resources to the application. The goal is to allow an individual job to &#x201c;span&#x201d; sub-clusters seamlessly.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-5220">YARN-5220</a> | <i>Major</i> | <b>Scheduling of OPPORTUNISTIC containers through YARN RM</b></li>
</ul>
<p>This extends the centralized YARN RM in to enable the scheduling of OPPORTUNISTIC containers in a centralized fashion. This way, users can use OPPORTUNISTIC containers to improve the cluster&#x2019;s utilization, without needing to enable distributed scheduling.</p>
</div>
</div>
<div class="clear">
<hr/>
</div>
<div id="footer">
<div class="xright">
&#169; 2008-2021
Apache Software Foundation
- <a href="http://maven.apache.org/privacy-policy.html">Privacy Policy</a>.
Apache Maven, Maven, Apache, the Apache feather logo, and the Apache Maven project logos are trademarks of The Apache Software Foundation.
</div>
<div class="clear">
<hr/>
</div>
</div>
</body>
</html>