Google Git
Sign in
apache / hadoop-site / 952d547359cafef280f0a0324055741b2ed55be1 / . / content / docs / r3.3.1 / hadoop-project-dist / hadoop-common / release / 3.0.0-alpha2 / RELEASENOTES.3.0.0-alpha2.html
blob: dd83cb4d2355a08862df3d4faf346ba45d89ab77 [file] [log] [blame]
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<!--
| Generated by Apache Maven Doxia at 2021-06-15
| Rendered using Apache Maven Stylus Skin 1.5
-->
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Apache Hadoop 3.3.1 &#x2013; Apache Hadoop 3.0.0-alpha2 Release Notes</title>
<style type="text/css" media="all">
@import url("../../css/maven-base.css");
@import url("../../css/maven-theme.css");
@import url("../../css/site.css");
</style>
<link rel="stylesheet" href="../../css/print.css" type="text/css" media="print" />
<meta name="Date-Revision-yyyymmdd" content="20210615" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body class="composite">
<div id="banner">
<a href="http://hadoop.apache.org/" id="bannerLeft">
<img src="http://hadoop.apache.org/images/hadoop-logo.jpg" alt="" />
</a>
<a href="http://www.apache.org/" id="bannerRight">
<img src="http://www.apache.org/images/asf_logo_wide.png" alt="" />
</a>
<div class="clear">
<hr/>
</div>
</div>
<div id="breadcrumbs">
<div class="xleft">
<a href="http://www.apache.org/" class="externalLink">Apache</a>
&gt;
<a href="http://hadoop.apache.org/" class="externalLink">Hadoop</a>
&gt;
<a href="../../../index.html">Apache Hadoop Project Dist POM</a>
&gt;
<a href="../../index.html">Apache Hadoop 3.3.1</a>
&gt;
Apache Hadoop 3.0.0-alpha2 Release Notes
</div>
<div class="xright"> <a href="http://wiki.apache.org/hadoop" class="externalLink">Wiki</a>
|
<a href="https://gitbox.apache.org/repos/asf/hadoop.git" class="externalLink">git</a>
|
<a href="http://hadoop.apache.org/" class="externalLink">Apache Hadoop</a>
&nbsp;| Last Published: 2021-06-15
&nbsp;| Version: 3.3.1
</div>
<div class="clear">
<hr/>
</div>
</div>
<div id="leftColumn">
<div id="navcolumn">
<h5>General</h5>
<ul>
<li class="none">
<a href="../../../../index.html">Overview</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/SingleCluster.html">Single Node Setup</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/ClusterSetup.html">Cluster Setup</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/CommandsManual.html">Commands Reference</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/FileSystemShell.html">FileSystem Shell</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/Compatibility.html">Compatibility Specification</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/DownstreamDev.html">Downstream Developer's Guide</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/AdminCompatibilityGuide.html">Admin Compatibility Guide</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/InterfaceClassification.html">Interface Classification</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/filesystem/index.html">FileSystem Specification</a>
</li>
</ul>
<h5>Common</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/CLIMiniCluster.html">CLI Mini Cluster</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/FairCallQueue.html">Fair Call Queue</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/NativeLibraries.html">Native Libraries</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/Superusers.html">Proxy User</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/RackAwareness.html">Rack Awareness</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/SecureMode.html">Secure Mode</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/ServiceLevelAuth.html">Service Level Authorization</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/HttpAuthentication.html">HTTP Authentication</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/CredentialProviderAPI.html">Credential Provider API</a>
</li>
<li class="none">
<a href="../../../../hadoop-kms/index.html">Hadoop KMS</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/Tracing.html">Tracing</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/UnixShellGuide.html">Unix Shell Guide</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/registry/index.html">Registry</a>
</li>
</ul>
<h5>HDFS</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsDesign.html">Architecture</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html">User Guide</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HDFSCommands.html">Commands Reference</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html">NameNode HA With QJM</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html">NameNode HA With NFS</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html">Observer NameNode</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/Federation.html">Federation</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/ViewFs.html">ViewFs</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/ViewFsOverloadScheme.html">ViewFsOverloadScheme</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html">Snapshots</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsEditsViewer.html">Edits Viewer</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html">Image Viewer</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html">Permissions and HDFS</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html">Quotas and HDFS</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/LibHdfs.html">libhdfs (C API)</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/WebHDFS.html">WebHDFS (REST API)</a>
</li>
<li class="none">
<a href="../../../../hadoop-hdfs-httpfs/index.html">HttpFS</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html">Short Circuit Local Reads</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html">Centralized Cache Management</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html">NFS Gateway</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html">Rolling Upgrade</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/ExtendedAttributes.html">Extended Attributes</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html">Transparent Encryption</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html">Multihoming</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html">Storage Policies</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/MemoryStorage.html">Memory Storage Support</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/SLGUserGuide.html">Synthetic Load Generator</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html">Erasure Coding</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HDFSDiskbalancer.html">Disk Balancer</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsUpgradeDomain.html">Upgrade Domain</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsDataNodeAdminGuide.html">DataNode Admin</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html">Router Federation</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsProvidedStorage.html">Provided Storage</a>
</li>
</ul>
<h5>MapReduce</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html">Tutorial</a>
</li>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html">Commands Reference</a>
</li>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html">Compatibility with 1.x</a>
</li>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html">Encrypted Shuffle</a>
</li>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html">Pluggable Shuffle/Sort</a>
</li>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html">Distributed Cache Deploy</a>
</li>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/SharedCacheSupport.html">Support for YARN Shared Cache</a>
</li>
</ul>
<h5>MapReduce REST APIs</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredAppMasterRest.html">MR Application Master</a>
</li>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-hs/HistoryServerRest.html">MR History Server</a>
</li>
</ul>
<h5>YARN</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/YARN.html">Architecture</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/YarnCommands.html">Commands Reference</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html">Capacity Scheduler</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/FairScheduler.html">Fair Scheduler</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html">ResourceManager Restart</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html">ResourceManager HA</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/ResourceModel.html">Resource Model</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/NodeLabel.html">Node Labels</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/NodeAttributes.html">Node Attributes</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html">Web Application Proxy</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html">Timeline Server</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html">Timeline Service V.2</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html">Writing YARN Applications</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html">YARN Application Security</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/NodeManager.html">NodeManager</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/DockerContainers.html">Running Applications in Docker Containers</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/RuncContainers.html">Running Applications in runC Containers</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/NodeManagerCgroups.html">Using CGroups</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/SecureContainer.html">Secure Containers</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/ReservationSystem.html">Reservation System</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html">Graceful Decommission</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html">Opportunistic Containers</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/Federation.html">YARN Federation</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/SharedCache.html">Shared Cache</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/UsingGpus.html">Using GPU</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/UsingFPGA.html">Using FPGA</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html">Placement Constraints</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/YarnUI2.html">YARN UI2</a>
</li>
</ul>
<h5>YARN REST APIs</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html">Introduction</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html">Resource Manager</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/NodeManagerRest.html">Node Manager</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html#Timeline_Server_REST_API_v1">Timeline Server</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html#Timeline_Service_v.2_REST_API">Timeline Service V.2</a>
</li>
</ul>
<h5>YARN Service</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/yarn-service/Overview.html">Overview</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/yarn-service/QuickStart.html">QuickStart</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/yarn-service/Concepts.html">Concepts</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/yarn-service/YarnServiceAPI.html">Yarn Service API</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/yarn-service/ServiceDiscovery.html">Service Discovery</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-site/yarn-service/SystemServices.html">System Services</a>
</li>
</ul>
<h5>Hadoop Compatible File Systems</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-aliyun/tools/hadoop-aliyun/index.html">Aliyun OSS</a>
</li>
<li class="none">
<a href="../../../../hadoop-aws/tools/hadoop-aws/index.html">Amazon S3</a>
</li>
<li class="none">
<a href="../../../../hadoop-azure/index.html">Azure Blob Storage</a>
</li>
<li class="none">
<a href="../../../../hadoop-azure-datalake/index.html">Azure Data Lake Storage</a>
</li>
<li class="none">
<a href="../../../../hadoop-openstack/index.html">OpenStack Swift</a>
</li>
<li class="none">
<a href="../../../../hadoop-cos/cloud-storage/index.html">Tencent COS</a>
</li>
</ul>
<h5>Auth</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-auth/index.html">Overview</a>
</li>
<li class="none">
<a href="../../../../hadoop-auth/Examples.html">Examples</a>
</li>
<li class="none">
<a href="../../../../hadoop-auth/Configuration.html">Configuration</a>
</li>
<li class="none">
<a href="../../../../hadoop-auth/BuildingIt.html">Building</a>
</li>
</ul>
<h5>Tools</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-streaming/HadoopStreaming.html">Hadoop Streaming</a>
</li>
<li class="none">
<a href="../../../../hadoop-archives/HadoopArchives.html">Hadoop Archives</a>
</li>
<li class="none">
<a href="../../../../hadoop-archive-logs/HadoopArchiveLogs.html">Hadoop Archive Logs</a>
</li>
<li class="none">
<a href="../../../../hadoop-distcp/DistCp.html">DistCp</a>
</li>
<li class="none">
<a href="../../../../hadoop-gridmix/GridMix.html">GridMix</a>
</li>
<li class="none">
<a href="../../../../hadoop-rumen/Rumen.html">Rumen</a>
</li>
<li class="none">
<a href="../../../../hadoop-resourceestimator/ResourceEstimator.html">Resource Estimator Service</a>
</li>
<li class="none">
<a href="../../../../hadoop-sls/SchedulerLoadSimulator.html">Scheduler Load Simulator</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/Benchmarking.html">Hadoop Benchmarking</a>
</li>
<li class="none">
<a href="../../../../hadoop-dynamometer/Dynamometer.html">Dynamometer</a>
</li>
</ul>
<h5>Reference</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/release/">Changelog and Release Notes</a>
</li>
<li class="none">
<a href="../../../../api/index.html">Java API docs</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/UnixShellAPI.html">Unix Shell API</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/Metrics.html">Metrics</a>
</li>
</ul>
<h5>Configuration</h5>
<ul>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/core-default.xml">core-default.xml</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs/hdfs-default.xml">hdfs-default.xml</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-hdfs-rbf/hdfs-rbf-default.xml">hdfs-rbf-default.xml</a>
</li>
<li class="none">
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml">mapred-default.xml</a>
</li>
<li class="none">
<a href="../../../../hadoop-yarn/hadoop-yarn-common/yarn-default.xml">yarn-default.xml</a>
</li>
<li class="none">
<a href="../../../../hadoop-kms/kms-default.html">kms-default.xml</a>
</li>
<li class="none">
<a href="../../../../hadoop-hdfs-httpfs/httpfs-default.html">httpfs-default.xml</a>
</li>
<li class="none">
<a href="../../../../hadoop-project-dist/hadoop-common/DeprecatedProperties.html">Deprecated Properties</a>
</li>
</ul>
<a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
<img alt="Built by Maven" src="../../images/logos/maven-feather.png"/>
</a>
</div>
</div>
<div id="bodyColumn">
<div id="contentBox">
<!---
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-->
<h1>Apache Hadoop 3.0.0-alpha2 Release Notes</h1>
<p>These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-12756">HADOOP-12756</a> | <i>Major</i> | <b>Incorporate Aliyun OSS file system implementation</b></li>
</ul>
<p>Aliyun OSS is widely used among China&#x2019;s cloud users and this work implemented a new Hadoop compatible filesystem AliyunOSSFileSystem with oss scheme, similar to the s3a and azure support.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-10760">HDFS-10760</a> | <i>Major</i> | <b>DataXceiver#run() should not log InvalidToken exception as an error</b></li>
</ul>
<p>Log InvalidTokenException at trace level in DataXceiver#run().</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13361">HADOOP-13361</a> | <i>Major</i> | <b>Modify hadoop_verify_user to be consistent with hadoop_subcommand_opts (ie more granularity)</b></li>
</ul>
<p>Users:</p>
<p>In Apache Hadoop 3.0.0-alpha1, verification required environment variables with the format of HADOOP_(subcommand)_USER where subcommand was lowercase applied globally. This changes the format to be (command)_(subcommand)_USER where all are uppercase to be consistent with the _OPTS functionality as well as being able to set per-command options. Additionally, the check is now happening sooner, which should make it faster to fail.</p>
<p>Developers:</p>
<p>This changes hadoop_verify_user to require the program&#x2019;s name as part of the function call. This is incompatible with Apache Hadoop 3.0.0-alpha1.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-5549">YARN-5549</a> | <i>Critical</i> | <b>AMLauncher#createAMContainerLaunchContext() should not log the command to be launched indiscriminately</b></li>
</ul>
<p>Introduces a new configuration property, yarn.resourcemanager.amlauncher.log.command. If this property is set to true, then the AM command being launched will be masked in the RM log.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-6962">HDFS-6962</a> | <i>Critical</i> | <b>ACL inheritance conflicts with umaskmode</b></li>
</ul><!-- markdown -->
<p>The original implementation of HDFS ACLs applied the client&#x2019;s umask to the permissions when inheriting a default ACL defined on a parent directory. This behavior is a deviation from the POSIX ACL specification, which states that the umask has no influence when a default ACL propagates from parent to child. HDFS now offers the capability to ignore the umask in this case for improved compliance with POSIX. This change is considered backward-incompatible, so the new behavior is off by default and must be explicitly configured by setting dfs.namenode.posix.acl.inheritance.enabled to true in hdfs-site.xml. Please see the HDFS Permissions Guide for further details.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13341">HADOOP-13341</a> | <i>Major</i> | <b>Deprecate HADOOP_SERVERNAME_OPTS; replace with (command)_(subcommand)_OPTS</b></li>
</ul><!-- markdown -->
<p>Users: * Ability to set per-command+sub-command options from the command line. * Makes daemon environment variable options consistent across the project. (See deprecation list below) * HADOOP_CLIENT_OPTS is now honored for every non-daemon sub-command. Prior to this change, many sub-commands did not use it.</p>
<p>Developers: * No longer need to do custom handling for options in the case section of the shell scripts. * Consolidates all _OPTS handling into hadoop-functions.sh to enable future projects. * All daemons running with secure mode features now get _SECURE_EXTRA_OPTS support.</p>
<p>_OPTS Changes:</p>
<table border="0" class="bodyTable">
<thead>
<tr class="a">
<th align="left"> Old </th>
<th align="left"> New </th></tr>
</thead><tbody>
<tr class="b">
<td align="left"> HADOOP_BALANCER_OPTS </td>
<td align="left"> HDFS_BALANCER_OPTS </td></tr>
<tr class="a">
<td align="left"> HADOOP_DATANODE_OPTS </td>
<td align="left"> HDFS_DATANODE_OPTS </td></tr>
<tr class="b">
<td align="left"> HADOOP_DN_SECURE_EXTRA_OPTS </td>
<td align="left"> HDFS_DATANODE_SECURE_EXTRA_OPTS </td></tr>
<tr class="a">
<td align="left"> HADOOP_JOB_HISTORYSERVER_OPTS </td>
<td align="left"> MAPRED_HISTORYSERVER_OPTS </td></tr>
<tr class="b">
<td align="left"> HADOOP_JOURNALNODE_OPTS </td>
<td align="left"> HDFS_JOURNALNODE_OPTS </td></tr>
<tr class="a">
<td align="left"> HADOOP_MOVER_OPTS </td>
<td align="left"> HDFS_MOVER_OPTS </td></tr>
<tr class="b">
<td align="left"> HADOOP_NAMENODE_OPTS </td>
<td align="left"> HDFS_NAMENODE_OPTS </td></tr>
<tr class="a">
<td align="left"> HADOOP_NFS3_OPTS </td>
<td align="left"> HDFS_NFS3_OPTS </td></tr>
<tr class="b">
<td align="left"> HADOOP_NFS3_SECURE_EXTRA_OPTS </td>
<td align="left"> HDFS_NFS3_SECURE_EXTRA_OPTS </td></tr>
<tr class="a">
<td align="left"> HADOOP_PORTMAP_OPTS </td>
<td align="left"> HDFS_PORTMAP_OPTS </td></tr>
<tr class="b">
<td align="left"> HADOOP_SECONDARYNAMENODE_OPTS </td>
<td align="left"> HDFS_SECONDARYNAMENODE_OPTS </td></tr>
<tr class="a">
<td align="left"> HADOOP_ZKFC_OPTS </td>
<td align="left"> HDFS_ZKFC_OPTS </td></tr>
</tbody>
</table><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13588">HADOOP-13588</a> | <i>Major</i> | <b>ConfServlet should respect Accept request header</b></li>
</ul>
<p>Conf HTTP service should set response&#x2019;s content type according to the Accept header in the request.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-10636">HDFS-10636</a> | <i>Major</i> | <b>Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.</b></li>
</ul>
<p><b>WARNING: No release note provided for this change.</b></p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13218">HADOOP-13218</a> | <i>Major</i> | <b>Migrate other Hadoop side tests to prepare for removing WritableRPCEngine</b></li>
</ul>
<p><b>WARNING: No release note provided for this change.</b></p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-10489">HDFS-10489</a> | <i>Minor</i> | <b>Deprecate dfs.encryption.key.provider.uri for HDFS encryption zones</b></li>
</ul>
<p>The configuration dfs.encryption.key.provider.uri is deprecated. To configure key provider in HDFS, please use hadoop.security.key.provider.path.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-10877">HDFS-10877</a> | <i>Major</i> | <b>Make RemoteEditLogManifest.committedTxnId optional in Protocol Buffers</b></li>
</ul>
<p>A new protobuf field added to RemoteEditLogManifest was mistakenly marked as required. This changes the field to optional, preserving compatibility with 2.x releases but breaking compatibility with 3.0.0-alpha1.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-10914">HDFS-10914</a> | <i>Critical</i> | <b>Move remnants of oah.hdfs.client to hadoop-hdfs-client</b></li>
</ul>
<p>The remaining classes in the org.apache.hadoop.hdfs.client package have been moved from hadoop-hdfs to hadoop-hdfs-client.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13681">HADOOP-13681</a> | <i>Major</i> | <b>Reduce Kafka dependencies in hadoop-kafka module</b></li>
</ul>
<p>Changed Apache Kafka dependency from kafka-2.10 to kafka-clients in hadoop-kafka module.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-12667">HADOOP-12667</a> | <i>Major</i> | <b>s3a: Support createNonRecursive API</b></li>
</ul>
<p>S3A now provides a working implementation of the FileSystem#createNonRecursive method.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-10609">HDFS-10609</a> | <i>Major</i> | <b>Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications</b></li>
</ul>
<p>If pipeline recovery fails due to expired encryption key, attempt to refresh the key and retry.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13678">HADOOP-13678</a> | <i>Major</i> | <b>Update jackson from 1.9.13 to 2.x in hadoop-tools</b></li>
</ul>
<p>Jackson 1.9.13 dependency was removed from hadoop-tools module.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/MAPREDUCE-6776">MAPREDUCE-6776</a> | <i>Major</i> | <b>yarn.app.mapreduce.client.job.max-retries should have a more useful default</b></li>
</ul>
<p>The default value of yarn.app.mapreduce.client.job.max-retries has been changed from 0 to 3. This will help protect clients from failures that are transient. True failures may take slightly longer now due to the retries.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-10797">HDFS-10797</a> | <i>Major</i> | <b>Disk usage summary of snapshots causes renamed blocks to get counted twice</b></li>
</ul>
<p>Disk usage summaries previously incorrectly counted files twice if they had been renamed (including files moved to Trash) since being snapshotted. Summaries now include current data plus snapshotted data that is no longer under the directory either due to deletion or being moved outside of the directory.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13699">HADOOP-13699</a> | <i>Critical</i> | <b>Configuration does not substitute multiple references to the same var</b></li>
</ul>
<p>This changes the config var cycle detection introduced in 3.0.0-alpha1 by HADOOP-6871 such that it detects single-variable but not multi-variable loops. This also fixes resolution of multiple specifications of the same variable in a config value.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-10637">HDFS-10637</a> | <i>Major</i> | <b>Modifications to remove the assumption that FsVolumes are backed by java.io.File.</b></li>
</ul>
<p><b>WARNING: No release note provided for this change.</b></p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-10916">HDFS-10916</a> | <i>Major</i> | <b>Switch from &#x201c;raw&#x201d; to &#x201c;system&#x201d; xattr namespace for erasure coding policy</b></li>
</ul>
<p>EC policy is now stored in the &#x201c;system&#x201d; extended attribute namespace rather than &#x201c;raw&#x201d;. This means the EC policy extended attribute is no longer directly accessible by users or preserved across a distcp that preserves raw extended attributes.</p>
<p>Users can instead use HdfsAdmin#setErasureCodingPolicy and HdfsAdmin#getErasureCodingPolicy to set and get the EC policy for a path.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-4464">YARN-4464</a> | <i>Blocker</i> | <b>Lower the default max applications stored in the RM and store</b></li>
</ul>
<p>The maximum applications the RM stores in memory and in the state-store by default has been lowered from 10,000 to 1,000. This should ease the pressure on the state-store. However, installations relying on the default to be 10,000 are affected.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-10883">HDFS-10883</a> | <i>Major</i> | <b>`getTrashRoot`&#x2019;s behavior is not consistent in DFS after enabling EZ.</b></li>
</ul>
<p>If root path / is an encryption zone, the old DistributedFileSystem#getTrashRoot(new Path(&#x201c;/&#x201d;)) returns /user/$USER/.Trash which is a wrong behavior. The correct value should be /.Trash/$USER</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13721">HADOOP-13721</a> | <i>Minor</i> | <b>Remove stale method ViewFileSystem#getTrashCanLocation</b></li>
</ul>
<p>The unused method getTrashCanLocation has been removed. This method has long been superceded by FileSystem#getTrashRoot.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13661">HADOOP-13661</a> | <i>Major</i> | <b>Upgrade HTrace version</b></li>
</ul>
<p>Bump HTrace version from 4.0.1-incubating to 4.1.0-incubating.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-10957">HDFS-10957</a> | <i>Major</i> | <b>Retire BKJM from trunk</b></li>
</ul>
<p>The BookkeeperJournalManager implementation has been removed. Users are encouraged to use QuorumJournalManager instead.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13522">HADOOP-13522</a> | <i>Major</i> | <b>Add %A and %a formats for fs -stat command to print permissions</b></li>
</ul>
<p>Added permissions to the fs stat command. They are now available as symbolic (%A) and octal (%a) formats, which are in line with Linux.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-5718">YARN-5718</a> | <i>Major</i> | <b>TimelineClient (and other places in YARN) shouldn&#x2019;t over-write HDFS client retry settings which could cause unexpected behavior</b></li>
</ul>
<p><b>WARNING: No release note provided for this change.</b></p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13560">HADOOP-13560</a> | <i>Major</i> | <b>S3ABlockOutputStream to support huge (many GB) file writes</b></li>
</ul>
<p>This mechanism replaces the (experimental) fast output stream of Hadoop 2.7.x, combining better scalability options with instrumentation. Consult the S3A documentation to see the extra configuration operations.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/MAPREDUCE-6791">MAPREDUCE-6791</a> | <i>Minor</i> | <b>remove unnecessary dependency from hadoop-mapreduce-client-jobclient to hadoop-mapreduce-client-shuffle</b></li>
</ul>
<p>An unnecessary dependency on hadoop-mapreduce-client-shuffle in hadoop-mapreduce-client-jobclient has been removed.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-7352">HADOOP-7352</a> | <i>Major</i> | <b>FileSystem#listStatus should throw IOE upon access error</b></li>
</ul>
<p>Change FileSystem#listStatus contract to never return null. Local filesystems prior to 3.0.0 returned null upon access error. It is considered erroneous. We should expect FileSystem#listStatus to throw IOException upon access error.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13693">HADOOP-13693</a> | <i>Minor</i> | <b>Remove the message about HTTP OPTIONS in SPNEGO initialization message from kms audit log</b></li>
</ul>
<p>kms-audit.log used to show an UNAUTHENTICATED message even for successful operations, because of the OPTIONS HTTP request during SPNEGO initial handshake. This message brings more confusion than help, and has hence been removed.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-11018">HDFS-11018</a> | <i>Major</i> | <b>Incorrect check and message in FsDatasetImpl#invalidate</b></li>
</ul>
<p>Improves the error message when datanode removes a replica which is not found.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-10976">HDFS-10976</a> | <i>Major</i> | <b>Report erasure coding policy of EC files in Fsck</b></li>
</ul>
<p>Fsck now reports whether a file is replicated and erasure-coded. If it is replicated, fsck reports replication factor of the file. If it is erasure coded, fsck reports the erasure coding policy of the file.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-10975">HDFS-10975</a> | <i>Major</i> | <b>fsck -list-corruptfileblocks does not report corrupt EC files</b></li>
</ul>
<p>Fixed a bug that made fsck -list-corruptfileblocks counts corrupt erasure coded files incorrectly.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-5388">YARN-5388</a> | <i>Critical</i> | <b>Deprecate and remove DockerContainerExecutor</b></li>
</ul>
<p>DockerContainerExecutor is deprecated starting 2.9.0 and removed from 3.0.0. Please use LinuxContainerExecutor with the DockerRuntime to run Docker containers on YARN clusters.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-11798">HADOOP-11798</a> | <i>Major</i> | <b>Native raw erasure coder in XOR codes</b></li>
</ul>
<p>This provides a native implementation of XOR codec by leveraging Intel ISA-L library function to achieve a better performance.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13659">HADOOP-13659</a> | <i>Major</i> | <b>Upgrade jaxb-api version</b></li>
</ul>
<p>Bump the version of third party dependency jaxb-api to 2.2.11.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-3732">YARN-3732</a> | <i>Minor</i> | <b>Change NodeHeartbeatResponse.java and RegisterNodeManagerResponse.java as abstract classes</b></li>
</ul>
<p>Interface classes has been changed to Abstract class to maintain consistency across all other protos.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-5767">YARN-5767</a> | <i>Major</i> | <b>Fix the order that resources are cleaned up from the local Public/Private caches</b></li>
</ul>
<p>This issue fixes a bug in how resources are evicted from the PUBLIC and PRIVATE yarn local caches used by the node manager for resource localization. In summary, the caches are now properly cleaned based on an LRU policy across both the public and private caches.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-11048">HDFS-11048</a> | <i>Major</i> | <b>Audit Log should escape control characters</b></li>
</ul>
<p>HDFS audit logs are formatted as individual lines, each of which has a few of key-value pair fields. Some of the values come from client request (e.g. src, dst). Before this patch the control characters including \t \n etc are not escaped in audit logs. That may break lines unexpectedly or introduce additional table character (in the worst case, both) within a field. Tools that parse audit logs had to deal with this case carefully. After this patch, the control characters in the src/dst fields are escaped.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-8500">HADOOP-8500</a> | <i>Minor</i> | <b>Fix javadoc jars to not contain entire target directory</b></li>
</ul>
<p>Hadoop&#x2019;s javadoc jars should be significantly smaller, and contain only javadoc.</p>
<p>As a related cleanup, the dummy hadoop-dist-* jars are no longer generated as part of the build.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13792">HADOOP-13792</a> | <i>Major</i> | <b>Stackoverflow for schemeless defaultFS with trailing slash</b></li>
</ul>
<p>FileSystem#getDefaultUri will throw IllegalArgumentException if default FS has no scheme and can not be fixed.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-10756">HDFS-10756</a> | <i>Major</i> | <b>Expose getTrashRoot to HTTPFS and WebHDFS</b></li>
</ul>
<p>&#x201c;getTrashRoot&#x201d; returns a trash root for a path. Currently in DFS if the path &#x201c;/foo&#x201d; is a normal path, it returns &#x201c;/user/$USER/.Trash&#x201d; for &#x201c;/foo&#x201d; and if &#x201c;/foo&#x201d; is an encrypted zone, it returns &#x201c;/foo/.Trash/$USER&#x201d; for the child file/dir of &#x201c;/foo&#x201d;. This patch is about to override the old &#x201c;getTrashRoot&#x201d; of httpfs and webhdfs, so that the behavior of returning trash root in httpfs and webhdfs are consistent with DFS.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-10970">HDFS-10970</a> | <i>Major</i> | <b>Update jackson from 1.9.13 to 2.x in hadoop-hdfs</b></li>
</ul>
<p>Removed jackson 1.9.13 dependency from hadoop-hdfs-project module.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-5847">YARN-5847</a> | <i>Major</i> | <b>Revert health check exit code check</b></li>
</ul>
<p>This change reverts YARN-5567 from 3.0.0-alpha1. The exit codes of the health check script are once again ignored.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-9337">HDFS-9337</a> | <i>Major</i> | <b>Validate required params for WebHDFS requests</b></li>
</ul>
<p>Strict validations will be done for mandatory parameters for WebHDFS REST requests.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-11116">HDFS-11116</a> | <i>Minor</i> | <b>Fix javac warnings caused by deprecation of APIs in TestViewFsDefaultValue</b></li>
</ul>
<p>ViewFileSystem#getServerDefaults(Path) throws NotInMountException instead of FileNotFoundException for unmounted path.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-12718">HADOOP-12718</a> | <i>Major</i> | <b>Incorrect error message by fs -put local dir without permission</b></li>
</ul><!-- markdown -->
<p>The <tt>hadoop fs -ls</tt> command now prints &#x201c;Permission denied&#x201d; rather than &#x201c;No such file or directory&#x201d; when the user doesn&#x2019;t have permission to traverse the path.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-11056">HDFS-11056</a> | <i>Major</i> | <b>Concurrent append and read operations lead to checksum error</b></li>
</ul>
<p>Load last partial chunk checksum properly into memory when converting a finalized/temporary replica to rbw replica. This ensures concurrent reader reads the correct checksum that matches the data before the update.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-5765">YARN-5765</a> | <i>Blocker</i> | <b>Revert CHMOD on the new dirs created-LinuxContainerExecutor creates appcache and its subdirectories with wrong group owner.</b></li>
</ul>
<p>This change reverts YARN-5287 from 3.0.0-alpha1. chmod clears the set-group-ID bit of a regular file hence folder was getting reset with the rights.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13660">HADOOP-13660</a> | <i>Major</i> | <b>Upgrade commons-configuration version to 2.1</b></li>
</ul>
<p>Bump commons-configuration version from 1.6 to 2.1</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-12705">HADOOP-12705</a> | <i>Major</i> | <b>Upgrade Jackson 2.2.3 to 2.7.8</b></li>
</ul>
<p>We are sorry for causing pain for everyone for whom this Jackson update causes problems, but it was proving impossible to stay on the older version: too much code had moved past it, and by staying back we were limiting what Hadoop could do, and giving everyone who wanted an up to date version of Jackson a different set of problems. We&#x2019;ve selected Jackson 2.7.8 as it fixed fix a security issue in XML parsing, yet proved compatible at the API level with the Hadoop codebase &#x2013;and hopefully everything downstream.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-5713">YARN-5713</a> | <i>Major</i> | <b>Update jackson from 1.9.13 to 2.x in hadoop-yarn</b></li>
</ul>
<p>Jackson 1.9.13 dependency was removed from hadoop-yarn-project.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13050">HADOOP-13050</a> | <i>Blocker</i> | <b>Upgrade to AWS SDK 1.11.45</b></li>
</ul>
<p>The dependency on the AWS SDK has been bumped to 1.11.45.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-1381">HADOOP-1381</a> | <i>Major</i> | <b>The distance between sync blocks in SequenceFiles should be configurable</b></li>
</ul>
<p>The default sync interval within new SequenceFile writes is now 100KB, up from the older default of 2000B. The sync interval is now also manually configurable via the SequenceFile.Writer API.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-10994">HDFS-10994</a> | <i>Major</i> | <b>Support an XOR policy XOR-2-1-64k in HDFS</b></li>
</ul>
<p>This introduced a new erasure coding policy named XOR-2-1-64k using the simple XOR codec, and it can be used to evaluate HDFS erasure coding feature in a small cluster (only 2 + 1 datanodes needed). The policy isn&#x2019;t recommended to be used in a production cluster.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/MAPREDUCE-6743">MAPREDUCE-6743</a> | <i>Major</i> | <b>nativetask unit tests need to provide usable output; fix link errors during mvn test</b></li>
</ul>
<p>As part of this patch, the Google test framework code was updated to v1.8.0</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13706">HADOOP-13706</a> | <i>Major</i> | <b>Update jackson from 1.9.13 to 2.x in hadoop-common-project</b></li>
</ul>
<p>Removed Jackson 1.9.13 dependency from hadoop-common module.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13812">HADOOP-13812</a> | <i>Blocker</i> | <b>Upgrade Tomcat to 6.0.48</b></li>
</ul>
<p>Tomcat 6.0.46 starts to filter weak ciphers. Some old SSL clients may be affected. It is recommended to upgrade the SSL client. Run the SSL client against <a class="externalLink" href="https://www.howsmyssl.com/a/check">https://www.howsmyssl.com/a/check</a> to find out its TLS version and cipher suites.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-5517">HDFS-5517</a> | <i>Major</i> | <b>Lower the default maximum number of blocks per file</b></li>
</ul>
<p>The default value of &#x201c;dfs.namenode.fs-limits.max-blocks-per-file&#x201d; has been reduced from 1M to 10K.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13827">HADOOP-13827</a> | <i>Major</i> | <b>Add reencryptEncryptedKey interface to KMS</b></li>
</ul>
<p>A reencryptEncryptedKey interface is added to the KMS, to re-encrypt an encrypted key with the latest version of encryption key.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13842">HADOOP-13842</a> | <i>Minor</i> | <b>Update jackson from 1.9.13 to 2.x in hadoop-maven-plugins</b></li>
</ul>
<p>Jackson 1.9.13 dependency was removed from hadoop-maven-plugins module.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/MAPREDUCE-4683">MAPREDUCE-4683</a> | <i>Critical</i> | <b>Create and distribute hadoop-mapreduce-client-core-tests.jar</b></li>
</ul>
<p>hadoop-mapreduce-client-core module now creates and distributes test jar.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-11217">HDFS-11217</a> | <i>Major</i> | <b>Annotate NameNode and DataNode MXBean interfaces as Private/Stable</b></li>
</ul>
<p>The DataNode and NameNode MXBean interfaces have been marked as Private and Stable to indicate that although users should not be implementing these interfaces directly, the information exposed by these interfaces is part of the HDFS public API.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-11229">HDFS-11229</a> | <i>Blocker</i> | <b>HDFS-11056 failed to close meta file</b></li>
</ul>
<p>The fix for HDFS-11056 reads meta file to load last partial chunk checksum when a block is converted from finalized/temporary to rbw. However, it did not close the file explicitly, which may cause number of open files reaching system limit. This jira fixes it by closing the file explicitly after the meta file is read.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-11804">HADOOP-11804</a> | <i>Major</i> | <b>Shaded Hadoop client artifacts and minicluster</b></li>
</ul><!-- markdown -->
<p>The <tt>hadoop-client</tt> Maven artifact available in 2.x releases pulls Hadoop&#x2019;s transitive dependencies onto a Hadoop application&#x2019;s classpath. This can be problematic if the versions of these transitive dependencies conflict with the versions used by the application.</p>
<p><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-11804">HADOOP-11804</a> adds new <tt>hadoop-client-api</tt> and <tt>hadoop-client-runtime</tt> artifacts that shade Hadoop&#x2019;s dependencies into a single jar. This avoids leaking Hadoop&#x2019;s dependencies onto the application&#x2019;s classpath.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-11160">HDFS-11160</a> | <i>Major</i> | <b>VolumeScanner reports write-in-progress replicas as corrupt incorrectly</b></li>
</ul>
<p>Fixed a race condition that caused VolumeScanner to recognize a good replica as a bad one if the replica is also being written concurrently.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13597">HADOOP-13597</a> | <i>Major</i> | <b>Switch KMS from Tomcat to Jetty</b></li>
</ul><!-- markdown -->
<p>The following environment variables are deprecated. Set the corresponding configuration properties instead.</p>
<table border="0" class="bodyTable">
<thead>
<tr class="a">
<th>Environment Variable </th>
<th> Configuration Property </th>
<th> Configuration File</th></tr>
</thead><tbody>
<tr class="b">
<td>KMS_HTTP_PORT </td>
<td> hadoop.kms.http.port </td>
<td> kms-site.xml</td></tr>
<tr class="a">
<td>KMS_MAX_HTTP_HEADER_SIZE </td>
<td> hadoop.http.max.request.header.size and hadoop.http.max.response.header.size </td>
<td> kms-site.xml</td></tr>
<tr class="b">
<td>KMS_MAX_THREADS </td>
<td> hadoop.http.max.threads </td>
<td> kms-site.xml</td></tr>
<tr class="a">
<td>KMS_SSL_ENABLED </td>
<td> hadoop.kms.ssl.enabled </td>
<td> kms-site.xml</td></tr>
<tr class="b">
<td>KMS_SSL_KEYSTORE_FILE </td>
<td> ssl.server.keystore.location </td>
<td> ssl-server.xml</td></tr>
<tr class="a">
<td>KMS_SSL_KEYSTORE_PASS </td>
<td> ssl.server.keystore.password </td>
<td> ssl-server.xml</td></tr>
<tr class="b">
<td>KMS_TEMP </td>
<td> hadoop.http.temp.dir </td>
<td> kms-site.xml</td></tr>
</tbody>
</table>
<p>These default HTTP Services have been added.</p>
<table border="0" class="bodyTable">
<thead>
<tr class="a">
<th>Name </th>
<th> Description</th></tr>
</thead><tbody>
<tr class="b">
<td>/conf </td>
<td> Display configuration properties</td></tr>
<tr class="a">
<td>/jmx </td>
<td> Java JMX management interface</td></tr>
<tr class="b">
<td>/logLevel </td>
<td> Get or set log level per class</td></tr>
<tr class="a">
<td>/logs </td>
<td> Display log files</td></tr>
<tr class="b">
<td>/stacks </td>
<td> Display JVM stacks</td></tr>
<tr class="a">
<td>/static/index.html </td>
<td> The static home page</td></tr>
</tbody>
</table>
<p>The JMX path has been changed from /kms/jmx to /jmx.</p>
<p>Script kms.sh has been deprecated, use <tt>hadoop kms</tt> instead. The new scripts are based on the Hadoop shell scripting framework. <tt>hadoop daemonlog</tt> is supported. SSL configurations are read from ssl-server.xml.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13953">HADOOP-13953</a> | <i>Major</i> | <b>Make FTPFileSystem&#x2019;s data connection mode and transfer mode configurable</b></li>
</ul>
<p>Added two configuration key fs.ftp.data.connection.mode and fs.ftp.transfer.mode, and configure FTP data connection mode and transfer mode accordingly.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-6071">YARN-6071</a> | <i>Blocker</i> | <b>Fix incompatible API change on AM-RM protocol due to YARN-3866 (trunk only)</b></li>
</ul>
<p><b>WARNING: No release note provided for this change.</b></p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13673">HADOOP-13673</a> | <i>Major</i> | <b>Update scripts to be smarter when running with privilege</b></li>
</ul>
<p>Apache Hadoop is now able to switch to the appropriate user prior to launching commands so long as the command is being run with a privileged user and the appropriate set of _USER variables are defined. This re-enables sbin/start-all.sh and sbin/stop-all.sh as well as fixes the sbin/start-dfs.sh and sbin/stop-dfs.sh to work with both secure and unsecure systems.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13964">HADOOP-13964</a> | <i>Major</i> | <b>Remove vestigal templates directories creation</b></li>
</ul>
<p>This patch removes share/hadoop/{hadoop,hdfs,mapred,yarn}/templates directories and contents.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-5271">YARN-5271</a> | <i>Major</i> | <b>ATS client doesn&#x2019;t work with Jersey 2 on the classpath</b></li>
</ul>
<p>A workaround to avoid dependency conflict with Spark2, before a full classpath isolation solution is implemented. Skip instantiating a Timeline Service client if encountering NoClassDefFoundError.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13037">HADOOP-13037</a> | <i>Major</i> | <b>Refactor Azure Data Lake Store as an independent FileSystem</b></li>
</ul>
<p>Hadoop now supports integration with Azure Data Lake as an alternative Hadoop-compatible file system. Please refer to the Hadoop site documentation of Azure Data Lake for details on usage and configuration.</p><hr />
<ul>
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-2877">YARN-2877</a> | <i>Major</i> | <b>Extend YARN to support distributed scheduling</b></li>
</ul>
<p>With this JIRA we are introducing distributed scheduling in YARN. In particular, we make the following contributions: - Introduce the notion of container types. GUARANTEED containers follow the semantics of the existing YARN containers. OPPORTUNISTIC ones can be seen as lower priority containers, and can be preempted in order to make space for GUARANTEED containers to run. - Queuing of tasks at the NMs. This enables us to send more containers in an NM than its available resources. At the moment we are allowing queuing of OPPORTUNISTIC containers. Once resources become available at the NM, such containers can immediately start their execution. - Introduce the AMRMProxy. This is a service running at each node, intercepting the requests between the AM and the RM. It is instrumental for both distributed scheduling and YARN Federation (YARN-2915). - Enable distributed scheduling. To minimize their allocation latency, OPPORTUNISTIC containers are dispatched immediately to NMs in a distributed fashion by using the AMRMProxy of the node where the corresponding AM resides, without needing to go through the ResourceManager.</p>
<p>All the functionality introduced in this JIRA is disabled by default, so it will not affect the behavior of existing applications. We have introduced parameters in YarnConfiguration to enable NM queuing (yarn.nodemanager.container-queuing-enabled), distributed scheduling (yarn.distributed-scheduling.enabled) and the AMRMProxy service (yarn.nodemanager.amrmproxy.enable). AMs currently need to specify the type of container to be requested for each task. We are in the process of adding in the MapReduce AM the ability to randomly request OPPORTUNISTIC containers for a specified percentage of a job&#x2019;s tasks, so that users can experiment with the new features.</p>
</div>
</div>
<div class="clear">
<hr/>
</div>
<div id="footer">
<div class="xright">
&#169; 2008-2021
Apache Software Foundation
- <a href="http://maven.apache.org/privacy-policy.html">Privacy Policy</a>.
Apache Maven, Maven, Apache, the Apache feather logo, and the Apache Maven project logos are trademarks of The Apache Software Foundation.
</div>
<div class="clear">
<hr/>
</div>
</div>
</body>
</html>