blob: 29da38394e022789d09370c9b10703d3e3971070 [file] [log] [blame]
<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Hadoop 2.6.1 Release Notes</title>
<STYLE type="text/css">
H1 {font-family: sans-serif}
H2 {font-family: sans-serif; margin-left: 7mm}
TABLE {margin-left: 7mm}
</STYLE>
</head>
<body>
<h1>Hadoop 2.6.1 Release Notes</h1>
These release notes include new developer and user-facing incompatibilities, features, and major improvements.
<a name="changes"/>
<h2>Changes since Hadoop 2.6.0</h2>
<ul>
<li> <a href="https://issues.apache.org/jira/browse/YARN-4047">YARN-4047</a>.
Major bug reported by Jason Lowe and fixed by Jason Lowe (resourcemanager)<br>
<b>ClientRMService getApplications has high scheduler lock contention</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3999">YARN-3999</a>.
Major bug reported by Jian He and fixed by Jian He <br>
<b>RM hangs on draining events</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3990">YARN-3990</a>.
Critical bug reported by Rohith Sharma K S and fixed by Bibin A Chundatt (resourcemanager)<br>
<b>AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3978">YARN-3978</a>.
Major improvement reported by Eric Payne and fixed by Eric Payne (timelineserver , yarn)<br>
<b>Configurably turn off the saving of container info in Generic AHS</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3850">YARN-3850</a>.
Blocker bug reported by Varun Saxena and fixed by Varun Saxena (log-aggregation , nodemanager)<br>
<b>NM fails to read files from full disks which can lead to container logs being lost and other issues</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3832">YARN-3832</a>.
Critical bug reported by Ranga Swamy and fixed by Brahma Reddy Battula (nodemanager)<br>
<b>Resource Localization fails on a cluster due to existing cache directories</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3733">YARN-3733</a>.
Blocker bug reported by Bibin A Chundatt and fixed by Rohith Sharma K S (resourcemanager)<br>
<b>Fix DominantRC#compare() does not work as expected if cluster resource is empty</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3725">YARN-3725</a>.
Blocker bug reported by Zhijie Shen and fixed by Zhijie Shen (resourcemanager , timelineserver)<br>
<b>App submission via REST API is broken in secure mode due to Timeline DT service address is empty</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3700">YARN-3700</a>.
Major sub-task reported by Xuan Gong and fixed by Xuan Gong (resourcemanager , webapp , yarn)<br>
<b>ATS Web Performance issue at load time when large number of jobs</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3585">YARN-3585</a>.
Critical bug reported by Peng Zhang and fixed by Rohith Sharma K S <br>
<b>NodeManager cannot exit on SHUTDOWN event triggered and NM recovery is enabled</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3526">YARN-3526</a>.
Major bug reported by Weiwei Yang and fixed by Weiwei Yang (resourcemanager , webapp)<br>
<b>ApplicationMaster tracking URL is incorrectly redirected on a QJM cluster</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3493">YARN-3493</a>.
Critical bug reported by Sumana Sathish and fixed by Jian He (yarn)<br>
<b>RM fails to come up with error "Failed to load/recover state" when mem settings are changed</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3487">YARN-3487</a>.
Critical sub-task reported by Jason Lowe and fixed by Jason Lowe (capacityscheduler)<br>
<b>CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3464">YARN-3464</a>.
Critical bug reported by zhihai xu and fixed by zhihai xu (nodemanager)<br>
<b>Race condition in LocalizerRunner kills localizer before localizing all resources</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3393">YARN-3393</a>.
Critical bug reported by Zhijie Shen and fixed by Zhijie Shen (timelineserver)<br>
<b>Getting application(s) goes wrong when app finishes before starting the attempt</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3369">YARN-3369</a>.
Blocker bug reported by Giovanni Matteo Fumarola and fixed by Brahma Reddy Battula (resourcemanager)<br>
<b>Missing NullPointer check in AppSchedulingInfo causes RM to die </b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3287">YARN-3287</a>.
Major bug reported by Jonathan Eagles and fixed by Daryn Sharp <br>
<b>TimelineClient kerberos authentication failure uses wrong login context.</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3267">YARN-3267</a>.
Major bug reported by Prakash Ramachandran and fixed by Chang Li <br>
<b>Timelineserver applies the ACL rules after applying the limit on the number of records</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3251">YARN-3251</a>.
Blocker bug reported by Jason Lowe and fixed by Craig Welch <br>
<b>Fix CapacityScheduler deadlock when computing absolute max avail capacity (short term fix for 2.6.1)</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3249">YARN-3249</a>.
Minor improvement reported by Ryu Kobayashi and fixed by Ryu Kobayashi (resourcemanager)<br>
<b>Add a "kill application" button to Resource Manager's Web UI</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3242">YARN-3242</a>.
Critical bug reported by zhihai xu and fixed by zhihai xu (resourcemanager)<br>
<b>Asynchrony in ZK-close can lead to ZKRMStateStore watcher receiving events for old client</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3239">YARN-3239</a>.
Major bug reported by Hitesh Shah and fixed by Jian He <br>
<b>WebAppProxy does not support a final tracking url which has query fragments and params </b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3238">YARN-3238</a>.
Blocker bug reported by Jason Lowe and fixed by Jason Lowe <br>
<b>Connection timeouts to nodemanagers are retried at multiple levels</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3231">YARN-3231</a>.
Critical bug reported by Siqi Li and fixed by Siqi Li <br>
<b>FairScheduler: Changing queueMaxRunningApps interferes with pending jobs</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3230">YARN-3230</a>.
Major improvement reported by Jian He and fixed by Jian He <br>
<b>Clarify application states on the web UI</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3227">YARN-3227</a>.
Critical bug reported by Jonathan Eagles and fixed by Zhijie Shen <br>
<b>Timeline renew delegation token fails when RM user's TGT is expired</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3222">YARN-3222</a>.
Critical bug reported by Rohith Sharma K S and fixed by Rohith Sharma K S (resourcemanager)<br>
<b>RMNodeImpl#ReconnectNodeTransition should send scheduler events in sequential order</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3207">YARN-3207</a>.
Major bug reported by Prakash Ramachandran and fixed by Zhijie Shen (timelineserver)<br>
<b>secondary filter matches entites which do not have the key being filtered for.</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3124">YARN-3124</a>.
Major sub-task reported by Wangda Tan and fixed by Wangda Tan (api , client , resourcemanager)<br>
<b>Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track capacities-by-label</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3103">YARN-3103</a>.
Blocker bug reported by Jason Lowe and fixed by Jason Lowe (client)<br>
<b>AMRMClientImpl does not update AMRM token properly</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3099">YARN-3099</a>.
Major sub-task reported by Wangda Tan and fixed by Wangda Tan (api , client , resourcemanager)<br>
<b>Capacity Scheduler LeafQueue/ParentQueue should use ResourceUsage to track used-resources-by-label.</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3098">YARN-3098</a>.
Major sub-task reported by Wangda Tan and fixed by Wangda Tan (capacityscheduler)<br>
<b>Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3094">YARN-3094</a>.
Major bug reported by Jun Gong and fixed by Jun Gong (resourcemanager)<br>
<b>reset timer for liveness monitors after RM recovery</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3092">YARN-3092</a>.
Major sub-task reported by Wangda Tan and fixed by Wangda Tan (api , client , resourcemanager)<br>
<b>Create common ResourceUsage class to track labeled resource usages in Capacity Scheduler</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3055">YARN-3055</a>.
Blocker bug reported by Yi Liu and fixed by Daryn Sharp (security)<br>
<b>The token is not renewed properly if it's shared by jobs (oozie) in DelegationTokenRenewer</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3024">YARN-3024</a>.
Major bug reported by Chengbing Liu and fixed by Chengbing Liu (nodemanager)<br>
<b>LocalizerRunner should give DIE action when all resources are localized</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-3011">YARN-3011</a>.
Major sub-task reported by Wang Hao and fixed by Varun Saxena (nodemanager)<br>
<b>NM dies because of the failure of resource localization</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-2997">YARN-2997</a>.
Major bug reported by Chengbing Liu and fixed by Chengbing Liu (nodemanager)<br>
<b>NM keeps sending already-sent completed containers to RM until containers are removed from context</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-2992">YARN-2992</a>.
Blocker bug reported by Karthik Kambatla and fixed by Karthik Kambatla (resourcemanager)<br>
<b>ZKRMStateStore crashes due to session expiry</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-2978">YARN-2978</a>.
Critical bug reported by Jason Tufo and fixed by Varun Saxena <br>
<b>ResourceManager crashes with NPE while getting queue info</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-2964">YARN-2964</a>.
Blocker bug reported by Daryn Sharp and fixed by Jian He (resourcemanager)<br>
<b>RM prematurely cancels tokens for jobs that submit jobs (oozie)</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-2952">YARN-2952</a>.
Major bug reported by Jian He and fixed by Rohith Sharma K S <br>
<b>Incorrect version check in RMStateStore</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-2922">YARN-2922</a>.
Major bug reported by Jason Tufo and fixed by Rohith Sharma K S (capacityscheduler , resourcemanager , scheduler)<br>
<b>ConcurrentModificationException in CapacityScheduler's LeafQueue</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-2920">YARN-2920</a>.
Major sub-task reported by Wangda Tan and fixed by Wangda Tan <br>
<b>CapacityScheduler should be notified when labels on nodes changed</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-2918">YARN-2918</a>.
Major sub-task reported by Rohith Sharma K S and fixed by Wangda Tan (resourcemanager)<br>
<b>Don't fail RM if queue's configured labels are not existed in cluster-node-labels</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-2917">YARN-2917</a>.
Critical bug reported by Rohith Sharma K S and fixed by Rohith Sharma K S (resourcemanager)<br>
<b>Potential deadlock in AsyncDispatcher when system.exit called in AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-2910">YARN-2910</a>.
Major bug reported by Wilfred Spiegelenburg and fixed by Wilfred Spiegelenburg (fairscheduler)<br>
<b>FSLeafQueue can throw ConcurrentModificationException</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-2906">YARN-2906</a>.
Major bug reported by Jason Lowe and fixed by Jason Lowe (capacityscheduler)<br>
<b>CapacitySchedulerPage shows HTML tags for a queue's Active Users</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-2905">YARN-2905</a>.
Blocker bug reported by Jason Lowe and fixed by Varun Saxena <br>
<b>AggregatedLogsBlock page can infinitely loop if the aggregated log file is corrupted</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-2900">YARN-2900</a>.
Major sub-task reported by Jonathan Eagles and fixed by Mit Desai (timelineserver)<br>
<b>Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500)</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-2894">YARN-2894</a>.
Major bug reported by Rohith Sharma K S and fixed by Rohith Sharma K S (resourcemanager)<br>
<b>When ACL's are enabled, if RM switches then application can not be viewed from web.</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-2890">YARN-2890</a>.
Major bug reported by Mit Desai and fixed by Mit Desai <br>
<b>MiniYarnCluster should turn on timeline service if configured to do so</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-2874">YARN-2874</a>.
Blocker bug reported by Naganarasimha G R and fixed by Naganarasimha G R (resourcemanager)<br>
<b>Dead lock in "DelegationTokenRenewer" which blocks RM to execute any further apps</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-2865">YARN-2865</a>.
Critical bug reported by Rohith Sharma K S and fixed by Rohith Sharma K S (resourcemanager)<br>
<b>Application recovery continuously fails with "Application with id already present. Cannot duplicate"</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-2856">YARN-2856</a>.
Critical bug reported by Rohith Sharma K S and fixed by Rohith Sharma K S (resourcemanager)<br>
<b>Application recovery throw InvalidStateTransitonException: Invalid event: ATTEMPT_KILLED at ACCEPTED</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-2816">YARN-2816</a>.
Major bug reported by zhihai xu and fixed by zhihai xu (nodemanager)<br>
<b>NM fail to start with NPE during container recovery</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-2766">YARN-2766</a>.
Major sub-task reported by Robert Kanter and fixed by Robert Kanter (timelineserver)<br>
<b> ApplicationHistoryManager is expected to return a sorted list of apps/attempts/containers</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-2694">YARN-2694</a>.
Major sub-task reported by Wangda Tan and fixed by Wangda Tan (capacityscheduler , resourcemanager)<br>
<b>Ensure only single node labels specified in resource request / host, and node label expression only specified when resourceName=ANY</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-2637">YARN-2637</a>.
Critical bug reported by Wangda Tan and fixed by Craig Welch (resourcemanager)<br>
<b>maximum-am-resource-percent could be respected for both LeafQueue/User when trying to activate applications.</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-2414">YARN-2414</a>.
Major bug reported by Zhijie Shen and fixed by Wangda Tan (webapp)<br>
<b>RM web UI: app page will crash if app is failed before any attempt has been created</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-2340">YARN-2340</a>.
Critical bug reported by Nishan Shetty and fixed by Rohith Sharma K S (resourcemanager , scheduler)<br>
<b>NPE thrown when RM restart after queue is STOPPED. There after RM can not recovery application's and remain in standby</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-2301">YARN-2301</a>.
Major improvement reported by Jian He and fixed by Naganarasimha G R <br>
<b>Improve yarn container command</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-2246">YARN-2246</a>.
Major bug reported by Devaraj K and fixed by Devaraj K (webapp)<br>
<b>Job History Link in RM UI is redirecting to the URL which contains Job Id twice</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-1984">YARN-1984</a>.
Major sub-task reported by Jason Lowe and fixed by Varun Saxena <br>
<b>LeveldbTimelineStore does not handle db exceptions properly</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/YARN-1809">YARN-1809</a>.
Major sub-task reported by Zhijie Shen and fixed by Xuan Gong <br>
<b>Synchronize RM and Generic History Service Web-UIs</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/MAPREDUCE-6361">MAPREDUCE-6361</a>.
Critical bug reported by Junping Du and fixed by Junping Du <br>
<b>NPE issue in shuffle caused by concurrent issue between copySucceeded() in one thread and copyFailed() in another thread on the same host</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/MAPREDUCE-6324">MAPREDUCE-6324</a>.
Blocker bug reported by Jason Lowe and fixed by Jason Lowe (mr-am)<br>
<b>Uber jobs fail to update AMRM token when it rolls over</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/MAPREDUCE-6303">MAPREDUCE-6303</a>.
Blocker bug reported by Jason Lowe and fixed by Jason Lowe <br>
<b>Read timeout when retrying a fetch error can be fatal to a reducer</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/MAPREDUCE-6300">MAPREDUCE-6300</a>.
Minor bug reported by Siqi Li and fixed by Siqi Li <br>
<b>Task list sort by task id broken</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/MAPREDUCE-6267">MAPREDUCE-6267</a>.
Minor improvement reported by Chris Trezzo and fixed by Chris Trezzo <br>
<b>Refactor JobSubmitter#copyAndConfigureFiles into it's own class</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/MAPREDUCE-6238">MAPREDUCE-6238</a>.
Critical bug reported by zhihai xu and fixed by zhihai xu (mrv2)<br>
<b>MR2 can't run local jobs with -libjars command options which is a regression from MR1</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/MAPREDUCE-6237">MAPREDUCE-6237</a>.
Major bug reported by Kannan Rajah and fixed by Kannan Rajah (mrv2)<br>
<b>Multiple mappers with DBInputFormat don't work because of reusing conections</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/MAPREDUCE-6230">MAPREDUCE-6230</a>.
Blocker bug reported by Jason Lowe and fixed by Jason Lowe (mr-am)<br>
<b>MR AM does not survive RM restart if RM activated a new AMRM secret key</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/MAPREDUCE-6166">MAPREDUCE-6166</a>.
Major bug reported by Eric Payne and fixed by Eric Payne (mrv2)<br>
<b>Reducers do not validate checksum of map outputs when fetching directly to disk</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/MAPREDUCE-5649">MAPREDUCE-5649</a>.
Major bug reported by stanley shi and fixed by Gera Shegalov (mrv2)<br>
<b>Reduce cannot use more than 2G memory for the final merge</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-8863">HDFS-8863</a>.
Critical bug reported by Kihwal Lee and fixed by Kihwal Lee <br>
<b>The remaining space check in BlockPlacementPolicyDefault is flawed</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-8846">HDFS-8846</a>.
Major bug reported by Zhe Zhang and fixed by Zhe Zhang (namenode)<br>
<b>Add a unit test for INotify functionality across a layout version upgrade</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-8486">HDFS-8486</a>.
Blocker bug reported by Daryn Sharp and fixed by Daryn Sharp (datanode)<br>
<b>DN startup may cause severe data loss</b><br>
<blockquote>Public service notice:
- Every restart of a 2.6.x or 2.7.0 DN incurs a risk of unwanted block deletion.
- Apply this patch if you are running a pre-2.7.1 release.</blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-8480">HDFS-8480</a>.
Critical bug reported by Zhe Zhang and fixed by Zhe Zhang <br>
<b>Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve old edit logs instead of copying them</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-8431">HDFS-8431</a>.
Critical bug reported by Sumana Sathish and fixed by Anu Engineer (HDFS)<br>
<b>hdfs crypto class not found in Windows</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-8404">HDFS-8404</a>.
Major bug reported by Nathan Roberts and fixed by Nathan Roberts (namenode)<br>
<b>Pending block replication can get stuck using older genstamp</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-8384">HDFS-8384</a>.
Minor improvement reported by Tsz Wo Nicholas Sze and fixed by Jing Zhao (namenode)<br>
<b>Allow NN to startup if there are files having a lease but are not under construction</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-8270">HDFS-8270</a>.
Major bug reported by Andrey Stepachev and fixed by J.Andreina (hdfs-client)<br>
<b>create() always retried with hardcoded timeout when file already exists with open lease</b><br>
<blockquote>Proxy level retries will not be done on AlreadyBeingCreatedExeption for create() op.</blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-8245">HDFS-8245</a>.
Major bug reported by Rushabh S Shah and fixed by Rushabh S Shah <br>
<b>Standby namenode doesn't process DELETED_BLOCK if the add block request is in edit log.</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-8219">HDFS-8219</a>.
Major bug reported by Peter Shi and fixed by Surendra Singh Lilhore <br>
<b>setStoragePolicy with folder behavior is different after cluster restart</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-8127">HDFS-8127</a>.
Blocker bug reported by Jing Zhao and fixed by Jing Zhao (ha)<br>
<b>NameNode Failover during HA upgrade can cause DataNode to finalize upgrade</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-8072">HDFS-8072</a>.
Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal (datanode)<br>
<b>Reserved RBW space is not released if client terminates while writing block</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-8070">HDFS-8070</a>.
Blocker bug reported by Gopal V and fixed by Colin Patrick McCabe (caching)<br>
<b>Pre-HDFS-7915 DFSClient cannot use short circuit on post-HDFS-7915 DataNode</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-8046">HDFS-8046</a>.
Major bug reported by Kihwal Lee and fixed by Kihwal Lee <br>
<b>Allow better control of getContentSummary</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7999">HDFS-7999</a>.
Major bug reported by zhouyingchao and fixed by zhouyingchao <br>
<b>FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7980">HDFS-7980</a>.
Major bug reported by Hui Zheng and fixed by Walter Su <br>
<b>Incremental BlockReport will dramatically slow down the startup of a namenode</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7960">HDFS-7960</a>.
Critical bug reported by Lei (Eddy) Xu and fixed by Colin Patrick McCabe <br>
<b>The full block report should prune zombie storages even if they're not empty</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7930">HDFS-7930</a>.
Blocker bug reported by Konstantin Shvachko and fixed by Yi Liu (namenode)<br>
<b>commitBlockSynchronization() does not remove locations</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7929">HDFS-7929</a>.
Major bug reported by Zhe Zhang and fixed by Zhe Zhang <br>
<b>inotify unable fetch pre-upgrade edit log segments once upgrade starts</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7915">HDFS-7915</a>.
Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe <br>
<b>The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7894">HDFS-7894</a>.
Critical bug reported by Kihwal Lee and fixed by Brahma Reddy Battula <br>
<b>Rolling upgrade readiness is not updated in jmx until query command is issued.</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7885">HDFS-7885</a>.
Critical bug reported by vitthal (Suhas) Gogate and fixed by Tsz Wo Nicholas Sze (datanode)<br>
<b>Datanode should not trust the generation stamp provided by client</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7884">HDFS-7884</a>.
Blocker bug reported by Tsz Wo Nicholas Sze and fixed by Brahma Reddy Battula (datanode)<br>
<b>NullPointerException in BlockSender</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7871">HDFS-7871</a>.
Critical bug reported by Jing Zhao and fixed by Jing Zhao <br>
<b>NameNodeEditLogRoller can keep printing "Swallowing exception" message</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7830">HDFS-7830</a>.
Major bug reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (datanode)<br>
<b>DataNode does not release the volume lock when adding a volume fails.</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7788">HDFS-7788</a>.
Blocker bug reported by Kihwal Lee and fixed by Rushabh S Shah <br>
<b>Post-2.6 namenode may not start up with an image containing inodes created with an old release.</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7763">HDFS-7763</a>.
Major bug reported by Liang Xie and fixed by Liang Xie (ha)<br>
<b>fix zkfc hung issue due to not catching exception in a corner case</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7742">HDFS-7742</a>.
Major bug reported by Nathan Roberts and fixed by Nathan Roberts (namenode)<br>
<b>favoring decommissioning node for replication can cause a block to stay underreplicated for long periods</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7733">HDFS-7733</a>.
Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal (nfs)<br>
<b>NFS: readdir/readdirplus return null directory attribute on failure</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7714">HDFS-7714</a>.
Major bug reported by Chris Nauroth and fixed by Vinayakumar B (datanode)<br>
<b>Simultaneous restart of HA NameNodes and DataNode can cause DataNode to register successfully with only one NameNode.</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7707">HDFS-7707</a>.
Major bug reported by Yongjun Zhang and fixed by Yongjun Zhang (namenode)<br>
<b>Edit log corruption due to delayed block removal again</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7610">HDFS-7610</a>.
Major bug reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (datanode)<br>
<b>Fix removal of dynamically added DN volumes</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7609">HDFS-7609</a>.
Critical bug reported by Carrey Zhan and fixed by Ming Ma (namenode)<br>
<b>Avoid retry cache collision when Standby NameNode loading edits</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7596">HDFS-7596</a>.
Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal (namenode)<br>
<b>NameNode should prune dead storages from storageMap</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7587">HDFS-7587</a>.
Blocker bug reported by Kihwal Lee and fixed by Jing Zhao (namenode)<br>
<b>Edit log corruption can happen if append fails with a quota violation</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7579">HDFS-7579</a>.
Minor improvement reported by Charles Lamb and fixed by Charles Lamb (datanode)<br>
<b>Improve log reporting during block report rpc failure</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7575">HDFS-7575</a>.
Critical bug reported by Lars Francke and fixed by Arpit Agarwal <br>
<b>Upgrade should generate a unique storage ID for each volume</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7552">HDFS-7552</a>.
Major bug reported by Liang Xie and fixed by Liang Xie (datanode , test)<br>
<b>change FsVolumeList toString() to fix TestDataNodeVolumeFailureToleration </b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7533">HDFS-7533</a>.
Major bug reported by Kihwal Lee and fixed by Eric Payne <br>
<b>Datanode sometimes does not shutdown on receiving upgrade shutdown command</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7531">HDFS-7531</a>.
Major improvement reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (datanode)<br>
<b>Improve the concurrent access on FsVolumeList</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7503">HDFS-7503</a>.
Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal (namenode)<br>
<b>Namenode restart after large deletions can cause slow processReport (due to logging)</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7489">HDFS-7489</a>.
Critical bug reported by Noah Lorang and fixed by Noah Lorang (datanode)<br>
<b>Incorrect locking in FsVolumeList#checkDirs can hang datanodes</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7470">HDFS-7470</a>.
Major bug reported by zhaoyunjiong and fixed by zhaoyunjiong (namenode)<br>
<b>SecondaryNameNode need twice memory when calling reloadFromImageFile</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7446">HDFS-7446</a>.
Major improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (hdfs-client)<br>
<b>HDFS inotify should have the ability to determine what txid it has read up to</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7443">HDFS-7443</a>.
Blocker bug reported by Kihwal Lee and fixed by Colin Patrick McCabe <br>
<b>Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are present in the same volume</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7425">HDFS-7425</a>.
Minor bug reported by Chris Nauroth and fixed by Chris Nauroth (namenode)<br>
<b>NameNode block deletion logging uses incorrect appender.</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7314">HDFS-7314</a>.
Major improvement reported by Ming Ma and fixed by Ming Ma <br>
<b>When the DFSClient lease cannot be renewed, abort open-for-write files rather than the entire DFSClient</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7278">HDFS-7278</a>.
Major improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (datanode)<br>
<b>Add a command that allows sysadmins to manually trigger full block reports from a DN</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7263">HDFS-7263</a>.
Major bug reported by Konstantin Shvachko and fixed by Tao Luo (hdfs-client)<br>
<b>Snapshot read can reveal future bytes for appended files.</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7235">HDFS-7235</a>.
Major bug reported by Yongjun Zhang and fixed by Yongjun Zhang (datanode , namenode)<br>
<b>DataNode#transferBlock should report blocks that don't exist using reportBadBlock</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7225">HDFS-7225</a>.
Major bug reported by Zhe Zhang and fixed by Zhe Zhang (namenode)<br>
<b>Remove stale block invalidation work when DN re-registers with different UUID</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7213">HDFS-7213</a>.
Critical bug reported by Daryn Sharp and fixed by Eric Payne (namenode)<br>
<b>processIncrementalBlockReport performance degradation</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7182">HDFS-7182</a>.
Major improvement reported by Ming Ma and fixed by Ming Ma <br>
<b>JMX metrics aren't accessible when NN is busy</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7035">HDFS-7035</a>.
Major sub-task reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (datanode)<br>
<b>Make adding a new data directory to the DataNode an atomic operation and improve error handling</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-7009">HDFS-7009</a>.
Major bug reported by Ming Ma and fixed by Ming Ma (datanode)<br>
<b>Active NN and standby NN have different live nodes</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-4882">HDFS-4882</a>.
Critical bug reported by Zesheng Wu and fixed by Ravi Prakash (hdfs-client , namenode)<br>
<b>Prevent the Namenode's LeaseManager from looping forever in checkLeases</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HDFS-3443">HDFS-3443</a>.
Major bug reported by suja s and fixed by Vinayakumar B (auto-failover , ha)<br>
<b>Fix NPE when namenode transition to active during startup by adding checkNNStartup() in NameNodeRpcServer</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HADOOP-12280">HADOOP-12280</a>.
Minor improvement reported by Masatake Iwasaki and fixed by Masatake Iwasaki (test)<br>
<b>Skip unit tests based on maven profile rather than NativeCodeLoader.isNativeCodeLoaded</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HADOOP-11934">HADOOP-11934</a>.
Blocker bug reported by Mike Yoder and fixed by Larry McCay (security)<br>
<b>Use of JavaKeyStoreProvider in LdapGroupsMapping causes infinite loop</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HADOOP-11932">HADOOP-11932</a>.
Critical bug reported by Jian He and fixed by Brahma Reddy Battula <br>
<b> MetricsSinkAdapter hangs when being stopped</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HADOOP-11812">HADOOP-11812</a>.
Blocker improvement reported by Gera Shegalov and fixed by Gera Shegalov (fs)<br>
<b>Implement listLocatedStatus for ViewFileSystem to speed up split calculation</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HADOOP-11802">HADOOP-11802</a>.
Major bug reported by Eric Payne and fixed by Colin Patrick McCabe <br>
<b>DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShm</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HADOOP-11730">HADOOP-11730</a>.
Major bug reported by Takenori Sato and fixed by Takenori Sato (fs/s3)<br>
<b>Regression: s3n read failure recovery broken</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HADOOP-11710">HADOOP-11710</a>.
Critical sub-task reported by Sean Busbey and fixed by Sean Busbey (fs)<br>
<b>Make CryptoOutputStream behave like DFSOutputStream wrt synchronization</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HADOOP-11674">HADOOP-11674</a>.
Critical bug reported by Sean Busbey and fixed by Sean Busbey (io)<br>
<b>oneByteBuf in CryptoInputStream and CryptoOutputStream should be non static</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HADOOP-11604">HADOOP-11604</a>.
Critical bug reported by Liang Xie and fixed by Chris Nauroth (net)<br>
<b>Prevent ConcurrentModificationException while closing domain sockets during shutdown of DomainSocketWatcher thread.</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HADOOP-11506">HADOOP-11506</a>.
Major improvement reported by Dmitriy V. Ryaboy and fixed by Gera Shegalov (conf)<br>
<b>Configuration variable expansion regex expensive for long values</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HADOOP-11491">HADOOP-11491</a>.
Critical bug reported by Gera Shegalov and fixed by Brahma Reddy Battula (fs)<br>
<b>HarFs incorrectly declared as requiring an authority</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HADOOP-11482">HADOOP-11482</a>.
Major bug reported by Arun Suresh and fixed by Arun Suresh <br>
<b>Use correct UGI when KMSClientProvider is called by a proxy user</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HADOOP-11466">HADOOP-11466</a>.
Minor improvement reported by Suman Somasundar and fixed by Suman Somasundar (io , performance , util)<br>
<b>FastByteComparisons: do not use UNSAFE_COMPARER on the SPARC architecture because it is slower there</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HADOOP-11368">HADOOP-11368</a>.
Major bug reported by Arun Suresh and fixed by Arun Suresh (kms)<br>
<b>Fix SSLFactory truststore reloader thread leak in KMSClientProvider</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HADOOP-11350">HADOOP-11350</a>.
Major bug reported by Benoy Antony and fixed by Benoy Antony (security)<br>
<b>The size of header buffer of HttpServer is too small when HTTPS is enabled</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HADOOP-11343">HADOOP-11343</a>.
Blocker bug reported by Jerry Chen and fixed by Jerry Chen (security)<br>
<b>Overflow is not properly handled in caclulating final iv for AES CTR</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HADOOP-11333">HADOOP-11333</a>.
Major bug reported by zhaoyunjiong and fixed by zhaoyunjiong <br>
<b>Fix deadlock in DomainSocketWatcher when the notification pipe is full</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HADOOP-11316">HADOOP-11316</a>.
Blocker bug reported by Tsuyoshi Ozawa and fixed by Tsuyoshi Ozawa <br>
<b>"mvn package -Pdist,docs -DskipTests -Dtar" fails because of non-ascii characters</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HADOOP-11295">HADOOP-11295</a>.
Major bug reported by Ming Ma and fixed by Ming Ma <br>
<b>RPC Server Reader thread can't shutdown if RPCCallQueue is full</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HADOOP-11238">HADOOP-11238</a>.
Minor bug reported by Chris Li and fixed by Chris Li <br>
<b>Update the NameNode's Group Cache in the background when possible</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HADOOP-10786">HADOOP-10786</a>.
Major bug reported by Tobi Vollebregt and fixed by Stephen Chu (security)<br>
<b>Fix UGI#reloginFromKeytab on Java 8</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HADOOP-8151">HADOOP-8151</a>.
Major bug reported by Todd Lipcon and fixed by Matt Foley (io , native)<br>
<b>Error handling in snappy decompressor throws invalid exceptions</b><br>
<blockquote></blockquote></li>
<li> <a href="https://issues.apache.org/jira/browse/HADOOP-7139">HADOOP-7139</a>.
Major improvement reported by Stephen Rose and fixed by Kanaka Kumar Avvaru (io)<br>
<b>Allow appending to existing SequenceFiles</b><br>
<blockquote>Existing sequence files can be appended.</blockquote></li>
</ul>
</body></html>