Apache Hadoop Changelog

Release 2.6.4 - 2016-02-11

INCOMPATIBLE CHANGES:

JIRASummaryPriorityComponentReporterContributor
YARN-3154Should not upload partial logs for MR jobs or other "short-running' applicationsBlockernodemanager, resourcemanagerXuan GongXuan Gong

IMPORTANT ISSUES:

JIRASummaryPriorityComponentReporterContributor

NEW FEATURES:

JIRASummaryPriorityComponentReporterContributor

IMPROVEMENTS:

JIRASummaryPriorityComponentReporterContributor
HDFS-9415Document dfs.cluster.administrators and dfs.permissions.superusergroupMajordocumentationArpit AgarwalXiaobing Zhou
HDFS-9314Improve BlockPlacementPolicyDefault's picking of excess replicasMajor.Ming MaXiao Chen
HDFS-8722Optimize datanode writes for small writes and flushesCritical.Kihwal LeeKihwal Lee
HDFS-8647Abstract BlockManager's rack policy into BlockPlacementPolicyMajor.Ming MaBrahma Reddy Battula
HDFS-7694FSDataInputStream should support “unbuffer”Major.Colin Patrick McCabeColin Patrick McCabe
MAPREDUCE-6436JobHistory cache issueBlocker.Ryu KobayashiKai Sasaki

BUG FIXES:

JIRASummaryPriorityComponentReporterContributor
HADOOP-12706TestLocalFsFCStatistics#testStatisticsThreadLocalDataCleanUp times out occasionallyMajortestJason LoweSangjin Lee
HADOOP-12107long running apps may have a huge number of StatisticsData instances under FileSystemCriticalfsSangjin LeeSangjin Lee
HADOOP-11252RPC client does not time out by defaultCriticalipcWilfred SpiegelenburgMasatake Iwasaki
HDFS-9600do not check replication if the block is under constructionCritical.Phil YangPhil Yang
HDFS-9574Reduce client failures during datanode restartMajor.Kihwal LeeKihwal Lee
HDFS-9445Datanode may deadlock while handling a bad volumeBlocker.Kihwal LeeWalter Su
HDFS-9313Possible NullPointerException in BlockManager if no excess replica can be chosenMajor.Ming MaMing Ma
HDFS-9294DFSClient deadlock when close file and failed to renew leaseBlockerhdfs-clientDENG FEIBrahma Reddy Battula
HDFS-9220Reading small file (< 512 bytes) that is open for append fails due to incorrect checksumBlocker.Bogdan RaducanuJing Zhao
HDFS-9178Slow datanode I/O can cause a wrong node to be marked badCritical.Kihwal LeeKihwal Lee
HDFS-8767RawLocalFileSystem.listStatus() returns null for UNIX pipefileCritical.Haohui MaiKanaka Kumar Avvaru
HDFS-6945BlockManager should remove a block from excessReplicateMap and decrement ExcessBlocks metric when the block is removedCriticalnamenodeAkira AJISAKAAkira AJISAKA
HDFS-4660Block corruption can happen during pipeline recoveryBlockerdatanodePeng ZhangKihwal Lee
MAPREDUCE-6621Memory Leak in JobClient#submitJobInternal()Major.Xuan GongXuan Gong
MAPREDUCE-6619HADOOP_CLASSPATH is overwritten in MR containerMajormrv2shanyu zhaoJunping Du
MAPREDUCE-6618YarnClientProtocolProvider leaking the YarnClient thread.Major.Xuan GongXuan Gong
MAPREDUCE-6577MR AM unable to load native library without MR_AM_ADMIN_USER_ENV setCriticalmr-amSangjin LeeSangjin Lee
MAPREDUCE-6554MRAppMaster servicestart failing with NPE in MRAppMaster#parsePreviousJobHistoryCritical.Bibin A ChundattBibin A Chundatt
MAPREDUCE-6492AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTaskCritical.Bibin A ChundattBibin A Chundatt
MAPREDUCE-6363[NNBench] Lease mismatch error when running with multiple mappersCriticalbenchmarksBrahma Reddy BattulaBibin A Chundatt
MAPREDUCE-5982Task attempts that fail from the ASSIGNED state can disappearMajormr-amJason LoweChang Li
YARN-4598Invalid event: RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILLMajornodemanagertangshangwentangshangwen
YARN-4581AHS writer thread leak makes RM crash while RM is recoveringMajorresourcemanagersandfleesandflee
YARN-4546ResourceManager crash due to scheduling opportunity overflowCriticalresourcemanagerJason LoweJason Lowe
YARN-4452NPE when submit Unmanaged applicationCritical.Naganarasimha G RNaganarasimha G R
YARN-4414Nodemanager connection errors are retried at multiple levelsMajornodemanagerJason LoweChang Li
YARN-4380TestResourceLocalizationService.testDownloadingResourcesOnContainerKill fails intermittentlyMajortestTsuyoshi OzawaVarun Saxena
YARN-4354Public resource localization fails with NPEBlockernodemanagerJason LoweJason Lowe
YARN-4180AMLauncher does not retry on failures when talking to NMCriticalresourcemanagerAnubhav DhootAnubhav Dhoot
YARN-3857Memory leak in ResourceManager with SIMPLE modeCriticalresourcemanagermujunchaomujunchao
YARN-3849Too much of preemption activity causing continuos killing of containers across queuesCriticalcapacityschedulerSunil GSunil G
YARN-3842NMProxy should retry on NMNotYetReadyExceptionCritical.Karthik KambatlaRobert Kanter
YARN-3697FairScheduler: ContinuousSchedulingThread can fail to shutdownCriticalfairschedulerzhihai xuzhihai xu
YARN-3695ServerProxy (NMProxy, etc.) shouldn't retry forever for non network exception.Major.Junping DuRaju Bairishetti
YARN-3535Scheduler must re-request container resources when RMContainer transitions from ALLOCATED to KILLEDCriticalcapacityscheduler, fairscheduler, resourcemanagerPeng ZhangPeng Zhang
YARN-2975FSLeafQueue app lists are accessed without required locksBlocker.Karthik KambatlaKarthik Kambatla

TESTS:

JIRASummaryPriorityComponentReporterContributor
HADOOP-12736TestTimedOutTestsListener#testThreadDumpAndDeadlocks sometimes times outMajor.Xiao ChenXiao Chen

SUB-TASKS:

JIRASummaryPriorityComponentReporterContributor
YARN-3893Both RM in active state when Admin#transitionToActive failure from refeshAll()CriticalresourcemanagerBibin A ChundattBibin A Chundatt
YARN-2902Killing a container that is localizing can orphan resources in the DOWNLOADING stateMajornodemanagerJason LoweVarun Saxena

OTHER:

JIRASummaryPriorityComponentReporterContributor