Apache Hadoop Changelog

Release 2.7.1 - 2015-07-06

INCOMPATIBLE CHANGES:

JIRASummaryPriorityComponentReporterContributor
HDFS-8226Non-HA rollback compatibility brokenBlocker.J.AndreinaJ.Andreina

IMPORTANT ISSUES:

JIRASummaryPriorityComponentReporterContributor

NEW FEATURES:

JIRASummaryPriorityComponentReporterContributor

IMPROVEMENTS:

JIRASummaryPriorityComponentReporterContributor
HADOOP-11851s3n to swallow IOEs on inner stream closeMinorfs/s3Steve LoughranTakenori Sato
HDFS-8521Add @VisibleForTesting annotation to {{BlockPoolSlice#selectReplicaToDelete}}Trivial.Colin Patrick McCabeColin Patrick McCabe
HDFS-8361Choose SSD over DISK in block placementMajornamenodeTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HDFS-8204Mover/Balancer should not schedule two replicas to the same DNMinorbalancer & moverWalter SuWalter Su
HDFS-7770Need document for storage type label of data node storage locations under dfs.datanode.data.dirMajordocumentationXiaoyu YaoXiaoyu Yao
HDFS-7546Document, and set an accepting default for dfs.namenode.kerberos.principal.patternMinorsecurityHarsh JHarsh J
YARN-3539Compatibility doc to state that ATS v1 is a stable REST APIMajordocumentationSteve LoughranSteve Loughran
YARN-3489RMServerUtils.validateResourceRequests should only obtain queue info onceMajorresourcemanagerJason LoweVarun Saxena
YARN-3469ZKRMStateStore: Avoid setting watches that are not requiredMinor.Jun GongJun Gong
YARN-3193When visit standby RM webui, it will redirect to the active RM webui slowly.MinorwebappJaps_123Steve Loughran

BUG FIXES:

JIRASummaryPriorityComponentReporterContributor
HADOOP-12103Small refactoring of DelegationTokenAuthenticationFilter to allow code sharingMinorsecurityYongjun ZhangYongjun Zhang
HADOOP-12100ImmutableFsPermission should not override applyUmask since that method doesn't modify the FsPermissionMajor.Robert KanterBibin A Chundatt
HADOOP-12078The default retry policy does not handle RetriableException correctlyCriticalipcArpit AgarwalArpit Agarwal
HADOOP-12058Fix dead links to DistCp and Hadoop Archives pages.Minordocumentation, siteKazuho FujiiKazuho Fujii
HADOOP-11973Ensure ZkDelegationTokenSecretManager namespace znodes get created with ACLsMajorsecurityGregory ChananGregory Chanan
HADOOP-11966Variable cygwin is undefined in hadoop-config.sh when executed through hadoop-daemon.sh.CriticalscriptsChris NaurothChris Nauroth
HADOOP-11934Use of JavaKeyStoreProvider in LdapGroupsMapping causes infinite loopBlockersecurityMike YoderLarry McCay
HADOOP-11891OsSecureRandom should lazily fill its reservoirMajorsecurityArun SureshArun Suresh
HADOOP-11872“hadoop dfs” command prints message about using “yarn jar” on Windows(branch-2 only)MinorscriptsVarun VasudevVarun Vasudev
HADOOP-11868Invalid user logins trigger large backtraces in server logMajor.Chang LiChang Li
HADOOP-11802DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShmMajor.Eric PayneColin Patrick McCabe
HADOOP-11730Regression: s3n read failure recovery brokenMajorfs/s3Takenori SatoTakenori Sato
HADOOP-11663Remove description about Java 6 from docsMinordocumentationMasatake IwasakiMasatake Iwasaki
HADOOP-9658SnappyCodec#checkNativeCodeLoaded may unexpectedly fail when native code is not loadedMajor.Zhijie ShenZhijie Shen
HDFS-8681BlockScanner is incorrectly disabled by defaultBlockerdatanodeAndrew WangArpit Agarwal
HDFS-8633Fix setting of dfs.datanode.readahead.bytes in hdfs-default.xml to match DFSConfigKeysMinordatanodeRay ChiangRay Chiang
HDFS-8626Reserved RBW space is not released if creation of RBW File failsBlocker.Kanaka Kumar AvvaruKanaka Kumar Avvaru
HDFS-8600TestWebHdfsFileSystemContract.testGetFileBlockLocations fails in branch-2.7MajorwebhdfsArpit AgarwalArpit Agarwal
HDFS-8596TestDistributedFileSystem et al tests are broken in branch-2 due to incorrect setting of “datanode” attributeBlockerdatanodeYongjun ZhangYongjun Zhang
HDFS-8595TestCommitBlockSynchronization fails in branch-2.7MajortestArpit AgarwalArpit Agarwal
HDFS-8583Document that NFS gateway does not work with rpcbind on SLES 11MajordocumentationArpit AgarwalArpit Agarwal
HDFS-8576Lease recovery should return true if the lease can be released and the file can be closedMajornamenodeJ.AndreinaJ.Andreina
HDFS-8572DN always uses HTTP/localhost@REALM principals in SPNEGOBlocker.Haohui MaiHaohui Mai
HDFS-8566HDFS documentation about debug commands wrongly identifies them as “hdfs dfs” commandsMajordocumentationSurendra Singh LilhoreSurendra Singh Lilhore
HDFS-8544Incorrect port specified in HFTP Guide document in branch-2MajordocumentationBrahma Reddy BattulaBrahma Reddy Battula
HDFS-8523Remove usage information on unsupported operation “fsck -showprogress” from branch-2MajordocumentationJ.AndreinaJ.Andreina
HDFS-8522Change heavily recorded NN logs from INFO to DEBUG levelMajornamenodeXiaoyu YaoXiaoyu Yao
HDFS-8486DN startup may cause severe data lossBlockerdatanodeDaryn SharpDaryn Sharp
HDFS-8480Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve old edit logs instead of copying themCritical.Zhe ZhangZhe Zhang
HDFS-8451DFSClient probe for encryption testing interprets empty URI property for “enabled”BlockerencryptionSteve LoughranSteve Loughran
HDFS-8405Fix a typo in NamenodeFsckMinornamenodeTsz Wo Nicholas SzeTakanobu Asanuma
HDFS-8404Pending block replication can get stuck using older genstampMajornamenodeNathan RobertsNathan Roberts
HDFS-8305HDFS INotify: the destination field of RenameOp should always end with the file nameMajor.Colin Patrick McCabeColin Patrick McCabe
HDFS-8273FSNamesystem#Delete() should not call logSync() when holding the lockBlockernamenodeJing ZhaoHaohui Mai
HDFS-8270create() always retried with hardcoded timeout when file already exists with open leaseMajorhdfs-clientAndrey StepachevJ.Andreina
HDFS-8269getBlockLocations() does not resolve the .reserved path and generates incorrect edit logs when updating the atimeBlocker.Yesha VoraHaohui Mai
HDFS-8245Standby namenode doesn't process DELETED_BLOCK if the add block request is in edit log.Major.Rushabh S ShahRushabh S Shah
HDFS-8213DFSClient should use hdfs.client.htrace HTrace configuration prefix rather than hadoop.htraceCritical.Billie RinaldiColin Patrick McCabe
HDFS-8179DFSClient#getServerDefaults returns null within 1 hour of system startBlocker.Xiaoyu YaoXiaoyu Yao
HDFS-8163Using monotonicNow for block report scheduling causes test failures on recently restarted systemsBlockerdatanodeArpit AgarwalArpit Agarwal
HDFS-8153Error Message points to wrong parent directory in case of path component name length errorMajornamenodeAnu EngineerAnu Engineer
HDFS-8151Always use snapshot path as source when invalid snapshot names are used for diff based distcpMinordistcpSushmitha SreenivasanJing Zhao
HDFS-8149The footer of the Web UI “Hadoop, 2014” is oldMajor.Akira AJISAKABrahma Reddy Battula
HDFS-8147Mover should not schedule two replicas to the same DN storageMajorbalancer & moverSurendra Singh LilhoreSurendra Singh Lilhore
HDFS-8127NameNode Failover during HA upgrade can cause DataNode to finalize upgradeBlockerhaJing ZhaoJing Zhao
HDFS-8091ACLStatus and XAttributes not properly presented to INodeAttributesProvider before returning to clientMajornamenodeArun SureshArun Suresh
HDFS-8081Split getAdditionalBlock() into two methods.Major.Konstantin ShvachkoKonstantin Shvachko
HDFS-8070Pre-HDFS-7915 DFSClient cannot use short circuit on post-HDFS-7915 DataNodeBlockercachingGopal VColin Patrick McCabe
HDFS-7980Incremental BlockReport will dramatically slow down the startup of a namenodeMajor.Hui ZhengWalter Su
HDFS-7934Update RollingUpgrade rollback documentation: should use bootstrapstandby for standby NNCriticaldocumentationJ.AndreinaJ.Andreina
HDFS-7931DistributedFIleSystem should not look for keyProvider in cache if Encryption is disabledMinorhdfs-clientArun SureshArun Suresh
HDFS-7916‘reportBadBlocks’ from datanodes to standby Node BPServiceActor goes for infinite loopCriticaldatanodeVinayakumar BRushabh S Shah
HDFS-7894Rolling upgrade readiness is not updated in jmx until query command is issued.Critical.Kihwal LeeBrahma Reddy Battula
HDFS-6300Prevent multiple balancers from running simultaneouslyCriticalbalancer & moverRakesh RRakesh R
HDFS-5215dfs.datanode.du.reserved is not considered while computing available spaceMajordatanodeBrahma Reddy BattulaBrahma Reddy Battula
HDFS-4660Block corruption can happen during pipeline recoveryBlockerdatanodePeng ZhangKihwal Lee
MAPREDUCE-6410Aggregated Logs Deletion doesnt work after refreshing Log Retention Settings in secure clusterCritical.Zhang WeiVarun Saxena
MAPREDUCE-6387Serialize the recently added Task#encryptedSpillKey field at the endMinor.Arun SureshArun Suresh
MAPREDUCE-6361NPE issue in shuffle caused by concurrent issue between copySucceeded() in one thread and copyFailed() in another thread on the same hostCritical.Junping DuJunping Du
MAPREDUCE-6339Job history file is not flushed correctly because isTimerActive flag is not set true when flushTimerTask is scheduled.Criticalmrv2zhihai xuzhihai xu
MAPREDUCE-6334Fetcher#copyMapOutput is leaking usedMemory upon IOException during InMemoryMapOutput shuffle handlerBlocker.Eric PayneEric Payne
MAPREDUCE-6324Uber jobs fail to update AMRM token when it rolls overBlockermr-amJason LoweJason Lowe
MAPREDUCE-6300Task list sort by task id brokenMinor.Siqi LiSiqi Li
MAPREDUCE-6259IllegalArgumentException due to missing job submit timeMajorjobhistoryserverzhihai xuzhihai xu
MAPREDUCE-6252JobHistoryServer should not fail when encountering a missing directoryMajorjobhistoryserverCraig WelchCraig Welch
MAPREDUCE-6251JobClient needs additional retries at a higher level to address not-immediately-consistent dfs corner casesMajorjobhistoryserver, mrv2Craig WelchCraig Welch
MAPREDUCE-6238MR2 can't run local jobs with -libjars command options which is a regression from MR1Criticalmrv2zhihai xuzhihai xu
YARN-3850NM fails to read files from full disks which can lead to container logs being lost and other issuesBlockerlog-aggregation, nodemanagerVarun SaxenaVarun Saxena
YARN-3842NMProxy should retry on NMNotYetReadyExceptionCritical.Karthik KambatlaRobert Kanter
YARN-3832Resource Localization fails on a cluster due to existing cache directoriesCriticalnodemanagerRanga SwamyBrahma Reddy Battula
YARN-3809Failed to launch new attempts because ApplicationMasterLauncher's threads all hangMajorresourcemanagerJun GongJun Gong
YARN-3804Both RM are on standBy state when kerberos user not in yarn.admin.aclCriticalresourcemanagerBibin A ChundattVarun Saxena
YARN-3764CapacityScheduler should forbid moving LeafQueue from one parent to anotherBlocker.Wangda TanWangda Tan
YARN-3753RM failed to come up with “java.io.IOException: Wait for ZKClient creation timed out”CriticalyarnSumana SathishJian He
YARN-3733Fix DominantRC#compare() does not work as expected if cluster resource is emptyBlockerresourcemanagerBibin A ChundattRohith Sharma K S
YARN-3725App submission via REST API is broken in secure mode due to Timeline DT service address is emptyBlockerresourcemanager, timelineserverZhijie ShenZhijie Shen
YARN-3723Need to clearly document primaryFilter and otherInfo value typeCriticaltimelineserverZhijie ShenZhijie Shen
YARN-3701Isolating the error of generating a single app report when getting all apps from generic history serviceBlockertimelineserverZhijie ShenZhijie Shen
YARN-3694Fix dead link for TimelineServer REST APIMinordocumentationAkira AJISAKAJagadesh Kiran N
YARN-3681yarn cmd says “could not find main class ‘queue’” in windowsBlockeryarnSumana SathishVarun Saxena
YARN-3677Fix findbugs warnings in yarn-server-resourcemanagerMinorresourcemanagerAkira AJISAKAVinod Kumar Vavilapalli
YARN-3675FairScheduler: RM quits when node removal races with continousscheduling on the same nodeCriticalfairschedulerAnubhav DhootAnubhav Dhoot
YARN-3646Applications are getting stuck some times in case of retry policy foreverMajorclientRaju BairishettiRaju Bairishetti
YARN-3626On Windows localized resources are not moved to the front of the classpath when they should beMajoryarnCraig WelchCraig Welch
YARN-3614FileSystemRMStateStore throw exception when failed to remove application, that cause resourcemanager to crashCriticalresourcemanagerlachisis
YARN-3601Fix UT TestRMFailover.testRMWebAppRedirectCriticalresourcemanager, webappWeiwei YangWeiwei Yang
YARN-3585NodeManager cannot exit on SHUTDOWN event triggered and NM recovery is enabledCritical.Peng ZhangRohith Sharma K S
YARN-3554Default value for maximum nodemanager connect wait time is too highMajor.Jason LoweNaganarasimha G R
YARN-3537NPE when NodeManager.serviceInit fails and stopRecoveryStore invokedMajornodemanagerBrahma Reddy BattulaBrahma Reddy Battula
YARN-3526ApplicationMaster tracking URL is incorrectly redirected on a QJM clusterMajorresourcemanager, webappWeiwei YangWeiwei Yang
YARN-3522DistributedShell uses the wrong user to put timeline dataBlockertimelineserverZhijie ShenZhijie Shen
YARN-3516killing ContainerLocalizer action doesn't take effect when private localizer receives FETCH_FAILURE status.Minornodemanagerzhihai xuzhihai xu
YARN-3497ContainerManagementProtocolProxy modifies IPC timeout conf without making a copyMajorclientJason LoweJason Lowe
YARN-3493RM fails to come up with error “Failed to load/recover state” when mem settings are changedCriticalyarnSumana SathishJian He
YARN-3485FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policiesCriticalfairschedulerKarthik KambatlaKarthik Kambatla
YARN-3476Nodemanager can fail to delete local logs if log aggregation failsMajorlog-aggregation, nodemanagerJason LoweRohith Sharma K S
YARN-3472Possible leak in DelegationTokenRenewer#allTokensMajor.Jian HeRohith Sharma K S
YARN-3466Fix RM nodes web page to sort by node HTTP-address, #containers and node-label columnMajorresourcemanager, webappJason LoweJason Lowe
YARN-3465Use LinkedHashMap to preserve order of resource requestsMajornodemanagerzhihai xuzhihai xu
YARN-3464Race condition in LocalizerRunner kills localizer before localizing all resourcesCriticalnodemanagerzhihai xuzhihai xu
YARN-3462Patches applied for YARN-2424 are inconsistent between trunk and branch-2Major.Sidharta SeethanaNaganarasimha G R
YARN-3457NPE when NodeManager.serviceInit fails and stopRecoveryStore calledMinornodemanagerBibin A ChundattBibin A Chundatt
YARN-3434Interaction between reservations and userlimit can result in significant ULF violationMajorcapacityschedulerThomas GravesThomas Graves
YARN-3385Race condition: KeeperException$NoNodeException will cause RM shutdown during ZK node deletion.Criticalresourcemanagerzhihai xuzhihai xu
YARN-3382Some of UserMetricsInfo metrics are incorrectly set to root queue metricsMajorwebappRohit AgarwalRohit Agarwal
YARN-3358Audit log not present while refreshing Service ACLsMinorresourcemanagerVarun SaxenaVarun Saxena
YARN-3351AppMaster tracking URL is broken in HAMajorwebappAnubhav DhootAnubhav Dhoot
YARN-3243CapacityScheduler should pass headroom from parent to children to make sure ParentQueue obey its capacity limits.Majorcapacityscheduler, resourcemanagerWangda TanWangda Tan
YARN-2238filtering on UI sticks even if I move away from the pageMajorwebappSangjin LeeJian He

TESTS:

JIRASummaryPriorityComponentReporterContributor

SUB-TASKS:

JIRASummaryPriorityComponentReporterContributor
HDFS-8597Fix TestFSImage#testZeroBlockSize on WindowsMajordatanode, testXiaoyu YaoXiaoyu Yao
HDFS-7164Feature documentation for HDFS-6581MajordocumentationArpit AgarwalArpit Agarwal
YARN-3711Documentation of ResourceManager HA should explain configurations about listen addressesMinordocumentationMasatake IwasakiMasatake Iwasaki
YARN-3686CapacityScheduler should trim default_node_label_expressionCriticalapi, client, resourcemanagerWangda TanSunil G
YARN-3609Move load labels from storage from serviceInit to serviceStart to make it works with RM HA case.MajorresourcemanagerWangda TanWangda Tan
YARN-3544AM logs link missing in the RM UI for a completed appBlocker.Hitesh ShahXuan Gong
YARN-3487CapacityScheduler scheduler lock obtained unnecessarily when calling getQueueCriticalcapacityschedulerJason LoweJason Lowe
YARN-3301Fix the format issue of the new RM web UI and AHS web UI after YARN-3272 / YARN-3262MajorresourcemanagerXuan GongXuan Gong
YARN-3006Improve the error message when attempting manual failover with auto-failover enabledMinor.Akira AJISAKAAkira AJISAKA
YARN-2918Don‘t fail RM if queue’s configured labels are not existed in cluster-node-labelsMajorresourcemanagerRohith Sharma K SWangda Tan
YARN-2900Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500)MajortimelineserverJonathan EaglesMit Desai
YARN-2605[RM HA] Rest api endpoints doing redirect incorrectlyMajorresourcemanagerbc WongXuan Gong

OTHER:

JIRASummaryPriorityComponentReporterContributor