Apache Hadoop Changelog

Release 2.7.2 - 2016-01-25

INCOMPATIBLE CHANGES:

JIRASummaryPriorityComponentReporterContributor
HDFS-7645Rolling upgrade is restoring blocks from trash multiple timesMajordatanodeNathan RobertsKeisuke Ogiwara

IMPORTANT ISSUES:

JIRASummaryPriorityComponentReporterContributor

NEW FEATURES:

JIRASummaryPriorityComponentReporterContributor

IMPROVEMENTS:

JIRASummaryPriorityComponentReporterContributor
HADOOP-12825Log slow name resolutionsMajor.Sidharta SeethanaSidharta Seethana
HADOOP-12413AccessControlList should avoid calling getGroupNames in isUserInList with empty groups.Majorsecurityzhihai xuzhihai xu
HADOOP-12280Skip unit tests based on maven profile rather than NativeCodeLoader.isNativeCodeLoadedMinortestMasatake IwasakiMasatake Iwasaki
HADOOP-12232Upgrade Tomcat dependency to 6.0.44.MajorbuildChris NaurothChris Nauroth
HADOOP-11812Implement listLocatedStatus for ViewFileSystem to speed up split calculationBlockerfsGera ShegalovGera Shegalov
HADOOP-7139Allow appending to existing SequenceFilesMajorioStephen RoseKanaka Kumar Avvaru
HADOOP-5323Trash documentation should describe its directory structure and configurationsMinordocumentationSuman SehgalWeiwei Yang
HDFS-9434Recommission a datanode with 500k blocks may pause NN for 30 secondsMajornamenodeTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HDFS-9221HdfsServerConstants#ReplicaState#getState should avoid calling values() since it creates a temporary arrayMajorperformanceStaffan FribergStaffan Friberg
HDFS-8722Optimize datanode writes for small writes and flushesCritical.Kihwal LeeKihwal Lee
HDFS-8659Block scanner INFO message is spamming logsMajordatanodeYongjun ZhangYongjun Zhang
HDFS-8384Allow NN to startup if there are files having a lease but are not under constructionMinornamenodeTsz Wo Nicholas SzeJing Zhao
HDFS-7314When the DFSClient lease cannot be renewed, abort open-for-write files rather than the entire DFSClientMajor.Ming MaMing Ma
YARN-4158Remove duplicate close for LogWriter in AppLogAggregatorImpl#uploadLogsForContainersMinornodemanagerzhihai xuzhihai xu
YARN-3978Configurably turn off the saving of container info in Generic AHSMajortimelineserver, yarnEric PayneEric Payne
YARN-3727For better error recovery, check if the directory exists before using it for localization.Majornodemanagerzhihai xuzhihai xu
YARN-3248Display count of nodes blacklisted by apps in the web UIMajorcapacityscheduler, resourcemanagerVarun VasudevVarun Vasudev
YARN-3170YARN architecture document needs updatingMajordocumentationAllen WittenauerBrahma Reddy Battula

BUG FIXES:

JIRASummaryPriorityComponentReporterContributor
HADOOP-12577Bump up commons-collections version to 3.2.2 to address a security flawBlockerbuild, securityWei-Chiu ChuangWei-Chiu Chuang
HADOOP-12526[Branch-2] there are duplicate dependency definitions in pom'sMajorbuildSangjin LeeSangjin Lee
HADOOP-12465Incorrect javadoc in WritableUtils.javaMinordocumentationMartin PetricekJagadesh Kiran N
HADOOP-12464Interrupted client may try to fail-over and retryMajoripcKihwal LeeKihwal Lee
HADOOP-12451[Branch-2] Setting HADOOP_HOME explicitly should be allowedBlockerscriptsKarthik KambatlaKarthik Kambatla
HADOOP-12415hdfs and nfs builds broken on -missing compile-time dependency on nettyMajornfsKonstantin BoudnikTom Zeng
HADOOP-12359hadoop fs -getmerge doc is wrongMajordocumentationDaniel TempletonJagadesh Kiran N
HADOOP-12304Applications using FileContext fail with the default file system configured to be wasb/s3/etc.BlockerfsChris NaurothChris Nauroth
HADOOP-12230hadoop-project declares duplicate, conflicting curator dependenciesMinorbuildSteve LoughranRakesh R
HADOOP-12213Interrupted exception can occur when Client#stop is calledMinor.Oleg ZhurakouskyKuhu Shukla
HADOOP-12191Bzip2Factory is not thread safeMajorioJason LoweBrahma Reddy Battula
HADOOP-12186ActiveStandbyElector shouldn't call monitorLockNodeAsync multiple timesMajorhazhihai xuzhihai xu
HADOOP-12006Remove unimplemented option for hadoop fs -ls from document in branch-2.7Major.Akira AJISAKAAkira AJISAKA
HADOOP-11932MetricsSinkAdapter hangs when being stoppedCritical.Jian HeBrahma Reddy Battula
HADOOP-11491HarFs incorrectly declared as requiring an authorityCriticalfsGera ShegalovBrahma Reddy Battula
HADOOP-10365BufferedOutputStream in FileUtil#unpackEntries() should be closed in finally blockMinorutilTed YuKiran Kumar M R
HADOOP-9242Duplicate surefire plugin config in hadoop-commonMajortestAndrey KlochkovAndrey Klochkov
HADOOP-8151Error handling in snappy decompressor throws invalid exceptionsMajorio, nativeTodd LipconMatt Foley
HDFS-9574Reduce client failures during datanode restartMajor.Kihwal LeeKihwal Lee
HDFS-9470Encryption zone on root not loaded from fsimage after NN restartCritical.Xiao ChenXiao Chen
HDFS-9445Datanode may deadlock while handling a bad volumeBlocker.Kihwal LeeWalter Su
HDFS-9431DistributedFileSystem#concat fails if the target path is relative.Majorhdfs-clientKazuho FujiiKazuho Fujii
HDFS-9426Rollingupgrade finalization is not backward compatibleBlocker.Kihwal LeeKihwal Lee
HDFS-9413getContentSummary() on standby should throw StandbyExceptionCritical.Brahma Reddy BattulaBrahma Reddy Battula
HDFS-9317Document fsck -blockId and -storagepolicy options in branch-2.7MajordocumentationAkira AJISAKAAkira AJISAKA
HDFS-9305Delayed heartbeat processing causes storm of subsequent heartbeatsMajordatanodeChris NaurothArpit Agarwal
HDFS-9294DFSClient deadlock when close file and failed to renew leaseBlockerhdfs-clientDENG FEIBrahma Reddy Battula
HDFS-9290DFSClient#callAppend() is not backward compatible for slightly older NameNodesBlocker.Tony WuTony Wu
HDFS-9289Make DataStreamer#block thread safe and verify genStamp in commitBlockCritical.Chang LiChang Li
HDFS-9273ACLs on root directory may be lost after NN restartCriticalnamenodeXiao ChenXiao Chen
HDFS-9220Reading small file (< 512 bytes) that is open for append fails due to incorrect checksumBlocker.Bogdan RaducanuJing Zhao
HDFS-9178Slow datanode I/O can cause a wrong node to be marked badCritical.Kihwal LeeKihwal Lee
HDFS-9106Transfer failure during pipeline recovery causes permanent write failuresCritical.Kihwal LeeKihwal Lee
HDFS-9083Replication violates block placement policy.BlockernamenodeRushabh S ShahRushabh S Shah
HDFS-9043Doc updation for commands in HDFS FederationMinordocumentationJ.AndreinaJ.Andreina
HDFS-9042Update document for the Storage policy nameMinordocumentationJ.AndreinaJ.Andreina
HDFS-9033dfsadmin -metasave prints “NaN” for cache used%Major.Archana TBrahma Reddy Battula
HDFS-8995Flaw in registration bookeeping can make DN die on reconnectCritical.Kihwal LeeKihwal Lee
HDFS-8950NameNode refresh doesn't remove DataNodes that are no longer in the allowed listMajordatanode, namenodeDaniel TempletonDaniel Templeton
HDFS-8891HDFS concat should keep srcs orderBlocker.Yong ZhangYong Zhang
HDFS-8879Quota by storage type usage incorrectly initialized upon namenode restartMajornamenodeKihwal LeeXiaoyu Yao
HDFS-8867Enable optimized block reportsMajor.Rushabh S ShahDaryn Sharp
HDFS-8863The remaining space check in BlockPlacementPolicyDefault is flawedCritical.Kihwal LeeKihwal Lee
HDFS-8852HDFS architecture documentation of version 2.x is outdated about append write supportMajordocumentationHong Dai ThanhAjith S
HDFS-8850VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocksMajordatanodeColin Patrick McCabeColin Patrick McCabe
HDFS-8846Add a unit test for INotify functionality across a layout version upgradeMajornamenodeZhe ZhangZhe Zhang
HDFS-8806Inconsistent metrics: number of missing blocks with replication factor 1 not properly clearedMajor.Zhe ZhangZhe Zhang
HDFS-8767RawLocalFileSystem.listStatus() returns null for UNIX pipefileCritical.Haohui MaiKanaka Kumar Avvaru
HDFS-8676Delayed rolling upgrade finalization can cause heartbeat expiration and write failuresCritical.Kihwal LeeWalter Su
HDFS-8656Preserve compatibility of ClientProtocol#rollingUpgrade after finalizationCriticalrolling upgradesAndrew WangAndrew Wang
HDFS-8615Correct HTTP method in WebHDFS documentMajordocumentationAkira AJISAKABrahma Reddy Battula
HDFS-8431hdfs crypto class not found in WindowsCriticalscriptsSumana SathishAnu Engineer
HDFS-8219setStoragePolicy with folder behavior is different after cluster restartMajor.Peter ShiSurendra Singh Lilhore
HDFS-8099Change “DFSInputStream has been closed already” message to debug log levelMinorhdfs-clientCharles LambCharles Lamb
HDFS-8046Allow better control of getContentSummaryMajor.Kihwal LeeKihwal Lee
HDFS-7725Incorrect “nodes in service” metrics caused all writes to failMajor.Ming MaMing Ma
HDFS-7609Avoid retry cache collision when Standby NameNode loading editsCriticalnamenodeCarrey ZhanMing Ma
HDFS-6945BlockManager should remove a block from excessReplicateMap and decrement ExcessBlocks metric when the block is removedCriticalnamenodeAkira AJISAKAAkira AJISAKA
MAPREDUCE-6549multibyte delimiters with LineRecordReader cause duplicate recordsMajormrv1, mrv2Dustin CoteWilfred Spiegelenburg
MAPREDUCE-6540TestMRTimelineEventHandling failsMajortestSangjin LeeSangjin Lee
MAPREDUCE-6528Memory leak for HistoryFileManager.getJobSummary()CriticaljobhistoryserverJunping DuJunping Du
MAPREDUCE-6518Set SO_KEEPALIVE on shuffle connectionsMajormrv2, nodemanagerNathan RobertsChang Li
MAPREDUCE-6497Fix wrong value of JOB_FINISHED event in JobHistoryEventHandlerMajor.Shinichi YamashitaShinichi Yamashita
MAPREDUCE-6492AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTaskCritical.Bibin A ChundattBibin A Chundatt
MAPREDUCE-6481LineRecordReader may give incomplete record and wrong position/key information for uncompressed input sometimes.Criticalmrv2zhihai xuzhihai xu
MAPREDUCE-6474ShuffleHandler can possibly exhaust nodemanager file descriptorsMajormrv2, nodemanagerNathan RobertsKuhu Shukla
MAPREDUCE-6472MapReduce AM should have java.io.tmpdir=./tmp to be consistent with tasksMajormr-amJason LoweNaganarasimha G R
MAPREDUCE-6454MapReduce doesn't set the HADOOP_CLASSPATH for jar lib in distributed cache.Critical.Junping DuJunping Du
MAPREDUCE-6451DistCp has incorrect chunkFilePath for multiple jobs when strategy is dynamicMajordistcpKuhu ShuklaKuhu Shukla
MAPREDUCE-6442Stack trace is missing when error occurs in client protocol provider's constructorMajorclientChang LiChang Li
MAPREDUCE-6439AM may fail instead of retrying if RM shuts down during the allocate callCritical.Anubhav DhootAnubhav Dhoot
MAPREDUCE-6426TestShuffleHandler#testGetMapOutputInfo is failingMajortestDevaraj Kzhihai xu
MAPREDUCE-6425ShuffleHandler passes wrong “base” parameter to getMapOutputInfo if mapId is not in the cache.Majormrv2, nodemanagerzhihai xuzhihai xu
MAPREDUCE-6377JHS sorting on state column not working in webUiMinorjobhistoryserverBibin A Chundattzhihai xu
MAPREDUCE-6273HistoryFileManager should check whether summaryFile exists to avoid FileNotFoundException causing HistoryFileInfo into MOVE_FAILED stateMinorjobhistoryserverzhihai xuzhihai xu
MAPREDUCE-5982Task attempts that fail from the ASSIGNED state can disappearMajormr-amJason LoweChang Li
MAPREDUCE-5948org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record delimiters wellCritical.Kris GeusebroekAkira AJISAKA
MAPREDUCE-5883“Total megabyte-seconds” in job counters is slightly misleadingMinor.Nathan RobertsNathan Roberts
MAPREDUCE-5649Reduce cannot use more than 2G memory for the final mergeMajormrv2stanley shiGera Shegalov
YARN-4434NodeManager Disk Checker parameter documentation is not correctMinordocumentation, nodemanagerTakashi OhnishiWeiwei Yang
YARN-4424Fix deadlock in RMAppImplBlocker.Yesha VoraJian He
YARN-4365FileSystemNodeLabelStore should check for root dir existence on startupMajorresourcemanagerJason LoweKuhu Shukla
YARN-4354Public resource localization fails with NPEBlockernodemanagerJason LoweJason Lowe
YARN-4348ZKRMStateStore.syncInternal shouldn‘t wait for sync completion for avoiding blocking ZK’s event threadBlocker.Tsuyoshi OzawaTsuyoshi Ozawa
YARN-4344NMs reconnecting with changed capabilities can lead to wrong cluster resource calculationsCriticalresourcemanagerVarun VasudevVarun Vasudev
YARN-4326Fix TestDistributedShell timeout as AHS in MiniYarnCluster no longer binds to default port 8188Major.MENG DINGMENG DING
YARN-4321Incessant retries if NoAuthException is thrown by Zookeeper in non HA modeMajorresourcemanagerVarun SaxenaVarun Saxena
YARN-4320TestJobHistoryEventHandler fails as AHS in MiniYarnCluster no longer binds to default port 8188Major.Varun SaxenaVarun Saxena
YARN-4313Race condition in MiniMRYarnCluster when getting history server addressMajor.Jian HeJian He
YARN-4312TestSubmitApplicationWithRMHA fails on branch-2.7 and branch-2.6 as some of the test cases time outMajor.Varun SaxenaVarun Saxena
YARN-42812.7 RM app page is brokenBlocker.Chang LiChang Li
YARN-4241Fix typo of property name in yarn-default.xmlMajordocumentationAnthony RojasAnthony Rojas
YARN-4209RMStateStore FENCED state doesn’t work due to updateFencedState called by stateMachine.doTransitionCriticalresourcemanagerzhihai xuzhihai xu
YARN-4180AMLauncher does not retry on failures when talking to NMCriticalresourcemanagerAnubhav DhootAnubhav Dhoot
YARN-4127RM fail with noAuth error if switched from failover mode to non-failover modeMajorresourcemanagerJian HeVarun Saxena
YARN-4105Capacity Scheduler headroom for DRF is wrongMajorcapacityschedulerChang LiChang Li
YARN-4103RM WebServices missing scheme for appattempts logLinksMajor.Jonathan EaglesJonathan Eagles
YARN-4096App local logs are leaked if log aggregation fails to initialize for the appMajorlog-aggregation, nodemanagerJason LoweJason Lowe
YARN-4087Followup fixes after YARN-2019 regarding RM behavior when state-store error occursMajor.Jian HeJian He
YARN-4047ClientRMService getApplications has high scheduler lock contentionMajorresourcemanagerJason LoweJason Lowe
YARN-4041Slow delegation token renewal can severely prolong RM recoveryMajorresourcemanagerJason LoweSunil G
YARN-4009CORS support for ResourceManager REST APIMajor.Prakash RamachandranVarun Vasudev
YARN-4005Completed container whose app is finished is not removed from NMStateStoreMajor.Jun GongJun Gong
YARN-4000RM crashes with NPE if leaf queue becomes parent queue during restartMajorcapacityscheduler, resourcemanagerJason LoweVarun Saxena
YARN-3999RM hangs on draining eventsMajor.Jian HeJian He
YARN-3990AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnectedCriticalresourcemanagerRohith Sharma K SBibin A Chundatt
YARN-3975WebAppProxyServlet should not redirect to RM page if AHS is enabledMajor.Mit DesaiMit Desai
YARN-3967Fetch the application report from the AHS if the RM does not know about itMajor.Mit DesaiMit Desai
YARN-3925ContainerLogsUtils#getContainerLogFile fails to read container log files from full disks.Criticalnodemanagerzhihai xuzhihai xu
YARN-3905Application History Server UI NPEs when accessing apps run after RM restartMajortimelineserverEric PayneEric Payne
YARN-3896RMNode transitioned from RUNNING to REBOOTED because its response id had not been reset synchronouslyMajorresourcemanagerJun GongJun Gong
YARN-3878AsyncDispatcher can hang while stopping if it is configured for draining events on stopCritical.Varun SaxenaVarun Saxena
YARN-3857Memory leak in ResourceManager with SIMPLE modeCriticalresourcemanagermujunchaomujunchao
YARN-3802Two RMNodes for the same NodeId are used in RM sometimes after NM is reconnected.Majorresourcemanagerzhihai xuzhihai xu
YARN-3798ZKRMStateStore shouldn't create new session without occurrance of SESSIONEXPIEDBlockerresourcemanagerBibin A ChundattVarun Saxena
YARN-3793Several NPEs when deleting local files on NM recoveryMajornodemanagerKarthik KambatlaVarun Saxena
YARN-3780Should use equals when compare Resource in RMNodeImpl#ReconnectNodeTransitionMinorresourcemanagerzhihai xuzhihai xu
YARN-3697FairScheduler: ContinuousSchedulingThread can fail to shutdownCriticalfairschedulerzhihai xuzhihai xu
YARN-3690[JDK8] ‘mvn site’ failsMajorapi, siteAkira AJISAKABrahma Reddy Battula
YARN-3624ApplicationHistoryServer reverses the order of the filters it getsMajortimelineserverMit DesaiMit Desai
YARN-3619ContainerMetrics unregisters during getMetrics and leads to ConcurrentModificationExceptionMajornodemanagerJason Lowezhihai xu
YARN-3535Scheduler must re-request container resources when RMContainer transitions from ALLOCATED to KILLEDCriticalcapacityscheduler, fairscheduler, resourcemanagerPeng ZhangPeng Zhang
YARN-3508Prevent processing preemption events on the main RM dispatcherMajorresourcemanager, schedulerJason LoweVarun Saxena
YARN-2890MiniYarnCluster should turn on timeline service if configured to do soMajor.Mit DesaiMit Desai
YARN-2859ApplicationHistoryServer binds to default port 8188 in MiniYARNClusterCriticaltimelineserverHitesh ShahVinod Kumar Vavilapalli
YARN-2019Retrospect on decision of making RM crashed if any exception throw in ZKRMStateStoreCritical.Junping DuJian He

TESTS:

JIRASummaryPriorityComponentReporterContributor
YARN-3580[JDK 8] TestClientRMService.testGetLabelsToNodes failsMajortestRobert KanterRobert Kanter

SUB-TASKS:

JIRASummaryPriorityComponentReporterContributor
YARN-4101RM should print alert messages if Zookeeper and Resourcemanager gets connection issueCriticalyarnYesha VoraXuan Gong
YARN-4092RM HA UI redirection needs to be fixed when both RMs are in standby modeMajorresourcemanagerXuan GongXuan Gong
YARN-3969Allow jobs to be submitted to reservation that is active but does not have any allocationsMajorcapacityscheduler, fairscheduler, resourcemanagerSubru KrishnanSubru Krishnan
YARN-3893Both RM in active state when Admin#transitionToActive failure from refeshAll()CriticalresourcemanagerBibin A ChundattBibin A Chundatt
YARN-3740Fixed the typo with the configuration name: APPLICATION_HISTORY_PREFIX_MAX_APPSMajorresourcemanager, webapp, yarnXuan GongXuan Gong
YARN-3700ATS Web Performance issue at load time when large number of jobsMajorresourcemanager, webapp, yarnXuan GongXuan Gong
YARN-3136getTransferredContainers can be a bottleneck during AM registrationMajorschedulerJason LoweSunil G
YARN-2902Killing a container that is localizing can orphan resources in the DOWNLOADING stateMajornodemanagerJason LoweVarun Saxena
YARN-2801Add documentation for node labels featureMajordocumentationGururaj ShettyWangda Tan
YARN-2513Host framework UIs in YARN for use with the ATSMajortimelineserverJonathan EaglesJonathan Eagles

OTHER:

JIRASummaryPriorityComponentReporterContributor