Apache Hadoop Changelog

Release 2.6.1 - 2015-09-23

INCOMPATIBLE CHANGES:

JIRASummaryPriorityComponentReporterContributor

IMPORTANT ISSUES:

JIRASummaryPriorityComponentReporterContributor

NEW FEATURES:

JIRASummaryPriorityComponentReporterContributor

IMPROVEMENTS:

JIRASummaryPriorityComponentReporterContributor
HADOOP-12280Skip unit tests based on maven profile rather than NativeCodeLoader.isNativeCodeLoadedMinortestMasatake IwasakiMasatake Iwasaki
HADOOP-11812Implement listLocatedStatus for ViewFileSystem to speed up split calculationBlockerfsGera ShegalovGera Shegalov
HADOOP-11506Configuration variable expansion regex expensive for long valuesMajorconfDmitriy V. RyaboyGera Shegalov
HADOOP-11466FastByteComparisons: do not use UNSAFE_COMPARER on the SPARC architecture because it is slower thereMinorio, performance, utilSuman SomasundarSuman Somasundar
HADOOP-7139Allow appending to existing SequenceFilesMajorioStephen RoseKanaka Kumar Avvaru
HDFS-8384Allow NN to startup if there are files having a lease but are not under constructionMinornamenodeTsz Wo Nicholas SzeJing Zhao
HDFS-7579Improve log reporting during block report rpc failureMinordatanodeCharles LambCharles Lamb
HDFS-7531Improve the concurrent access on FsVolumeListMajordatanodeLei (Eddy) XuLei (Eddy) Xu
HDFS-7446HDFS inotify should have the ability to determine what txid it has read up toMajorhdfs-clientColin Patrick McCabeColin Patrick McCabe
HDFS-7314When the DFSClient lease cannot be renewed, abort open-for-write files rather than the entire DFSClientMajor.Ming MaMing Ma
HDFS-7278Add a command that allows sysadmins to manually trigger full block reports from a DNMajordatanodeColin Patrick McCabeColin Patrick McCabe
HDFS-7182JMX metrics aren't accessible when NN is busyMajor.Ming MaMing Ma
MAPREDUCE-6267Refactor JobSubmitter#copyAndConfigureFiles into it's own classMinor.Chris TrezzoChris Trezzo
YARN-3978Configurably turn off the saving of container info in Generic AHSMajortimelineserver, yarnEric PayneEric Payne
YARN-3249Add a “kill application” button to Resource Manager's Web UIMinorresourcemanagerRyu KobayashiRyu Kobayashi
YARN-3248Display count of nodes blacklisted by apps in the web UIMajorcapacityscheduler, resourcemanagerVarun VasudevVarun Vasudev
YARN-3230Clarify application states on the web UIMajor.Jian HeJian He
YARN-2301Improve yarn container commandMajor.Jian HeNaganarasimha G R

BUG FIXES:

JIRASummaryPriorityComponentReporterContributor
HADOOP-11934Use of JavaKeyStoreProvider in LdapGroupsMapping causes infinite loopBlockersecurityMike YoderLarry McCay
HADOOP-11932MetricsSinkAdapter hangs when being stoppedCritical.Jian HeBrahma Reddy Battula
HADOOP-11802DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShmMajor.Eric PayneColin Patrick McCabe
HADOOP-11730Regression: s3n read failure recovery brokenMajorfs/s3Takenori SatoTakenori Sato
HADOOP-11674oneByteBuf in CryptoInputStream and CryptoOutputStream should be non staticCriticalioSean BusbeySean Busbey
HADOOP-11604Prevent ConcurrentModificationException while closing domain sockets during shutdown of DomainSocketWatcher thread.CriticalnetLiang XieChris Nauroth
HADOOP-11491HarFs incorrectly declared as requiring an authorityCriticalfsGera ShegalovBrahma Reddy Battula
HADOOP-11482Use correct UGI when KMSClientProvider is called by a proxy userMajor.Arun SureshArun Suresh
HADOOP-11368Fix SSLFactory truststore reloader thread leak in KMSClientProviderMajorkmsArun SureshArun Suresh
HADOOP-11350The size of header buffer of HttpServer is too small when HTTPS is enabledMajorsecurityBenoy AntonyBenoy Antony
HADOOP-11343Overflow is not properly handled in caclulating final iv for AES CTRBlockersecurityJerry ChenJerry Chen
HADOOP-11333Fix deadlock in DomainSocketWatcher when the notification pipe is fullMajor.zhaoyunjiongzhaoyunjiong
HADOOP-11316“mvn package -Pdist,docs -DskipTests -Dtar” fails because of non-ascii charactersBlocker.Tsuyoshi OzawaTsuyoshi Ozawa
HADOOP-11295RPC Server Reader thread can't shutdown if RPCCallQueue is fullMajor.Ming MaMing Ma
HADOOP-11238Update the NameNode's Group Cache in the background when possibleMinor.Chris LiChris Li
HADOOP-10786Fix UGI#reloginFromKeytab on Java 8MajorsecurityTobi VollebregtStephen Chu
HADOOP-8151Error handling in snappy decompressor throws invalid exceptionsMajorio, nativeTodd LipconMatt Foley
HDFS-8863The remaining space check in BlockPlacementPolicyDefault is flawedCritical.Kihwal LeeKihwal Lee
HDFS-8846Add a unit test for INotify functionality across a layout version upgradeMajornamenodeZhe ZhangZhe Zhang
HDFS-8486DN startup may cause severe data lossBlockerdatanodeDaryn SharpDaryn Sharp
HDFS-8480Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve old edit logs instead of copying themCritical.Zhe ZhangZhe Zhang
HDFS-8431hdfs crypto class not found in WindowsCriticalscriptsSumana SathishAnu Engineer
HDFS-8404Pending block replication can get stuck using older genstampMajornamenodeNathan RobertsNathan Roberts
HDFS-8270create() always retried with hardcoded timeout when file already exists with open leaseMajorhdfs-clientAndrey StepachevJ.Andreina
HDFS-8245Standby namenode doesn't process DELETED_BLOCK if the add block request is in edit log.Major.Rushabh S ShahRushabh S Shah
HDFS-8219setStoragePolicy with folder behavior is different after cluster restartMajor.Peter ShiSurendra Singh Lilhore
HDFS-8127NameNode Failover during HA upgrade can cause DataNode to finalize upgradeBlockerhaJing ZhaoJing Zhao
HDFS-8072Reserved RBW space is not released if client terminates while writing blockMajordatanodeArpit AgarwalArpit Agarwal
HDFS-8070Pre-HDFS-7915 DFSClient cannot use short circuit on post-HDFS-7915 DataNodeBlockercachingGopal VColin Patrick McCabe
HDFS-8046Allow better control of getContentSummaryMajor.Kihwal LeeKihwal Lee
HDFS-7999FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long timeMajor.zhouyingchaozhouyingchao
HDFS-7980Incremental BlockReport will dramatically slow down the startup of a namenodeMajor.Hui ZhengWalter Su
HDFS-7960The full block report should prune zombie storages even if they're not emptyCritical.Lei (Eddy) XuColin Patrick McCabe
HDFS-7930commitBlockSynchronization() does not remove locationsBlockernamenodeKonstantin ShvachkoYi Liu
HDFS-7929inotify unable fetch pre-upgrade edit log segments once upgrade startsMajor.Zhe ZhangZhe Zhang
HDFS-7915The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network errorMajor.Colin Patrick McCabeColin Patrick McCabe
HDFS-7894Rolling upgrade readiness is not updated in jmx until query command is issued.Critical.Kihwal LeeBrahma Reddy Battula
HDFS-7885Datanode should not trust the generation stamp provided by clientCriticaldatanodevitthal (Suhas) GogateTsz Wo Nicholas Sze
HDFS-7884NullPointerException in BlockSenderBlockerdatanodeTsz Wo Nicholas SzeBrahma Reddy Battula
HDFS-7871NameNodeEditLogRoller can keep printing “Swallowing exception” messageCritical.Jing ZhaoJing Zhao
HDFS-7830DataNode does not release the volume lock when adding a volume fails.MajordatanodeLei (Eddy) XuLei (Eddy) Xu
HDFS-7788Post-2.6 namenode may not start up with an image containing inodes created with an old release.Blocker.Kihwal LeeRushabh S Shah
HDFS-7763fix zkfc hung issue due to not catching exception in a corner caseMajorhaLiang XieLiang Xie
HDFS-7742favoring decommissioning node for replication can cause a block to stay underreplicated for long periodsMajornamenodeNathan RobertsNathan Roberts
HDFS-7733NFS: readdir/readdirplus return null directory attribute on failureMajornfsArpit AgarwalArpit Agarwal
HDFS-7714Simultaneous restart of HA NameNodes and DataNode can cause DataNode to register successfully with only one NameNode.MajordatanodeChris NaurothVinayakumar B
HDFS-7707Edit log corruption due to delayed block removal againMajornamenodeYongjun ZhangYongjun Zhang
HDFS-7610Fix removal of dynamically added DN volumesMajordatanodeLei (Eddy) XuLei (Eddy) Xu
HDFS-7609Avoid retry cache collision when Standby NameNode loading editsCriticalnamenodeCarrey ZhanMing Ma
HDFS-7596NameNode should prune dead storages from storageMapMajornamenodeArpit AgarwalArpit Agarwal
HDFS-7587Edit log corruption can happen if append fails with a quota violationBlockernamenodeKihwal LeeJing Zhao
HDFS-7575Upgrade should generate a unique storage ID for each volumeCritical.Lars FranckeArpit Agarwal
HDFS-7552change FsVolumeList toString() to fix TestDataNodeVolumeFailureTolerationMajordatanode, testLiang XieLiang Xie
HDFS-7533Datanode sometimes does not shutdown on receiving upgrade shutdown commandMajor.Kihwal LeeEric Payne
HDFS-7503Namenode restart after large deletions can cause slow processReport (due to logging)MajornamenodeArpit AgarwalArpit Agarwal
HDFS-7489Incorrect locking in FsVolumeList#checkDirs can hang datanodesCriticaldatanodeNoah LorangNoah Lorang
HDFS-7470SecondaryNameNode need twice memory when calling reloadFromImageFileMajornamenodezhaoyunjiongzhaoyunjiong
HDFS-7443Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are present in the same volumeBlocker.Kihwal LeeColin Patrick McCabe
HDFS-7425NameNode block deletion logging uses incorrect appender.MinornamenodeChris NaurothChris Nauroth
HDFS-7263Snapshot read can reveal future bytes for appended files.Majorhdfs-clientKonstantin ShvachkoTao Luo
HDFS-7235DataNode#transferBlock should report blocks that don't exist using reportBadBlockMajordatanode, namenodeYongjun ZhangYongjun Zhang
HDFS-7225Remove stale block invalidation work when DN re-registers with different UUIDMajornamenodeZhe ZhangZhe Zhang
HDFS-7213processIncrementalBlockReport performance degradationCriticalnamenodeDaryn SharpEric Payne
HDFS-7009Active NN and standby NN have different live nodesMajordatanodeMing MaMing Ma
HDFS-4882Prevent the Namenode's LeaseManager from looping forever in checkLeasesCriticalhdfs-client, namenodeZesheng WuRavi Prakash
HDFS-3443Fix NPE when namenode transition to active during startup by adding checkNNStartup() in NameNodeRpcServerMajorauto-failover, hasuja sVinayakumar B
MAPREDUCE-6361NPE issue in shuffle caused by concurrent issue between copySucceeded() in one thread and copyFailed() in another thread on the same hostCritical.Junping DuJunping Du
MAPREDUCE-6324Uber jobs fail to update AMRM token when it rolls overBlockermr-amJason LoweJason Lowe
MAPREDUCE-6303Read timeout when retrying a fetch error can be fatal to a reducerBlocker.Jason LoweJason Lowe
MAPREDUCE-6300Task list sort by task id brokenMinor.Siqi LiSiqi Li
MAPREDUCE-6238MR2 can't run local jobs with -libjars command options which is a regression from MR1Criticalmrv2zhihai xuzhihai xu
MAPREDUCE-6237Multiple mappers with DBInputFormat don't work because of reusing conectionsMajormrv2Kannan RajahKannan Rajah
MAPREDUCE-6230MR AM does not survive RM restart if RM activated a new AMRM secret keyBlockermr-amJason LoweJason Lowe
MAPREDUCE-6166Reducers do not validate checksum of map outputs when fetching directly to diskMajormrv2Eric PayneEric Payne
MAPREDUCE-5649Reduce cannot use more than 2G memory for the final mergeMajormrv2stanley shiGera Shegalov
YARN-4047ClientRMService getApplications has high scheduler lock contentionMajorresourcemanagerJason LoweJason Lowe
YARN-3999RM hangs on draining eventsMajor.Jian HeJian He
YARN-3990AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnectedCriticalresourcemanagerRohith Sharma K SBibin A Chundatt
YARN-3850NM fails to read files from full disks which can lead to container logs being lost and other issuesBlockerlog-aggregation, nodemanagerVarun SaxenaVarun Saxena
YARN-3832Resource Localization fails on a cluster due to existing cache directoriesCriticalnodemanagerRanga SwamyBrahma Reddy Battula
YARN-3733Fix DominantRC#compare() does not work as expected if cluster resource is emptyBlockerresourcemanagerBibin A ChundattRohith Sharma K S
YARN-3725App submission via REST API is broken in secure mode due to Timeline DT service address is emptyBlockerresourcemanager, timelineserverZhijie ShenZhijie Shen
YARN-3585NodeManager cannot exit on SHUTDOWN event triggered and NM recovery is enabledCritical.Peng ZhangRohith Sharma K S
YARN-3526ApplicationMaster tracking URL is incorrectly redirected on a QJM clusterMajorresourcemanager, webappWeiwei YangWeiwei Yang
YARN-3493RM fails to come up with error “Failed to load/recover state” when mem settings are changedCriticalyarnSumana SathishJian He
YARN-3464Race condition in LocalizerRunner kills localizer before localizing all resourcesCriticalnodemanagerzhihai xuzhihai xu
YARN-3393Getting application(s) goes wrong when app finishes before starting the attemptCriticaltimelineserverZhijie ShenZhijie Shen
YARN-3369Missing NullPointer check in AppSchedulingInfo causes RM to dieBlockerresourcemanagerGiovanni Matteo FumarolaBrahma Reddy Battula
YARN-3287TimelineClient kerberos authentication failure uses wrong login context.Major.Jonathan EaglesDaryn Sharp
YARN-3267Timelineserver applies the ACL rules after applying the limit on the number of recordsMajor.Prakash RamachandranChang Li
YARN-3251Fix CapacityScheduler deadlock when computing absolute max avail capacity (short term fix for 2.6.1)Blocker.Jason LoweCraig Welch
YARN-3242Asynchrony in ZK-close can lead to ZKRMStateStore watcher receiving events for old clientCriticalresourcemanagerzhihai xuzhihai xu
YARN-3239WebAppProxy does not support a final tracking url which has query fragments and paramsMajor.Hitesh ShahJian He
YARN-3238Connection timeouts to nodemanagers are retried at multiple levelsBlocker.Jason LoweJason Lowe
YARN-3231FairScheduler: Changing queueMaxRunningApps interferes with pending jobsCritical.Siqi LiSiqi Li
YARN-3227Timeline renew delegation token fails when RM user's TGT is expiredCritical.Jonathan EaglesZhijie Shen
YARN-3222RMNodeImpl#ReconnectNodeTransition should send scheduler events in sequential orderCriticalresourcemanagerRohith Sharma K SRohith Sharma K S
YARN-3207secondary filter matches entites which do not have the key being filtered for.MajortimelineserverPrakash RamachandranZhijie Shen
YARN-3103AMRMClientImpl does not update AMRM token properlyBlockerclientJason LoweJason Lowe
YARN-3094reset timer for liveness monitors after RM recoveryMajorresourcemanagerJun GongJun Gong
YARN-3055The token is not renewed properly if it's shared by jobs (oozie) in DelegationTokenRenewerBlockersecurityYi LiuDaryn Sharp
YARN-3024LocalizerRunner should give DIE action when all resources are localizedMajornodemanagerChengbing LiuChengbing Liu
YARN-2997NM keeps sending already-sent completed containers to RM until containers are removed from contextMajornodemanagerChengbing LiuChengbing Liu
YARN-2992ZKRMStateStore crashes due to session expiryBlockerresourcemanagerKarthik KambatlaKarthik Kambatla
YARN-2978ResourceManager crashes with NPE while getting queue infoCritical.Jason TufoVarun Saxena
YARN-2964RM prematurely cancels tokens for jobs that submit jobs (oozie)BlockerresourcemanagerDaryn SharpJian He
YARN-2952Incorrect version check in RMStateStoreMajor.Jian HeRohith Sharma K S
YARN-2922ConcurrentModificationException in CapacityScheduler's LeafQueueMajorcapacityscheduler, resourcemanager, schedulerJason TufoRohith Sharma K S
YARN-2917Potential deadlock in AsyncDispatcher when system.exit called in AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hookCriticalresourcemanagerRohith Sharma K SRohith Sharma K S
YARN-2910FSLeafQueue can throw ConcurrentModificationExceptionMajorfairschedulerWilfred SpiegelenburgWilfred Spiegelenburg
YARN-2906CapacitySchedulerPage shows HTML tags for a queue's Active UsersMajorcapacityschedulerJason LoweJason Lowe
YARN-2905AggregatedLogsBlock page can infinitely loop if the aggregated log file is corruptedBlocker.Jason LoweVarun Saxena
YARN-2894When ACL's are enabled, if RM switches then application can not be viewed from web.MajorresourcemanagerRohith Sharma K SRohith Sharma K S
YARN-2890MiniYarnCluster should turn on timeline service if configured to do soMajor.Mit DesaiMit Desai
YARN-2874Dead lock in “DelegationTokenRenewer” which blocks RM to execute any further appsBlockerresourcemanagerNaganarasimha G RNaganarasimha G R
YARN-2865Application recovery continuously fails with “Application with id already present. Cannot duplicate”CriticalresourcemanagerRohith Sharma K SRohith Sharma K S
YARN-2856Application recovery throw InvalidStateTransitonException: Invalid event: ATTEMPT_KILLED at ACCEPTEDCriticalresourcemanagerRohith Sharma K SRohith Sharma K S
YARN-2816NM fail to start with NPE during container recoveryMajornodemanagerzhihai xuzhihai xu
YARN-2637maximum-am-resource-percent could be respected for both LeafQueue/User when trying to activate applications.CriticalresourcemanagerWangda TanCraig Welch
YARN-2414RM web UI: app page will crash if app is failed before any attempt has been createdMajorwebappZhijie ShenWangda Tan
YARN-2340NPE thrown when RM restart after queue is STOPPED. There after RM can not recovery application's and remain in standbyCriticalresourcemanager, schedulerNishan ShettyRohith Sharma K S
YARN-2246Job History Link in RM UI is redirecting to the URL which contains Job Id twiceMajorwebappDevaraj KDevaraj K

TESTS:

JIRASummaryPriorityComponentReporterContributor

SUB-TASKS:

JIRASummaryPriorityComponentReporterContributor
HADOOP-11710Make CryptoOutputStream behave like DFSOutputStream wrt synchronizationCriticalfsSean BusbeySean Busbey
HDFS-7035Make adding a new data directory to the DataNode an atomic operation and improve error handlingMajordatanodeLei (Eddy) XuLei (Eddy) Xu
YARN-3740Fixed the typo with the configuration name: APPLICATION_HISTORY_PREFIX_MAX_APPSMajorresourcemanager, webapp, yarnXuan GongXuan Gong
YARN-3700ATS Web Performance issue at load time when large number of jobsMajorresourcemanager, webapp, yarnXuan GongXuan Gong
YARN-3544AM logs link missing in the RM UI for a completed appBlocker.Hitesh ShahXuan Gong
YARN-3487CapacityScheduler scheduler lock obtained unnecessarily when calling getQueueCriticalcapacityschedulerJason LoweJason Lowe
YARN-3171Sort by Application id, AppAttempt & ContainerID doesn't work in ATS / RM web uiMinortimelineserverJeff ZhangNaganarasimha G R
YARN-3124Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track capacities-by-labelMajorapi, client, resourcemanagerWangda TanWangda Tan
YARN-3099Capacity Scheduler LeafQueue/ParentQueue should use ResourceUsage to track used-resources-by-label.Majorapi, client, resourcemanagerWangda TanWangda Tan
YARN-3098Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queuesMajorcapacityschedulerWangda TanWangda Tan
YARN-3092Create common ResourceUsage class to track labeled resource usages in Capacity SchedulerMajorapi, client, resourcemanagerWangda TanWangda Tan
YARN-3011NM dies because of the failure of resource localizationMajornodemanagerWang HaoVarun Saxena
YARN-2920CapacityScheduler should be notified when labels on nodes changedMajor.Wangda TanWangda Tan
YARN-2918Don‘t fail RM if queue’s configured labels are not existed in cluster-node-labelsMajorresourcemanagerRohith Sharma K SWangda Tan
YARN-2900Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500)MajortimelineserverJonathan EaglesMit Desai
YARN-2766ApplicationHistoryManager is expected to return a sorted list of apps/attempts/containersMajortimelineserverRobert KanterRobert Kanter
YARN-2694Ensure only single node labels specified in resource request / host, and node label expression only specified when resourceName=ANYMajorcapacityscheduler, resourcemanagerWangda TanWangda Tan
YARN-1984LeveldbTimelineStore does not handle db exceptions properlyMajor.Jason LoweVarun Saxena
YARN-1884ContainerReport should have nodeHttpAddressMajor.Zhijie ShenXuan Gong
YARN-1809Synchronize RM and Generic History Service Web-UIsMajor.Zhijie ShenXuan Gong

OTHER:

JIRASummaryPriorityComponentReporterContributor