“Apache Hadoop” Changelog

Release 2.8.2 - 2017-10-19

INCOMPATIBLE CHANGES:

JIRASummaryPriorityComponentReporterContributor
HADOOP-14174Set default ADLS access token provider type to ClientCredentialMajorfs/adlJohn ZhugeJohn Zhuge

IMPORTANT ISSUES:

JIRASummaryPriorityComponentReporterContributor

NEW FEATURES:

JIRASummaryPriorityComponentReporterContributor

IMPROVEMENTS:

JIRASummaryPriorityComponentReporterContributor
HDFS-7764DirectoryScanner shouldn't abort the scan if one directory had an errorMajordatanodeRakesh RRakesh R
HDFS-7541Upgrade Domains in HDFSMajor.Ming MaMing Ma
HDFS-10683Make class Token$PrivateToken privateMinor.John ZhugeJohn Zhuge
HADOOP-13688Stop bundling HTML source code in javadoc JARsMajorbuildAndrew WangAndrew Wang
HADOOP-13737Cleanup DiskChecker interfaceMajorutilArpit AgarwalArpit Agarwal
HADOOP-13496Include file lengths in Mismatch in length error for distcpMinor.Ted YuTed Yu
HDFS-10534NameNode WebUI should display DataNode usage histogramMajornamenode, uiZhe ZhangKai Sasaki
HADOOP-14050Add process name to kms processMinorkms, scriptsRushabh S ShahRushabh S Shah
HDFS-11390Add process name to httpfs processMinorhttpfs, scriptsJohn ZhugeWeiwei Yang
HDFS-11333Print a user friendly error message when plugins are not foundMinornamenodeWei-Chiu ChuangWei-Chiu Chuang
HDFS-11466Change dfs.namenode.write-lock-reporting-threshold-ms default from 1000ms to 5000msMajornamenodeAndrew WangAndrew Wang
HDFS-11432Federation : Support fully qualified path for Quota/Snapshot/cacheadmin/cryptoadmin commandsMajorfederationBrahma Reddy BattulaBrahma Reddy Battula
HADOOP-14169Implement listStatusIterator, listLocatedStatus for ViewFsMinorviewfsErik KrogenErik Krogen
HADOOP-14233Delay construction of PreCondition.check failure message in Configuration#setMajor.Jonathan EaglesJonathan Eagles
HADOOP-14240Configuration#get return value optimizationMajor.Jonathan EaglesJonathan Eagles
YARN-6339Improve performance for createAndGetApplicationReportMajor.yunjiong zhaoyunjiong zhao
HDFS-9705Refine the behaviour of getFileChecksum when length = 0Minor.Kai ZhengSammiChen
HDFS-11628Clarify the behavior of HDFS Mover in documentationMajordocumentationXiaobing ZhouXiaobing Zhou
HADOOP-14104Client should always ask namenode for kms provider path.MajorkmsRushabh S ShahRushabh S Shah
HADOOP-14276Add a nanosecond API to Time/Timer/FakeTimerMinorutilErik KrogenErik Krogen
HDFS-11558BPServiceActor thread name is too longMinordatanodeTsz Wo Nicholas SzeXiaobing Zhou
HDFS-11648Lazy construct the IIP pathnameMajor.Daryn SharpDaryn Sharp
HDFS-11634Optimize BlockIterator when iterating starts in the middle.Major.Konstantin ShvachkoKonstantin Shvachko
HDFS-11384Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spikeMajorbalancer & moveryunjiong zhaoKonstantin Shvachko
YARN-6457Allow custom SSL configuration to be supplied in WebAppsMajorwebapp, yarnSanjay M PujareSanjay M Pujare
HDFS-11641Reduce cost of audit logging by using FileStatus instead of HdfsFileStatusMajorhdfsDaryn SharpDaryn Sharp
YARN-6493Print requested node partition in assignContainer logsMajor.Jonathan HungJonathan Hung
HDFS-11891DU#refresh should print the path of the directory when an exception is caughtMinor.Chen LiangChen Liang
HADOOP-14440Add metrics for connections droppedMajor.Eric BadgerEric Badger
HDFS-11861ipc.Client.Connection#sendRpcRequest should log request nameTrivialipcJohn ZhugeJohn Zhuge
HDFS-11345Document the configuration key for FSNamesystem lock fairnessMinordocumentation, namenodeZhe ZhangErik Krogen
YARN-6738LevelDBCacheTimelineStore should reuse ObjectMapper instancesMajortimelineserverZoltan HaindrichZoltan Haindrich
HADOOP-14515Specifically configure zookeeper-related log levels in KMS log4jMajorkmsXiao ChenXiao Chen
HDFS-11881NameNode consumes a lot of memory for snapshot diff report generationMajorhdfs, snapshotsManoj GovindassamyManoj Govindassamy
HDFS-12042Lazy initialize AbstractINodeDiffList#diffs for snapshots to reduce memory consumptionMajor.Misha DmitrievMisha Dmitriev
HDFS-12078Add time unit to the description of property dfs.namenode.stale.datanode.interval in hdfs-default.xmlMinordocumentation, hdfsWeiwei YangWeiwei Yang
HADOOP-14629Improve exception checking in FileContext related JUnit testsMajorfs, testAndras BokorAndras Bokor
HDFS-12137DN dataset lock should be fairCriticaldatanodeDaryn SharpDaryn Sharp
HADOOP-14659UGI getShortUserName does not need to search the SubjectMajorcommonDaryn SharpDaryn Sharp
YARN-6768Improve performance of yarn api record toString and fromStringMajor.Jonathan EaglesJonathan Eagles
YARN-5892Support user-specific minimum user limit percentage in Capacity SchedulerMajorcapacityschedulerEric PayneEric Payne
YARN-6917Queue path is recomputed from scratch on every allocationMinorcapacityschedulerJason LoweEric Payne
HDFS-12301NN File Browser UI: Navigate to a path when enter is pressedTrivialuiRavi PrakashRavi Prakash
HADOOP-14251Credential provider should handle property key deprecationCriticalsecurityJohn ZhugeJohn Zhuge
MAPREDUCE-6937Backport MAPREDUCE-6870 to branch-2 while preserving compatibilityMajor.Zhe ZhangPeter Bacsko
YARN-5547NMLeveldbStateStore should be more tolerant of unknown keysMajornodemanagerJason LoweAjith S
YARN-6930Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManagerMajornodemanagerVinod Kumar VavilapalliShane Kumpf

BUG FIXES:

JIRASummaryPriorityComponentReporterContributor
YARN-524TestYarnVersionInfo failing if generated properties doesn't include an SVN URLMinorapiSteve LoughranSteve Loughran
YARN-1471The SLS simulator is not running the preemption policy for CapacitySchedulerMinor.Carlo CurinoCarlo Curino
HADOOP-11703git should ignore .DS_Store files on Mac OS XMajor.Abin ShahabAbin Shahab
YARN-4612Fix rumen and scheduler load simulator handle killed tasks properlyMajor.Ming MaMing Ma
YARN-4594container-executor fails to remove directory tree when chmod requiredMajornodemanagerColin P. McCabeColin P. McCabe
YARN-4731container-executor should not follow symlinks in recursive_unlink_childrenBlocker.Bibin A ChundattColin P. McCabe
YARN-4812TestFairScheduler#testContinuousScheduling fails intermittentlyMajorfairschedulerKarthik KambatlaKarthik Kambatla
YARN-4927TestRMHA#testTransitionedToActiveRefreshFail fails with FairSchedulerMajortestKarthik KambatlaBibin A Chundatt
YARN-4562YARN WebApp ignores the configuration passed to it for keystore settingsMajor.Sergey ShelukhinSergey Shelukhin
HDFS-9276Failed to Update HDFS Delegation Token for long running application in HA modeMajorfs, ha, securityLiangliang GuLiangliang Gu
YARN-5333Some recovered apps are put into default queue when RM HAMajor.Jun GongJun Gong
HADOOP-13437KMS should reload whitelist and default key ACLs when hot-reloadingMajorkmsXiao ChenXiao Chen
HDFS-8312Trash does not descent into child directories to check for permissionsCriticalfs, securityEric YangWeiwei Yang
YARN-5920Fix deadlock in TestRMHA.testTransitionedToStandbyShouldNotHangMajortestRohith Sharma K SVarun Saxena
HADOOP-13867FilterFileSystem should override rename(.., options) to take effect of Rename options called via FilterFileSystem implementationsMajor.Vinayakumar BVinayakumar B
HADOOP-13508FsPermission string constructor does not recognize sticky bitMajor.Atul SikariaAtul Sikaria
YARN-5988RM unable to start in secure setupBlocker.Ajith SAjith S
YARN-6054TimelineServer fails to start when some LevelDb state files are missing.Critical.Ravi PrakashRavi Prakash
HADOOP-13928TestAdlFileContextMainOperationsLive.testGetFileContext1 runtime errorMajorfs/adl, testJohn ZhugeJohn Zhuge
HADOOP-13976Path globbing does not match newlinesMajor.Eric BadgerEric Badger
HDFS-11316TestDataNodeVolumeFailure#testUnderReplicationAfterVolFailure fails in trunkMinor.Yiqun LinYiqun Lin
HADOOP-14044Synchronization issue in delegation token cancel functionalityMajor.Hrishikesh GadreHrishikesh Gadre
HDFS-11377Balancer hung due to no available mover threadsMajorbalancer & moveryunjiong zhaoyunjiong zhao
YARN-6031Application recovery has failed when node label feature is turned off during RM recoveryMinorschedulerYing ZhangYing Zhang
YARN-6137Yarn client implicitly invoke ATS client which accesses HDFSMajor.Yesha VoraLi Lu
HADOOP-13433Race in UGI.reloginFromKeytabMajorsecurityDuo ZhangDuo Zhang
HADOOP-13119Add ability to secure log servlet using proxy usersMajor.Jeffrey E RodriguezYuanbo Liu
HADOOP-14058Fix NativeS3FileSystemContractBaseTest#testDirWithDifferentMarkersWorksMinorfs/s3, testAkira AjisakaYiqun Lin
HDFS-11084Add a regression test for sticky bit support of OIV ReverseXML processorMajortoolsWei-Chiu ChuangWei-Chiu Chuang
HDFS-11391Numeric usernames do no work with WebHDFS FS (write access)MajorwebhdfsPierre VillardPierre Villard
HDFS-11177‘storagepolicies -getStoragePolicy’ command should accept URI based path.MajorshellSurendra Singh LilhoreSurendra Singh Lilhore
HDFS-11404Increase timeout on TestShortCircuitLocalRead.testDeprecatedGetBlockLocalPathInfoRpcMajor.Eric BadgerEric Badger
HADOOP-14114S3A can no longer handle unencoded + in URIsMinorfs/s3Sean MackrorySean Mackrory
MAPREDUCE-6841Fix dead link in MapReduce tutorial documentMinordocumentationAkira AjisakaVictor Nee
MAPREDUCE-6753Variable in byte printed directly in mapreduce clientMajorclientNemo ChenKai Sasaki
YARN-6263NMTokenSecretManagerInRM.createAndGetNMToken is not thread safeMajoryarnHaibo ChenHaibo Chen
HADOOP-14026start-build-env.sh: invalid docker image nameMajorbuildGergő PásztorGergő Pásztor
HDFS-11441Add escaping to error message in KMS web UIMinorsecurityAaron T. MyersAaron T. Myers
MAPREDUCE-6855Specify charset when create String in CredentialsTestJobMinor.Akira AjisakaKai Sasaki
YARN-6165Intra-queue preemption occurs even when preemption is turned off for a specific queue.Majorcapacity scheduler, scheduler preemptionEric PayneEric Payne
YARN-6310OutputStreams in AggregatedLogFormat.LogWriter can be left open upon exceptionsMajoryarnHaibo ChenHaibo Chen
YARN-6321TestResources test timeouts are too aggressiveMajortestJason LoweEric Badger
HDFS-11512Increase timeout on TestShortCircuitLocalRead#testSkipWithVerifyChecksumMinor.Eric BadgerEric Badger
HDFS-11499Decommissioning stuck because of failing recoveryMajorhdfs, namenodeLukas MajercakLukas Majercak
HDFS-11395RequestHedgingProxyProvider#RequestHedgingInvocationHandler hides the Exception thrown from NameNodeMajorhaNanda kumarNanda kumar
YARN-4051ContainerKillEvent lost when container is still recovering and application finishesCriticalnodemanagersandfleesandflee
YARN-6217TestLocalCacheDirectoryManager test timeout is too aggressiveMajortestJason LoweMiklos Szegedi
HDFS-11132Allow AccessControlException in contract tests when getFileStatus on subdirectory of existing filesMajorfs/adl, testVishwajeet DusaneVishwajeet Dusane
HADOOP-14204S3A multipart commit failing, “UnsupportedOperationException at java.util.Collections$UnmodifiableList.sort”Criticalfs/s3Steve LoughranSteve Loughran
HADOOP-14205No FileSystem for scheme: adlMajorfs/adlJohn ZhugeJohn Zhuge
HDFS-11561HttpFS doc errorsTrivialdocumentation, httpfs, testYuanbo LiuYuanbo Liu
HADOOP-9631ViewFs should use underlying FileSystem's server side defaultsMajorfs, viewfsLohit VijayarenuErik Krogen
HADOOP-14214DomainSocketWatcher::add()/delete() should not self interrupt while looping await()Criticalhdfs-clientMingliang LiuMingliang Liu
HADOOP-14195CredentialProviderFactory$getProviders is not thread-safeMajorsecurityVihang KarajgaonkarVihang Karajgaonkar
HADOOP-14211FilterFs and ChRootedFs are too aggressive about enforcing “authorityNeeded”MajorviewfsErik KrogenErik Krogen
MAPREDUCE-6866Fix getNumMapTasks() documentation in JobConfMinordocumentationJoe MészárosJoe Mészáros
MAPREDUCE-6868License check for jdiff output files should be ignoredMajorbuildAkira AjisakaAkira Ajisaka
HDFS-10506OIV's ReverseXML processor cannot reconstruct some snapshot detailsMajortoolsColin P. McCabeAkira Ajisaka
HDFS-11486Client close() should not fail fast if the last block is being decommissionedMajor.Wei-Chiu ChuangWei-Chiu Chuang
YARN-6359TestRM#testApplicationKillAtAcceptedState fails rarely due to race conditionMajortestRobert KanterRobert Kanter
YARN-5368Memory leak in timeline serverCriticaltimelineserverWataru YukawaJonathan Eagles
HADOOP-14247FileContextMainOperationsBaseTest should clean up test root pathMinorfs, testMingliang LiuMingliang Liu
YARN-6352Header injections are possible in application proxy servletMajorresourcemanager, securityNaganarasimha G RNaganarasimha G R
MAPREDUCE-6873MR Job Submission Fails if MR framework application path not on defaultFSMinormrv2Erik KrogenErik Krogen
HADOOP-14256[S3A DOC] Correct the format for “Seoul” exampleMinordocumentation, fs/s3Brahma Reddy BattulaBrahma Reddy Battula
MAPREDUCE-6850Shuffle Handler keep-alive connections are closed from the server sideMajor.Jonathan EaglesJonathan Eagles
HDFS-11592Closing a file has a wasteful preconditions in NameNodeMajornamenodeEric BadgerEric Badger
YARN-6354LeveldbRMStateStore can parse invalid keys when recovering reservationsMajorresourcemanagerJason LoweJason Lowe
HADOOP-14268Fix markdown itemization in hadoop-aws documentsMinordocumentation, fs/s3Akira AjisakaAkira Ajisaka
YARN-6436TestSchedulingPolicy#testParseSchedulingPolicy timeout is too lowMajortestJason LoweEric Badger
YARN-6420RM startup failure due to wrong order in nodelabel editlogCritical.Bibin A ChundattBibin A Chundatt
YARN-6403Invalid local resource request can raise NPE and make NM exitMajornodemanagerTao YangTao Yang
HDFS-11538Move ClientProtocol HA proxies into hadoop-hdfs-clientBlockerhdfs-clientAndrew WangHuafeng Wang
YARN-6437TestSignalContainer#testSignalRequestDeliveryToNM fails intermittentlyMajortestJason LoweJason Lowe
HDFS-11629Revert HDFS-11431 hadoop-hdfs-client JAR does not include ConfiguredFailoverProxyProvider.Major.Andrew WangAndrew Wang
HADOOP-13996Fix some release build issuesBlockerbuildAndrew WangAndrew Wang
HDFS-11608HDFS write crashed with block size greater than 2 GBCriticalhdfs-clientXiaobing ZhouXiaobing Zhou
YARN-6288Exceptions during aggregated log writes are mishandledCriticallog-aggregationAkira AjisakaAkira Ajisaka
HADOOP-14066VersionInfo should be marked as public APICriticalcommonThejas M NairAkira Ajisaka
HADOOP-14293Initialize FakeTimer with a less trivial valueMajortestAndrew WangAndrew Wang
YARN-6461TestRMAdminCLI has very low test timeoutsMajortestJason LoweEric Badger
HDFS-11163Mover should move the file blocks to default storage once policy is unsetMajorbalancer & moverSurendra Singh LilhoreSurendra Singh Lilhore
YARN-6450TestContainerManagerWithLCE requires override for each new test added to ContainerManagerTestMajortestJason LoweJason Lowe
YARN-3760FSDataOutputStream leak in AggregatedLogFormat.LogWriter.close()CriticalnodemanagerDaryn SharpHaibo Chen
YARN-5994TestCapacityScheduler.testAMLimitUsage fails intermittentlyMajor.Eric BadgerEric Badger
YARN-6480Timeout is too aggressive for TestAMRestart.testPreemptedAMRestartOnRMRestartMajor.Eric BadgerEric Badger
HADOOP-14311Add python2.7-dev to DockerfileMajor.Allen WittenauerAllen Wittenauer
YARN-6304Skip rm.transitionToActive call to RM if RM is already active.MajorresourcemanagerRohith Sharma K SRohith Sharma K S
HDFS-11615FSNamesystemLock metrics can be inaccurate due to millisecond precisionMajorhdfsErik KrogenErik Krogen
HADOOP-14318Remove non-existent setfattr command option from FileSystemShell.mdMinordocumentationDoris GuDoris Gu
HADOOP-14315Python example in the rack awareness document doesn't work due to bad indentationMinordocumentationKengo SekiKengo Seki
HDFS-11689New exception thrown by DFSClient#isHDFSEncryptionEnabled broke hacky hive codeMajor.Yongjun ZhangYongjun Zhang
HDFS-11691Add a proper scheme to the datanode links in NN web UIMajor.Kihwal LeeKihwal Lee
HADOOP-14341Support multi-line value for ssl.server.exclude.cipher.listMajor.John ZhugeJohn Zhuge
YARN-5617AMs only intended to run one attempt can be run more than onceMajorresourcemanagerJason LoweJason Lowe
HADOOP-14346CryptoOutputStream throws IOException on flush() if stream is closedMajor.Pierre LacavePierre Lacave
HDFS-11709StandbyCheckpointer should handle an non-existing legacyOivImageDir gracefullyCriticalha, namenodeZhe ZhangErik Krogen
YARN-5894fixed license warning caused by de.ruedigermoeller:fst:jar:2.24BlockeryarnHaibo ChenHaibo Chen
HADOOP-14320TestIPC.testIpcWithReaderQueuing fails intermittentlyMajor.Eric BadgerEric Badger
YARN-6536TestAMRMClient.testAMRMClientWithSaslEncryption fails intermittentlyMajor.Eric BadgerJason Lowe
HDFS-11609Some blocks can be permanently lost if nodes are decommissioned while deadBlockernamenodeKihwal LeeKihwal Lee
HDFS-8498Blocks can be committed with wrong sizeCriticalhdfs-clientDaryn SharpJing Zhao
HDFS-11714Newly added NN storage directory won't get initialized and cause space exhaustionCritical.Kihwal LeeKihwal Lee
HADOOP-14371License error in TestLoadBalancingKMSClientProvider.javaMajor.hu xiaodonghu xiaodong
HADOOP-14369NetworkTopology calls expensive toString() when loggingMajor.Íñigo GoiriÍñigo Goiri
HADOOP-14306TestLocalFileSystem tests have very low timeoutsMajor.Eric BadgerEric Badger
HADOOP-14372TestSymlinkLocalFS timeouts are too lowMajor.Eric BadgerEric Badger
HADOOP-14207“dfsadmin -refreshCallQueue” fails with DecayRpcSchedulerBlockerrpc-serverSurendra Singh LilhoreSurendra Singh Lilhore
HDFS-11702Remove indefinite caching of key provider uri in DFSClientMajorhdfs-clientRushabh S ShahRushabh S Shah
HADOOP-14374License error in GridmixTestUtils.javaMajor.lixinglonglixinglong
HADOOP-14100Upgrade Jsch jar to latest version to fix vulnerability in old versionsCritical.Vinayakumar BVinayakumar B
HADOOP-14377Increase Common test timeouts from 1 second to 10 secondsMajor.Eric BadgerEric Badger
HADOOP-14373License error In org.apache.hadoop.metrics2.util.ServersMajor.hu xiaodonghu xiaodong
YARN-6552Increase YARN test timeouts from 1 second to 10 secondsMajor.Eric BadgerEric Badger
MAPREDUCE-6882Increase MapReduce test timeouts from 1 second to 10 secondsMajor.Eric BadgerEric Badger
HADOOP-14405Fix performance regression due to incorrect use of DataChecksumMajornative, performanceLiXin GeLiXin Ge
HDFS-11745Increase HDFS test timeouts from 1 second to 10 secondsMajor.Eric BadgerEric Badger
HDFS-11755Underconstruction blocks can be considered missingMajor.Nathan RobertsNathan Roberts
YARN-5543ResourceManager SchedulingMonitor could potentially terminate the preemption checker threadMajorcapacityscheduler, resourcemanagerMin ShenMin Shen
HDFS-11674reserveSpaceForReplicas is not released if append request failed due to mirror down and replica recoveredCriticaldatanodeVinayakumar BVinayakumar B
HADOOP-14376Memory leak when reading a compressed file using the native libraryMajorcommon, ioEli AcherkanEli Acherkan
HDFS-11818TestBlockManager.testSufficientlyReplBlocksUsesNewRack fails intermittentlyMajor.Eric BadgerNathan Roberts
YARN-6598History server getApplicationReport NPE when fetching report for pre-2.8 jobBlockertimelineserverJason LoweJason Lowe
YARN-6603NPE in RMAppsBlockMajorresourcemanagerJason LoweJason Lowe
HDFS-11833HDFS architecture documentation describes outdated placement policyMinor.dudChen Liang
HADOOP-14412HostsFileReader#getHostDetails is very expensive on large clustersMajorutilJason LoweJason Lowe
YARN-6577Remove unused ContainerLocalization classesMinornodemanagerZhangBing LinZhangBing Lin
YARN-6618TestNMLeveldbStateStoreService#testCompactionCycle can fail if compaction occurs more than onceMinortestJason LoweJason Lowe
HDFS-11849JournalNode startup failure exception should be logged in log fileMajorjournal-nodeSurendra Singh LilhoreSurendra Singh Lilhore
HDFS-11864Document Metrics to track usage of memory for writesMajordocumentationBrahma Reddy BattulaYiqun Lin
YARN-6615AmIpFilter drops query parameters on redirectMajor.Wilfred SpiegelenburgWilfred Spiegelenburg
HADOOP-14449The ASF Header in ComparableVersion.java and SSLHostnameVerifier.java is not correctMinorcommon, documentationZhangBing LinZhangBing Lin
HADOOP-14166Reset the DecayRpcScheduler AvgResponseTime metric to zero when queue is not usedMajorcommonSurendra Singh LilhoreSurendra Singh Lilhore
HDFS-11661GetContentSummary uses excessive amounts of memoryBlockernamenodeNathan RobertsWei-Chiu Chuang
YARN-6141ppc64le on Linux doesn't trigger __linux get_executable codepathMajornodemanagerSonia GarudiAyappan
HDFS-11445FSCK shows overall health status as corrupt even one replica is corruptCritical.Brahma Reddy BattulaBrahma Reddy Battula
YARN-6643TestRMFailover fails rarely due to port conflictMajortestRobert KanterRobert Kanter
HDFS-11817A faulty node can cause a lease leak and NPE on accessing dataCritical.Kihwal LeeKihwal Lee
YARN-6641Non-public resource localization on a bad disk causes subsequent containers failureMajor.Kuhu ShuklaKuhu Shukla
HDFS-11078Fix NPE in LazyPersistFileScrubberMajor.Íñigo GoiriÍñigo Goiri
HADOOP-14464hadoop-aws doc header warning #5 line wrappedTrivialdocumentation, fs/s3John ZhugeJohn Zhuge
HDFS-5042Completed files lost after power failureCritical.Dave LathamVinayakumar B
YARN-6649RollingLevelDBTimelineServer throws RuntimeException if object decoding ever fails runtime exceptionCritical.Jonathan EaglesJonathan Eagles
HDFS-11893Fix TestDFSShell.testMoveWithTargetPortEmpty failure.MajortestKonstantin ShvachkoBrahma Reddy Battula
HDFS-11741Long running balancer may fail due to expired DataEncryptionKeyMajorbalancer & moverWei-Chiu ChuangWei-Chiu Chuang
HDFS-11856Ability to re-add Upgrading Nodes (remote) to pipeline for future pipeline updatesMajorhdfs-client, rolling upgradesVinayakumar BVinayakumar B
HADOOP-14474Use OpenJDK 7 instead of Oracle JDK 7 to avoid oracle-java7-installer failuresMajorbuildAkira AjisakaAkira Ajisaka
HDFS-10816TestComputeInvalidateWork#testDatanodeReRegistration fails due to race between test and replication monitorMajor.Eric BadgerEric Badger
HDFS-11932BPServiceActor thread name is not correctly setMajorhdfsChen LiangChen Liang
HDFS-11708Positional read will fail if replicas moved to different DNs after stream is openedCriticalhdfs-clientVinayakumar BVinayakumar B
HDFS-11711DN should not delete the block On “Too many open files” ExceptionCriticaldatanodeBrahma Reddy BattulaBrahma Reddy Battula
MAPREDUCE-6676NNBench should Throw IOException when rename and delete failsMajor.Brahma Reddy BattulaBrahma Reddy Battula
HADOOP-14208Fix typo in the top page in branch-2.8Trivialcommon, documentationAkira AjisakaWenxin He
HDFS-11945Internal lease recovery may not be retried for a long timeMajornamenodeKihwal LeeKihwal Lee
HADOOP-14511WritableRpcEngine.Invocation#toString NPE on null parametersMinoripcJohn ZhugeJohn Zhuge
HADOOP-14512WASB atomic rename should not throw exception if the file is neither in src nor in dst when doing the renameMajorfs/azureDuo XuDuo Xu
YARN-6585RM fails to start when upgrading from 2.7 to 2.8 for clusters with node labels.Blocker.Eric PayneSunil G
HDFS-11967TestJMXGet fails occasionallyMajor.Arpit AgarwalArpit Agarwal
YARN-6719Fix findbugs warnings in SLSCapacityScheduler.javaMajor.Akira AjisakaAkira Ajisaka
HADOOP-14540Replace MRv1 specific terms in HostsFileReaderMinordocumentationAkira Ajisakahu xiaodong
HDFS-11995HDFS Architecture documentation incorrectly describes writing to a local temporary file.MinordocumentationChris NaurothNanda kumar
HDFS-11736OIV tests should not write outside ‘target’ directory.Major.Konstantin ShvachkoYiqun Lin
YARN-6713Fix dead link in the Javadoc of FairSchedulerEventLog.javaMinordocumentationAkira AjisakaWeiwei Yang
HADOOP-14533Size of args cannot be less than zero in TraceAdmin#run as its linkedlistTrivialcommon, tracingWeisen HanWeisen Han
HDFS-11960Successfully closed files can stay under-replicated.Critical.Kihwal LeeKihwal Lee
HADOOP-14146KerberosAuthenticationHandler should authenticate with SPN in AP-REQMajorsecurityDaryn SharpDaryn Sharp
YARN-6467CSQueueMetrics needs to update the current metrics for default partition onlyMajorcapacity schedulerNaganarasimha G RManikandan R
HADOOP-14024KMS JMX endpoint throws ClassNotFoundExceptionCriticalkmsAndrew WangJohn Zhuge
YARN-6749TestAppSchedulingInfo.testPriorityAccounting fails consistentlyMajor.Eric BadgerNaganarasimha G R
MAPREDUCE-6905Fix meaningless operations in TestDFSIO in some situation.Majortools/rumenLiXin GeLiXin Ge
MAPREDUCE-6909LocalJobRunner fails when run on a node from multiple usersBlockerclientJason LoweJason Lowe
HADOOP-13414Hide Jetty Server version header in HTTP responsesMajorsecurityVinayakumar BSurendra Singh Lilhore
HDFS-12089Fix ambiguous NN retry log messageMajorwebhdfsEric BadgerEric Badger
MAPREDUCE-6911TestMapreduceConfigFields.testCompareXmlAgainstConfigurationClass fails consistentlyMajor.Eric BadgerEric Badger
YARN-6708Nodemanager container crash after ext3 folder limitCritical.Bibin A ChundattBibin A Chundatt
HADOOP-14563LoadBalancingKMSClientProvider#warmUpEncryptedKeys swallows IOExceptionMajor.Rushabh S ShahRushabh S Shah
MAPREDUCE-6246DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2Majormrv1, mrv2ramtinGergely Novák
YARN-6428Queue AM limit is not honored in CS alwaysMajor.Bibin A ChundattBibin A Chundatt
YARN-6770[Docs] A small mistake in the example of TimelineClientTrivialdocsJinjiang LingJinjiang Ling
HADOOP-10829Iteration on CredentialProviderFactory.serviceLoader is thread-unsafeMajorsecurityBenoy AntonyBenoy Antony
YARN-6809Fix typo in ResourceManagerHA.mdTrivialdocumentationAkira AjisakaYeliang Cang
MAPREDUCE-5621mr-jobhistory-daemon.sh doesn't have to execute mkdir and chown all the timeMinorjobhistoryserverShinichi YamashitaShinichi Yamashita
YARN-6797TimelineWriter does not fully consume the POST responseMajortimelineclientJason LoweJason Lowe
HDFS-11502Datanode UI should display hostname based on JMX bean instead of window.location.hostnameMajorhdfsJeffrey E RodriguezJeffrey E Rodriguez
HADOOP-14646FileContextMainOperationsBaseTest#testListStatusFilterWithSomeMatches never runsMinortestAndras BokorAndras Bokor
MAPREDUCE-6697Concurrent task limits should only be applied when necessaryMajormrv2Jason LoweNathan Roberts
YARN-6654RollingLevelDBTimelineStore backwards incompatible after fst upgradeBlocker.Jonathan EaglesJonathan Eagles
YARN-6805NPE in LinuxContainerExecutor due to null PrivilegedOperationException exit codeMajornodemanagerJason LoweJason Lowe
YARN-3260AM attempt fail to register before RM processes launch eventCriticalresourcemanagerJason LoweBibin A Chundatt
HDFS-12140Remove BPOfferService lock contention to get block pool idCriticaldatanodeDaryn SharpDaryn Sharp
HDFS-12112TestBlockManager#testBlockManagerMachinesArray sometimes fails with NPEMinor.Wei-Chiu ChuangWei-Chiu Chuang
MAPREDUCE-6910MapReduceTrackingUriPlugin can not return the right URI of history server with HTTPSMajorjobhistoryserverLantao JinLantao Jin
HDFS-12158Secondary Namenode's web interface lack configs for X-FRAME-OPTIONS protectionMajornamenodeMukul Kumar SinghMukul Kumar Singh
YARN-6837Null LocalResource visibility or resource type can crash the nodemanagerMajor.Jinjiang LingJinjiang Ling
HDFS-11472Fix inconsistent replica size after a data pipeline failureCriticaldatanodeWei-Chiu ChuangErik Krogen
HDFS-12177NameNode exits due to setting BlockPlacementPolicy loglevel to DebugMajorblock placementJiandan YangJiandan Yang
HDFS-11742Improve balancer usability after HDFS-8818Blocker.Kihwal LeeKihwal Lee
HDFS-11896Non-dfsUsed will be doubled on dead node re-registrationBlocker.Brahma Reddy BattulaBrahma Reddy Battula
YARN-5728TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization timeoutMajortestAkira AjisakaAkira Ajisaka
YARN-6628Unexpected jackson-core-2.2.3 dependency introducedBlockertimelineserverJason LoweJonathan Eagles
YARN-5731Preemption calculation is not accurate when reserved containers are present in queue.Majorcapacity schedulerSunil GWangda Tan
HADOOP-14683FileStatus.compareTo binary compatible issueBlocker.Sergey ShelukhinAkira Ajisaka
YARN-6872Ensure apps could run given NodeLabels are disabled post RM switchover/restartMajorresourcemanagerSunil GSunil G
YARN-6846Nodemanager can fail to fully delete application local directories when applications are killedCriticalnodemanagerJason LoweJason Lowe
MAPREDUCE-6927MR job should only set tracking url if history was successfully writtenMajor.Eric BadgerEric Badger
YARN-6890If UI is not secured, we allow user to kill other users' job even yarn cluster is secured.Critical.Sumana SathishJunping Du
HDFS-10326Disable setting tcp socket send/receive buffers for write pipelinesMajordatanode, hdfsDaryn SharpDaryn Sharp
HDFS-12157Do fsyncDirectory(..) outside of FSDataset lockCriticaldatanodeVinayakumar BVinayakumar B
HDFS-12278LeaseManager operations are inefficient in 2.8.BlockernamenodeRushabh S ShahRushabh S Shah
YARN-6987Log app attempt during InvalidStateTransitionMajor.Jonathan EaglesJonathan Eagles
YARN-7020TestAMRMProxy#testAMRMProxyTokenRenewal is flakeyMajor.Robert KanterRobert Kanter
YARN-2416InvalidStateTransitonException in ResourceManager if AMLauncher does not receive response for startContainers() call in timeCriticalresourcemanagerJian FangJonathan Eagles
YARN-7048Fix tests faking kerberos to explicitly set ugi auth typeMajoryarnDaryn SharpDaryn Sharp
HADOOP-14687AuthenticatedURL will reuse bad/expired session cookiesCriticalcommonDaryn SharpDaryn Sharp
YARN-6640AM heartbeat stuck when responseId overflows MAX_INTBlocker.Botong HuangBotong Huang
HDFS-12319DirectoryScanner will throw IllegalStateException when Multiple BP's are presentBlocker.Brahma Reddy BattulaBrahma Reddy Battula
HDFS-12299Race Between update pipeline and DN Re-RegistrationCritical.Brahma Reddy BattulaBrahma Reddy Battula
YARN-7052RM SchedulingMonitor gives no indication why the spawned thread crashed.CriticalyarnEric PayneEric Payne
YARN-7087NM failed to perform log aggregation due to absent containerBlockerlog-aggregationJason LoweJason Lowe
YARN-7051Avoid concurrent modification exception in FifoIntraQueuePreemptionPluginCriticalcapacity scheduler, scheduler preemption, yarnEric PayneEric Payne
HDFS-12364[branch-2.8.2] Fix the Compile Error after HDFS-12299BlockerhdfsJiandan YangJiandan Yang
YARN-7112TestAMRMProxy is failing with invalid requestMajor.Jason LoweJason Lowe
YARN-7076yarn application -list -appTypes <appType> is not workingBlocker.Jian HeJian He
MAPREDUCE-6641TestTaskAttempt fails in trunkMajortestTsuyoshi OzawaHaibo Chen
YARN-7083Log aggregation deletes/renames while file is openCriticalnodemanagerDaryn SharpJason Lowe
HADOOP-14814Fix incompatible API change on FsServerDefaults to HADOOP-14104Blocker.Junping DuJunping Du
YARN-7023Incorrect ReservationId.compareTo() implementationMinorreservation systemOleg DanilovOleg Danilov
HADOOP-14842Hadoop 2.8.2 release build process get stuck due to java issueBlockerbuildJunping DuJunping Du
YARN-5195RM intermittently crashed with NPE while handling APP_ATTEMPT_REMOVED event when async-scheduling enabled in CapacitySchedulerMajorresourcemanagerKaram Singhsandflee
YARN-7034DefaultLinuxContainerRuntime and DockerLinuxContainerRuntime sends client environment variables to container-executorBlockernodemanagerMiklos SzegediMiklos Szegedi
YARN-7249Fix CapacityScheduler NPE issue when a container preempted while the node is being removedBlocker.Wangda TanWangda Tan
YARN-7325Remove unused container variable in DockerLinuxContainerRuntimeMinornodemanagerShane KumpfShane Kumpf
YARN-7246Fix the default docker binary pathBlockernodemanagerShane KumpfShane Kumpf
YARN-7333container-executor fails to remove entries from a directory that is not writable or executableCritical.Jason LoweJason Lowe
YARN-7230Document DockerContainerRuntime for branch-2.8 with proper scope and claim as an experimental featureBlockerdocumentationJunping DuShane Kumpf
HADOOP-14958CLONE - Fix source-level compatibility after HADOOP-11252Blocker.Junping DuJunping Du

TESTS:

JIRASummaryPriorityComponentReporterContributor
HDFS-9300TestDirectoryScanner.testThrottle() is still a little flakeyMajorbalancer & mover, testDaniel TempletonDaniel Templeton
HADOOP-13178TestShellBasedIdMapping.testStaticMapUpdate doesn't work on OS XMajortestAllen WittenauerKai Sasaki
YARN-5343TestContinuousScheduling#testSortedNodes fails intermittentlyMinor.sandfleeYufei Gu
MAPREDUCE-6831Flaky test TestJobImpl.testKilledDuringKillAbortMajormrv2Peter BacskoPeter Bacsko
HDFS-11290TestFSNameSystemMBean should wait until JMX cache is clearedMajortestAkira AjisakaErik Krogen
YARN-5349TestWorkPreservingRMRestart#testUAMRecoveryOnRMWorkPreservingRestart fail intermittentlyMinor.sandfleeJason Lowe

SUB-TASKS:

JIRASummaryPriorityComponentReporterContributor
HDFS-9754Avoid unnecessary getBlockCollection calls in BlockManagerMajornamenodeJing ZhaoJing Zhao
HADOOP-14032Reduce fair call queue priority inversionMajoripcDaryn SharpDaryn Sharp
HADOOP-14034Allow ipc layer exceptions to selectively close connectionsMajoripcDaryn SharpDaryn Sharp
HADOOP-14033Reduce fair call queue lock contentionMajoripcDaryn SharpDaryn Sharp
HADOOP-14135Remove URI parameter in AWSCredentialProvider constructorsMajorfs/s3Mingliang LiuMingliang Liu
HADOOP-14196Azure Data Lake doc is missing required config entryMajorfs/adlAtul SikariaAtul Sikaria
HADOOP-14197Fix ADLS doc for credential providerMajordocumentation, fs/adlJohn ZhugeJohn Zhuge
HADOOP-14230TestAdlFileSystemContractLive fails to clean upMinorfs/adl, testJohn ZhugeJohn Zhuge
HADOOP-14038Rename ADLS credential propertiesMinorfs/adlJohn ZhugeJohn Zhuge
HADOOP-14321Explicitly exclude S3A root dir ITests from parallel runsMinorfs/s3, testSteve LoughranSteve Loughran
HADOOP-14241Add ADLS sensitive config keys to default listMinorfs, fs/adl, securityJohn ZhugeJohn Zhuge
HADOOP-14349Rename ADLS CONTRACT_ENABLE_KEYMinorfs/adlMingliang LiuMingliang Liu
HDFS-9005Provide configuration support for upgrade domainMajor.Ming MaMing Ma
HDFS-9016Display upgrade domain information in fsckMajor.Ming MaMing Ma
HDFS-9922Upgrade Domain placement policy status marks a good block in violation when there are decommissioned nodesMinor.Chris TrezzoChris Trezzo
HADOOP-14035Reduce fair call queue backoff's impact on clientsMajoripcDaryn SharpDaryn Sharp
YARN-6682Improve performance of AssignmentInformation datastructuresMajor.Daryn SharpDaryn Sharp
YARN-6680Avoid locking overhead for NO_LABEL lookupsMajorresourcemanagerDaryn SharpDaryn Sharp
YARN-6681Eliminate double-copy of child queues in canAssignToThisQueueMajorresourcemanagerDaryn SharpDaryn Sharp
YARN-2113Add cross-user preemption within CapacityScheduler's leaf-queueMajorcapacity schedulerVinod Kumar VavilapalliSunil G
YARN-6775CapacityScheduler: Improvements to assignContainers, avoid unnecessary canAssignToUser/Queue callsMajorcapacityschedulerNathan RobertsNathan Roberts
YARN-6988container-executor fails for docker when command length > 4096 BMajoryarnEric BadgerEric Badger
HDFS-12473Change hosts JSON file formatMajor.Ming MaMing Ma

OTHER:

JIRASummaryPriorityComponentReporterContributor
HADOOP-14344Revert HADOOP-13606 swift FS to add a service load metadata fileMajor.John ZhugeJohn Zhuge
HDFS-11717Add unit test for HDFS-11709 StandbyCheckpointer should handle non-existing legacyOivImageDir gracefullyMinorha, namenodeErik KrogenErik Krogen