“Apache Hadoop” Changelog

Release 2.10.0 - 2019-10-22

INCOMPATIBLE CHANGES:

JIRASummaryPriorityComponentReporterContributor
HDFS-12883RBF: Document Router and State Store metricsMajordocumentationYiqun LinYiqun Lin
HDFS-12895RBF: Add ACL support for mount tableMajor.Yiqun LinYiqun Lin
HDFS-13099RBF: Use the ZooKeeper as the default State StoreMinordocumentationYiqun LinYiqun Lin
HADOOP-16055Upgrade AWS SDK to 1.11.271 in branch-2Blockerfs/s3Akira AjisakaAkira Ajisaka
HADOOP-16053Backport HADOOP-14816 to branch-2MajorbuildAkira AjisakaAkira Ajisaka

IMPORTANT ISSUES:

JIRASummaryPriorityComponentReporterContributor
HDFS-13083RBF: Fix doc error setting up clientMajorfederationtartarustartarus

NEW FEATURES:

JIRASummaryPriorityComponentReporterContributor
HDFS-13283Percentage based Reserved Space Calculation for DataNodeMajordatanode, hdfsLukas MajercakLukas Majercak
HDFS-13553RBF: Support global quotaMajor.Íñigo GoiriYiqun Lin
HADOOP-15950Failover for LdapGroupsMappingMajorcommon, securityLukas MajercakLukas Majercak
YARN-9761Allow overriding application submissions based on server side configsMajor.Jonathan Hungpralabhkumar
YARN-9760Support configuring application priorities on a workflow levelMajor.Jonathan HungVarun Saxena

IMPROVEMENTS:

JIRASummaryPriorityComponentReporterContributor
HADOOP-14987Improve KMSClientProvider log around delegation token checkingMajor.Xiaoyu YaoXiaoyu Yao
HADOOP-14872CryptoInputStream should implement unbufferMajorfs, securityJohn ZhugeJohn Zhuge
HADOOP-14960Add GC time percentage monitor/alerterMajor.Misha DmitrievMisha Dmitriev
HADOOP-15023ValueQueue should also validate (lowWatermark * numValues) > 0 on constructionMinor.Xiao ChenXiao Chen
YARN-6851Capacity Scheduler: document configs for controlling # containers allowed to be allocated per node heartbeatMinor.Wei YanWei Yan
YARN-7495Improve robustness of the AggregatedLogDeletionServiceMajorlog-aggregationJonathan Turner EaglesJonathan Turner Eagles
HADOOP-15056Fix TestUnbuffer#testUnbufferException failureMinortestJack BeardenJack Bearden
HADOOP-15012Add readahead, dropbehind, and unbuffer to StreamCapabilitiesMajorfsJohn ZhugeJohn Zhuge
HADOOP-15104AliyunOSS: change the default value of max error retryMajorfs/osswujinhuwujinhu
YARN-7274Ability to disable elasticity at leaf queue levelMajorcapacityschedulerScott BrokawZian Chen
YARN-7642Add test case to verify context update after container promotion or demotion with or without auto updateMinornodemanagerWeiwei YangWeiwei Yang
HADOOP-15111AliyunOSS: backport HADOOP-14993 to branch-2Majorfs/ossGenmao YuGenmao Yu
HDFS-12818Support multiple storages in DataNodeCluster / SimulatedFSDatasetMinordatanode, testErik KrogenErik Krogen
HDFS-9023When NN is not able to identify DN for replication, reason behind it can be loggedCriticalhdfs-client, namenodeSurendra Singh LilhoreXiao Chen
YARN-7678Ability to enable logging of container memory statsMajornodemanagerJim BrennanJim Brennan
HDFS-12945Switch to ClientProtocol instead of NamenodeProtocols in NamenodeWebHdfsMethodsMinor.Wei YanWei Yan
YARN-7622Allow fair-scheduler configuration on HDFSMinorfairscheduler, resourcemanagerGreg PhillipsGreg Phillips
YARN-7590Improve container-executor validation checkMajorsecurity, yarnEric YangEric Yang
MAPREDUCE-7029FileOutputCommitter is slow on filesystems lacking recursive deleteMinor.Karthik PalaniappanKarthik Palaniappan
MAPREDUCE-6984MR AM to clean up temporary files from previous attempt in case of no recoveryMajorapplicationmasterGergo RepasGergo Repas
HADOOP-15189backport HADOOP-15039 to branch-2 and branch-3Blocker.Genmao YuGenmao Yu
HADOOP-15212Add independent secret manager method for logging expired tokensMajorsecurityDaryn SharpDaryn Sharp
YARN-7728Expose container preemptions related information in Capacity Scheduler queue metricsMajor.Eric PayneEric Payne
MAPREDUCE-7048Uber AM can crash due to unknown task in statusUpdateMajormr-amPeter BacskoPeter Bacsko
HADOOP-13972ADLS to support per-store configurationMajorfs/adlJohn ZhugeSharad Sonker
YARN-7813Capacity Scheduler Intra-queue Preemption should be configurable for each queueMajorcapacity scheduler, scheduler preemptionEric PayneEric Payne
HADOOP-15235Authentication Tokens should use HMAC instead of MACMajorsecurityRobert KanterRobert Kanter
HDFS-11187Optimize disk access for last partial chunk checksum of Finalized replicaMajordatanodeWei-Chiu ChuangGabor Bota
HADOOP-15266[branch-2] Upper/Lower case conversion support for group names in LdapGroupsMappingMajor.Nanda kumarNanda kumar
HADOOP-15279increase maven heap size recommendationsMinorbuild, documentation, testAllen WittenauerAllen Wittenauer
HDFS-12884BlockUnderConstructionFeature.truncateBlock should be of type BlockInfoMajornamenodeKonstantin Shvachkochencan
HADOOP-15334Upgrade Maven surefire pluginMajorbuildArpit AgarwalArpit Agarwal
HADOOP-15312Undocumented KeyProvider configuration keysMajor.Wei-Chiu ChuangLiXin Ge
YARN-7623Fix the CapacityScheduler Queue configuration documentationMajor.Arun SureshJonathan Hung
HDFS-13314NameNode should optionally exit if it detects FsImage corruptionMajornamenodeArpit AgarwalArpit Agarwal
HDFS-13418NetworkTopology should be configurable when enable DFSNetworkTopologyMajor.Tao JieTao Jie
HADOOP-15394Backport PowerShell NodeFencer HADOOP-14309 to branch-2Minor.Íñigo GoiriÍñigo Goiri
HDFS-13462Add BIND_HOST configuration for JournalNode's HTTP and RPC ServersMajorhdfs, journal-nodeLukas MajercakLukas Majercak
HDFS-13492Limit httpfs binds to certain IP addresses in branch-2MajorhttpfsWei-Chiu ChuangWei-Chiu Chuang
HDFS-12981renameSnapshot a Non-Existent snapshot to itself should throw errorMinorhdfsSailesh PatelKitti Nanasi
HADOOP-15441Log kms url and token service at debug level.Minor.Wei-Chiu ChuangGabor Bota
HDFS-13544Improve logging for JournalNode in federated clusterMajorfederation, hdfsHanisha KoneruHanisha Koneru
YARN-8249Few REST api's in RMWebServices are missing static user checkCriticalwebapp, yarnSunil GSunil G
HADOOP-15486Make NetworkTopology#netLock fairMajornetNanda kumarNanda kumar
HADOOP-15449Increase default timeout of ZK session to avoid frequent NameNode failoverCriticalcommonKarthik PalanisamyKarthik Palanisamy
HDFS-13602Add checkOperation(WRITE) checks in FSNamesystemMajorha, namenodeErik KrogenChao Sun
HDFS-13644Backport HDFS-10376 to branch-2Major.Yiqun LinZsolt Venczel
HDFS-13653Make dfs.client.failover.random.order a per nameservice configurationMajorfederationEkanth SethuramalingamEkanth Sethuramalingam
HDFS-13686Add overall metrics for FSNamesystemLockMajorhdfs, namenodeLukas MajercakLukas Majercak
HDFS-13714Fix TestNameNodePrunesMissingStorages test failures on WindowsMajorhdfs, namenode, testLukas MajercakLukas Majercak
HDFS-13719Docs around dfs.image.transfer.timeout are misleadingMajordocumentationKitti NanasiKitti Nanasi
HDFS-11060make DEFAULT_MAX_CORRUPT_FILEBLOCKS_RETURNED configurableMinorhdfsLantao JinLantao Jin
YARN-8155Improve ATSv2 client logging in RM and NM publisherMajor.Rohith Sharma K SAbhishek Modi
HDFS-13814Remove super user privilege requirement for NameNode.getServiceStatusMinornamenodeChao SunChao Sun
YARN-8559Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpointMajorresourcemanagerAnna SavarinWeiwei Yang
HDFS-13813Exit NameNode if dangling child inode is detected when saving FsImageMajorhdfs, namenodeSiyao MengSiyao Meng
HDFS-13821RBF: Add dfs.federation.router.mount-table.cache.enable so that users can disable cacheMajorhdfsFei HuiFei Hui
HADOOP-15689Add “*.patch” into .gitignore file of branch-2Major.Rui GaoRui Gao
HDFS-13831Make block increment deletion number configurableMajor.Yiqun LinRyan Wu
YARN-8051TestRMEmbeddedElector#testCallbackSynchronization is flakeyMajortestRobert KanterRobert Kanter
HADOOP-15547WASB: improve listStatus performanceMajorfs/azureThomas MarqardtThomas Marqardt
HDFS-13857RBF: Choose to enable the default nameservice to read/write filesMajorfederation, hdfsyanghuafengyanghuafeng
HDFS-13812Fix the inconsistent default refresh interval on Caching documentationTrivialdocumentationDavid MollitorHrishikesh Gadre
HADOOP-15657Registering MutableQuantiles via Metric annotationMajormetricsSushil KsSushil Ks
HDFS-13902Add JMX, conf and stacks menus to the datanode pageMinordatanodefengchuangfengchuang
HADOOP-15726Create utility to limit frequency of log statementsMajorcommon, utilErik KrogenErik Krogen
YARN-7974Allow updating application tracking url after registrationMajor.Jonathan HungJonathan Hung
YARN-8750Refactor TestQueueMetricsMinorresourcemanagerSzilard NemethSzilard Nemeth
YARN-8896Limit the maximum number of container assignments per heartbeatMajor.Weiwei YangZhankun Tang
HADOOP-15804upgrade to commons-compress 1.18Major.PJ FanningAkira Ajisaka
YARN-8915Update the doc about the default value of “maximum-container-assignments” for capacity schedulerMinor.Zhankun TangZhankun Tang
YARN-7225Add queue and partition info to RM audit logMajorresourcemanagerJonathan HungEric Payne
HADOOP-15919AliyunOSS: Enable Yarn to use OSSMajorfs/osswujinhuwujinhu
HADOOP-15943AliyunOSS: add missing owner & group attributes for oss FileStatusMajorfs/osswujinhuwujinhu
YARN-9036Escape newlines in health report in YARN UIMajor.Jonathan HungKeqiu Hu
YARN-9085Add Guaranteed and MaxCapacity to CSQueueMetricsMajor.Jonathan HungJonathan Hung
HDFS-14171Performance improvement in Tailing EditLogMajornamenodeKenneth YangKenneth Yang
HADOOP-15481Emit FairCallQueue stats as metricsMajormetrics, rpc-serverErik KrogenChristopher Gregorian
HADOOP-15617Node.js and npm package loading in the Dockerfile failing on branch-2MajorbuildAllen WittenauerAkhil PB
HADOOP-16089AliyunOSS: update oss-sdk version to 3.4.1Majorfs/osswujinhuwujinhu
YARN-7171RM UI should sort memory / cores numericallyMajor.Eric MaynardAhmed Hussein
YARN-9282Typo in javadoc of class LinuxContainerExecutor: hadoop.security.authetication should be ‘authentication’Trivial.Szilard NemethCharan Hebri
HADOOP-16126ipc.Client.stop() may sleep too long to wait for all connectionsMajoripcTsz-wo SzeTsz-wo Sze
HDFS-14247Repeat adding node description into network topologyMinordatanodeHuangTaoHuangTao
YARN-9150Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2MajorATSv2Sushil KsSushil Ks
HDFS-14366Improve HDFS append performanceMajorhdfsChao SunChao Sun
HDFS-14205Backport HDFS-6440 to branch-2Major.Chen LiangChao Sun
HDFS-14391Backport HDFS-9659 to branch-2Minor.Chao SunChao Sun
HDFS-14415Backport HDFS-13799 to branch-2Trivial.Chao SunChao Sun
HDFS-14432dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xmlMinorhdfspuleya7puleya7
YARN-9529Log correct cpu controller path on error while initializing CGroups.MajornodemanagerJonathan HungJonathan Hung
HDFS-14502keepResults option in NNThroughputBenchmark should call saveNamespace()Majorbenchmarks, hdfsKonstantin ShvachkoKonstantin Shvachko
HADOOP-16323https everywhere in Maven settingsMinorbuildAkira AjisakaAkira Ajisaka
YARN-9563Resource report REST API could return NaN or InfMinor.Ahmed HusseinAhmed Hussein
HDFS-14513FSImage which is saving should be clean while NameNode shutdownMajornamenodeXiaoqiao HeXiaoqiao He
HADOOP-16369Fix zstandard shortname misspelled as ztsMajor.Jonathan Turner EaglesJonathan Turner Eagles
HADOOP-16359Bundle ZSTD native in branch-2MajornativeChao SunChao Sun
HADOOP-15253Should update maxQueueSize when refresh call queueMinor.Tao JieTao Jie
HADOOP-16266Add more fine-grained processing time metrics to the RPC layerMinoripcChristopher GregorianErik Krogen
HDFS-13694Making md5 computing being in parallel with image loadingMajor.zhouyingchaoLisheng Sun
HDFS-14632Reduce useless #getNumLiveDataNodes call in SafeModeMonitorMajornamenodeXiaoqiao HeXiaoqiao He
HDFS-14547DirectoryWithQuotaFeature.quota costs additional memory even the storage type quota is not set.Major.JinglunJinglun
HDFS-14697Backport HDFS-14513 to branch-2MinornamenodeXiaoqiao HeXiaoqiao He
YARN-8045Reduce log output from container status callsMajor.Shane KumpfCraig Condit
HDFS-14313Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/duMajordatanode, performanceLisheng SunLisheng Sun
HDFS-14696Backport HDFS-11273 to branch-2 (Move TransferFsImage#doGetUrl function to a Util class)Major.Siyao MengSiyao Meng
HDFS-14370Edit log tailing fast-path should allow for backoffMajornamenode, qjmErik KrogenErik Krogen
YARN-9442container working directory has group read permissionsMinoryarnJim BrennanJim Brennan
HADOOP-16459Backport [HADOOP-16266] “Add more fine-grained processing time metrics to the RPC layer” to branch-2Major.Erik KrogenErik Krogen
HDFS-14707Add JAVA_LIBRARY_PATH to HTTPFS startup options in branch-2MajorhttpfsMasatake IwasakiMasatake Iwasaki
HDFS-14723Add helper method FSNamesystem#setBlockManagerForTesting() in branch-2BlockernamenodeWei-Chiu ChuangWei-Chiu Chuang
HDFS-14276[SBN read] Reduce tailing overheadMajorha, namenodeWei-Chiu ChuangAyush Saxena
HDFS-14617Improve fsimage load time by writing sub-sections to the fsimage indexMajornamenodeStephen O'DonnellStephen O'Donnell
YARN-9756Create metric that sums total memory/vcores preempted per roundMajorcapacity schedulerEric PayneManikandan R
HDFS-14633The StorageType quota and consume in QuotaFeature is not handled for renameMajor.JinglunJinglun
HADOOP-16439Upgrade bundled Tomcat in branch-2Majorhttpfs, kmsMasatake IwasakiMasatake Iwasaki
YARN-9810Add queue capacity/maxcapacity percentage metricsMajor.Jonathan HungShubham Gupta
YARN-9763Print application tags in application summaryMajor.Jonathan HungManoj Kumar
YARN-9764Print application submission context label in application summaryMajor.Jonathan HungManoj Kumar
HADOOP-16530Update xercesImpl in branch-2Major.Wei-Chiu ChuangWei-Chiu Chuang
YARN-9824Fall back to configured queue ordering policy class nameMajor.Jonathan HungJonathan Hung
YARN-9825Changes for initializing placement rules with ResourceScheduler in branch-2Major.Jonathan HungJonathan Hung
YARN-9762Add submission context label to audit logsMajor.Jonathan HungManoj Kumar
HDFS-14667Backport [HDFS-14403] “Cost-based FairCallQueue” to branch-2Major.Erik KrogenErik Krogen

BUG FIXES:

JIRASummaryPriorityComponentReporterContributor
MAPREDUCE-6896Document wrong spelling in usage of MapredTestDriver tools.MajordocumentationLiXin GeLiXin Ge
HDFS-12052Set SWEBHDFS delegation token kind when ssl is enabled in HttpFSMajorhttpfs, webhdfsZoran DimitrijevicZoran Dimitrijevic
HDFS-12318Fix IOException condition for openInfo in DFSInputStreamMajor.legendlegend
HDFS-12614FSPermissionChecker#getINodeAttrs() throws NPE when INodeAttributesProvider configuredMajor.Manoj GovindassamyManoj Govindassamy
YARN-7396NPE when accessing container logs due to null dirsHandlerMajor.Jonathan HungJonathan Hung
YARN-7370Preemption properties should be refreshableMajorcapacity scheduler, scheduler preemptionEric PayneGergely Novák
YARN-7428Add containerId to Localizer failed logsMinornodemanagerPrabhu JosephPrabhu Joseph
YARN-7410Cleanup FixedValueResource to avoid dependency to ResourceUtilsMajorresourcemanagerSunil GWangda Tan
HDFS-12783[branch-2] “dfsrouter” should use hdfsScriptMajor.Brahma Reddy BattulaBrahma Reddy Battula
HDFS-12788Reset the upload button when file upload failsCriticalui, webhdfsBrahma Reddy BattulaBrahma Reddy Battula
YARN-7469Capacity Scheduler Intra-queue preemption: User can starve if newest app is exactly at user limitMajorcapacity scheduler, yarnEric PayneEric Payne
HADOOP-14982Clients using FailoverOnNetworkExceptionRetry can go into a loop if they're used without authenticating with kerberos in HA envMajorcommonPeter BacskoPeter Bacsko
YARN-7489ConcurrentModificationException in RMAppImpl#getRMAppMetricsMajorcapacityschedulerTao YangTao Yang
YARN-7525Incorrect query parameters in cluster nodes REST API documentMinordocumentationTao YangTao Yang
HDFS-12813RequestHedgingProxyProvider can hide Exception thrown from the Namenode for proxy size of 1MajorhaMukul Kumar SinghMukul Kumar Singh
HADOOP-15045ISA-L build options are documented in branch-2Majorbuild, documentationAkira AjisakaAkira Ajisaka
HADOOP-15067GC time percentage reported in JvmMetrics should be a gauge, not counterMajor.Misha DmitrievMisha Dmitriev
YARN-7363ContainerLocalizer doesn't have a valid log4j config when using LinuxContainerExecutorMajornodemanagerYufei GuYufei Gu
HDFS-12754Lease renewal can hit a deadlockMajor.Kuhu ShuklaKuhu Shukla
HDFS-12832INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exitCriticalnamenodeDENG FEIKonstantin Shvachko
HDFS-11754Make FsServerDefaults cache configurable.Minor.Rushabh ShahMikhail Erofeev
HADOOP-15042Azure PageBlobInputStream.skip() can return negative value when numberOfPagesRemaining is 0Minorfs/azureRajesh BalamohanRajesh Balamohan
HDFS-12638Delete copy-on-truncate block along with the original block, when deleting a file being truncatedBlockerhdfsJiandan YangKonstantin Shvachko
YARN-4813TestRMWebServicesDelegationTokenAuthentication.testDoAs fails intermittentlyMajorresourcemanagerDaniel TempletonGergo Repas
MAPREDUCE-5124AM lacks flow control for task eventsMajormr-amJason Darrell LowePeter Bacsko
YARN-7455quote_and_append_arg can overflow bufferMajornodemanagerJason Darrell LoweJim Brennan
YARN-7594TestNMWebServices#testGetNMResourceInfo fails on trunkMajornodemanager, webappGergely NovákGergely Novák
YARN-5594Handle old RMDelegationToken format when recovering RMMajorresourcemanagerTatyana ButRobert Kanter
HADOOP-14985Remove subversion related code from VersionInfoMojo.javaMinorbuildAkira AjisakaAjay Kumar
HDFS-11751DFSZKFailoverController daemon exits with wrong status codeMajorauto-failoverDoris GuBharat Viswanadham
HDFS-12889Router UI is missing robots.txt fileMajor.Bharat ViswanadhamBharat Viswanadham
YARN-7607Remove the trailing duplicated timestamp in container diagnostics messageMinornodemanagerWeiwei YangWeiwei Yang
HADOOP-15080Aliyun OSS: update oss sdk from 2.8.1 to 2.8.3 to remove its dependency on Cat-x “json-lib”Blockerfs/ossChristopher DouglasSammi Chen
YARN-7608Incorrect sTarget column causing DataTable warning on RM application and scheduler web pageMajorresourcemanager, webappWeiwei YangGergely Novák
HDFS-12833Distcp : Update the usage of delete option for dependency with update and overwrite optionMinordistcp, hdfsHarshakiran Reddyusharani
YARN-7647NM print inappropriate error log when node-labels is enabledMinor.Yang WangYang Wang
HDFS-12907Allow read-only access to reserved raw for non-superusersMajornamenodeDaryn SharpRushabh Shah
HDFS-12881Output streams closed with IOUtils suppressing write errorsMajor.Jason Darrell LoweAjay Kumar
YARN-7595Container launching code suppresses close exceptions after writesMajornodemanagerJason Darrell LoweJim Brennan
HADOOP-15085Output streams closed with IOUtils suppressing write errorsMajor.Jason Darrell LoweJim Brennan
HADOOP-15123KDiag tries to load krb5.conf from KRB5CCNAME instead of KRB5_CONFIGMinorsecurityVipin RathorVipin Rathor
YARN-7661NodeManager metrics return wrong value after update node resourceMajor.Yang WangYang Wang
HDFS-12347TestBalancerRPCDelay#testBalancerRPCDelay fails very frequentlyCriticaltestXiao ChenBharat Viswanadham
YARN-7662[Atsv2] Define new set of configurations for reader and collectors to bind.Major.Rohith Sharma K SRohith Sharma K S
YARN-7674Update Timeline Reader web app address in UI2Major.Rohith Sharma K SSunil G
YARN-7542Fix issue that causes some Running Opportunistic Containers to be recovered as PAUSEDMajor.Arun SureshSampada Dehankar
HADOOP-15143NPE due to Invalid KerberosTicket in UGIMajor.Jitendra Nath PandeyMukul Kumar Singh
YARN-7692Skip validating priority acls while recovering applicationsBlockerresourcemanagerCharan HebriSunil G
MAPREDUCE-7028Concurrent task progress updates causing NPE in Application MasterBlockermr-amGergo RepasGergo Repas
YARN-7619Max AM Resource value in Capacity Scheduler UI has to be refreshed for every userMajorcapacity scheduler, yarnEric PayneEric Payne
YARN-7699queueUsagePercentage is coming as INF for getApp REST api callMajorwebappSunil GSunil G
HDFS-12985NameNode crashes during restart after an OpenForWrite file present in the Snapshot got deletedMajorhdfsManoj GovindassamyManoj Govindassamy
YARN-4227Ignore expired containers from removed nodes in FairSchedulerCriticalfairschedulerWilfred SpiegelenburgWilfred Spiegelenburg
YARN-7508NPE in FiCaSchedulerApp when debug log enabled in async-scheduling modeMajorcapacityschedulerTao YangTao Yang
YARN-7663RMAppImpl:Invalid event: START at KILLEDMajorresourcemanagerlujielujie
YARN-6948Invalid event: ATTEMPT_ADDED at FINAL_SAVINGMajoryarnlujielujie
HADOOP-15060TestShellBasedUnixGroupsMapping.testFiniteGroupResolutionTime flakyMajor.Miklos SzegediMiklos Szegedi
YARN-7735Fix typo in YARN documentationMinordocumentationTakanobu AsanumaTakanobu Asanuma
YARN-7727Incorrect log levels in few logs with QueuePriorityContainerCandidateSelectorMinoryarnPrabhu JosephPrabhu Joseph
HDFS-11915Sync rbw dir on the first hsync() to avoid file lost on power failureCritical.Kanaka Kumar AvvaruVinayakumar B
YARN-7705Create the container log directory with correct sticky bit in C codeMajornodemanagerYufei GuYufei Gu
HDFS-9049Make Datanode Netty reverse proxy port to be configurableMajordatanodeVinayakumar BVinayakumar B
YARN-7758Add an additional check to the validity of container and application ids passed to container-executorMajornodemanagerMiklos SzegediYufei Gu
HADOOP-15150in FsShell, UGI params should be overidden through env vars(-D arg)Major.Brahma Reddy BattulaBrahma Reddy Battula
HADOOP-15181Typo in SecureMode.mdTrivialdocumentationMasahiro TanakaMasahiro Tanaka
YARN-7806Distributed Shell should use timeline async api'sMajordistributed-shellSumana SathishRohith Sharma K S
HADOOP-15121Encounter NullPointerException when using DecayRpcSchedulerMajor.Tao JieTao Jie
MAPREDUCE-7015Possible race condition in JHS if the job is not loadedMajorjobhistoryserverPeter BacskoPeter Bacsko
YARN-7737prelaunch.err file not found exception on container failureMajor.Jonathan HungKeqiu Hu
HDFS-13063Fix the incorrect spelling in HDFSHighAvailabilityWithQJM.mdTrivialdocumentationJianfei JiangJianfei Jiang
YARN-7102NM heartbeat stuck when responseId overflows MAX_INTCritical.Botong HuangBotong Huang
MAPREDUCE-7041MR should not try to clean up at first job attemptMajor.Takanobu AsanumaGergo Repas
MAPREDUCE-7020Task timeout in uber mode can crash AMMajormr-amAkira AjisakaPeter Bacsko
YARN-7765[Atsv2] GSSException: No valid credentials provided - Failed to find any Kerberos tgt thrown by Timelinev2Client & HBaseClient in NMBlocker.Sumana SathishRohith Sharma K S
HDFS-12974Exception message is not printed when creating an encryption zone fails with AuthorizationExceptionMinorencryptionfang zhenyifang zhenyi
YARN-7698A misleading variable's name in ApplicationAttemptEventDispatcherMinorresourcemanagerJinjiang LingJinjiang Ling
HDFS-12528Add an option to not disable short-circuit reads on failuresMajorhdfs-client, performanceAndre AraujoXiao Chen
HDFS-13100Handle IllegalArgumentException when GETSERVERDEFAULTS is not implemented in webhdfs.Criticalhdfs, webhdfsYongjun ZhangYongjun Zhang
YARN-7849TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization fails due to heartbeat sync errorMajortestJason Darrell LoweBotong Huang
YARN-7801AmFilterInitializer should addFilter after fill all parametersCritical.Sumana SathishWangda Tan
YARN-7890NPE during container relaunchMajor.Billie RinaldiJason Darrell Lowe
HDFS-13115In getNumUnderConstructionBlocks(), ignore the inodeIds for which the inodes have been deletedMajor.Yongjun ZhangYongjun Zhang
HDFS-12935Get ambiguous result for DFSAdmin command in HA mode when only one namenode is upMajortoolsJianfei JiangJianfei Jiang
HDFS-13120Snapshot diff could be corrupted after concatMajornamenode, snapshotsXiaoyu YaoXiaoyu Yao
HDFS-10453ReplicationMonitor thread could stuck for long time due to the race between replication and delete of same file in a large cluster.MajornamenodeXiaoqiao HeXiaoqiao He
HDFS-8693refreshNamenodes does not support adding a new standby to a running DNCriticaldatanode, haJian FangAjith S
MAPREDUCE-7052TestFixedLengthInputFormat#testFormatCompressedIn is flakyMajorclient, testPeter BacskoPeter Bacsko
HDFS-13112Token expiration edits may cause log corruption or deadlockCriticalnamenodeDaryn SharpDaryn Sharp
MAPREDUCE-7053Timed out tasks can fail to produce thread dumpMajor.Jason Darrell LoweJason Darrell Lowe
HADOOP-15206BZip2 drops and duplicates records when input split size is smallMajor.Aki TanakaAki Tanaka
YARN-7937Fix http method name in Cluster Application Timeout Update API example requestMinordocs, documentationCharan HebriCharan Hebri
YARN-7947Capacity Scheduler intra-queue preemption can NPE for non-schedulable appsMajorcapacity scheduler, scheduler preemptionEric PayneEric Payne
YARN-7945Java Doc error in UnmanagedAMPoolManager for branch-2Major.Rohith Sharma K SBotong Huang
HADOOP-14903Add json-smart explicitly to pom.xmlMajorcommonRay ChiangRay Chiang
HADOOP-15236Fix typo in RequestHedgingProxyProvider and RequestHedgingRMFailoverProxyProviderTrivialdocumentationAkira AjisakaGabor Bota
MAPREDUCE-7027HadoopArchiveLogs shouldn't delete the original logs if the HAR creation failsCriticalharchiveGergely NovákGergely Novák
HDFS-12781After Datanode down, In Namenode UI Datanode tab is throwing warning message.MajordatanodeHarshakiran ReddyBrahma Reddy Battula
HDFS-12070Failed block recovery leaves files open indefinitely and at risk for data lossMajor.Daryn SharpKihwal Lee
HADOOP-15251Backport HADOOP-13514 (surefire upgrade) to branch-2MajortestChristopher DouglasChristopher Douglas
HDFS-13194CachePool permissions incorrectly checkedMajor.Yiqun LinJianfei Jiang
HADOOP-15276branch-2 site not building after ADL troubleshooting doc addedMajordocumentationSteve LoughranSteve Loughran
YARN-7835[Atsv2] Race condition in NM while publishing events if second attempt is launched on the same nodeCritical.Rohith Sharma K SRohith Sharma K S
HADOOP-15275Incorrect javadoc for return type of RetryPolicy#shouldRetryMinordocumentationNanda kumarNanda kumar
YARN-7511NPE in ContainerLocalizer when localization failed for running containerMajornodemanagerTao YangTao Yang
MAPREDUCE-7023TestHadoopArchiveLogs.testCheckFilesAndSeedApps fails on rerunMinortestGergely NovákGergely Novák
HADOOP-15283Upgrade from findbugs 3.0.1 to spotbugs 3.1.2 in branch-2 to fix docker image buildMajor.Xiao ChenAkira Ajisaka
HADOOP-15286Remove unused imports from TestKMSWithZK.javaMinortestAkira AjisakaAjay Kumar
HDFS-13040Kerberized inotify client fails despite kinit properlyMajornamenodeWei-Chiu ChuangXiao Chen
YARN-7736Fix itemization in YARN federation documentMinordocumentationAkira AjisakaSen Zhao
HDFS-13164File not closed if streamer fail with DSQuotaExceededExceptionMajorhdfs-clientXiao ChenXiao Chen
HDFS-13109Support fully qualified hdfs path in EZ commandsMajorhdfsHanisha KoneruHanisha Koneru
MAPREDUCE-6930mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores are both present twice in mapred-default.xmlMajormrv2Daniel TempletonSen Zhao
HDFS-10618TestPendingReconstruction#testPendingAndInvalidate is flaky due to race conditionMajor.Eric BadgerEric Badger
HDFS-10803TestBalancerWithMultipleNameNodes#testBalancing2OutOf3Blockpools fails intermittently due to no free space availableMajor.Yiqun LinYiqun Lin
HDFS-12156TestFSImage fails without -PnativeMajortestAkira AjisakaAkira Ajisaka
HDFS-13261Fix incorrect null value checkMinorhdfsJianfei JiangJianfei Jiang
HDFS-12886Ignore minReplication for block recoveryMajorhdfs, namenodeLukas MajercakLukas Majercak
YARN-8039Clean up log dir configuration in TestLinuxContainerExecutorWithMocks.testStartLocalizerMinor.Miklos SzegediMiklos Szegedi
HDFS-13296GenericTestUtils generates paths with drive letter in Windows and fail webhdfs related test casesMajor.Xiao LiangXiao Liang
HDFS-13268TestWebHdfsFileContextMainOperations fails on WindowsMajor.Íñigo GoiriXiao Liang
YARN-8054Improve robustness of the LocalDirsHandlerService MonitoringTimerTask threadMajor.Jonathan Turner EaglesJonathan Turner Eagles
YARN-7873Revert YARN-6078Blocker.Billie RinaldiBillie Rinaldi
HDFS-13195DataNode conf page cannot display the current value after reconfigMinordatanodemaobaolongmaobaolong
YARN-8063DistributedShellTimelinePlugin wrongly check for entityId instead of entityTypeMajor.Rohith Sharma K SRohith Sharma K S
YARN-8068Application Priority field causes NPE in app timeline publish when Hadoop 2.7 based clients to 2.8+BlockeryarnSunil GSunil G
HADOOP-12862LDAP Group Mapping over SSL can not specify trust storeMajor.Wei-Chiu ChuangWei-Chiu Chuang
HADOOP-15317Improve NetworkTopology chooseRandom's loopMajor.Xiao ChenXiao Chen
HADOOP-15355TestCommonConfigurationFields is broken by HADOOP-15312MajortestKonstantin ShvachkoLiXin Ge
HDFS-13176WebHdfs file path gets truncated when having semicolon (;) insideMajorwebhdfsZsolt VenczelZsolt Venczel
HADOOP-15375Branch-2 pre-commit failed to build docker imageMajor.Xiao ChenXiao Chen
HADOOP-15357Configuration.getPropsWithPrefix no longer does variable substitutionMajor.Jim BrennanJim Brennan
MAPREDUCE-7062Update mapreduce.job.tags description for making use for ATSv2 purpose.Major.Charan HebriCharan Hebri
YARN-8073TimelineClientImpl doesn't honor yarn.timeline-service.versions configurationMajor.Rohith Sharma K SRohith Sharma K S
YARN-6629NPE occurred when container allocation proposal is applied but its resource requests are removed beforeCritical.Tao YangTao Yang
HDFS-13427Fix the section titles of transparent encryption documentMinordocumentationAkira AjisakaAkira Ajisaka
YARN-7527Over-allocate node resource in async-scheduling mode of CapacitySchedulerMajorcapacityschedulerTao YangTao Yang
HDFS-7101Potential null dereference in DFSck#doWork()Minor.Ted Yuskrho
YARN-8120JVM can crash with SIGSEGV when exiting due to custom leveldb loggerMajornodemanager, resourcemanagerJason Darrell LoweJason Darrell Lowe
YARN-8147TestClientRMService#testGetApplications sporadically failsMajortestJason Darrell LoweJason Darrell Lowe
HADOOP-14970MiniHadoopClusterManager doesn't respect lack of format optionMinor.Erik KrogenErik Krogen
YARN-8156Increase the default value of yarn.timeline-service.app-collector.linger-period.msMajor.Rohith Sharma K SCharan Hebri
YARN-8165Incorrect queue name logging in AbstractContainerAllocatorTrivialcapacityschedulerWeiwei YangWeiwei Yang
YARN-8164Fix a potential NPE in AbstractSchedulerPlanFollowerMajor.lujielujie
HDFS-12828OIV ReverseXML Processor fails with escaped charactersCriticalhdfsErik KrogenErik Krogen
HADOOP-15180branch-2 : daemon processes' sysout overwrites ‘ulimit -a’ in daemon's out fileMinorscriptsRanith SardarRanith Sardar
HADOOP-15396Some java source files are executableMinor.Akira AjisakaShashikant Banerjee
YARN-6827[ATS1/1.5] NPE exception while publishing recovering applications into ATS during RM restart.MajorresourcemanagerRohith Sharma K SRohith Sharma K S
YARN-7786NullPointerException while launching ApplicationMasterMajor.lujielujie
HDFS-13408MiniDFSCluster to support being built on randomized base directoryMajortestXiao LiangXiao Liang
HADOOP-15390Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokensCritical.Xiao ChenXiao Chen
HDFS-13336Test cases of TestWriteToReplica failed in windowsMajor.Xiao LiangXiao Liang
YARN-7598Document how to use classpath isolation for aux-services in YARNMajor.Xuan GongXuan Gong
YARN-8183Fix ConcurrentModificationException inside RMAppAttemptMetrics#convertAtomicLongMaptoLongMapCriticalyarnSumana SathishSuma Shivaprasad
HADOOP-15385Many tests are failing in hadoop-distcp project in branch-2Criticaltools/distcpRushabh ShahJason Darrell Lowe
MAPREDUCE-7042Killed MR job data does not move to mapreduce.jobhistory.done-dir when ATS v2 is enabledMajor.Yesha VoraXuan Gong
YARN-8205Application State is not updated to ATS if AM launching is delayed.Critical.Sumana SathishRohith Sharma K S
MAPREDUCE-7072mapred job -history prints duplicate counter in human outputMajorclientWilfred SpiegelenburgWilfred Spiegelenburg
YARN-8221RMWebServices also need to honor yarn.resourcemanager.display.per-user-appsMajorwebappSunil GSunil G
HDFS-13509Bug fix for breakHardlinks() of ReplicaInfo/LocalReplica, and fix TestFileAppend failures on WindowsMajor.Xiao LiangXiao Liang
MAPREDUCE-7073Optimize TokenCache#obtainTokensForNamenodesInternalMajor.Bibin ChundattBibin Chundatt
YARN-8025UsersManangers#getComputedResourceLimitForActiveUsers throws NPE due to preComputedActiveUserLimit is emptyMajoryarnJiandan YangTao Yang
YARN-8232RMContainer lost queue name when RM HA happensMajorresourcemanagerHu ZiqianHu Ziqian
HDFS-13537TestHdfsHelper does not generate jceks path properly for relative path in WindowsMajor.Xiao LiangXiao Liang
HADOOP-15446WASB: PageBlobInputStream.skip breaks HBASE replicationMajorfs/azureThomas MarqardtThomas Marqardt
YARN-8244TestContainerSchedulerQueuing.testStartMultipleContainers failedMajor.Miklos SzegediJim Brennan
HDFS-13586Fsync fails on directories on WindowsCriticaldatanode, hdfsLukas MajercakLukas Majercak
HDFS-13590Backport HDFS-12378 to branch-2Majordatanode, hdfs, testLukas MajercakLukas Majercak
HADOOP-15478WASB: hflush() and hsync() regressionMajorfs/azureThomas MarqardtThomas Marqardt
HADOOP-15450Avoid fsync storm triggered by DiskChecker and handle disk full situationBlocker.Kihwal LeeArpit Agarwal
HDFS-13601Optimize ByteString conversions in PBHelperMajor.Andrew WangAndrew Wang
HDFS-13588Fix TestFsDatasetImpl test failures on WindowsMajor.Xiao LiangXiao Liang
YARN-8310Handle old NMTokenIdentifier, AMRMTokenIdentifier, and ContainerTokenIdentifier formatsMajor.Robert KanterRobert Kanter
YARN-8344Missing nm.stop() in TestNodeManagerResync to fix testKillContainersOnResyncMajor.Giovanni Matteo FumarolaGiovanni Matteo Fumarola
YARN-8327Fix TestAggregatedLogFormat#testReadAcontainerLogs1 on WindowsMajorlog-aggregationGiovanni Matteo FumarolaGiovanni Matteo Fumarola
YARN-8346Upgrading to 3.1 kills running containers with error “Opportunistic container queue is full”Blocker.Rohith Sharma K SJason Darrell Lowe
HDFS-13611Unsafe use of Text as a ConcurrentHashMap key in PBHelperClientMajor.Andrew WangAndrew Wang
HDFS-13618Fix TestDataNodeFaultInjector test failures on WindowsMajortestXiao LiangXiao Liang
HADOOP-15473Configure serialFilter in KeyProvider to avoid UnrecoverableKeyException caused by JDK-8189997CriticalkmsGabor BotaGabor Bota
HDFS-13626Fix incorrect username when deny the setOwner operationMinornamenodeluhuachaoZsolt Venczel
MAPREDUCE-7103Fix TestHistoryViewerPrinter on windows due to a mismatch line separatorMinor.Giovanni Matteo FumarolaGiovanni Matteo Fumarola
YARN-8359Exclude containermanager.linux test classes on WindowsMajor.Giovanni Matteo FumarolaJason Darrell Lowe
HDFS-13664Refactor ConfiguredFailoverProxyProvider to make inheritance easierMinorhdfs-clientChao SunChao Sun
YARN-8405RM zk-state-store.parent-path ACLs has been changed since HADOOP-14773Major.Rohith Sharma K SÍñigo Goiri
MAPREDUCE-7108TestFileOutputCommitter fails on WindowsMinortestZuoming ZhangZuoming Zhang
MAPREDUCE-7101Add config parameter to allow JHS to alway scan user dir irrespective of modTimeCritical.Wangda TanThomas Marqardt
YARN-8404Timeline event publish need to be async to avoid Dispatcher thread leak in case ATS is downBlocker.Rohith Sharma K SRohith Sharma K S
HDFS-13675Speed up TestDFSAdminWithHAMajorhdfs, namenodeLukas MajercakLukas Majercak
HDFS-13673TestNameNodeMetrics fails on WindowsMinortestZuoming ZhangZuoming Zhang
HDFS-13676TestEditLogRace fails on WindowsMinortestZuoming ZhangZuoming Zhang
HADOOP-15523Shell command timeout given is in seconds whereas it is taken as millisec while schedulingMajor.Bilwa S TBilwa S T
YARN-8444NodeResourceMonitor crashes on bad swapFree valueMajor.Jim BrennanJim Brennan
YARN-8457Compilation is broken with -Pyarn-uiMajorwebappSunil GSunil G
YARN-8401[UI2] new ui is not accessible with out internet connectionBlocker.Bibin ChundattBibin Chundatt
YARN-8451Multiple NM heartbeat thread created when a slow NM resync with RMMajornodemanagerBotong HuangBotong Huang
HADOOP-15548Randomize local dirsMinor.Jim BrennanJim Brennan
YARN-8473Containers being launched as app tears down can leave containers in NEW stateMajornodemanagerJason Darrell LoweJason Darrell Lowe
HDFS-13729Fix broken links to RBF documentationMinordocumentationjwhitterGabor Bota
YARN-8515container-executor can crash with SIGPIPE after nodemanager restartMajor.Jim BrennanJim Brennan
YARN-8421when moving app, activeUsers is increased, even though app does not have outstanding requestMajor.kyungwan nam
HADOOP-15614TestGroupsCaching.testExceptionOnBackgroundRefreshHandled reliably failsMajor.Kihwal LeeWeiwei Yang
YARN-4606CapacityScheduler: applications could get starved because computation of #activeUsers considers pending appsCriticalcapacity scheduler, capacityschedulerKaram SinghManikandan R
HADOOP-15637LocalFs#listLocatedStatus does not filter out hidden .crc filesMinorfsErik KrogenErik Krogen
HADOOP-15644Hadoop Docker Image Pip Install Fails on branch-2CriticalbuildHaibo ChenHaibo Chen
YARN-6966NodeManager metrics may return wrong negative values when NM restartMajor.Yang WangSzilard Nemeth
YARN-8331Race condition in NM container launched after doneMajor.Yang WangPradeep Ambati
HDFS-13758DatanodeManager should throw exception if it has BlockRecoveryCommand but the block is not under constructionMajornamenodeWei-Chiu Chuangchencan
YARN-8612Fix NM Collector Service Port issue in YarnConfigurationMajorATSv2Prabha ManepalliPrabha Manepalli
HADOOP-15674Test failure TestSSLHttpServer.testExcludedCiphers with TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 cipher suiteMajorcommonGabor BotaSzilard Nemeth
YARN-8640Restore previous state in container-executor after failureMajor.Jim BrennanJim Brennan
YARN-8679[ATSv2] If HBase cluster is down for long time, high chances that NM ContainerManager dispatcher get blockedMajor.Rohith Sharma K SWangda Tan
HADOOP-14314The OpenSolaris taxonomy link is dead in InterfaceClassification.mdMajordocumentationDaniel TempletonRui Gao
YARN-8649NPE in localizer hearbeat processing if a container is killed while localizingMajor.lujielujie
HADOOP-10219ipc.Client.setupIOstreams() needs to check for ClientCache.stopClient requested shutdownsMajoripcSteve LoughranKihwal Lee
MAPREDUCE-7131Job History Server has race condition where it moves files from intermediate to finished but thinks file is in intermediateMajor.Anthony HsuAnthony Hsu
HDFS-13836RBF: Handle mount table znode with null valueMajorfederation, hdfsyanghuafengyanghuafeng
HDFS-12716‘dfs.datanode.failed.volumes.tolerated’ to support minimum number of volumes to be availableMajordatanodeusharaniRanith Sardar
YARN-8709CS preemption monitor always fails since one under-served queue was deletedMajorcapacityscheduler, scheduler preemptionTao YangTao Yang
HDFS-13051Fix dead lock during async editlog rolling if edit queue is fullMajornamenodezhangweiDaryn Sharp
HDFS-13914Fix DN UI logs link broken when https is enabled after HDFS-13902MinordatanodeJianfei JiangJianfei Jiang
MAPREDUCE-7133History Server task attempts REST API returns invalid dataMajorjobhistoryserverOleksandr ShevchenkoOleksandr Shevchenko
YARN-8720CapacityScheduler does not enforce max resource allocation check at queue levelMajorcapacity scheduler, capacityscheduler, resourcemanagerTarun ParimiTarun Parimi
HDFS-13844Fix the fmt_bytes function in the dfs-dust.jsMinorhdfs, uiyanghuafengyanghuafeng
HADOOP-15755StringUtils#createStartupShutdownMessage throws NPE when args is nullMajor.Lokesh JainDinesh Chitlangia
MAPREDUCE-3801org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testExponentialEstimator fails intermittentlyMajormrv2Robert Joseph EvansJason Darrell Lowe
MAPREDUCE-7137MRAppBenchmark.benchmark1() fails with NullPointerExceptionMinortestOleksandr ShevchenkoOleksandr Shevchenko
MAPREDUCE-7138ThrottledContainerAllocator in MRAppBenchmark should implement RMHeartbeatHandlerMinortestOleksandr ShevchenkoOleksandr Shevchenko
HDFS-13908TestDataNodeMultipleRegistrations is flakyMajor.Íñigo GoiriAyush Saxena
HADOOP-15772Remove the ‘Path ... should be specified as a URI’ warnings on startupMajorconfArpit AgarwalAyush Saxena
YARN-8804resourceLimits may be wrongly calculated when leaf-queue is blocked in cluster with 3+ level queuesCriticalcapacityschedulerTao YangTao Yang
YARN-8774Memory leak when CapacityScheduler allocates from reserved container with non-default labelCriticalcapacityschedulerTao YangTao Yang
YARN-8840Add missing cleanupSSLConfig() call for TestTimelineClient testMinortest, timelineclientAki TanakaAki Tanaka
HADOOP-15817Reuse Object Mapper in KMSJSONReaderMajorkmsJonathan Turner EaglesJonathan Turner Eagles
HADOOP-15820ZStandardDecompressor native code sets an integer field as a longBlocker.Jason Darrell LoweJason Darrell Lowe
HDFS-13964RBF: TestRouterWebHDFSContractAppend fails with No Active Namenode under nameserviceMajor.Ayush SaxenaAyush Saxena
HDFS-13768Adding replicas to volume map makes DataNode start slowlyMajor.Yiqun LinSurendra Singh Lilhore
HADOOP-15818Fix deprecated maven-surefire-plugin configuration in hadoop-kms moduleMinorkmsAkira AjisakaVidura Bhathiya Mudalige
HDFS-13802RBF: Remove FSCK from Router Web UIMajor.Fei HuiFei Hui
HADOOP-15859ZStandardDecompressor.c mistakes a class for an instanceBlocker.Ben LauJason Darrell Lowe
HADOOP-15850CopyCommitter#concatFileChunks should check that the blocks per chunk is not 0Criticaltools/distcpTed YuTed Yu
YARN-7502Nodemanager restart docs should describe nodemanager supervised propertyMajordocumentationJason Darrell LoweSuma Shivaprasad
HADOOP-15866Renamed HADOOP_SECURITY_GROUP_SHELL_COMMAND_TIMEOUT keys break compatibilityBlocker.Wei-Chiu ChuangWei-Chiu Chuang
YARN-8826Fix lingering timeline collector after serviceStop in TimelineCollectorManagerTrivialATSv2Prabha ManepalliPrabha Manepalli
HADOOP-15822zstd compressor can fail with a small output bufferMajor.Jason Darrell LoweJason Darrell Lowe
HDFS-13959TestUpgradeDomainBlockPlacementPolicy is flakyMajor.Ayush SaxenaAyush Saxena
HADOOP-15899Update AWS Java SDK versions in NOTICE.txtMajor.Akira AjisakaAkira Ajisaka
HADOOP-15900Update JSch versions in LICENSE.txtMajor.Akira AjisakaAkira Ajisaka
HDFS-14043Tolerate corrupted seen_txid fileMajorhdfs, namenodeLukas MajercakLukas Majercak
YARN-8858CapacityScheduler should respect maximum node resource when per-queue maximum-allocation is being used.Major.Sumana SathishWangda Tan
HDFS-14048DFSOutputStream close() throws exception on subsequent call after DataNode restartMajorhdfs-clientErik KrogenErik Krogen
MAPREDUCE-7156NullPointerException when reaching max shuffle connectionsMajormrv2Peter BacskoPeter Bacsko
YARN-8233NPE in CapacityScheduler#tryCommit when handling allocate/reserve proposal whose allocatedOrReservedContainer is nullCriticalcapacityschedulerTao YangTao Yang
HADOOP-15923create-release script should set max-cache-ttl as well as default-cache-ttl for gpg-agentBlockerbuildAkira AjisakaAkira Ajisaka
HADOOP-15930Exclude MD5 checksum files from release artifactCriticalbuildAkira AjisakaAkira Ajisaka
HADOOP-15925The config and log of gpg-agent are removed in create-release scriptMajorbuildAkira AjisakaDinesh Chitlangia
HDFS-14056Fix error messages in HDFS-12716MinorhdfsAdam AntalAyush Saxena
YARN-7794SLSRunner is not loading timeline service jars causing failureBlockerscheduler-load-simulatorSunil GYufei Gu
HADOOP-16008Fix typo in CommandsManual.mdMinor.Akira AjisakaShweta
HADOOP-15973Configuration: Included properties are not cached if resource is a streamCritical.Eric PayneEric Payne
YARN-9175Null resources check in ResourceInfo for branch-3.0Major.Jonathan HungJonathan Hung
HADOOP-15992JSON License is included in the transitive dependency of aliyun-sdk-oss 3.0.0Blocker.Akira AjisakaAkira Ajisaka
HADOOP-16030AliyunOSS: bring fixes back from HADOOP-15671Blockerfs/osswujinhuwujinhu
YARN-9162Fix TestRMAdminCLI#testHelpMajorresourcemanager, testAyush SaxenaAyush Saxena
YARN-8833Avoid potential integer overflow when computing fair sharesMajorfairschedulerliyakunliyakun
HADOOP-16016TestSSLFactory#testServerWeakCiphers sporadically fails in precommit buildsMajorsecurity, testJason Darrell LoweAkira Ajisaka
HADOOP-16013DecayRpcScheduler decay thread should run as a daemonMajoripcErik KrogenErik Krogen
YARN-8747[UI2] YARN UI2 page loading failed due to js error under some time zone configurationCriticalwebappcollinmacollinma
YARN-9210RM nodes web page can not display node infoBlockeryarnJiandan YangJiandan Yang
YARN-7088Add application launch time to Resource Manager REST APIMajor.Abdullah YousufiKanwaljeet Sachdev
YARN-9222Print launchTime in ApplicationSummaryMajor.Jonathan HungJonathan Hung
YARN-6616YARN AHS shows submitTime for jobs same as startTimeMinor.Prabhu JosephPrabhu Joseph
MAPREDUCE-7177Disable speculative execution in TestDFSIOMajor.Kihwal LeeZhaohui Xin
YARN-9206RMServerUtils does not count SHUTDOWN as an accepted stateMajor.Kuhu ShuklaKuhu Shukla
YARN-9283Javadoc of LinuxContainerExecutor#addSchedPriorityCommand has a wrong property name as referenceMinordocumentationSzilard NemethAdam Antal
HADOOP-15813Enable more reliable SSL connection reuseMajorcommonDaryn SharpDaryn Sharp
HDFS-14314fullBlockReportLeaseId should be reset after registering to NNCriticaldatanodestarstar
YARN-5714ContainerExecutor does not order environment mapTrivialnodemanagerRemi CatherinotRemi Catherinot
HADOOP-16192CallQueue backoff bug fixes: doesn‘t perform backoff when add() is used, and doesn’t update backoff when refreshedMajoripcErik KrogenErik Krogen
HDFS-14399Backport HDFS-10536 to branch-2Critical.Chao SunChao Sun
HDFS-14407Fix misuse of SLF4j logging API in DatasetVolumeChecker#checkAllVolumesMinor.Wanqiang JiWanqiang Ji
HDFS-14414Clean up findbugs warning in branch-2Major.Wei-Chiu ChuangDinesh Chitlangia
HADOOP-14544DistCp documentation for command line options is misaligned.MinordocumentationChris NaurothMasatake Iwasaki
HDFS-10477Stop decommission a rack of DataNodes caused NameNode fail over to standbyMajornamenodeyunjiong zhaoyunjiong zhao
HDFS-13677Dynamic refresh Disk configuration results in overwriting VolumeMapBlocker.xuzqxuzq
YARN-9285RM UI progress column is of wrong typeMinoryarnAhmed HusseinAhmed Hussein
HDFS-14500NameNode StartupProgress continues to report edit log segments after the LOADING_EDITS phase is finishedMajornamenodeErik KrogenErik Krogen
HADOOP-16331Fix ASF License check in pom.xmlMajor.Wanqiang JiAkira Ajisaka
HDFS-14514Actual read size of open file in encryption zone still larger than listing size even after enabling HDFS-11402 in Hadoop 2Majorencryption, hdfs, snapshotsSiyao MengSiyao Meng
HDFS-14512ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes)Major.Shen YinjieAyush Saxena
HDFS-14521Suppress setReplication logging.Major.Kihwal LeeKihwal Lee
YARN-8625Aggregate Resource Allocation for each job is not present in ATSMajorATSv2Prabhu JosephPrabhu Joseph
HADOOP-16345Potential NPE when instantiating FairCallQueue metricsMajoripcErik KrogenErik Krogen
HDFS-14494Move Server logging of StatedId inside receiveRequestState()Major.Konstantin ShvachkoShweta
HDFS-14535The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut readMajorhdfs-clientZheng HuZheng Hu
HDFS-13730BlockReaderRemote.sendReadResult throws NPEMajorhdfs-clientWei-Chiu ChuangYuanbo Liu
HDFS-13770dfsadmin -report does not always decrease “missing blocks (with replication factor 1)” metrics when file is deletedMajorhdfsKitti NanasiKitti Nanasi
HDFS-14101Random failure of testListCorruptFilesCorruptedBlockMajortestKihwal LeeZsolt Venczel
HDFS-14465When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit.Major.Yicong CaiYicong Cai
HDFS-14541When evictableMmapped or evictable size is zero, do not throw NoSuchElementExceptionMajorhdfs-client, performanceZheng HuLisheng Sun
HDFS-14629Property value Hard Coded in DNConf.javaTrivial.hemanthboyinahemanthboyina
HDFS-12703Exceptions are fatal to decommissioning monitorCriticalnamenodeDaryn SharpXiaoqiao He
HDFS-12748NameNode memory leak when accessing webhdfs GETHOMEDIRECTORYMajorhdfsJiandan YangWeiwei Yang
HADOOP-16386FindBugs warning in branch-2: GlobalStorageStatistics defines non-transient non-serializable instance field mapMajorfsWei-Chiu ChuangMasatake Iwasaki
MAPREDUCE-6521MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local filesystem in unit testMajortestMasatake IwasakiMasatake Iwasaki
MAPREDUCE-7076TestNNBench#testNNBenchCreateReadAndDelete failing in our internal buildMinortestRushabh Shahkevin su
YARN-9668UGI conf doesn't read user overridden configurations on RM and NM startupMajor.Jonathan HungJonathan Hung
HADOOP-9844NPE when trying to create an error message response of SASL RPCMajoripcSteve LoughranSteve Loughran
HADOOP-16245Enabling SSL within LdapGroupsMapping can break system SSL configsMajorcommon, securityErik KrogenErik Krogen
HDFS-14660[SBN Read] ObserverNameNode should throw StandbyException for requests not from ObserverProxyProviderMajor.Chao SunChao Sun
HDFS-14429Block remain in COMMITTED but not COMPLETE caused by DecommissionMajor.Yicong CaiYicong Cai
HDFS-14672Backport HDFS-12703 to branch-2MajornamenodeXiaoqiao HeXiaoqiao He
HADOOP-16435RpcMetrics should not be retained foreverCriticalrpc-serverZoltan HaindrichZoltan Haindrich
HDFS-14464Remove unnecessary log message from DFSInputStreamTrivial.Kihwal LeeChao Sun
HDFS-14569Result of crypto -listZones is not formatted properlyMajor.hemanthboyinahemanthboyina
YARN-9596QueueMetrics has incorrect metrics when labelled partitions are involvedMajorcapacity schedulerMuhammad Samir KhanMuhammad Samir Khan
HADOOP-15237In KMS docs there should be one space between KMS_LOG and NOTEMinorkmsSnigdhanjali MishraSnigdhanjali Mishra
HDFS-14462WebHDFS throws “Error writing request body to server” instead of DSQuotaExceededExceptionMajorwebhdfsErik KrogenSimbarashe Dzinamarira
HDFS-14631The DirectoryScanner doesn't fix the wrongly placed replica.Major.JinglunJinglun
HDFS-14724Fix JDK7 compatibility in branch-2Blocker.Wei-Chiu ChuangChen Liang
HDFS-14423Percent (%) and plus (+) characters no longer work in WebHDFSMajorwebhdfsJing WangMasatake Iwasaki
HDFS-13101Yet another fsimage corruption related to snapshotMajorsnapshotsYongjun ZhangShashikant Banerjee
HDFS-14311Multi-threading conflict at layoutVersion when loading block pool storageMajorrolling upgradesYicong CaiYicong Cai
HADOOP-16494Add SHA-256 or SHA-512 checksum to release artifacts to comply with the release distribution policyBlockerbuildAkira AjisakaAkira Ajisaka
HDFS-13977NameNode can kill itself if it tries to send too many txns to a QJM simultaneouslyMajornamenode, qjmErik KrogenErik Krogen
YARN-9438launchTime not written to state store for running applicationsMajor.Jonathan HungJonathan Hung
YARN-7585NodeManager should go unhealthy when state store throws DBExceptionMajornodemanagerWilfred SpiegelenburgWilfred Spiegelenburg
HDFS-14726Fix JN incompatibility issue in branch-2 due to backport of HDFS-10519Blockerjournal-nodeChen LiangChen Liang
YARN-9806TestNMSimulator#testNMSimulator fails in branch-2Major.Jonathan HungJonathan Hung
YARN-9820RM logs InvalidStateTransitionException when app is submittedCritical.Rohith Sharma K SPrabhu Joseph
HDFS-14303check block directory logic not correct when there is only meta file, print no meaning warn logMinordatanode, hdfsqiang Liuqiang Liu
HADOOP-16582LocalFileSystem's mkdirs() does not work as expected under viewfs.Major.Kihwal LeeKihwal Lee
HADOOP-16581ValueQueue does not trigger an async refill when number of values falls below watermarkMajorcommon, kmsYuval DeganiYuval Degani
HDFS-14853NPE in DFSNetworkTopology#chooseRandomWithStorageType() when the excludedNode is not presentMajor.Ranith SardarRanith Sardar
YARN-9858Optimize RMContext getExclusiveEnforcedPartitionsMajor.Jonathan HungJonathan Hung
HDFS-14655[SBN Read] Namenode crashes if one of The JN is downCritical.Harshakiran ReddyAyush Saxena
HDFS-14245Class cast error in GetGroups with ObserverReadProxyProviderMajor.Shen YinjieErik Krogen
HDFS-14509DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.xBlocker.Yuxuan WangYuxuan Wang
HADOOP-16655Change cipher suite when fetching tomcat tarball for branch-2Major.Jonathan HungJonathan Hung

TESTS:

JIRASummaryPriorityComponentReporterContributor
HDFS-13337Backport HDFS-4275 to branch-2.9Minor.Íñigo GoiriXiao Liang
HDFS-13503Fix TestFsck test failures on WindowsMajorhdfsXiao LiangXiao Liang
HDFS-13542TestBlockManager#testNeededReplicationWhileAppending fails due to improper cluster shutdown in TestBlockManager#testBlockManagerMachinesArray on WindowsMinor.Anbang HuAnbang Hu
HDFS-13551TestMiniDFSCluster#testClusterSetStorageCapacity does not shut down clusterMinor.Anbang HuAnbang Hu
HDFS-11700TestHDFSServerPorts#testBackupNodePorts doesn't pass on WindowsMinor.Anbang HuAnbang Hu
HDFS-13548TestResolveHdfsSymlink#testFcResolveAfs fails on WindowsMinor.Anbang HuAnbang Hu
HDFS-13567TestNameNodeMetrics#testGenerateEDEKTime,TestNameNodeMetrics#testResourceCheck should use a different cluster basedirMinor.Anbang HuAnbang Hu
HDFS-13557TestDFSAdmin#testListOpenFiles fails on WindowsMinor.Anbang HuAnbang Hu
HDFS-13550TestDebugAdmin#testComputeMetaCommand fails on WindowsMinor.Anbang HuAnbang Hu
HDFS-13559TestBlockScanner does not close TestContext properlyMinor.Anbang HuAnbang Hu
HDFS-13570TestQuotaByStorageType,TestQuota,TestDFSOutputStream fail on WindowsMinor.Anbang HuAnbang Hu
HDFS-13558TestDatanodeHttpXFrame does not shut down clusterMinor.Anbang HuAnbang Hu
HDFS-13554TestDatanodeRegistration#testForcedRegistration does not shut down clusterMinor.Anbang HuAnbang Hu
HDFS-13556TestNestedEncryptionZones does not shut down clusterMinor.Anbang HuAnbang Hu
HDFS-13560Insufficient system resources exist to complete the requested service for some tests on WindowsMinor.Anbang HuAnbang Hu
HDFS-13592TestNameNodePrunesMissingStorages#testNameNodePrunesUnreportedStorages does not shut down cluster properlyMinor.Anbang HuAnbang Hu
HDFS-13593TestBlockReaderLocalLegacy#testBlockReaderLocalLegacyWithAppend fails on WindowsMinortestAnbang HuAnbang Hu
HDFS-13587TestQuorumJournalManager fails on WindowsMinor.Anbang HuAnbang Hu
HDFS-13620Randomize the test directory path for TestHDFSFileSystemContractMinor.Anbang HuAnbang Hu
HDFS-13591TestDFSShell#testSetrepLow fails on WindowsMinor.Anbang HuAnbang Hu
HDFS-13632Randomize baseDir for MiniJournalCluster in MiniQJMHACluster for TestDFSAdminWithHAMinor.Anbang HuAnbang Hu
MAPREDUCE-7102Fix TestJavaSerialization for Windows due a mismatch line separatorMinor.Giovanni Matteo FumarolaGiovanni Matteo Fumarola
HDFS-13652Randomize baseDir for MiniDFSCluster in TestBlockScannerMinor.Anbang HuAnbang Hu
YARN-8370Some Node Manager tests fail on Windows due to improper path/file separatorMinor.Anbang HuAnbang Hu
YARN-8422TestAMSimulator failing with NPEMinor.Giovanni Matteo FumarolaGiovanni Matteo Fumarola
HADOOP-15532TestBasicDiskValidator fails with NoSuchFileExceptionMinor.Íñigo GoiriGiovanni Matteo Fumarola
HDFS-13563TestDFSAdminWithHA times out on WindowsMinor.Anbang HuLukas Majercak
HDFS-13681Fix TestStartup.testNNFailToStartOnReadOnlyNNDir test failure on WindowsMajortestXiao LiangXiao Liang
YARN-8944TestContainerAllocation.testUserLimitAllocationMultipleContainers failure after YARN-8896Minorcapacity schedulerWilfred SpiegelenburgWilfred Spiegelenburg
HDFS-11950Disable libhdfs zerocopy test on MacMinorlibhdfsJohn ZhugeAkira Ajisaka

SUB-TASKS:

JIRASummaryPriorityComponentReporterContributor
YARN-4081Add support for multiple resource types in the Resource classMajorresourcemanagerVarun VasudevVarun Vasudev
YARN-4172Extend DominantResourceCalculator to account for all resourcesMajorresourcemanagerVarun VasudevVarun Vasudev
YARN-4715Add support to read resource types from a config fileMajornodemanager, resourcemanagerVarun VasudevVarun Vasudev
YARN-4829Add support for binary unitsMajornodemanager, resourcemanagerVarun VasudevVarun Vasudev
YARN-4830Add support for resource types in the nodemanagerMajornodemanagerVarun VasudevVarun Vasudev
YARN-5242Update DominantResourceCalculator to consider all resource types in calculationsMajorresourcemanagerVarun VasudevVarun Vasudev
YARN-5586Update the Resources class to consider all resource typesMajornodemanager, resourcemanagerVarun VasudevVarun Vasudev
YARN-6761Fix build for YARN-3926 branchMajornodemanager, resourcemanagerVarun VasudevVarun Vasudev
YARN-6786ResourcePBImpl imports cleanupTrivialresourcemanagerDaniel TempletonYeliang Cang
YARN-6788Improve performance of resource profile branchBlockernodemanager, resourcemanagerSunil GSunil G
YARN-6994Remove last uses of Long from resource types codeMinorresourcemanagerDaniel TempletonDaniel Templeton
YARN-6892Improve API implementation in Resources and DominantResourceCalculator classMajornodemanager, resourcemanagerSunil GSunil G
YARN-6610DominantResourceCalculator#getResourceAsValue dominant param is updated to handle multiple resourcesCriticalresourcemanagerDaniel TempletonDaniel Templeton
YARN-7030Performance optimizations in Resource and ResourceUtils classCriticalnodemanager, resourcemanagerWangda TanWangda Tan
YARN-7042Clean up unit tests after YARN-6610MajortestDaniel TempletonDaniel Templeton
YARN-6789Add Client API to get all supported resource types from RMMajornodemanager, resourcemanagerSunil GSunil G
YARN-6781ResourceUtils#initializeResourcesMap takes an unnecessary Map parameterMinorresourcemanagerDaniel TempletonYu-Tang Lin
YARN-7067Optimize ResourceType information display in UICriticalnodemanager, resourcemanagerWangda TanWangda Tan
YARN-7039Fix javac and javadoc errors in YARN-3926 branchMajornodemanager, resourcemanagerSunil GSunil G
YARN-7093Improve log message in ResourceUtilsTrivialnodemanager, resourcemanagerSunil GSunil G
YARN-6933ResourceUtils.DISALLOWED_NAMES check is duplicatedMajorresourcemanagerDaniel TempletonManikandan R
YARN-7137Move newly added APIs to unstable in YARN-3926 branchBlockernodemanager, resourcemanagerWangda TanWangda Tan
HADOOP-14799Update nimbus-jose-jwt to 4.41.1Major.Ray ChiangRay Chiang
YARN-7345GPU Isolation: Incorrect minor device numbers written to devices.deny fileMajor.Jonathan HungJonathan Hung
HADOOP-14997Add hadoop-aliyun as dependency of hadoop-cloud-storageMinorfs/ossGenmao YuGenmao Yu
YARN-7143FileNotFound handling in ResourceUtils is inconsistentMajorresourcemanagerDaniel TempletonDaniel Templeton
YARN-6909Use LightWeightedResource when number of resource types more than twoCriticalresourcemanagerDaniel TempletonSunil G
HDFS-12801RBF: Set MountTableResolver as default file resolverMinor.Íñigo GoiriÍñigo Goiri
YARN-7430Enable user re-mapping for Docker containers by defaultBlockersecurity, yarnEric YangEric Yang
HADOOP-15024AliyunOSS: support user agent configuration and include that & Hadoop version information to oss serverMajorfs, fs/ossSammi ChenSammi Chen
HDFS-12858RBF: Add router admin commands usage in HDFS commands reference docMinordocumentationYiqun LinYiqun Lin
HDFS-12835RBF: Fix Javadoc parameter errorsMinor.Wei YanWei Yan
YARN-7573Gpu Information page could be empty for nodes without GPUMajorwebapp, yarn-ui-v2Sunil GSunil G
HDFS-12396Webhdfs file system should get delegation token from kms provider.Majorencryption, kms, webhdfsRushabh ShahRushabh Shah
YARN-6704Add support for work preserving NM restart when FederationInterceptor is enabled in AMRMProxyServiceMajor.Botong HuangBotong Huang
HDFS-12875RBF: Complete logic for -readonly option of dfsrouteradmin add commandMajor.Yiqun LinÍñigo Goiri
YARN-7383Node resource is not parsed correctly for resource names containing dotMajornodemanager, resourcemanagerJonathan HungGergely Novák
YARN-7630Fix AMRMToken rollover handling in AMRMProxyMinor.Botong HuangBotong Huang
HDFS-12937RBF: Add more unit tests for router admin commandsMajortestYiqun LinYiqun Lin
YARN-7032[ATSv2] NPE while starting hbase co-processor when HBase authorization is enabled.Critical.Rohith Sharma K SRohith Sharma K S
HDFS-12988RBF: Mount table entries not properly updated in the local cacheMajor.Íñigo GoiriÍñigo Goiri
HDFS-12802RBF: Control MountTableResolver cache sizeMajor.Íñigo GoiriÍñigo Goiri
HDFS-12934RBF: Federation supports global quotaMajor.Yiqun LinYiqun Lin
HDFS-12972RBF: Display mount table quota info in Web UI and admin commandMajor.Yiqun LinYiqun Lin
YARN-6736Consider writing to both ats v1 & v2 from RM for smoother upgradesMajortimelineserverVrushali CAaron Gresch
HADOOP-15027AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performanceMajorfs/osswujinhuwujinhu
HDFS-12973RBF: Document global quota supporting in federationMajor.Yiqun LinYiqun Lin
HDFS-13028RBF: Fix spurious TestRouterRpc#testProxyGetStatsMinor.Íñigo GoiriÍñigo Goiri
HDFS-12772RBF: Federation Router State State Store internal APIMajor.Íñigo GoiriÍñigo Goiri
HDFS-13042RBF: Heartbeat Router StateMajor.Íñigo GoiriÍñigo Goiri
HDFS-13049RBF: Inconsistent Router OPTS config in branch-2 and branch-3Minor.Wei YanWei Yan
YARN-7817Add Resource reference to RM's NodeInfo object so REST API can get non memory/vcore resource usages.Major.Sumana SathishSunil G
HDFS-12574Add CryptoInputStream to WebHdfsFileSystem read call.Majorencryption, kms, webhdfsRushabh ShahRushabh Shah
HDFS-13044RBF: Add a safe mode for the RouterMajor.Íñigo GoiriÍñigo Goiri
HDFS-13043RBF: Expose the state of the Routers in the federationMajor.Íñigo GoiriÍñigo Goiri
HDFS-13068RBF: Add router admin option to manage safe modeMajor.Íñigo GoiriYiqun Lin
YARN-7860Fix UT failure TestRMWebServiceAppsNodelabel#testAppsRunningMajor.Weiwei YangSunil G
HDFS-13119RBF: Manage unavailable clustersMajor.Íñigo GoiriYiqun Lin
YARN-7223Document GPU isolation featureBlocker.Wangda TanWangda Tan
HDFS-13187RBF: Fix Routers information shown in the web UIMinor.Wei YanWei Yan
HDFS-13184RBF: Improve the unit test TestRouterRPCClientRetriesMinortestYiqun LinYiqun Lin
HDFS-13199RBF: Fix the hdfs router page missing label icon issueMajorfederation, hdfsmaobaolongmaobaolong
HADOOP-15090Add ADL troubleshooting docMajordocumentation, fs/adlSteve LoughranSteve Loughran
YARN-7919Refactor timelineservice-hbase module into submodulesMajortimelineserviceHaibo ChenHaibo Chen
YARN-8003Backport the code structure changes in YARN-7346 to branch-2Major.Haibo ChenHaibo Chen
HDFS-13214RBF: Complete document of Router configurationMajor.Tao JieYiqun Lin
HADOOP-15267S3A multipart upload fails when SSE-C encryption is enabledCriticalfs/s3Anis ElleuchAnis Elleuch
HDFS-13230RBF: ConnectionManager‘s cleanup task will compare each pool’s own active conns with its total connsMinor.Wei YanChao Sun
HDFS-13233RBF: MountTableResolver doesn't return the correct mount point of the given pathMajorhdfswangzhiyuanwangzhiyuan
HDFS-13212RBF: Fix router location cache issueMajorfederation, hdfsWu WeiweiWu Weiwei
HDFS-13232RBF: ConnectionPool should return first usable connectionMinor.Wei YanEkanth Sethuramalingam
HDFS-13240RBF: Update some inaccurate document descriptionsMinor.Yiqun LinYiqun Lin
HDFS-11399Many tests fails in Windows due to injecting disk failuresMajor.Yiqun LinYiqun Lin
HDFS-13241RBF: TestRouterSafemode failed if the port 8888 is in useMajorhdfs, testmaobaolongmaobaolong
HDFS-13253RBF: Quota management incorrect parent-child relationship judgementMajor.Yiqun LinYiqun Lin
HDFS-13226RBF: Throw the exception if mount table entry validated failedMajorhdfsmaobaolongmaobaolong
HADOOP-15308TestConfiguration fails on Windows because of pathsMajortestÍñigo GoiriXiao Liang
HDFS-12773RBF: Improve State Store FS implementationMajor.Íñigo GoiriÍñigo Goiri
HDFS-13198RBF: RouterHeartbeatService throws out CachedStateStore related exceptions when starting routerMinor.Wei YanWei Yan
HDFS-13224RBF: Resolvers to support mount points across multiple subclustersMajor.Íñigo GoiriÍñigo Goiri
HDFS-13299RBF : Fix compilation error in branch-2 (TestMultipleDestinationResolver)Blocker.Brahma Reddy BattulaBrahma Reddy Battula
HADOOP-15262AliyunOSS: move files under a directory in parallel when rename a directoryMajorfs/osswujinhuwujinhu
HDFS-13215RBF: Move Router to its own moduleMajor.Íñigo GoiriWei Yan
HDFS-13307RBF: Improve the use of setQuota commandMajor.liuhongtongliuhongtong
HDFS-13250RBF: Router to manage requests across multiple subclustersMajor.Íñigo GoiriÍñigo Goiri
HDFS-13318RBF: Fix FindBugs in hadoop-hdfs-rbfMinor.Íñigo GoiriEkanth Sethuramalingam
HDFS-12792RBF: Test Router-based federation using HDFSContractMajor.Íñigo GoiriÍñigo Goiri
YARN-7581HBase filters are not constructed correctly in ATSv2MajorATSv2Haibo ChenHaibo Chen
YARN-7986ATSv2 REST API queries do not return results for uppercase application tagsCritical.Charan HebriCharan Hebri
HDFS-12512RBF: Add WebHDFSMajorfsÍñigo GoiriWei Yan
HDFS-13291RBF: Implement available space based OrderResolverMajor.Yiqun LinYiqun Lin
HDFS-13204RBF: Optimize name service safe mode iconMinor.liuhongtongliuhongtong
HDFS-13352RBF: Add xsl stylesheet for hdfs-rbf-default.xmlMajordocumentationTakanobu AsanumaTakanobu Asanuma
YARN-8010Add config in FederationRMFailoverProxy to not bypass facade cache when failing overMinor.Botong HuangBotong Huang
HDFS-13347RBF: Cache datanode reportsMinor.Íñigo GoiriÍñigo Goiri
HDFS-13289RBF: TestConnectionManager#testCleanup() test case need correctionMinor.Dibyendu KarmakarDibyendu Karmakar
HDFS-13364RBF: Support NamenodeProtocol in the RouterMajor.Íñigo GoiriÍñigo Goiri
HADOOP-14651Update okhttp version to 2.7.5Majorfs/adlRay ChiangRay Chiang
YARN-6936[Atsv2] Retrospect storing entities into sub application table from client perspectiveMajor.Rohith Sharma K SRohith Sharma K S
HDFS-13353RBF: TestRouterWebHDFSContractCreate failedMajortestTakanobu AsanumaTakanobu Asanuma
YARN-8107Give an informative message when incorrect format is used in ATSv2 filter attributesMajorATSv2Charan HebriRohith Sharma K S
YARN-8110AMRMProxy recover should catch for all throwable to avoid premature exitMajor.Botong HuangBotong Huang
HDFS-13402RBF: Fix java doc for StateStoreFileSystemImplMinorhdfsYiran WuYiran Wu
HDFS-13380RBF: mv/rm fail after the directory exceeded the quota limitMajor.Wu WeiweiYiqun Lin
HDFS-13410RBF: Support federation with no subclustersMinor.Íñigo GoiriÍñigo Goiri
HDFS-13384RBF: Improve timeout RPC call mechanismMinor.Íñigo GoiriÍñigo Goiri
HDFS-13045RBF: Improve error message returned from subclusterMinor.Wei YanÍñigo Goiri
HDFS-13428RBF: Remove LinkedList From StateStoreFileImpl.javaTrivialfederationDavid MollitorDavid Mollitor
HADOOP-14999AliyunOSS: provide one asynchronous multi-part based uploading mechanismMajorfs/ossGenmao YuGenmao Yu
YARN-7810TestDockerContainerRuntime test failures due to UID lookup of a non-existent userMajor.Shane KumpfShane Kumpf
HDFS-13435RBF: Improve the error loggings for printing the stack traceMajor.Yiqun LinYiqun Lin
YARN-7189Container-executor doesn't remove Docker containers that error out earlyMajoryarnEric BadgerEric Badger
HDFS-13466RBF: Add more router-related information to the UIMinor.Wei YanWei Yan
HDFS-13453RBF: getMountPointDates should fetch latest subdir time/date when parent dir is not present but /parent/child dirs are present in mount tableMajor.Dibyendu KarmakarDibyendu Karmakar
HDFS-13478RBF: Disabled Nameservice store APIMajor.Íñigo GoiriÍñigo Goiri
HDFS-13490RBF: Fix setSafeMode in the RouterMajor.Íñigo GoiriÍñigo Goiri
HDFS-13484RBF: Disable Nameservices from the federationMajor.Íñigo GoiriÍñigo Goiri
HDFS-13326RBF: Improve the interfaces to modify and view mount tablesMinor.Wei YanGang Li
HDFS-13499RBF: Show disabled name services in the UIMinor.Íñigo GoiriÍñigo Goiri
YARN-8215ATS v2 returns invalid YARN_CONTAINER_ALLOCATED_HOST_HTTP_ADDRESS from NMCriticalATSv2Yesha VoraRohith Sharma K S
HDFS-13508RBF: Normalize paths (automatically) when adding, updating, removing or listing mount table entriesMinor.Ekanth SethuramalingamEkanth Sethuramalingam
HDFS-13434RBF: Fix dead links in RBF documentMajordocumentationAkira AjisakaChetna Chaudhari
HDFS-13488RBF: Reject requests when a Router is overloadedMajor.Íñigo GoiriÍñigo Goiri
HDFS-13525RBF: Add unit test TestStateStoreDisabledNameserviceMajor.Yiqun LinYiqun Lin
YARN-8253HTTPS Ats v2 api call fails with “bad HTTP parsed”CriticalATSv2Yesha VoraCharan Hebri
HADOOP-15454TestRollingFileSystemSinkWithLocal fails on WindowsMajortestXiao LiangXiao Liang
HDFS-13346RBF: Fix synchronization of router quota and nameservice quotaMajor.liuhongtongYiqun Lin
YARN-8247Incorrect HTTP status code returned by ATSv2 for non-whitelisted usersCriticalATSv2Charan HebriRohith Sharma K S
YARN-8130Race condition when container events are published for KILLED applicationsMajorATSv2Charan HebriRohith Sharma K S
YARN-7900[AMRMProxy] AMRMClientRelayer for stateful FederationInterceptorMajor.Botong HuangBotong Huang
HADOOP-15498TestHadoopArchiveLogs (#testGenerateScript, #testPrepareWorkingDir) fails on WindowsMinor.Anbang HuAnbang Hu
HADOOP-15497TestTrash should use proper test path to avoid failing on WindowsMinor.Anbang HuAnbang Hu
HDFS-13637RBF: Router fails when threadIndex (in ConnectionPool) wraps around Integer.MIN_VALUECriticalfederationCR HotaCR Hota
YARN-4781Support intra-queue preemption for fairness ordering policy.MajorschedulerWangda TanEric Payne
HADOOP-15506Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code blocksMinorfs/azureEsfandiar ManiiEsfandiar Manii
HADOOP-15529ContainerLaunch#testInvalidEnvVariableSubstitutionType is not supported in WindowsMinor.Giovanni Matteo FumarolaGiovanni Matteo Fumarola
HADOOP-15533Make WASB listStatus messages consistentTrivialfs/azureEsfandiar ManiiEsfandiar Manii
HADOOP-15458TestLocalFileSystem#testFSOutputStreamBuilder fails on WindowsMinortestXiao LiangXiao Liang
YARN-8481AMRMProxyPolicies should accept heartbeat response from new/unknown subclustersMinoramrmproxy, federationBotong HuangBotong Huang
HDFS-13528RBF: If a directory exceeds quota limit then quota usage is not refreshed for other mount entriesMajor.Dibyendu KarmakarDibyendu Karmakar
HDFS-13710RBF: setQuota and getQuotaUsage should check the dfs.federation.router.quota.enableMajorfederation, hdfsyanghuafengyanghuafeng
HDFS-13726RBF: Fix RBF configuration linksMinordocumentationTakanobu AsanumaTakanobu Asanuma
HDFS-13475RBF: Admin cannot enforce Router enter SafeModeMajor.Wei YanChao Sun
HDFS-13733RBF: Add Web UI configurations and descriptions to RBF documentMinordocumentationTakanobu AsanumaTakanobu Asanuma
HDFS-13743RBF: Router throws NullPointerException due to the invalid initialization of MountTableResolverMajor.Takanobu AsanumaTakanobu Asanuma
HDFS-13583RBF: Router admin clrQuota is not synchronized with nameserviceMajor.Dibyendu KarmakarDibyendu Karmakar
HDFS-13750RBF: Router ID in RouterRpcClient is always nullMajor.Takanobu AsanumaTakanobu Asanuma
YARN-8129Improve error message for invalid value in fields attributeMinorATSv2Charan HebriAbhishek Modi
YARN-8581[AMRMProxy] Add sub-cluster timeout in LocalityMulticastAMRMProxyPolicyMajoramrmproxy, federationBotong HuangBotong Huang
YARN-8673[AMRMProxy] More robust responseId resync after an YarnRM master slave switchMajoramrmproxyBotong HuangBotong Huang
HDFS-13848Refactor NameNode failover proxy providersMajorha, hdfs-clientKonstantin ShvachkoKonstantin Shvachko
HDFS-13634RBF: Configurable value in xml for async connection request queue size.MajorfederationCR HotaCR Hota
HADOOP-15731TestDistributedShell fails on WindowsMajor.Botong HuangBotong Huang
HADOOP-15759AliyunOSS: update oss-sdk version to 3.0.0Majorfs/osswujinhuwujinhu
HADOOP-15748S3 listing inconsistency can raise NPE in globberMajorfsSteve LoughranSteve Loughran
YARN-8696[AMRMProxy] FederationInterceptor upgrade: home sub-cluster heartbeat asyncMajornodemanagerBotong HuangBotong Huang
HADOOP-15671AliyunOSS: Support Assume Roles in AliyunOSSMajorfs/osswujinhuwujinhu
HDFS-13790RBF: Move ClientProtocol APIs to its own moduleMajor.Íñigo GoiriChao Sun
YARN-7652Handle AM register requests asynchronously in FederationInterceptorMajoramrmproxy, federationSubramaniam KrishnanBotong Huang
YARN-6989Ensure timeline service v2 codebase gets UGI from HttpServletRequest in a consistent wayMajortimelineserverVrushali CAbhishek Modi
YARN-3879[Storage implementation] Create HDFS backing storage implementation for ATS readsMajortimelineserverTsuyoshi OzawaAbhishek Modi
HADOOP-15837DynamoDB table Update can fail S3A FS initMajorfs/s3Steve LoughranSteve Loughran
HADOOP-15607AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStreamCritical.wujinhuwujinhu
HADOOP-15868AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copyMajorfs/osswujinhuwujinhu
YARN-8893[AMRMProxy] Fix thread leak in AMRMClientRelayer and UAM clientMajoramrmproxy, federationBotong HuangBotong Huang
YARN-8905[Router] Add JvmMetricsInfo and pause monitorMinor.Bibin ChundattBilwa S T
HADOOP-15917AliyunOSS: fix incorrect ReadOps and WriteOps in statisticsMajorfs/osswujinhuwujinhu
HADOOP-16009Replace the url of the repository in Apache Hadoop source codeMajordocumentationAkira AjisakaAkira Ajisaka
HADOOP-15323AliyunOSS: Improve copy file performance for AliyunOSSFileSystemStoreMajorfs/osswujinhuwujinhu
YARN-9182Backport YARN-6445 resource profile performance improvements to branch-2Major.Jonathan HungJonathan Hung
YARN-9181Backport YARN-6232 for generic resource type usage to branch-2Major.Jonathan HungJonathan Hung
YARN-9177Use resource map for app metrics in TestCombinedSystemMetricsPublisher for branch-2Major.Jonathan HungJonathan Hung
YARN-9188Port YARN-7136 to branch-2Major.Jonathan HungJonathan Hung
YARN-9187Backport YARN-6852 for GPU-specific native changes to branch-2Major.Jonathan HungJonathan Hung
YARN-9180Port YARN-7033 NM recovery of assigned resources to branch-3.0/branch-2Major.Jonathan HungJonathan Hung
YARN-9280Backport YARN-6620 to YARN-8200/branch-2Major.Jonathan HungJonathan Hung
YARN-9174Backport YARN-7224 for refactoring of GpuDevice classMajor.Jonathan HungJonathan Hung
YARN-9289Backport YARN-7330 for GPU in UI to branch-2Major.Jonathan HungJonathan Hung
HDFS-14262[SBN read] Unclear Log.WARN message in GlobalStateIdContextMajorhdfsShwetaShweta
YARN-8549Adding a NoOp timeline writer and reader plugin classes for ATSv2MinorATSv2, timelineclient, timelineserverPrabha ManepalliPrabha Manepalli
HADOOP-16109Parquet reading S3AFileSystem causes EOFBlockerfs/s3Dave ChristiansonSteve Loughran
YARN-9397Fix empty NMResourceInfo object test failures in branch-2Major.Jonathan HungJonathan Hung
HADOOP-16191AliyunOSS: improvements for copyFile/copyDirectory and loggingMajorfs/osswujinhuwujinhu
YARN-9271Backport YARN-6927 for resource type support in MapReduceMajor.Jonathan HungJonathan Hung
YARN-9291Backport YARN-7637 to branch-2Major.Jonathan HungJonathan Hung
YARN-9409Port resource type changes from YARN-7237 to branch-3.0/branch-2Major.Jonathan HungJonathan Hung
YARN-9272Backport YARN-7738 for refreshing max allocation for multiple resource typesMajor.Jonathan HungJonathan Hung
HADOOP-16205Backporting ABFS driver from trunk to branch 2.0Majorfs/azureEsfandiar ManiiYuan Gao
HADOOP-16269ABFS: add listFileStatus with StartFromMajorfs/azureDa ZhouDa Zhou
HADOOP-16306AliyunOSS: Remove temporary files when upload small files to OSSMajorfs/osswujinhuwujinhu
HDFS-14034Support getQuotaUsage API in WebHDFSMajorfs, webhdfsErik KrogenChao Sun
YARN-9775RMWebServices /scheduler-conf GET returns all hadoop configurations for ZKConfigurationStoreMajorrestapiPrabhu JosephPrabhu Joseph
HDFS-14771Backport HDFS-14617 to branch-2 (Improve fsimage load time by writing sub-sections to the fsimage index)MajornamenodeXiaoqiao HeXiaoqiao He
HDFS-14822[SBN read] Revisit GlobalStateIdContext locking when getting server state idMajorhdfsChen LiangChen Liang
HDFS-14785[SBN read] Change client logging to be less aggressiveMajorhdfsChen LiangChen Liang
HDFS-14858[SBN read] Allow configurably enable/disable AlignmentContext on NameNodeMajorhdfsChen LiangChen Liang
HDFS-12979StandbyNode should upload FsImage to ObserverNode after checkpointing.MajorhdfsKonstantin ShvachkoChen Liang
HADOOP-16630Backport HADOOP-16548 - “ABFS: Config to enable/disable flush operation” to branch-2Minorfs/azureSneha VijayarajanSneha Vijayarajan
HADOOP-16631Backport HADOOP-16578 - “ABFS: fileSystemExists() should not call container level apis” to Branch-2Majorfs/azureSneha VijayarajanSneha Vijayarajan
HDFS-14162Balancer should work with ObserverNodeMajor.Konstantin ShvachkoErik Krogen

OTHER:

JIRASummaryPriorityComponentReporterContributor
HADOOP-15149CryptoOutputStream should implement StreamCapabilitiesMajorfsMike DrobXiao Chen
HADOOP-15177Update the release year to 2018BlockerbuildAkira AjisakaBharat Viswanadham
YARN-8412Move ResourceRequest.clone logic everywhere into a proper APIMinor.Botong HuangBotong Huang
HDFS-13870WebHDFS: Document ALLOWSNAPSHOT and DISALLOWSNAPSHOT API docMinordocumentation, webhdfsSiyao MengSiyao Meng
HDFS-12729Document special paths in HDFSMajordocumentationChristopher DouglasMasatake Iwasaki
HADOOP-15711Move branch-2 precommit/nightly test builds to java 8Critical.Jonathan HungJonathan Hung
HDFS-14510Backport HDFS-13087 to branch-2 (Snapshotted encryption zone information should be immutable)Majorencryption, snapshotsWei-Chiu ChuangSiyao Meng
HDFS-14585Backport HDFS-8901 Use ByteBuffer in DFSInputStream#read to branch2.9Major.Lisheng SunLisheng Sun
HDFS-14483Backport HDFS-14111,HDFS-3246 ByteBuffer pread interface to branch-2.9Major.Zheng HuLisheng Sun
YARN-9559Create AbstractContainersLauncher for pluggable ContainersLauncher logicMajor.Jonathan HungJonathan Hung
HDFS-14725Backport HDFS-12914 to branch-2 (Block report leases cause missing blocks until next report)MajornamenodeWei-Chiu ChuangXiaoqiao He
YARN-8200Backport resource types/GPU features to branch-3.0/branch-2Major.Jonathan HungJonathan Hung
HADOOP-16555Update commons-compress to 1.19Major.Wei-Chiu ChuangYiSheng Lien
YARN-9730Support forcing configured partitions to be exclusive based on app node labelMajor.Jonathan HungJonathan Hung
HADOOP-16544update io.netty in branch-2Major.Wei-Chiu ChuangMasatake Iwasaki
HADOOP-16588Update commons-beanutils version to 1.9.4 in branch-2Critical.Wei-Chiu ChuangWei-Chiu Chuang