Apache Hadoop Changelog

Release 3.0.1 - 2018-03-16

INCOMPATIBLE CHANGES:

JIRASummaryPriorityComponentReporterContributor
HDFS-12990Change default NameNode RPC port back to 8020CriticalnamenodeXiao ChenXiao Chen

IMPORTANT ISSUES:

JIRASummaryPriorityComponentReporterContributor
HDFS-13083RBF: Fix doc error setting up clientMajorfederationtartarustartarus

IMPROVEMENTS:

JIRASummaryPriorityComponentReporterContributor
HADOOP-14872CryptoInputStream should implement unbufferMajorfs, securityJohn ZhugeJohn Zhuge
YARN-7414FairScheduler#getAppWeight() should be moved into FSAppAttempt#getWeight()MinorfairschedulerDaniel TempletonSoumabrata Chakraborty
HADOOP-15023ValueQueue should also validate (lowWatermark * numValues) > 0 on constructionMinor.Xiao ChenXiao Chen
HDFS-12814Add blockId when warning slow mirror/disk in BlockReceiverTrivialhdfsJiandan YangJiandan Yang
YARN-7524Remove unused FairSchedulerEventLogMajorfairschedulerWilfred SpiegelenburgWilfred Spiegelenburg
YARN-7495Improve robustness of the AggregatedLogDeletionServiceMajorlog-aggregationJonathan EaglesJonathan Eagles
YARN-7611Node manager web UI should display container type in containers pageMajornodemanager, webappWeiwei YangWeiwei Yang
YARN-6483Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes returned to the AMMajorresourcemanagerJuan Rodríguez HortaláJuan Rodríguez Hortalá
HADOOP-15056Fix TestUnbuffer#testUnbufferException failureMinortestJack BeardenJack Bearden
HADOOP-15012Add readahead, dropbehind, and unbuffer to StreamCapabilitiesMajorfsJohn ZhugeJohn Zhuge
HADOOP-15104AliyunOSS: change the default value of max error retryMajorfs/osswujinhuwujinhu
HDFS-12910Secure Datanode Starter should log the port when it fails to bindMinordatanodeStephen O'DonnellStephen O'Donnell
HDFS-12819Setting/Unsetting EC policy shows warning if the directory is not emptyMinorerasure-codingLei (Eddy) XuLei (Eddy) Xu
HDFS-12927Update erasure coding doc to address unsupported APIsMajorerasure-codingLei (Eddy) XuLei (Eddy) Xu
YARN-5418When partial log aggregation is enabled, display the list of aggregated files on the container log pageMajor.Siddharth SethXuan Gong
HDFS-12818Support multiple storages in DataNodeCluster / SimulatedFSDatasetMinordatanode, testErik KrogenErik Krogen
HDFS-9023When NN is not able to identify DN for replication, reason behind it can be loggedCriticalhdfs-client, namenodeSurendra Singh LilhoreXiao Chen
HDFS-11847Enhance dfsadmin listOpenFiles command to list files blocking datanode decommissioningMajorhdfsManoj GovindassamyManoj Govindassamy
YARN-7678Ability to enable logging of container memory statsMajornodemanagerJim BrennanJim Brennan
HDFS-11848Enhance dfsadmin listOpenFiles command to list files under a given pathMajor.Manoj GovindassamyYiqun Lin
HDFS-12945Switch to ClientProtocol instead of NamenodeProtocols in NamenodeWebHdfsMethodsMinor.Wei YanWei Yan
YARN-7590Improve container-executor validation checkMajorsecurity, yarnEric YangEric Yang
MAPREDUCE-6984MR AM to clean up temporary files from previous attempt in case of no recoveryMajorapplicationmasterGergo RepasGergo Repas
HADOOP-15185Update adls connector to use the current version of ADLS SDKMajorfs/adlAtul SikariaAtul Sikaria
HADOOP-15189backport HADOOP-15039 to branch-2 and branch-3Blocker.Genmao YuGenmao Yu
HADOOP-15186Allow Azure Data Lake SDK dependency version to be set on the command lineMajorbuild, fs/adlVishwajeet DusaneVishwajeet Dusane
HDFS-13092Reduce verbosity for ThrottledAsyncChecker.java:scheduleMinordatanodeMukul Kumar SinghMukul Kumar Singh
HDFS-13062Provide support for JN to use separate journal disk per namespaceMajor.Bharat ViswanadhamBharat Viswanadham
HADOOP-15212Add independent secret manager method for logging expired tokensMajorsecurityDaryn SharpDaryn Sharp
YARN-7728Expose container preemptions related information in Capacity Scheduler queue metricsMajor.Eric PayneEric Payne
MAPREDUCE-7048Uber AM can crash due to unknown task in statusUpdateMajormr-amPeter BacskoPeter Bacsko
HADOOP-15204Add Configuration API for parsing storage sizesMinorconfAnu EngineerAnu Engineer

BUG FIXES:

JIRASummaryPriorityComponentReporterContributor
YARN-7361Improve the docker container runtime documentationMajor.Shane KumpfShane Kumpf
YARN-7489ConcurrentModificationException in RMAppImpl#getRMAppMetricsMajorcapacityschedulerTao YangTao Yang
YARN-7525Incorrect query parameters in cluster nodes REST API documentMinordocumentationTao YangTao Yang
HADOOP-15046Document Apache Hadoop does not support Java 9 in BUILDING.txtMajordocumentationAkira AjisakaHanisha Koneru
YARN-7531ResourceRequest.equal does not check ExecutionTypeRequest.enforceExecutionType()MajorapiHaibo ChenHaibo Chen
YARN-7513Remove scheduler lock in FSAppAttempt.getWeight()MinorfairschedulerWilfred SpiegelenburgWilfred Spiegelenburg
YARN-7390All reservation related test cases failed when TestYarnClient runs against Fair Scheduler.Majorfairscheduler, reservation systemYufei GuYufei Gu
YARN-7363ContainerLocalizer doesn't have a valid log4j config when using LinuxContainerExecutorMajornodemanagerYufei GuYufei Gu
HDFS-12832INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exitCriticalnamenodeDENG FEIKonstantin Shvachko
HADOOP-15042Azure PageBlobInputStream.skip() can return negative value when numberOfPagesRemaining is 0Minorfs/azureRajesh BalamohanRajesh Balamohan
YARN-7558“yarn logs” command fails to get logs for running containers if UI authentication is enabled.Critical.Namit MaheshwariXuan Gong
HDFS-12638Delete copy-on-truncate block along with the original block, when deleting a file being truncatedBlockerhdfsJiandan YangKonstantin Shvachko
HDFS-12836startTxId could be greater than endTxId when tailing in-progress edit logMajorhdfsChao SunChao Sun
MAPREDUCE-5124AM lacks flow control for task eventsMajormr-amJason LowePeter Bacsko
YARN-7589TestPBImplRecords fails with NullPointerExceptionMajor.Jason LoweDaniel Templeton
YARN-7455quote_and_append_arg can overflow bufferMajornodemanagerJason LoweJim Brennan
HADOOP-15058create-release site build outputs dummy shaded jars due to skipShadeBlocker.Andrew WangAndrew Wang
HADOOP-14985Remove subversion related code from VersionInfoMojo.javaMinorbuildAkira AjisakaAjay Kumar
HADOOP-15098TestClusterTopology#testChooseRandom fails intermittentlyMajortestZsolt VenczelZsolt Venczel
HDFS-12891Do not invalidate blocks if toInvalidate is emptyMajor.Zsolt VenczelZsolt Venczel
YARN-7647NM print inappropriate error log when node-labels is enabledMinor.Yang WangYang Wang
HDFS-12907Allow read-only access to reserved raw for non-superusersMajornamenodeDaryn SharpRushabh S Shah
HDFS-12881Output streams closed with IOUtils suppressing write errorsMajor.Jason LoweAjay Kumar
YARN-7595Container launching code suppresses close exceptions after writesMajornodemanagerJason LoweJim Brennan
HADOOP-15085Output streams closed with IOUtils suppressing write errorsMajor.Jason LoweJim Brennan
YARN-7629TestContainerLaunch# fails after YARN-7381Major.Jason LoweJason Lowe
YARN-7664Several javadoc errorsBlocker.Sean MackrorySean Mackrory
HADOOP-15123KDiag tries to load krb5.conf from KRB5CCNAME instead of KRB5_CONFIGMinorsecurityVipin RathorVipin Rathor
YARN-7661NodeManager metrics return wrong value after update node resourceMajor.Yang WangYang Wang
HDFS-12347TestBalancerRPCDelay#testBalancerRPCDelay fails very frequentlyCriticaltestXiao ChenBharat Viswanadham
HDFS-12930Remove the extra space in HdfsImageViewer.mdTrivialdocumentationYiqun LinRahul Pathak
YARN-7662[Atsv2] Define new set of configurations for reader and collectors to bind.Major.Rohith Sharma K SRohith Sharma K S
HDFS-12845JournalNode Test failuresMajorjournal-nodeBharat ViswanadhamBharat Viswanadham
YARN-7466ResourceRequest has a different default for allocationRequestId than ContainerMajor.Chandni SinghChandni Singh
YARN-7674Update Timeline Reader web app address in UI2Major.Rohith Sharma K SSunil G
HDFS-12938TestErasureCodigCLI testAll failing consistently.Majorerasure-coding, hdfsRushabh S ShahAjay Kumar
YARN-7542Fix issue that causes some Running Opportunistic Containers to be recovered as PAUSEDMajor.Arun SureshSampada Dehankar
HDFS-12915Fix findbugs warning in INodeFile$HeaderFormat.getBlockLayoutRedundancyMajornamenodeWei-Chiu ChuangChris Douglas
HADOOP-15122Lock down version of doxia-module-markdown pluginBlocker.Elek, MartonElek, Marton
HADOOP-15143NPE due to Invalid KerberosTicket in UGIMajor.Jitendra Nath PandeyMukul Kumar Singh
YARN-7692Skip validating priority acls while recovering applicationsBlockerresourcemanagerCharan HebriSunil G
MAPREDUCE-7028Concurrent task progress updates causing NPE in Application MasterBlockermr-amGergo RepasGergo Repas
YARN-7602NM should reference the singleton JvmMetrics instanceMajornodemanagerHaibo ChenHaibo Chen
HDFS-12913TestDNFencingWithReplication.testFencingStress fix mini cluster not yet active issueMajor.Zsolt VenczelZsolt Venczel
HDFS-12860StripedBlockUtil#getRangesInternalBlocks throws exception for the block group size larger than 2GBMajorerasure-codingLei (Eddy) XuLei (Eddy) Xu
YARN-7619Max AM Resource value in Capacity Scheduler UI has to be refreshed for every userMajorcapacity scheduler, yarnEric PayneEric Payne
YARN-7699queueUsagePercentage is coming as INF for getApp REST api callMajorwebappSunil GSunil G
HDFS-12985NameNode crashes during restart after an OpenForWrite file present in the Snapshot got deletedMajorhdfsManoj GovindassamyManoj Govindassamy
YARN-7508NPE in FiCaSchedulerApp when debug log enabled in async-scheduling modeMajorcapacityschedulerTao YangTao Yang
YARN-7663RMAppImpl:Invalid event: START at KILLEDMinorresourcemanagerlujielujie
YARN-6948Invalid event: ATTEMPT_ADDED at FINAL_SAVINGMinoryarnlujielujie
HDFS-12994TestReconstructStripedFile.testNNSendsErasureCodingTasks fails due to socket timeoutMajorerasure-codingLei (Eddy) XuLei (Eddy) Xu
YARN-7665Allow FS scheduler state dump to be turned on/off separately from FS debug logMajor.Wilfred SpiegelenburgWilfred Spiegelenburg
HADOOP-15060TestShellBasedUnixGroupsMapping.testFiniteGroupResolutionTime flakyMajor.Miklos SzegediMiklos Szegedi
YARN-7735Fix typo in YARN documentationMinordocumentationTakanobu AsanumaTakanobu Asanuma
YARN-7727Incorrect log levels in few logs with QueuePriorityContainerCandidateSelectorMinoryarnPrabhu JosephPrabhu Joseph
HDFS-11915Sync rbw dir on the first hsync() to avoid file lost on power failureCritical.Kanaka Kumar AvvaruVinayakumar B
YARN-7705Create the container log directory with correct sticky bit in C codeMajornodemanagerYufei GuYufei Gu
YARN-7479TestContainerManagerSecurity.testContainerManager[Simple] flaky in trunkMajortestBotong HuangAkira Ajisaka
HDFS-13004TestLeaseRecoveryStriped#testLeaseRecovery is failing when safeLength is 0MB or larger than the test fileMajorhdfsZsolt VenczelZsolt Venczel
HDFS-9049Make Datanode Netty reverse proxy port to be configurableMajordatanodeVinayakumar BVinayakumar B
YARN-7758Add an additional check to the validity of container and application ids passed to container-executorMajornodemanagerMiklos SzegediYufei Gu
HADOOP-15150in FsShell, UGI params should be overidden through env vars(-D arg)Major.Brahma Reddy BattulaBrahma Reddy Battula
HADOOP-15166CLI MiniCluster fails with ClassNotFoundException o.a.h.yarn.server.timelineservice.collector.TimelineCollectorManagerMajor.Gera ShegalovGera Shegalov
HDFS-13039StripedBlockReader#createBlockReader leaks socket on IOExceptionCriticaldatanode, erasure-codingLei (Eddy) XuLei (Eddy) Xu
HADOOP-15181Typo in SecureMode.mdTrivialdocumentationMasahiro TanakaMasahiro Tanaka
YARN-7796Container-executor fails with segfault on certain OS configurationsMajornodemanagerGergo RepasGergo Repas
YARN-7806Distributed Shell should use timeline async api'sMajordistributed-shellSumana SathishRohith Sharma K S
MAPREDUCE-7015Possible race condition in JHS if the job is not loadedMajorjobhistoryserverPeter BacskoPeter Bacsko
YARN-7737prelaunch.err file not found exception on container failureMajor.Jonathan HungKeqiu Hu
HDFS-13063Fix the incorrect spelling in HDFSHighAvailabilityWithQJM.mdTrivialdocumentationJianfei JiangJianfei Jiang
YARN-7102NM heartbeat stuck when responseId overflows MAX_INTCritical.Botong HuangBotong Huang
MAPREDUCE-7041MR should not try to clean up at first job attemptMajor.Takanobu AsanumaGergo Repas
HDFS-13054Handling PathIsNotEmptyDirectoryException in DFSClient delete callMajorhdfs-clientNanda kumarNanda kumar
MAPREDUCE-7020Task timeout in uber mode can crash AMMajormr-amAkira AjisakaPeter Bacsko
YARN-7765[Atsv2] GSSException: No valid credentials provided - Failed to find any Kerberos tgt thrown by Timelinev2Client & HBaseClient in NMBlocker.Sumana SathishRohith Sharma K S
HDFS-13065TestErasureCodingMultipleRacks#testSkewedRack3 is failingMajorhdfsGabor BotaGabor Bota
HDFS-12974Exception message is not printed when creating an encryption zone fails with AuthorizationExceptionMinorencryptionfang zhenyifang zhenyi
YARN-7698A misleading variable's name in ApplicationAttemptEventDispatcherMinorresourcemanagerJinjiang LingJinjiang Ling
YARN-7790Improve Capacity Scheduler Async Scheduling to better handle node failuresCritical.Sumana SathishWangda Tan
HDFS-12528Add an option to not disable short-circuit reads on failuresMajorhdfs-client, performanceAndre AraujoXiao Chen
HDFS-12897getErasureCodingPolicy should handle .snapshot dir betterMajorerasure-coding, hdfs, snapshotsHarshakiran ReddyLiXin Ge
MAPREDUCE-7033Map outputs implicitly rely on permissive umask for shuffleCriticalmrv2Jason LoweJason Lowe
HDFS-12942Synchronization issue in FSDataSetImpl#moveBlockMajor.Ajay KumarAjay Kumar
HDFS-13048LowRedundancyReplicatedBlocks metric can be negativeMajormetricsAkira AjisakaAkira Ajisaka
HDFS-13100Handle IllegalArgumentException when GETSERVERDEFAULTS is not implemented in webhdfs.Criticalhdfs, webhdfsYongjun ZhangYongjun Zhang
YARN-7849TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization fails due to heartbeat sync errorMajortestJason LoweBotong Huang
YARN-7801AmFilterInitializer should addFilter after fill all parametersCritical.Sumana SathishWangda Tan
YARN-7890NPE during container relaunchMajor.Billie RinaldiJason Lowe
YARN-7873Revert YARN-6078Blocker.Billie RinaldiBillie Rinaldi
HDFS-13115In getNumUnderConstructionBlocks(), ignore the inodeIds for which the inodes have been deletedMajor.Yongjun ZhangYongjun Zhang
HDFS-12935Get ambiguous result for DFSAdmin command in HA mode when only one namenode is upMajortoolsJianfei JiangJianfei Jiang
HDFS-13120Snapshot diff could be corrupted after concatMajornamenode, snapshotsXiaoyu YaoXiaoyu Yao
HDFS-10453ReplicationMonitor thread could stuck for long time due to the race between replication and delete of same file in a large cluster.MajornamenodeHe XiaoqiaoHe Xiaoqiao
HDFS-8693refreshNamenodes does not support adding a new standby to a running DNCriticaldatanode, haJian FangAjith S
MAPREDUCE-7052TestFixedLengthInputFormat#testFormatCompressedIn is flakyMajorclient, testPeter BacskoPeter Bacsko
HDFS-13112Token expiration edits may cause log corruption or deadlockCriticalnamenodeDaryn SharpDaryn Sharp
YARN-7937Fix http method name in Cluster Application Timeout Update API example requestMinordocs, documentationCharan HebriCharan Hebri
HADOOP-10571Use Log.*(Object, Throwable) overload to log exceptionsMajor.Arpit AgarwalAndras Bokor
HDFS-13164File not closed if streamer fail with DSQuotaExceededExceptionMajorhdfs-clientXiao ChenXiao Chen
HDFS-13244Add stack, conf, metrics links to utilities dropdown in NN webUIMajor.Bharat ViswanadhamBharat Viswanadham
YARN-8022ResourceManager UI cluster/app/<app-id> page fails to renderBlockerwebappTarun ParimiTarun Parimi

TESTS:

JIRASummaryPriorityComponentReporterContributor
MAPREDUCE-7011TestClientDistributedCacheManager::testDetermineCacheVisibilities assumes all parent dirs set other execTrivial.Chris DouglasChris Douglas

SUB-TASKS:

JIRASummaryPriorityComponentReporterContributor
HADOOP-14993AliyunOSS: Override listFiles and listLocatedStatusMajorfs/ossGenmao YuGenmao Yu
YARN-6953Clean up ResourceUtils.setMinimumAllocationForMandatoryResources() and setMaximumAllocationForMandatoryResources()MinorresourcemanagerDaniel TempletonManikandan R
HDFS-12801RBF: Set MountTableResolver as default file resolverMinor.Íñigo GoiriÍñigo Goiri
HADOOP-15024AliyunOSS: support user agent configuration and include that & Hadoop version information to oss serverMajorfs, fs/ossSammiChenSammiChen
HDFS-12858RBF: Add router admin commands usage in HDFS commands reference docMinordocumentationYiqun LinYiqun Lin
HDFS-12835RBF: Fix Javadoc parameter errorsMinor.Wei YanWei Yan
YARN-6907Node information page in the old web UI should report resource typesMajorresourcemanagerDaniel TempletonGergely Novák
HDFS-12396Webhdfs file system should get delegation token from kms provider.Majorencryption, kms, webhdfsRushabh S ShahRushabh S Shah
YARN-7610Extend Distributed Shell to support launching job with opportunistic containersMajorapplications/distributed-shellWeiwei YangWeiwei Yang
HDFS-12875RBF: Complete logic for -readonly option of dfsrouteradmin add commandMajor.Yiqun LinÍñigo Goiri
YARN-7383Node resource is not parsed correctly for resource names containing dotMajornodemanager, resourcemanagerJonathan HungGergely Novák
YARN-7119Support multiple resource types in rmadmin updateNodeResource commandMajornodemanager, resourcemanagerDaniel TempletonManikandan R
YARN-7617Add a flag in distributed shell to automatically PROMOTE opportunistic containers to guaranteed once they are startedMinorapplications/distributed-shellWeiwei YangWeiwei Yang
HDFS-12937RBF: Add more unit tests for router admin commandsMajortestYiqun LinYiqun Lin
YARN-7032[ATSv2] NPE while starting hbase co-processor when HBase authorization is enabled.Critical.Rohith Sharma K SRohith Sharma K S
HADOOP-14965s3a input stream “normal” fadvise mode to be adaptiveMajorfs/s3Steve LoughranSteve Loughran
HADOOP-15086NativeAzureFileSystem file rename is not atomicMajorfs/azureShixiong ZhuThomas Marquardt
HDFS-12988RBF: Mount table entries not properly updated in the local cacheMajor.Íñigo GoiriÍñigo Goiri
YARN-7716metricsTimeStart and metricsTimeEnd should be all lower case in the docMajortimelinereaderHaibo ChenHaibo Chen
HDFS-12802RBF: Control MountTableResolver cache sizeMajor.Íñigo GoiriÍñigo Goiri
HDFS-12919RBF: Support erasure coding methods in RouterRpcServerCritical.Íñigo GoiriÍñigo Goiri
YARN-6736Consider writing to both ats v1 & v2 from RM for smoother upgradesMajortimelineserverVrushali CAaron Gresch
HADOOP-15027AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performanceMajorfs/osswujinhuwujinhu
HDFS-13028RBF: Fix spurious TestRouterRpc#testProxyGetStatsMinor.Íñigo GoiriÍñigo Goiri
YARN-5094some YARN container events have timestamp of -1CriticaltimelineserverSangjin LeeHaibo Chen
YARN-7782Enable user re-mapping for Docker containers in yarn-default.xmlBlockersecurity, yarnEric YangEric Yang
HDFS-12772RBF: Federation Router State State Store internal APIMajor.Íñigo GoiriÍñigo Goiri
HDFS-13042RBF: Heartbeat Router StateMajor.Íñigo GoiriÍñigo Goiri
HDFS-13049RBF: Inconsistent Router OPTS config in branch-2 and branch-3Minor.Wei YanWei Yan
HDFS-12574Add CryptoInputStream to WebHdfsFileSystem read call.Majorencryption, kms, webhdfsRushabh S ShahRushabh S Shah
HDFS-13044RBF: Add a safe mode for the RouterMajor.Íñigo GoiriÍñigo Goiri
HDFS-13043RBF: Expose the state of the Routers in the federationMajor.Íñigo GoiriÍñigo Goiri
HDFS-12997Move logging to slf4j in BlockPoolSliceStorage and StorageMajor.Ajay KumarAjay Kumar
HDFS-13068RBF: Add router admin option to manage safe modeMajor.Íñigo GoiriYiqun Lin
HADOOP-15247Move commons-net up to 3.6MinorfsSteve LoughranSteve Loughran
HADOOP-15090Add ADL troubleshooting docMajordocumentation, fs/adlSteve LoughranSteve Loughran

OTHER:

JIRASummaryPriorityComponentReporterContributor
HADOOP-15149CryptoOutputStream should implement StreamCapabilitiesMajorfsMike DrobXiao Chen
YARN-7691Add Unit Tests for ContainersLauncherMajor.Sampada DehankarSampada Dehankar
HADOOP-15177Update the release year to 2018BlockerbuildAkira AjisakaBharat Viswanadham
HADOOP-15197Remove tomcat from the Hadoop-auth test bundleMajor.Xiao ChenXiao Chen