Apache Hadoop Changelog

Release 2.1.1-beta - 2013-09-16

INCOMPATIBLE CHANGES:

JIRASummaryPriorityComponentReporterContributor
HADOOP-9944RpcRequestHeaderProto defines callId as uint32 while ipc.Client.CONNECTION_CONTEXT_CALL_ID is signed (-3)Blocker.Arun C MurthyArun C Murthy
YARN-1170yarn proto definitions should specify package as ‘hadoop.yarn’Blocker.Arun C MurthyBinglin Chang
YARN-707Add user info in the YARN ClientTokenBlocker.Bikas SahaJason Lowe

IMPORTANT ISSUES:

JIRASummaryPriorityComponentReporterContributor

NEW FEATURES:

JIRASummaryPriorityComponentReporterContributor
HADOOP-9789Support server advertised kerberos principalsCriticalipc, securityDaryn SharpDaryn Sharp
HDFS-5118Provide testing support for DFSClient to drop RPC responsesMajor.Jing ZhaoJing Zhao
HDFS-5076Add MXBean methods to query NN‘s transaction information and JournalNode’s journal statusMinor.Jing ZhaoJing Zhao

IMPROVEMENTS:

JIRASummaryPriorityComponentReporterContributor
HADOOP-9962in order to avoid dependency divergence within Hadoop itself lets enable DependencyConvergenceMajorbuildRoman ShaposhnikRoman Shaposhnik
HADOOP-9945HAServiceState should have a state for stopped servicesMinorhaKarthik KambatlaKarthik Kambatla
HADOOP-9918Add addIfService() to CompositeServiceMinor.Karthik KambatlaKarthik Kambatla
HADOOP-9886Turn warning message in RetryInvocationHandler to debugMinor.Arpit GuptaArpit Gupta
HADOOP-9879Move the version info of zookeeper dependencies to hadoop-project/pomMinorbuildKarthik KambatlaKarthik Kambatla
HADOOP-9833move slf4j to version 1.7.5MinorbuildSteve LoughranKousuke Saruta
HADOOP-9831Make checknative shell command accessible on Windows.MinorbinChris NaurothChris Nauroth
HADOOP-9821ClientId should have getMsb/getLsb methodsMinor.Tsuyoshi OzawaTsuyoshi Ozawa
HADOOP-9803Add generic type parameter to RetryInvocationHandlerMinoripcTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HADOOP-9802Support Snappy codec on Windows.MajorioChris NaurothChris Nauroth
HADOOP-9686Easy access to final parameters in ConfigurationMajorconfJason LoweJason Lowe
HADOOP-9672Upgrade Avro dependency to 1.7.4Major.Sandy RyzaSandy Ryza
HADOOP-9669Reduce the number of byte array creations and copies in XDR data manipulationMajornfsTsz Wo Nicholas SzeHaohui Mai
HADOOP-9446Support Kerberos HTTP SPNEGO authentication for non-SUN JDKMajorsecurityYu GaoYu Gao
HADOOP-8814Inefficient comparison with the empty string. Use isEmpty() insteadMinorconf, fs, fs/s3, ha, io, metrics, performance, record, security, utilBrandon LiBrandon Li
HDFS-5128Allow multiple net interfaces to be used with HA namenode RPC serverCritical.Kihwal LeeKihwal Lee
HDFS-5061Make FSNameSystem#auditLoggers an unmodifiable listMajornamenodeArpit AgarwalArpit Agarwal
HDFS-5045Add more unit tests for retry cache to cover all AtMostOnce methodsMinor.Jing ZhaoJing Zhao
HDFS-4926namenode webserver's page has a tooltip that is inconsistent with the datanode HTML linkTrivialnamenodeJoseph LorenziniVivek Ganesan
HDFS-4905Add appendToFile command to “hdfs dfs”MinortoolsArpit AgarwalArpit Agarwal
HDFS-4513Clarify WebHDFS REST API that all JSON respsonses may contain additional propertiesMinordocumentation, webhdfsTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HDFS-3245Add metrics and web UI for cluster version summaryMajornamenodeTodd LipconRavi Prakash
MAPREDUCE-5478TeraInputFormat unnecessarily defines its own FileSplit subclassMinorexamplesSandy RyzaSandy Ryza
MAPREDUCE-5379Include token tracking ids in jobconfMajorjob submission, securitySandy RyzaKarthik Kambatla
MAPREDUCE-5367Local jobs all use same local working directoryMajor.Sandy RyzaSandy Ryza
MAPREDUCE-1981Improve getSplits performance by using listLocatedStatusMajorjob submissionHairong KuangHairong Kuang
YARN-1137Add support whitelist for system users to Yarn container-executor.cMajornodemanagerAlejandro AbdelnurRoman Shaposhnik
YARN-1117Improve help message for $ yarn applications and $yarn nodeMajorclientTassapol AthiapinyaXuan Gong
YARN-1081Minor improvement to output header for $ yarn node -listMinorclientTassapol AthiapinyaAkira AJISAKA
YARN-1080Improve help message for $ yarn logsMajorclientTassapol AthiapinyaXuan Gong
YARN-1074Clean up YARN CLI app list to show only running apps.MajorclientTassapol AthiapinyaXuan Gong
YARN-910Allow auxiliary services to listen for container starts and completionsMajornodemanagerSandy RyzaAlejandro Abdelnur
YARN-758Augment MockNM to use multiple coresMinor.Bikas SahaKarthik Kambatla
YARN-696Enable multiple states to to be specified in Resource Manager apps REST callMajorresourcemanagerTrevor LorimerTrevor Lorimer
YARN-589Expose a REST API for monitoring the fair schedulerMajorschedulerSandy RyzaSandy Ryza

BUG FIXES:

JIRASummaryPriorityComponentReporterContributor
HADOOP-9977Hadoop services won't start with different keypass and keystorepass when https is enabledMajorsecurityYesha VoraChris Nauroth
HADOOP-9961versions of a few transitive dependencies diverged between hadoop subprojectsMinorbuildRoman ShaposhnikRoman Shaposhnik
HADOOP-9960Upgrade Jersey version to 1.9Blocker.Brock NolandKarthik Kambatla
HADOOP-9958Add old constructor back to DelegationTokenInformation to unbreak downstream buildsMajor.Andrew WangAndrew Wang
HADOOP-9932Improper synchronization in RetryCacheBlocker.Kihwal LeeKihwal Lee
HADOOP-9924FileUtil.createJarWithClassPath() does not generate relative classpath correctlyMajorfsshanyu zhaoshanyu zhao
HADOOP-9916Race condition in ipc.Client causes TestIPC timeoutMinor.Binglin ChangBinglin Chang
HADOOP-9910proxy server start and stop documentation wrongMinor.André Kelpe
HADOOP-9906Move HAZKUtil to o.a.h.util.ZKUtil and make inner-classes publicMinorhaKarthik KambatlaKarthik Kambatla
HADOOP-9899Remove the debug message added by HADOOP-8855MinorsecurityTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HADOOP-9880SASL changes from HADOOP-9421 breaks Secure HA NNBlocker.Kihwal LeeDaryn Sharp
HADOOP-9868Server must not advertise kerberos realmBlockeripcDaryn SharpDaryn Sharp
HADOOP-9858Remove unused private RawLocalFileSystem#execCommand method from branch-2.TrivialfsChris NaurothChris Nauroth
HADOOP-9857Tests block and sometimes timeout on Windows due to invalid entropy source.Majorbuild, testChris NaurothChris Nauroth
HADOOP-9806PortmapInterface should check if the procedure is out-of-rangeMajornfsBrandon LiBrandon Li
HADOOP-9801Configuration#writeXml uses platform defaulting encoding, which may mishandle multi-byte characters.MajorconfChris NaurothChris Nauroth
HADOOP-9774RawLocalFileSystem.listStatus() return absolute paths when input path is relative on WindowsMajorfsshanyu zhaoshanyu zhao
HADOOP-9768chown and chgrp reject users and groups with spaces on platforms where spaces are otherwise acceptableMajorfsChris NaurothChris Nauroth
HADOOP-9757Har metadata cache can grow without limitMajorfsJason LoweCristina L. Abad
HADOOP-9557hadoop-client excludes commons-httpclientMajorbuildLohit VijayarenuLohit Vijayarenu
HADOOP-9435Support building the JNI code against the IBM JVMMajorbuildTian Hong WangTian Hong Wang
HADOOP-9381Document dfs cp -f optionTrivial.Keegan WittKeegan Witt
HADOOP-9315Port HADOOP-9249 hadoop-maven-plugins Clover fix to branch-2 to fix build failuresMajorbuildDennis YChris Nauroth
HDFS-5231Fix broken links in the document of HDFS FederationMinor.Haohui MaiHaohui Mai
HDFS-5219Add configuration keys for retry policy in WebHDFSFileSystemMajorwebhdfsHaohui MaiHaohui Mai
HDFS-5192NameNode may fail to start when dfs.client.test.drop.namenode.response.number is setMinor.Jing ZhaoJing Zhao
HDFS-5159Secondary NameNode fails to checkpoint if error occurs downloading edits on first checkpointMajornamenodeAaron T. MyersAaron T. Myers
HDFS-5150Allow per NN SPN for internal SPNEGO.Blocker.Kihwal LeeKihwal Lee
HDFS-5140Too many safemode monitor threads being created in the standby namenode causing it to fail with out of memory errorBlockerhaArpit GuptaJing Zhao
HDFS-5132Deadlock in NameNode between SafeModeMonitor#run and DatanodeManager#handleHeartbeatBlockernamenodeArpit GuptaKihwal Lee
HDFS-5124DelegationTokenSecretManager#retrievePassword can cause deadlock in NameNodeBlockernamenodeDeepesh KhandelwalDaryn Sharp
HDFS-5111Remove duplicated error message for snapshot commands when processing invalid argumentsMinorsnapshotsJing ZhaoJing Zhao
HDFS-5106TestDatanodeBlockScanner fails on Windows due to incorrect path formatMinortestChuan LiuChuan Liu
HDFS-5105TestFsck fails on WindowsMinor.Chuan LiuChuan Liu
HDFS-5103TestDirectoryScanner fails on WindowsMinortestChuan LiuChuan Liu
HDFS-5102Snapshot names should not be allowed to contain slash charactersMajorsnapshotsAaron T. MyersJing Zhao
HDFS-5100TestNamenodeRetryCache fails on Windows due to incorrect cleanupMinortestChuan LiuChuan Liu
HDFS-5099Namenode#copyEditLogSegmentsToSharedDir should close EditLogInputStreams upon finishingMajornamenodeChuan LiuChuan Liu
HDFS-5091Support for spnego keytab separate from the JournalNode keytab for secure HAMinor.Jing ZhaoJing Zhao
HDFS-5080BootstrapStandby not working with QJM when the existing NN is activeMajorha, qjmJing ZhaoJing Zhao
HDFS-5077NPE in FSNamesystem.commitBlockSynchronization()MajornamenodeKonstantin ShvachkoPlamen Jeliazkov
HDFS-5055nn fails to download checkpointed image from snn in some setupsBlockernamenodeAllen WittenauerVinayakumar B
HDFS-5047Supress logging of full stack trace of quota and lease exceptionsMajornamenodeKihwal LeeRobert Parker
HDFS-5043For HdfsFileStatus, set default value of childrenNum to -1 instead of 0 to avoid confusing applicationsMajor.Brandon LiBrandon Li
HDFS-5028LeaseRenewer throw java.util.ConcurrentModificationException when timeoutMajor.zhaoyunjiongzhaoyunjiong
HDFS-4993fsck can fail if a file is renamed or deletedMajor.Kihwal LeeRobert Parker
HDFS-4898BlockPlacementPolicyWithNodeGroup.chooseRemoteRack() fails to properly fallback to local rackMinornamenodeEric SirianniTsz Wo Nicholas Sze
HDFS-4680Audit logging of delegation tokens for MR tracingMajornamenode, securityAndrew WangAndrew Wang
HDFS-4632globStatus using backslash for escaping does not work on WindowsMajortestChris NaurothChuan Liu
HDFS-4594WebHDFS open sets Content-Length header to what is specified by length parameter rather than how much data is actually returned.MinorwebhdfsArpit GuptaChris Nauroth
HDFS-2994If lease soft limit is recovered successfully the append can failMajor.Todd LipconTao Luo
MAPREDUCE-5497‘5s sleep’ in MRAppMaster.shutDownJob is only needed before stopping ClientServiceMajor.Jian HeJian He
MAPREDUCE-5493In-memory map outputs can be leaked after shuffle completesBlockermrv2Jason LoweJason Lowe
MAPREDUCE-5483revert MAPREDUCE-5357MajordistcpAlejandro AbdelnurRobert Kanter
MAPREDUCE-5475MRClientService does not verify ACLs properlyBlockermr-am, mrv2Jason LoweJason Lowe
MAPREDUCE-5470LocalJobRunner does not work on Windows.Major.Chris NaurothSandy Ryza
MAPREDUCE-5468AM recovery does not work for map only jobsBlocker.Yesha VoraVinod Kumar Vavilapalli
MAPREDUCE-5454TestDFSIO fails intermittently on JDK7MajortestKarthik KambatlaKarthik Kambatla
MAPREDUCE-5446TestJobHistoryEvents and TestJobHistoryParsing have race conditionsMajormrv2, testJason LoweJason Lowe
MAPREDUCE-5440TestCopyCommitter Fails on JDK7Majormrv2Robert ParkerRobert Parker
MAPREDUCE-5428HistoryFileManager doesn't stop threads when service is stoppedMajorjobhistoryserver, mrv2Jason LoweKarthik Kambatla
MAPREDUCE-5425Junit in TestJobHistoryServer failing in jdk 7MajorjobhistoryserverAshwin ShankarRobert Parker
MAPREDUCE-5414TestTaskAttempt fails jdk7 with NullPointerExceptionMajortestNemon LouNemon Lou
MAPREDUCE-5385JobContext cache files api are brokenBlocker.Omkar Vinit JoshiOmkar Vinit Joshi
MAPREDUCE-5358MRAppMaster throws invalid transitions for JobImplMajormr-amDevaraj KDevaraj K
MAPREDUCE-5317Stale files left behind for failed jobsMajormrv2Ravi PrakashRavi Prakash
MAPREDUCE-5251Reducer should not implicate map attempt if it has insufficient space to fetch map outputMajormrv2Jason LoweAshwin Shankar
MAPREDUCE-5164command “mapred job” and “mapred queue” omit HADOOP_CLIENT_OPTSMajor.Nemon LouNemon Lou
MAPREDUCE-5020Compile failure with JDK8MajorclientTrevor RobinsonTrevor Robinson
MAPREDUCE-5001LocalJobRunner has race condition resulting in job failuresMajor.Brock NolandSandy Ryza
MAPREDUCE-3193FileInputFormat doesn't read files recursively in the input path dirMajormrv1, mrv2Ramgopal NDevaraj K
YARN-1194TestContainerLogsPage fails with native buildsMinornodemanagerRoman ShaposhnikRoman Shaposhnik
YARN-1189NMTokenSecretManagerInNM is not being told when applications have finishedBlocker.Jason LoweOmkar Vinit Joshi
YARN-1176RM web services ClusterMetricsInfo total nodes doesn't include unhealthy nodesCriticalresourcemanagerThomas GravesJonathan Eagles
YARN-1152Invalid key to HMAC computation error when getting application report for completed app attemptBlockerresourcemanagerJason LoweJason Lowe
YARN-1144Unmanaged AMs registering a tracking URI should not be proxy-fiedCriticalresourcemanagerAlejandro AbdelnurAlejandro Abdelnur
YARN-1124By default yarn application -list should display all the applications in a state other than FINISHED / FAILEDBlocker.Omkar Vinit JoshiXuan Gong
YARN-1120Make ApplicationConstants.Environment.USER definition OS neutralMinor.Chuan LiuChuan Liu
YARN-1107Job submitted with Delegation token in secured environment causes RM to fail during RM restartBlockerresourcemanagerArpit GuptaOmkar Vinit Joshi
YARN-1101Active nodes can be decremented below 0MajorresourcemanagerRobert ParkerRobert Parker
YARN-1094RM restart throws Null pointer Exception in Secure EnvBlocker.Yesha VoraVinod Kumar Vavilapalli
YARN-1093Corrections to Fair Scheduler documentationMajordocumentationWing Yew Poon
YARN-1083ResourceManager should fail when yarn.nm.liveness-monitor.expiry-interval-ms is set less than heartbeat intervalMajorresourcemanagerYesha VoraZhijie Shen
YARN-1082Secure RM with recovery enabled and rm state store on hdfs fails with gss exceptionBlockerresourcemanagerArpit GuptaVinod Kumar Vavilapalli
YARN-1078TestNodeManagerResync, TestNodeManagerShutdown, and TestNodeStatusUpdater fail on WindowsMinor.Chuan LiuChuan Liu
YARN-1077TestContainerLaunch fails on WindowsMinor.Chuan LiuChuan Liu
YARN-1025ResourceManager and NodeManager do not load native libraries on Windows.Majornodemanager, resourcemanagerChris NaurothChris Nauroth
YARN-1008MiniYARNCluster with multiple nodemanagers, all nodes have same key for allocationsMajornodemanagerAlejandro AbdelnurAlejandro Abdelnur
YARN-1006Nodes list web page on the RM web UI is brokenMajor.Jian HeXuan Gong
YARN-994HeartBeat thread in AMRMClientAsync does not handle runtime exception correctlyMajor.Xuan GongXuan Gong
YARN-981YARN/MR2/Job-history /logs link does not have correct contentMajor.Xuan GongJian He
YARN-966The thread of ContainerLaunch#call will fail without any signal if getLocalizedResources() is called when the container is not at LOCALIZEDMajor.Zhijie ShenZhijie Shen
YARN-957Capacity Scheduler tries to reserve the memory more than what node manager reports.Blocker.Omkar Vinit JoshiOmkar Vinit Joshi
YARN-948RM should validate the release container list before actually releasing themMajor.Omkar Vinit JoshiOmkar Vinit Joshi
YARN-942In Fair Scheduler documentation, inconsistency on which properties have prefixMajorschedulerSandy RyzaAkira AJISAKA
YARN-903DistributedShell throwing Errors in logs after successfull completionMajorapplications/distributed-shellAbhishek KapoorOmkar Vinit Joshi
YARN-881Priority#compareTo method seems to be wrong.Major.Jian HeJian He
YARN-643WHY appToken is removed both in BaseFinalTransition and AMUnregisteredTransition AND clientToken is removed in FinalTransition and not BaseFinalTransitionMajor.Jian HeXuan Gong
YARN-602NodeManager should mandatorily set some Environment variables into every containers that it launchesMajor.Xuan GongKenji Kikushima
YARN-337RM handles killed application tracking URL poorlyMajorresourcemanagerJason LoweJason Lowe
YARN-107ClientRMService.forceKillApplication() should handle the non-RUNNING applications properlyMajorresourcemanagerDevaraj KXuan Gong

TESTS:

JIRASummaryPriorityComponentReporterContributor

SUB-TASKS:

JIRASummaryPriorityComponentReporterContributor
HDFS-5249Fix dumper thread which may die silentlyMajornfsBrandon LiBrandon Li
HDFS-5234Move RpcFrameDecoder out of the public APIMinornfsHaohui MaiHaohui Mai
HDFS-5212Refactor RpcMessage and NFS3Response to support different types of authentication informationMajornfsJing ZhaoJing Zhao
HDFS-5199Add more debug trace for NFS READ and WRITETrivialnfsBrandon LiBrandon Li
HDFS-5136MNT EXPORT should give the full group list which can mount the exportsMajornfsBrandon LiBrandon Li
HDFS-5110Change FSDataOutputStream to HdfsDataOutputStream for opened streams to fix type cast errorMajornfsBrandon LiBrandon Li
HDFS-5107Fix array copy error in Readdir and Readdirplus responsesMajornfsBrandon LiBrandon Li
HDFS-5104Support dotdot name in NFS LOOKUP operationMajornfsBrandon LiBrandon Li
HDFS-5085Refactor o.a.h.nfs to support different types of authenticationsMajornfsBrandon LiJing Zhao
HDFS-5078Support file append in NFSv3 gateway to enable data streaming to HDFSMajornfsBrandon LiBrandon Li
HDFS-5071Change hdfs-nfs parent project to hadoop-projectMajornfsKihwal LeeBrandon Li
HDFS-5069Include hadoop-nfs and hadoop-hdfs-nfs into hadoop dist for NFS deploymentMajornfsBrandon LiBrandon Li
HDFS-5067Support symlink operationsMajornfsBrandon LiBrandon Li
HDFS-4971Move IO operations out of locking in OpenFileCtxMajornfsJing ZhaoJing Zhao
HDFS-4962Use enum for nfs constantsMinornfsTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HDFS-4947Add NFS server export table to control export by hostname or IP rangeMajornfsBrandon LiJing Zhao
HDFS-4763Add script changes/utility for starting NFS gatewayMajornfsBrandon LiBrandon Li
MAPREDUCE-5476Job can fail when RM restarts after staging dir is cleaned but before MR successfully unregister with RMBlocker.Jian HeJian He
MAPREDUCE-5466Historyserver does not refresh the result of restarted jobs after RM restartBlocker.Yesha VoraJian He
MAPREDUCE-5462In map-side sort, swap entire meta entries instead of indexes for better cache performanceMajorperformance, taskSandy RyzaSandy Ryza
MAPREDUCE-5441JobClient exit whenever RM issue Reboot command to 1st attempt App Master.Majorapplicationmaster, clientRohith Sharma K SJian He
YARN-1184ClassCastException is thrown during preemption When a huge job is submitted to a queue B whose resources is used by a job in queueAMajorcapacityscheduler, resourcemanagerJ.AndreinaChris Douglas
YARN-1116Populate AMRMTokens back to AMRMTokenSecretManager after RM restartsMajorresourcemanagerJian HeJian He
YARN-1085Yarn and MRv2 should do HTTP client authentication in kerberos setup.Blockernodemanager, resourcemanagerJaimin D JetlyOmkar Vinit Joshi
YARN-1049ContainerExistStatus should define a status for preempted containersBlockerapiAlejandro AbdelnurAlejandro Abdelnur
YARN-906Cancelling ContainerLaunch#call at KILLING causes that the container cannot be completedMajor.Zhijie ShenZhijie Shen
YARN-771AMRMClient support for resource blacklistingMajor.Bikas SahaJunping Du
YARN-573Shared data structures in Public Localizer and Private Localizer are not Thread safe.Critical.Omkar Vinit JoshiOmkar Vinit Joshi
YARN-540Race condition causing RM to potentially relaunch already unregistered AMs on RM restartMajorresourcemanagerJian HeJian He
YARN-502RM crash with NPE on NODE_REMOVED event with FairSchedulerMajor.Lohit VijayarenuMayank Bansal
YARN-292ResourceManager throws ArrayIndexOutOfBoundsException while handling CONTAINER_ALLOCATED for application attemptMajorresourcemanagerDevaraj KZhijie Shen

OTHER:

JIRASummaryPriorityComponentReporterContributor
YARN-1034Remove “experimental” in the Fair Scheduler documentationTrivialdocumentation, schedulerSandy RyzaKarthik Kambatla
YARN-1001YARN should provide per application-type and state statisticsBlockerapiSrimanth GunturiZhijie Shen