Apache Hadoop Changelog

Release 2.0.0-alpha - 2012-05-23

INCOMPATIBLE CHANGES:

JIRASummaryPriorityComponentReporterContributor
HADOOP-8314HttpServer#hasAdminAccess should return false if authorization is enabled but user is not authenticatedMajorsecurityAlejandro AbdelnurAlejandro Abdelnur
HADOOP-8270hadoop-daemon.sh stop action should return 0 for an already stopped serviceMinorscriptsRoman ShaposhnikRoman Shaposhnik
HADOOP-8184ProtoBuf RPC engine does not need it own reply packet - it can use the IPC layer reply packet.MajoripcSanjay RadiaSanjay Radia
HADOOP-8154DNS#getIPs shouldn't silently return the local host IP for bogus interface namesMajorconfEli CollinsEli Collins
HADOOP-8149cap space usage of default log4j rolling policyMajorconfPatrick HuntPatrick Hunt
HADOOP-7524Change RPC to allow multiple protocols including multiple versions of the same protocolMajoripcSanjay RadiaSanjay Radia
HDFS-3286When the threshold value for balancer is 0(zero) ,unexpected output is displayedMajorbalancer & moverJ.AndreinaAshish Singhi
HDFS-3164Move DatanodeInfo#hostName to DatanodeIDMajordatanodeEli CollinsEli Collins
HDFS-3144Refactor DatanodeID#getName by useMajordatanodeEli CollinsEli Collins
HDFS-3138Move DatanodeInfo#ipcPort to DatanodeIDMajor.Eli CollinsEli Collins
HDFS-3137Bump LAST_UPGRADABLE_LAYOUT_VERSION to -16MajornamenodeEli CollinsEli Collins
HDFS-3044fsck move should be non-destructive by defaultMajornamenodeEli CollinsColin Patrick McCabe
HDFS-2303Unbundle jsvcMajorbuild, scriptsRoman ShaposhnikMingjie Lai
HDFS-395DFS Scalability: Incremental block reportsMajordatanode, namenodedhruba borthakurTomasz Nykiel

IMPORTANT ISSUES:

JIRASummaryPriorityComponentReporterContributor

NEW FEATURES:

JIRASummaryPriorityComponentReporterContributor
HADOOP-8343Allow configuration of authorization for JmxJsonServlet and MetricsServletMajorutilPhilip ZeyligerAlejandro Abdelnur
HADOOP-8206Common portion of ZK-based failover controllerMajorhaTodd LipconTodd Lipcon
HADOOP-8121Active Directory Group Mapping ServiceMajorsecurityJonathan NatkinsJonathan Natkins
HADOOP-7876Allow access to BlockKey/DelegationKey encoded key for RPC over protobufMajoripcSuresh SrinivasSuresh Srinivas
HADOOP-7806Support binding to sub-interfacesMajorutilHarsh JHarsh J
HADOOP-7454Common side of High Availability Framework (HDFS-1623)Major.Aaron T. Myers
HADOOP-7030Add TableMapping topology implementation to read host to rack mapping from a fileMajor.Patrick AngelesTom White
HDFS-3167CLI-based driver for MiniDFSClusterMinortestHenry RobinsonHenry Robinson
HDFS-3148The client should be able to use multiple local interfaces for data transferMajorhdfs-client, performanceEli CollinsEli Collins
HDFS-3102Add CLI tool to initialize the shared-edits dirMajorha, namenodeTodd LipconAaron T. Myers
HDFS-3004Implement Recovery ModeMajortoolsColin Patrick McCabeColin Patrick McCabe
HDFS-3000Add a public API for setting quotasMajorhdfs-clientAaron T. MyersAaron T. Myers
HDFS-2941Add an administrative command to download a copy of the fsimage from the NNMajorhdfs-client, namenodeAaron T. MyersAaron T. Myers
HDFS-2731HA: Autopopulate standby name dirs if they're emptyMajorhaAaron T. MyersTodd Lipcon
HDFS-2430The number of failed or low-resource volumes the NN can tolerate should be configurableMajornamenodeAaron T. MyersAaron T. Myers
HDFS-1623High Availability Framework for HDFS NNMajor.Sanjay Radia
HDFS-234Integration with BookKeeper logging systemMajor.Luca TelloliIvan Kelly

IMPROVEMENTS:

JIRASummaryPriorityComponentReporterContributor
HADOOP-8366Use ProtoBuf for RpcResponseHeaderBlocker.Sanjay RadiaSanjay Radia
HADOOP-8356FileSystem service loading mechanism should print the FileSystem impl it is failing to loadMajorfsAlejandro AbdelnurAlejandro Abdelnur
HADOOP-8353hadoop-daemon.sh and yarn-daemon.sh can be misleading on stopMajorscriptsRoman ShaposhnikRoman Shaposhnik
HADOOP-8350Improve NetUtils.getInputStream to return a stream which has a tunable timeoutMajorutilTodd LipconTodd Lipcon
HADOOP-8285Use ProtoBuf for RpcPayLoadHeaderMajoripcSanjay RadiaSanjay Radia
HADOOP-8280Move VersionUtil/TestVersionUtil and GenericTestUtils from HDFS into Common.Majortest, utilAhmed RadwanAhmed Radwan
HADOOP-8236haadmin should have configurable timeouts for failover commandsMajorhaPhilip ZeyligerTodd Lipcon
HADOOP-8214make hadoop script recognize a full set of deprecated commandsMajorscriptsRoman ShaposhnikRoman Shaposhnik
HADOOP-8200Remove HADOOP_[JOBTRACKER|TASKTRACKER]_OPTSMinorconfEli CollinsEli Collins
HADOOP-8193Refactor FailoverController/HAAdmin code to add an abstract class for “target” servicesMajorhaTodd LipconTodd Lipcon
HADOOP-8185Update namenode -format documentation and add -nonInteractive and -forceMajordocumentationArpit GuptaArpit Gupta
HADOOP-8183Stop using “mapred.used.genericoptionsparser” to avoid unnecessary warningsMinorutilHarsh JHarsh J
HADOOP-8163Improve ActiveStandbyElector to provide hooks for fencing old activeMajorhaTodd LipconTodd Lipcon
HADOOP-8152Expand public APIs for security library classesMajorsecurityAaron T. MyersAaron T. Myers
HADOOP-8118Print the stack trace of InstanceAlreadyExistsException in trace levelMinormetricsTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HADOOP-8117Upgrade test build to Surefire 2.12Trivialbuild, testTodd LipconTodd Lipcon
HADOOP-8113Correction to BUILDING.txt: HDFS needs ProtocolBuffer, too (not just MapReduce)TrivialdocumentationEugene KoontzEugene Koontz
HADOOP-8098KerberosAuthenticatorHandler should use _HOST replacement to resolve principal nameMajorsecurityAlejandro AbdelnurAlejandro Abdelnur
HADOOP-8086KerberosName silently sets defaultRealm to "" if the Kerberos config is not found, it should log a WARNMinorsecurityAlejandro AbdelnurAlejandro Abdelnur
HADOOP-8085Add RPC metrics to ProtobufRpcEngineMajoripc, metricsSuresh SrinivasHari Mankude
HADOOP-8084Protobuf RPC engine can be optimized to not do copying for the RPC request/responseMajoripcDevaraj DasDevaraj Das
HADOOP-8077HA: fencing method should be able to be configured on a per-NN or per-NS basisMajorhaTodd LipconTodd Lipcon
HADOOP-8070Add standalone benchmark of protobuf IPCMajorbenchmarks, ipcTodd LipconTodd Lipcon
HADOOP-8007HA: use substitution token for fencing argumentMajorhaAaron T. MyersTodd Lipcon
HADOOP-7987Support setting the run-as user in unsecure modeMajorsecurityDevaraj DasJitendra Nath Pandey
HADOOP-7957Classes deriving GetGroupsBase should be able to override proxy creation.Major.Jitendra Nath PandeyJitendra Nath Pandey
HADOOP-7899Generate proto java files as part of the buildMajorbuildAlejandro AbdelnurAlejandro Abdelnur
HADOOP-7875Add helper class to unwrap RemoteException from ServiceException thrown on protobuf based RPCMajoripcSuresh SrinivasSuresh Srinivas
HADOOP-7729Send back valid HTTP response if user hits IPC port with HTTP GETMajoripcTodd LipconTodd Lipcon
HADOOP-7727fix some typos and tabs in CHANGES.TXTTrivialbuildSteve LoughranSteve Loughran
HADOOP-7717Move handling of concurrent client fail-overs to RetryInvocationHandlerMajoripcAaron T. MyersAaron T. Myers
HADOOP-7716RPC protocol registration on SS does not log the protocol name (only the class which may be different)Minor.Sanjay RadiaSanjay Radia
HADOOP-7693fix RPC.Server#addProtocol to work in AvroRpcEngineMajoripcDoug CuttingDoug Cutting
HADOOP-7687Make getProtocolSignature publicMinor.Sanjay RadiaSanjay Radia
HADOOP-7635RetryInvocationHandler should release underlying resources on closeMajoripcAaron T. MyersAaron T. Myers
HADOOP-7607Simplify the RPC proxy cleanup processMajoripcAaron T. MyersAaron T. Myers
HADOOP-7549Use JDK ServiceLoader mechanism to find FileSystem implementationsMajorfsAlejandro AbdelnurAlejandro Abdelnur
HADOOP-7350Use ServiceLoader to discover compression codec classesMajorconf, ioTom WhiteTom White
HDFS-3418Rename BlockWithLocationsProto datanodeIDs field to storageIDsMinor.Eli CollinsEli Collins
HDFS-3378Remove DFS_NAMENODE_SECONDARY_HTTPS_PORT_KEY and DEFAULTTrivial.Eli CollinsEli Collins
HDFS-3375Put client name in DataXceiver thread name for readBlock and keepaliveTrivialdatanodeTodd LipconTodd Lipcon
HDFS-3365Enable users to disable socket caching in DFS client configurationMinorhdfs-clientTodd LipconTodd Lipcon
HDFS-3319DFSOutputStream should not start a thread in constructorsMinorhdfs-clientTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HDFS-3294Fix indentation in NamenodeWebHdfsMethods and DatanodeWebHdfsMethodsTrivialdatanode, namenodeTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HDFS-3279One of the FSEditLog constructors should be moved to TestEditLogMinornamenodeTsz Wo Nicholas SzeArpit Gupta
HDFS-3263HttpFS should read HDFS config from Hadoop site.xml filesMajor.Alejandro AbdelnurAlejandro Abdelnur
HDFS-3259NameNode#initializeSharedEdits should populate shared edits dir with edit log segmentsMajorha, namenodeAaron T. MyersAaron T. Myers
HDFS-3249Use ToolRunner.confirmPrompt in NameNodeTrivialnamenodeTodd LipconTodd Lipcon
HDFS-3247Improve bootstrapStandby behavior when original NN is not activeMinorhaTodd LipconTodd Lipcon
HDFS-3244Remove dead writable code from hdfs/protocolMajor.Eli CollinsEli Collins
HDFS-3240Drop log level of “heartbeat: ...” in BPServiceActor to DEBUGTrivialdatanodeTodd LipconTodd Lipcon
HDFS-3238ServerCommand and friends don't need to be writablesMajor.Eli CollinsEli Collins
HDFS-3226Allow GetConf tool to print arbitrary keysMajortoolsTodd LipconTodd Lipcon
HDFS-3206Miscellaneous xml cleanups for OEVMinortoolsColin Patrick McCabeColin Patrick McCabe
HDFS-3204Minor modification to JournalProtocol.proto to make it genericMajornamenodeSuresh SrinivasSuresh Srinivas
HDFS-3179Improve the error message: DataStreamer throw an exception, “nodes.length != original.length + 1” on single datanode clusterMajorhdfs-clientZhanwei WangTsz Wo Nicholas Sze
HDFS-3172dfs.upgrade.permission is dead codeTrivialnamenodeEli CollinsEli Collins
HDFS-3171The DatanodeID “name” field is overloadedMajordatanodeEli CollinsEli Collins
HDFS-3169TestFsck should test multiple -move operations in a rowMinortestColin Patrick McCabeColin Patrick McCabe
HDFS-3158LiveNodes member of NameNodeMXBean should list non-DFS used space and capacity per DNMajornamenodeAaron T. MyersAaron T. Myers
HDFS-3120Enable hsync and hflush by defaultMajor.Eli CollinsEli Collins
HDFS-3094add -nonInteractive and -force option to namenode -format commandMajor.Arpit GuptaArpit Gupta
HDFS-3091Update the usage limitations of ReplaceDatanodeOnFailure policy in the config description for the smaller clusters.Majordatanode, hdfs-client, namenodeUma Maheswara Rao GTsz Wo Nicholas Sze
HDFS-3084FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:portMajorhaPhilip ZeyligerTodd Lipcon
HDFS-3071haadmin failover command does not provide enough detail for when target NN is not ready to be activeMajorhaPhilip ZeyligerTodd Lipcon
HDFS-3056Add an interface for DataBlockScanner loggingMajordatanodeTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HDFS-3050rework OEV to share more code with the NameNodeMinornamenodeColin Patrick McCabeColin Patrick McCabe
HDFS-3036Remove unused method DFSUtil#isDefaultNamenodeAddressTrivialnamenodeAaron T. MyersAaron T. Myers
HDFS-3030Remove getProtocolVersion and getProtocolSignature from translatorsMajor.Jitendra Nath PandeyJitendra Nath Pandey
HDFS-3021Use generic type to declare FSDatasetInterfaceMajordatanodeTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HDFS-3014FSEditLogOp and its subclasses should have toString() methodMajornamenodeSho ShimauchiSho Shimauchi
HDFS-3003Remove getHostPortString() from NameNode, replace it with NetUtils.getHostPortString()TrivialnamenodeBrandon LiBrandon Li
HDFS-2983Relax the build version check to permit rolling upgrades within a releaseMajor.Eli CollinsAaron T. Myers
HDFS-2895Remove Writable wire protocol related code that is no longer necessaryMajordatanode, namenodeSuresh SrinivasSuresh Srinivas
HDFS-2708Stats for the # of blocks per DNMinordatanode, namenodeEli CollinsAaron T. Myers
HDFS-2650Replace @inheritDoc with @OverrideMinor.Hari MankudeHari Mankude
HDFS-2564Cleanup unnecessary exceptions thrown and unnecessary castsMinordatanode, hdfs-client, namenodeHari MankudeHari Mankude
HDFS-2496Separate datatypes for DatanodeProtocolMajor.Suresh SrinivasSuresh Srinivas
HDFS-2413Add public APIs for safemodeMajorhdfs-clientTodd LipconHarsh J
HDFS-2410Further clean up hard-coded configuration keysMinordatanode, namenode, testSuresh SrinivasSuresh Srinivas
HDFS-2351Change Namenode and Datanode to register each of their protocols seperatelyMajor.Sanjay RadiaSanjay Radia
HDFS-2337DFSClient shouldn't keep multiple RPC proxy referencesMajorhdfs-clientAaron T. MyersAaron T. Myers
HDFS-2223Untangle depencencies between NN componentsMajornamenodeTodd LipconTodd Lipcon
HDFS-2188HDFS-1580: Make FSEditLog create its journals from a list of URIs rather than NNStorageMajor.Ivan KellyIvan Kelly
HDFS-1580Add interface for generic Write Ahead Logging mechanismsMajornamenodeIvan KellyJitendra Nath Pandey
HDFS-309FSEditLog should log progress during replayMajor.Todd LipconSho Shimauchi
MAPREDUCE-4219make default container-executor.conf.dir be a path relative to the container-executor binaryMajorsecurityRoman ShaposhnikRoman Shaposhnik
MAPREDUCE-4138Reduce memory usage of counters due to non-static nested classesMajor.Tom WhiteTom White
MAPREDUCE-4103Fix HA docs for changes to shell command fencer argsMajordocumentationTodd LipconTodd Lipcon
MAPREDUCE-4093Improve RM WebApp start up when proxy address is not setMajormrv2Devaraj KDevaraj K
MAPREDUCE-3991Streaming FAQ has some wrong instructions about input files splittingTrivialdocumentationHarsh JHarsh J
MAPREDUCE-3955Replace ProtoOverHadoopRpcEngine with ProtobufRpcEngine.Blockermrv2Jitendra Nath PandeyJitendra Nath Pandey
MAPREDUCE-3909javadoc the Service interfacesTrivialmrv2Steve LoughranSteve Loughran
MAPREDUCE-3885Apply the fix similar to HADOOP-8084Majormrv2Devaraj DasDevaraj Das
MAPREDUCE-3883Document yarn.nodemanager.delete.debug-delay-sec configuration propertyMinordocumentation, mrv2Eugene KoontzEugene Koontz
MAPREDUCE-2934MR portion of HADOOP-7607 - Simplify the RPC proxy cleanup processMajormrv2Aaron T. MyersAaron T. Myers
MAPREDUCE-2887MR changes to match HADOOP-7524 (multiple RPC protocols)Major.Sanjay RadiaSanjay Radia

BUG FIXES:

JIRASummaryPriorityComponentReporterContributor
HADOOP-8355SPNEGO filter throws/logs exception when authentication failsMinorsecurityAlejandro AbdelnurAlejandro Abdelnur
HADOOP-8349ViewFS doesn't work when the root of a file system is mountedMajorviewfsAaron T. MyersAaron T. Myers
HADOOP-8347Hadoop Common logs misspell ‘successful’MajorsecurityPhilip ZeyligerPhilip Zeyliger
HADOOP-8310FileContext#checkPath should handle URIs with no portMajorfsAaron T. MyersAaron T. Myers
HADOOP-8309Pseudo & Kerberos AuthenticationHandler should use getType() to create tokenMajorsecurityAlejandro AbdelnurAlejandro Abdelnur
HADOOP-8296hadoop/yarn daemonlog usage wrongMinor.Thomas GravesDevaraj K
HADOOP-8282start-all.sh refers incorrectly start-dfs.sh existence for starting start-yarn.shMinorscriptsDevaraj KDevaraj K
HADOOP-8275Range check DelegationKey lengthMinor.Colin Patrick McCabeColin Patrick McCabe
HADOOP-8264Remove irritating double double quotes in front of hostnameTrivial.Bernd FondermannBernd Fondermann
HADOOP-8263Stringification of IPC calls not usefulMinoripcTodd LipconTodd Lipcon
HADOOP-8261Har file system doesn't deal with FS URIs with a host but no portMajorfsAaron T. MyersAaron T. Myers
HADOOP-8251SecurityUtil.fetchServiceTicket broken after HADOOP-6941BlockersecurityTodd LipconTodd Lipcon
HADOOP-8243Security support broken in CLI (manual) failover controllerCriticalha, securityTodd LipconTodd Lipcon
HADOOP-8238NetUtils#getHostNameOfIP blows up if given ip:port string w/o portMajor.Eli CollinsEli Collins
HADOOP-8218RPC.closeProxy shouldn't throw error when closing a mockCriticalipc, testTodd LipconTodd Lipcon
HADOOP-8204TestHealthMonitor fails occasionallyMajor.Tom WhiteTodd Lipcon
HADOOP-8202stopproxy() is not closing the proxies correctlyMinoripcHari MankudeHari Mankude
HADOOP-8199Fix issues in start-all.sh and stop-all.shMajor.Nishan ShettyDevaraj K
HADOOP-8191SshFenceByTcpPort uses netcat incorrectlyMajorhaPhilip ZeyligerTodd Lipcon
HADOOP-8189LdapGroupsMapping shouldn't throw away IOExceptionMajorsecurityJonathan NatkinsJonathan Natkins
HADOOP-8169javadoc generation fails with java.lang.OutOfMemoryError: Java heap spaceCriticalbuildThomas GravesThomas Graves
HADOOP-8159NetworkTopology: getLeaf should check for invalid topologiesMajor.Colin Patrick McCabeColin Patrick McCabe
HADOOP-8119Fix javac warnings in TestAuthenticationFilterMinortestTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HADOOP-7988Upper case in hostname part of the principals doesn't work with kerberos.Major.Jitendra Nath PandeyJitendra Nath Pandey
HADOOP-7968Errant println left in RPC.getHighestSupportedProtocolMinoripcTodd LipconSho Shimauchi
HADOOP-7940method clear() in org.apache.hadoop.io.Text does not workMajorioAaron,Csaba Miklos
HADOOP-7931o.a.h.ipc.WritableRpcEngine should have a way to force initializationMajoripcAaron T. MyersAaron T. Myers
HADOOP-7920Remove Avro RPCMajoripcSuresh SrinivasSuresh Srinivas
HADOOP-7900LocalDirAllocator confChanged() accesses conf.get() twiceMajorfsRavi GummadiRavi Gummadi
HADOOP-7897ProtobufRPCEngine client side exception mechanism is not consistent with WritableRpcEngineMajoripcSuresh SrinivasSuresh Srinivas
HADOOP-7892IPC logs too verbose after “RpcKind” introductionTrivialipcTodd LipconTodd Lipcon
HADOOP-7888TestFailoverProxy fails intermittently on trunkMajortestJason LoweJason Lowe
HADOOP-7833Inner classes of org.apache.hadoop.ipc.protobuf.HadoopRpcProtos generates findbugs warnings which results in -1 for findbugsMajoripcJohn LeeJohn Lee
HADOOP-7827jsp pages missing DOCTYPETrivial.Dave VronayDave Vronay
HADOOP-7704JsonFactory can be created only once and used for every next request to create JsonGenerator inside JMXJsonServletMinor.Devaraj KDevaraj K
HADOOP-7695RPC.stopProxy can throw unintended exception while logging errorMajoripcAaron T. MyersAaron T. Myers
HADOOP-7669Fix newly introduced release audit warning.Minor.Uma Maheswara Rao GUma Maheswara Rao G
HADOOP-6941Support non-SUN JREs in UserGroupInformationMajor.Stephen WattDevaraj Das
HADOOP-6924Build fails with non-Sun JREs due to different pathing to the operating system architecture shared librariesMajor.Stephen WattDevaraj Das
HDFS-3396FUSE build fails on Ubuntu 12.04Minorfuse-dfsColin Patrick McCabeColin Patrick McCabe
HDFS-3395NN doesn't start with HA+security enabled and HTTP address set to 0.0.0.0MajornamenodeAaron T. MyersAaron T. Myers
HDFS-3376DFSClient fails to make connection to DN if there are many unusable cached socketsCriticalhdfs-clientTodd LipconTodd Lipcon
HDFS-3357DataXceiver reads from client socket with incorrect/no timeoutCriticaldatanodeTodd LipconTodd Lipcon
HDFS-3351NameNode#initializeGenericKeys should always set fs.defaultFS regardless of whether HA or Federation is enabledMajornamenodeAaron T. MyersAaron T. Myers
HDFS-3350findbugs warning: INodeFileUnderConstruction doesn't override INodeFile.equals(Object)MajornamenodeTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HDFS-3336hdfs launcher script will be better off not special casing namenode command with regards to hadoop.security.loggerMinorscriptsRoman ShaposhnikRoman Shaposhnik
HDFS-3332NullPointerException in DN when directoryscanner is trying to report bad blocksMajordatanodeamithamith
HDFS-3330If GetImageServlet throws an Error or RTE, response has HTTP “OK” statusCriticalnamenodeTodd LipconTodd Lipcon
HDFS-3328NPE in DataNode.getIpcPortMinordatanodeUma Maheswara Rao GEli Collins
HDFS-3326Append enabled log message uses the wrong variableTrivialnamenodeJ.AndreinaMatthew Jacobs
HDFS-3314HttpFS operation for getHomeDirectory is incorrectMajor.Alejandro AbdelnurAlejandro Abdelnur
HDFS-3309HttpFS (Hoop) chmod not supporting octal and sticky bit permissionsMajor.Romain RigauxAlejandro Abdelnur
HDFS-3305GetImageServlet should consider SBN a valid requestor in a secure HA setupMajorha, namenodeAaron T. MyersAaron T. Myers
HDFS-3303RemoteEditLogManifest doesn't need to implements WritableMinornamenodeBrandon LiBrandon Li
HDFS-3284bootstrapStandby fails in secure clusterMinorha, securityTodd LipconTodd Lipcon
HDFS-3280DFSOutputStream.sync should not be synchronizedCriticalhdfs-clientTodd LipconTodd Lipcon
HDFS-3268Hdfs mishandles token service & incompatible with HACriticalha, hdfs-clientDaryn SharpDaryn Sharp
HDFS-3260TestDatanodeRegistration should set minimum DN version in addition to minimum NN versionMajor.Aaron T. MyersAaron T. Myers
HDFS-3256HDFS considers blocks under-replicated if topology script is configured with only 1 rackMajor.Aaron T. MyersAaron T. Myers
HDFS-3255HA DFS returns wrong token serviceCriticalha, hdfs-clientDaryn SharpDaryn Sharp
HDFS-3254Branch-2 build broken due to wrong version number in fuse-dfs' pom.xmlMajorfuse-dfsAnupam SethAnupam Seth
HDFS-3248bootstrapstanby repeated twice in hdfs namenode usage messageMinor.Colin Patrick McCabeColin Patrick McCabe
HDFS-3236NameNode does not initialize generic conf keys when started with -initializeSharedEditsDirMinorha, namenodeAaron T. MyersAaron T. Myers
HDFS-3234Accidentally left log message in GetConf after HDFS-3226TrivialtoolsTodd LipconTodd Lipcon
HDFS-3222DFSInputStream#openInfo should not silently get the length as 0 when locations length is zero for last partial block.Majorhdfs-clientUma Maheswara Rao GUma Maheswara Rao G
HDFS-3214InterDatanodeProtocolServerSideTranslatorPB doesn't handle null response from initReplicaRecoveryBlockerdatanodeTodd LipconTodd Lipcon
HDFS-3210JsonUtil#toJsonMap for for a DatanodeInfo should use “ipAddr” instead of “name”Major.Eli CollinsEli Collins
HDFS-3208Bogus entries in hosts files are incorrectly displayed in the reportMajornamenodeEli CollinsEli Collins
HDFS-3202NamespaceInfo PB translation drops build versionMajordatanodeAaron T. MyersAaron T. Myers
HDFS-3199TestValidateConfigurationSettings is failingMajor.Eli CollinsTodd Lipcon
HDFS-3181testHardLeaseRecoveryAfterNameNodeRestart fails when length before restart is 1 byte less than CRC chunk sizeMinortestColin Patrick McCabeTsz Wo Nicholas Sze
HDFS-3156TestDFSHAAdmin is failing post HADOOP-8202MajortestAaron T. MyersAaron T. Myers
HDFS-3143TestGetBlocks.testGetBlocks is failingMajortestEli CollinsArpit Gupta
HDFS-3142TestHDFSCLI.testAll is failingBlockertestEli CollinsBrandon Li
HDFS-3132Findbugs warning on HDFS trunkMinornamenodeTodd LipconTodd Lipcon
HDFS-3119Overreplicated block is not deleted even after the replication factor is reduced after sync follwed by closing that fileMinornamenodeJ.AndreinaAshish Singhi
HDFS-3109Remove hsqldb exclusions from pom.xmlMajor.Ravi PrakashRavi Prakash
HDFS-3100failed to append dataMajordatanodeZhanwei WangBrandon Li
HDFS-3099SecondaryNameNode does not properly initialize metrics systemMajornamenodeAaron T. MyersAaron T. Myers
HDFS-3093TestAllowFormat is trying to be interactiveCritical.Todd LipconTodd Lipcon
HDFS-3083Cannot run an MR job with HA and security enabled when second-listed NN activeCriticalha, securityMingjie LaiAaron T. Myers
HDFS-3070HDFS balancer doesn't ensure that hdfs-site.xml is loadedMajorbalancer & moverStephen ChuAaron T. Myers
HDFS-3062Fail to submit mapred job on a secured-HA-HDFS: logic URI cannot be picked up by job submission.Criticalha, securityMingjie LaiMingjie Lai
HDFS-3057httpfs and hdfs launcher scripts should honor CATALINA_HOME and HADOOP_LIBEXEC_DIRMajorscriptsRoman ShaposhnikRoman Shaposhnik
HDFS-3038Add FSEditLog.metrics to findbugs exclude listTrivial.Todd LipconTodd Lipcon
HDFS-3026HA: Handle failure during HA state transitionMajorha, namenodeAaron T. MyersAaron T. Myers
HDFS-3020Auto-logSync based on edit log buffer size brokenCriticalnamenodeTodd LipconTodd Lipcon
HDFS-3005ConcurrentModificationException in FSDataset$FSVolume.getDfsUsed(..)MajordatanodeTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HDFS-2995start-dfs.sh should only start the 2NN for namenodes with dfs.namenode.secondary.http-address configuredMajorscriptsTodd LipconEli Collins
HDFS-2968Protocol translator for BlockRecoveryCommand broken when multiple blocks need recoveryBlockerdatanode, namenodeTodd LipconTodd Lipcon
HDFS-2878TestBlockRecovery does not compileBlockertestEli CollinsTodd Lipcon
HDFS-2799Trim fs.checkpoint.dir valuesMajornamenodeEli Collinsamith
HDFS-2768BackupNode stop can not close proxy connections because it is not a proxy instance.MajornamenodeUma Maheswara Rao GUma Maheswara Rao G
HDFS-2765TestNameEditsConfigs is incorrectly swallowing IOEMajortestAaron T. MyersAaron T. Myers
HDFS-2739SecondaryNameNode doesn't start upCritical.Sho ShimauchiJitendra Nath Pandey
HDFS-2700TestDataNodeMultipleRegistrations is failing in trunkMajor.Uma Maheswara Rao GUma Maheswara Rao G
HDFS-2696Fix the fuse-fds buildMajorbuild, fuse-dfsPetru DimulescuBruno Mahé
HDFS-2694Removal of Avro broke non-PB NN servicesMajornamenodeAaron T. MyersAaron T. Myers
HDFS-2676Remove Avro RPCMajor.Suresh SrinivasSuresh Srinivas
HDFS-2532TestDfsOverAvroRpc timing out in trunkCriticaltestTodd LipconUma Maheswara Rao G
HDFS-2526(Client)NamenodeProtocolTranslatorR23 do not need to keep a reference to rpcProxyWithoutRetryMajorhdfs-client, namenodeAaron T. MyersAaron T. Myers
HDFS-2497Fix TestBackupNode failureMajor.Suresh SrinivasSuresh Srinivas
HDFS-2481Unknown protocol: org.apache.hadoop.hdfs.protocol.ClientProtocolMajor.Tsz Wo Nicholas SzeSanjay Radia
HDFS-2405hadoop dfs command with webhdfs fails on secure hadoopCriticalwebhdfsArpit GuptaJitendra Nath Pandey
HDFS-1765Block Replication should respect under-replication block priorityMajornamenodeHairong KuangUma Maheswara Rao G
HDFS-891DataNode no longer needs to check for dfs.network.scriptMinordatanodeSteve LoughranHarsh J
MAPREDUCE-4231Update RAID to not to use FSInodeInfoMajorcontrib/raidTsz Wo Nicholas SzeTsz Wo Nicholas Sze
MAPREDUCE-4202TestYarnClientProtocolProvider is brokenMajortestDaryn SharpDaryn Sharp
MAPREDUCE-4193broken doc link for yarn-default.xml in site.xmlMajordocumentationPatrick HuntPatrick Hunt
MAPREDUCE-4147YARN should not have a compile-time dependency on HDFSMajor.Tom WhiteTom White
MAPREDUCE-4105Yarn RackResolver ignores rack configurationsMajormrv2Ahmed RadwanAhmed Radwan
MAPREDUCE-4098TestMRApps testSetClasspath failsMajortestAlejandro AbdelnurAlejandro Abdelnur
MAPREDUCE-4095TestJobInProgress#testLocality uses a bogus topologyMajor.Eli CollinsColin Patrick McCabe
MAPREDUCE-4081TestMROutputFormat.java does not compileBlockerbuild, mrv2Jason LoweJason Lowe
MAPREDUCE-4076Stream job fails with ZipException when use yarn jar commandBlockermrv2Devaraj KDevaraj K
MAPREDUCE-4066To get “yarn.app.mapreduce.am.staging-dir” value, should set the default valueMinorjob submission, mrv2xieguimingxieguiming
MAPREDUCE-4057Compilation error in RAIDMajorcontrib/raidTsz Wo Nicholas SzeDevaraj K
MAPREDUCE-4008ResourceManager throws MetricsException on start up saying QueueMetrics MBean already existsMajormrv2, schedulerDevaraj KDevaraj K
MAPREDUCE-4007JobClient getJob(JobID) should return NULL if the job does not exist (for backwards compatibility)Majormrv2Alejandro AbdelnurAlejandro Abdelnur
MAPREDUCE-3974TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 mergeBlocker.Arun C MurthyAaron T. Myers
MAPREDUCE-3958RM: Remove RMNodeState and replace it with NodeStateMajormrv2Bikas SahaBikas Saha
MAPREDUCE-3952In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split.Majormrv2Zhenxiao LuoBhallamudi Venkata Siva Kamesh
MAPREDUCE-3933Failures because MALLOC_ARENA_MAX is not setMajormrv2, testAhmed RadwanAhmed Radwan
MAPREDUCE-3916various issues with running yarn proxyserverCriticalmrv2, resourcemanager, webappsRoman ShaposhnikDevaraj K
MAPREDUCE-3869Distributed shell application fails with NoClassDefFoundErrorBlockermrv2Devaraj KDevaraj K
MAPREDUCE-3867MiniMRYarn/MiniYarn uses fixed portsMajortestAlejandro AbdelnurAlejandro Abdelnur
MAPREDUCE-3818Trunk MRV1 compilation is broken.Blockerbuild, testVinod Kumar VavilapalliSuresh Srinivas
MAPREDUCE-3740Mapreduce Trunk compilation failsBlockermrv2Devaraj KDevaraj K
MAPREDUCE-3578starting nodemanager as ‘root’ gives “Unknown -jvm option”MajornodemanagerGilad WolffTom White
MAPREDUCE-3545Remove Avro RPCMajor.Suresh SrinivasSuresh Srinivas
MAPREDUCE-3431NPE in Resource Manager shutdownMinorresourcemanagerSteve LoughranSteve Loughran
MAPREDUCE-3377Compatibility issue with 0.20.203.Major.Jane ChenJane Chen
MAPREDUCE-3353Need a RM->AM channel to inform AMs about faulty/unhealthy/lost nodesMajorapplicationmaster, mrv2, resourcemanagerVinod Kumar VavilapalliBikas Saha
MAPREDUCE-3173MRV2 UI doesn't work properly without internetCriticalmrv2Devaraj KDevaraj K
MAPREDUCE-2942TestNMAuditLogger.testNMAuditLoggerWithIP failingCritical.Vinod Kumar VavilapalliThomas Graves
MAPREDUCE-1740NPE in getMatchingLevelForNodes when node locations are variable depthMajorjobtrackerTodd LipconAhmed Radwan

TESTS:

JIRASummaryPriorityComponentReporterContributor
HADOOP-8157TestRPCCallBenchmark#testBenchmarkWithWritable fails with RTEMajor.Eli CollinsTodd Lipcon
HDFS-3129NetworkTopology: add test that getLeaf should check for invalid topologiesMinor.Colin Patrick McCabeColin Patrick McCabe

SUB-TASKS:

JIRASummaryPriorityComponentReporterContributor
HADOOP-8211Update commons-net version to 3.1Majorio, performanceEli CollinsEli Collins
HADOOP-8210Common side of HDFS-3148Majorio, performanceEli CollinsEli Collins
HADOOP-7994Remove getProtocolVersion and getProtocolSignature from the client side translator and server side implementationMajor.Jitendra Nath PandeyJitendra Nath Pandey
HADOOP-7965Support for protocol version and signature in PBMajoripcJitendra Nath PandeyJitendra Nath Pandey
HADOOP-7913Fix bug in ProtoBufRpcEngine -MajoripcSanjay RadiaSanjay Radia
HADOOP-7862Move the support for multiple protocols to lower layer so that Writable, PB and Avro can all use itMajoripcSanjay RadiaSanjay Radia
HADOOP-7776Make the Ipc-Header in a RPC-Payload an explicit headerMajoripcSanjay RadiaSanjay Radia
HADOOP-7773Add support for protocol buffer based RPC engineMajoripcSuresh SrinivasSuresh Srinivas
HADOOP-7557Make IPC header be extensibleMajor.Sanjay RadiaSanjay Radia
HDFS-3363blockmanagement should stop using INodeFile & INodeFileUCMinornamenodeJohn GeorgeJohn George
HDFS-3339change INode to package privateMinornamenodeJohn GeorgeJohn George
HDFS-3322Update file context to use HdfsDataInputStream and HdfsDataOutputStreamMajorhdfs-clientTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HDFS-3298Add HdfsDataOutputStream as a public APIMajorhdfs-clientTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HDFS-3282Add HdfsDataInputStream as a public APIMajorhdfs-clientUma Maheswara Rao GUma Maheswara Rao G
HDFS-3211JournalProtocol changes required for introducing epoch and fencingMajorha, namenodeSuresh SrinivasSuresh Srinivas
HDFS-3187Upgrade guava to 11.0.2MinorbuildTodd LipconTodd Lipcon
HDFS-3155Clean up FSDataset implemenation related code.MajordatanodeTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HDFS-3130Move FSDataset implemenation to a packageMajordatanodeTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HDFS-3126Journal stream from the namenode to backup needs to have a timeoutMajorha, namenodeHari MankudeHari Mankude
HDFS-3105Add DatanodeStorage information to block recoveryMajordatanode, hdfs-clientTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HDFS-3089Move FSDatasetInterface and other related classes/interfaces to a packageMajordatanodeTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HDFS-3088Move FSDatasetInterface inner classes to a packageMajordatanodeTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HDFS-3086Change Datanode not to send storage list in registration - it will be sent in block reportMajordatanodeTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HDFS-3082Clean up FSDatasetInterfaceMajordatanodeTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HDFS-2899Service protocol change to support multiple storages added in HDFS-2880Major.Suresh SrinivasSuresh Srinivas
HDFS-2880Protocol buffer changes to add support multiple storagesMajordatanode, namenodeSuresh SrinivasSuresh Srinivas
HDFS-2801Provide a method in client side translators to check for a methods supported in underlying protocol.Major.Jitendra Nath PandeyJitendra Nath Pandey
HDFS-2697Move RefreshAuthPolicy, RefreshUserMappings, GetUserMappings protocol to protocol buffersMajor.Suresh SrinivasJitendra Nath Pandey
HDFS-2687Tests are failing with ClassCastException, due to new protocol changesMajortestUma Maheswara Rao GSuresh Srinivas
HDFS-2669Enable protobuf rpc for ClientNamenodeProtocolMajor.Sanjay RadiaSanjay Radia
HDFS-2666TestBackupNode failsMajortestSuresh SrinivasSuresh Srinivas
HDFS-2663Optional parameters are not handled correctlyMajor.Suresh SrinivasSuresh Srinivas
HDFS-2661Enable protobuf RPC for DatanodeProtocolMajor.Jitendra Nath PandeyJitendra Nath Pandey
HDFS-2651ClientNameNodeProtocol Translators for Protocol BuffersMajor.Sanjay RadiaSanjay Radia
HDFS-2647Enable protobuf RPC for InterDatanodeProtocol, ClientDatanodeProtocol, JournalProtocol and NamenodeProtocolMajorbalancer & mover, datanode, hdfs-client, namenodeSuresh SrinivasSuresh Srinivas
HDFS-2642Protobuf translators for DatanodeProtocolMajor.Jitendra Nath PandeyJitendra Nath Pandey
HDFS-2636Implement protobuf service for ClientDatanodeProtocolMajor.Suresh SrinivasSuresh Srinivas
HDFS-2629Implement protobuf service for InterDatanodeProtocolMajordatanodeSuresh SrinivasSuresh Srinivas
HDFS-2618Implement protobuf service for NamenodeProtocolMajornamenodeSuresh SrinivasSuresh Srinivas
HDFS-2597ClientNameNodeProtocol in Protocol BuffersMajor.Sanjay RadiaSanjay Radia
HDFS-2581Implement protobuf service for JournalProtocolMajornamenodeSuresh SrinivasSuresh Srinivas
HDFS-2520Protobuf - Add protobuf service for InterDatanodeProtocolMajordatanodeSuresh SrinivasSuresh Srinivas
HDFS-2519Protobuf - Add protobuf service for DatanodeProtocolMajordatanode, namenodeSuresh SrinivasSuresh Srinivas
HDFS-2518Protobuf - Add protobuf service for NamenodeProtocolMajornamenodeSuresh SrinivasSuresh Srinivas
HDFS-2517Protobuf - Add protocol service for JournalProtocolMajornamenodeSuresh SrinivasSuresh Srinivas
HDFS-2499Fix RPC client creation bug from HDFS-2459MajornamenodeSuresh SrinivasSuresh Srinivas
HDFS-2495Increase granularity of write operations in ReplicationMonitor thus reducing contention for write lockMajornamenodeTomasz NykielTomasz Nykiel
HDFS-2489Move commands Finalize and Register out of DatanodeCommand class.Major.Suresh SrinivasSuresh Srinivas
HDFS-2488Separate datatypes for InterDatanodeProtocolMajordatanodeSuresh SrinivasSuresh Srinivas
HDFS-2480Separate datatypes for NamenodeProtocolMajor.Suresh SrinivasSuresh Srinivas
HDFS-2479HDFS Client Data Types in Protocol BuffersMajor.Sanjay RadiaSanjay Radia
HDFS-2477Optimize computing the diff between a block report and the namenode state.MajornamenodeTomasz NykielTomasz Nykiel
HDFS-2476More CPU efficient data structure for under-replicated/over-replicated/invalidate blocksMajornamenodeTomasz NykielTomasz Nykiel
HDFS-2459Separate datatypes for Journal protocolMajor.Suresh SrinivasSuresh Srinivas
HDFS-2334Add Closeable to JournalManagerMajornamenodeIvan KellyIvan Kelly
HDFS-2181Separate HDFS Client wire protocol data typesMajor.Sanjay RadiaSanjay Radia
HDFS-2158Add JournalSet to manage the set of journals.Major.Jitendra Nath PandeyJitendra Nath Pandey
HDFS-20181073: Move all journal stream management code into one placeMajor.Ivan KellyIvan Kelly
MAPREDUCE-4113Fix tests org.apache.hadoop.mapred.TestClusterMRNotificationMajormrv2, testDevaraj KDevaraj K
MAPREDUCE-4112Fix tests org.apache.hadoop.mapred.TestClusterMapReduceTestCaseMajormrv2, testDevaraj KDevaraj K
MAPREDUCE-4111Fix tests in org.apache.hadoop.mapred.TestJobNameMajormrv2, testDevaraj KDevaraj K
MAPREDUCE-4110Fix tests in org.apache.hadoop.mapred.TestMiniMRClasspath & org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsersMajormrv2, testDevaraj KDevaraj K
MAPREDUCE-4108Fix tests in org.apache.hadoop.util.TestRunJarMajormrv2Devaraj KDevaraj K
MAPREDUCE-4107Fix tests in org.apache.hadoop.ipc.TestSocketFactoryMajormrv2Devaraj KDevaraj K

OTHER:

JIRASummaryPriorityComponentReporterContributor
HDFS-860fuse-dfs truncate behavior causes issues with scpMinorfuse-dfsBrian BockelmanBrian Bockelman
HADOOP-8359Clear up javadoc warnings in hadoop-common-projectTrivialconfHarsh JAnupam Seth
HADOOP-8142Update versions from 0.23.2 to 0.23.3MajorbuildTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HDFS-3111Missing license headers in trunkTrivial.Todd LipconUma Maheswara Rao G