Apache Hadoop Changelog

Release 0.23.2 - Unreleased (as of 2016-03-04)

INCOMPATIBLE CHANGES:

JIRASummaryPriorityComponentReporterContributor
HADOOP-8164Handle paths using back slash as path separator for windows onlyMajorfsSuresh SrinivasDaryn Sharp
HADOOP-8131FsShell put doesn't correctly handle a non-existent dirCritical.Daryn SharpDaryn Sharp
HDFS-2887Define a FSVolume interfaceMajordatanodeTsz Wo Nicholas SzeTsz Wo Nicholas Sze

IMPORTANT ISSUES:

JIRASummaryPriorityComponentReporterContributor

NEW FEATURES:

JIRASummaryPriorityComponentReporterContributor
HDFS-2943Expose last checkpoint time and transaction stats as JMX metricsMajornamenodeAaron T. MyersAaron T. Myers

IMPROVEMENTS:

JIRASummaryPriorityComponentReporterContributor
HADOOP-8071Avoid an extra packet in client code when nagling is disabledMinoripcTodd LipconTodd Lipcon
HADOOP-8048Allow merging of CredentialsMajorutilDaryn SharpDaryn Sharp
HDFS-3066cap space usage of default log4j rolling policy (hdfs specific changes)MajorscriptsPatrick HuntPatrick Hunt
HDFS-3024Improve performance of stringification in addStoredBlockMinornamenodeTodd LipconTodd Lipcon
HDFS-2985Improve logging when replicas are marked as corruptMinornamenodeTodd LipconTodd Lipcon
HDFS-2981The default value of dfs.client.block.write.replace-datanode-on-failure.enable should be trueMajor.Tsz Wo Nicholas SzeTsz Wo Nicholas Sze
HDFS-2907Make FSDataset in Datanode PluggableMinor.Sanjay RadiaTsz Wo Nicholas Sze
HDFS-2655BlockReaderLocal#skip performs unnecessary IOMajordatanodeEli CollinsBrandon Li
HDFS-2506Umbrella jira for tracking separation of wire protocol datatypes from the implementation typesMajordatanode, namenodeSuresh SrinivasSuresh Srinivas
HDFS-1217Some methods in the NameNdoe should not be publicMajornamenodeTsz Wo Nicholas SzeLaxman
MAPREDUCE-3989cap space usage of default log4j rolling policy (mr specific changes)Major.Patrick HuntPatrick Hunt
MAPREDUCE-3922Fix the potential problem compiling 32 bit binaries on a x86_64 host.Minorbuild, mrv2Eugene KoontzHitesh Shah
MAPREDUCE-3901lazy load JobHistory Task and TaskAttempt detailsMajorjobhistoryserver, mrv2Siddharth SethSiddharth Seth
MAPREDUCE-3864Fix cluster setup docs for correct SNN HTTPS parametersMinordocumentation, securityTodd LipconTodd Lipcon
MAPREDUCE-3849Change TokenCache's reading of the binary token fileMajorsecurityDaryn SharpDaryn Sharp
MAPREDUCE-3730Allow restarted NM to rejoin cluster before RM expires itMinormrv2, resourcemanagerJason LoweJason Lowe

BUG FIXES:

JIRASummaryPriorityComponentReporterContributor
HADOOP-8176Disambiguate the destination of FsShell copiesMajorfsDaryn SharpDaryn Sharp
HADOOP-8146FsShell commands cannot be interruptedMajorfsDaryn SharpDaryn Sharp
HADOOP-8140dfs -getmerge should process its argments betterMajor.arkady borkovskyDaryn Sharp
HADOOP-8137Site side links for commands manual (MAPREDUCE-3497)MajordocumentationVinod Kumar VavilapalliThomas Graves
HADOOP-8123hadoop-project invalid pom warnings prevent transitive dependency resolutionCriticalbuildJonathan EaglesJonathan Eagles
HADOOP-8083javadoc generation for some modules is not done under target/MajorbuildAlejandro AbdelnurAlejandro Abdelnur
HADOOP-8082add hadoop-client and hadoop-minicluster to the dependency-management sectionMajorbuildAlejandro AbdelnurAlejandro Abdelnur
HADOOP-8074Small bug in hadoop error message for unknown commandsTrivialscriptsEli CollinsColin Patrick McCabe
HADOOP-8066The full docs build intermittently failsMajorbuildAaron T. MyersAndrew Bayer
HADOOP-8064Remove unnecessary dependency on w3c.org in document processingMajorbuildKihwal LeeKihwal Lee
HADOOP-8057hadoop-setup-conf.sh not working because of some extra spaces.MajorscriptsVinayakumar BVinayakumar B
HADOOP-8051HttpFS documentation it is not wired to the generated siteMajordocumentationAlejandro AbdelnurAlejandro Abdelnur
HADOOP-8050Deadlock in metricsMajormetricsKihwal LeeKihwal Lee
HADOOP-8046Revert StaticMapping semantics to the existing ones, add DNS mapping diagnostics in progressMinor.Steve LoughranSteve Loughran
HADOOP-8042When copying a file out of HDFS, modifying it, and uploading it back into HDFS, the put fails due to a CRC mismatchCriticalfsKevin J. PriceDaryn Sharp
HADOOP-8036TestViewFsTrash assumes the user's home directory is 2 levels deepMajorfs, testEli CollinsColin Patrick McCabe
HADOOP-8035Hadoop Maven site is inefficient and runs phases redundantlyMinorbuildAndrew BayerAndrew Bayer
HADOOP-7874native libs should be under lib/native/ dirMajorbuildAlejandro AbdelnurAlejandro Abdelnur
HADOOP-7660Maven generated .classpath doesnot includes “target/generated-test-source/java” as source directory.MinorbuildLaxmanLaxman
HADOOP-6502DistributedFileSystem#listStatus is very slow when listing a directory with a size of 1300CriticalutilHairong KuangSharad Agarwal
HDFS-3853Port MiniDFSCluster enableManagedDfsDirsRedundancy option to branch-2MinornamenodeColin Patrick McCabeColin Patrick McCabe
HDFS-3160httpfs should exec catalina instead of forking itMajorscriptsRoman ShaposhnikRoman Shaposhnik
HDFS-3101cannot read empty file using webhdfsMajorwebhdfsZhanwei WangTsz Wo Nicholas Sze
HDFS-3032Lease renewer tries forever even if renewal is not possibleMajorhdfs-clientKihwal LeeKihwal Lee
HDFS-3012Exception while renewing delegation tokenCritical.Ramya SunilRobert Joseph Evans
HDFS-3008Negative caching of local addrs doesn't workMajorhdfs-clientEli CollinsEli Collins
HDFS-3006Webhdfs “SETOWNER” call returns incorrect content-typeMajorwebhdfsbc WongTsz Wo Nicholas Sze
HDFS-2969ExtendedBlock.equals is incorrectly implementedCriticaldatanodeTodd LipconTodd Lipcon
HDFS-2950Secondary NN HTTPS address should be listed as a NAMESERVICE_SPECIFIC_KEYMinornamenodeTodd LipconTodd Lipcon
HDFS-2944Typo in hdfs-default.xml causes dfs.client.block.write.replace-datanode-on-failure.enable to be mistakenly disabledMajorhdfs-clientAaron T. MyersAaron T. Myers
HDFS-2938Recursive delete of a large directory makes namenode unresponsiveMajornamenodeSuresh SrinivasHari Mankude
HDFS-2815Namenode is not coming out of safemode when we perform ( NN crash + restart ) . Also FSCK report shows blocks missed.CriticalnamenodeUma Maheswara Rao GUma Maheswara Rao G
HDFS-2764TestBackupNode is racyMajornamenode, testAaron T. MyersAaron T. Myers
HDFS-2725hdfs script usage information is missing the information about “dfs” commandMajorhdfs-clientPrashant Sharma
HDFS-2525Race between BlockPoolSliceScanner and appendCriticaldatanodeTodd LipconBrandon Li
HDFS-776Fix exception handling in BalancerCriticalbalancer & moverOwen O'MalleyUma Maheswara Rao G
MAPREDUCE-4061RM only has 1 AM launcher threadBlockermrv2Thomas GravesThomas Graves
MAPREDUCE-4043Secret keys set in Credentials are not seen by tasksBlockermrv2, securityJason LoweJason Lowe
MAPREDUCE-4034Unable to view task logs on history server with mapreduce.job.acl-view-job=*Blockermrv2Jason LoweJason Lowe
MAPREDUCE-4025AM can crash if task attempt reports bogus progress valueBlockermr-am, mrv2Jason LoweJason Lowe
MAPREDUCE-4006history server container log web UI sometimes combines stderr/stdout/syslog contents togetherMajorjobhistoryserver, mrv2Jason LoweSiddharth Seth
MAPREDUCE-4005AM container logs URL is broken for completed apps when log aggregation is enabledMajormrv2Jason LoweJason Lowe
MAPREDUCE-3982TestEmptyJob fails with FileNotFoundCriticalmrv2Robert Joseph EvansRobert Joseph Evans
MAPREDUCE-3977LogAggregationService leaks log aggregator objectsCriticalmrv2, nodemanagerJason LoweJason Lowe
MAPREDUCE-3976TestRMContainerAllocator failingMajormrv2Bikas SahaJason Lowe
MAPREDUCE-3975Default value not set for Configuration parameter mapreduce.job.local.dirBlockermrv2Eric PayneEric Payne
MAPREDUCE-3964ResourceManager does not have JVM metricsCriticalmrv2, resourcemanagerJason LoweJason Lowe
MAPREDUCE-3961Map/ReduceSlotMillis computation incorrectMajormrv2Siddharth SethSiddharth Seth
MAPREDUCE-3960web proxy doesn't forward request to AM with configured hostname/IPCriticalmrv2Thomas GravesThomas Graves
MAPREDUCE-3954Clean up passing HEAPSIZE to yarn and mapred commands.Blockermrv2Robert Joseph EvansRobert Joseph Evans
MAPREDUCE-3931MR tasks failing due to changing timestamps on Resources to downloadMajormrv2Vinod Kumar VavilapalliSiddharth Seth
MAPREDUCE-3930The AM page for a Reducer that has not been launched causes an NPECriticalmrv2Robert Joseph EvansRobert Joseph Evans
MAPREDUCE-3929output of mapred -showacl is not clearMajormrv2John GeorgeJohn George
MAPREDUCE-3920Revise yarn default port number selectionMajornodemanager, resourcemanagerDave ThompsonDave Thompson
MAPREDUCE-3918proc_historyserver no longer in command line arguments for HistoryServerMajormrv2Jonathan EaglesJonathan Eagles
MAPREDUCE-3913RM application webpage is unresponsive after 2000 jobsCriticalmrv2, webappsJason LoweJason Lowe
MAPREDUCE-3910user not allowed to submit jobs even though queue -showacls shows it allowsBlockermrv2John GeorgeJohn George
MAPREDUCE-3904[NPE] Job history produced with mapreduce.cluster.acls.enabled false can not be viewed with mapreduce.cluster.acls.enabled trueMajormrv2Jonathan EaglesJonathan Eagles
MAPREDUCE-3903no admin override to view jobs on mr app master and job history serverCriticalmrv2Thomas GravesThomas Graves
MAPREDUCE-3897capacity scheduler - maxActiveApplicationsPerUser calculation can be wrongCriticalmrv2Thomas GravesEric Payne
MAPREDUCE-3896pig job through oozie hangsBlockerjobhistoryserver, mrv2John GeorgeVinod Kumar Vavilapalli
MAPREDUCE-3884PWD should be first in the classpath of MR tasksCriticalmrv2Alejandro AbdelnurAlejandro Abdelnur
MAPREDUCE-3878Null user on filtered jobhistory job pageCriticalmrv2Jonathan EaglesJonathan Eagles
MAPREDUCE-3866bin/yarn prints the command line unnecessarilyMinormrv2Vinod Kumar VavilapalliVinod Kumar Vavilapalli
MAPREDUCE-3862Nodemanager can appear to hang on shutdown due to lingering DeletionService threadsMajormrv2, nodemanagerJason LoweJason Lowe
MAPREDUCE-3852test TestLinuxResourceCalculatorPlugin failingBlockermrv2Thomas GravesThomas Graves
MAPREDUCE-3816capacity scheduler web ui bar graphs for used capacity wrongCriticalmrv2Thomas GravesThomas Graves
MAPREDUCE-3792job -list displays only the jobs submitted by a particular userCriticalmrv2Ramya SunilJason Lowe
MAPREDUCE-3790Broken pipe on streaming job can lead to truncated output for a successful jobMajorcontrib/streaming, mrv2Jason LoweJason Lowe
MAPREDUCE-3738NM can hang during shutdown if AppLogAggregatorImpl thread dies unexpectedlyCriticalmrv2, nodemanagerJason LoweJason Lowe
MAPREDUCE-3706HTTP Circular redirect error on the job attempts pageCriticalmrv2Thomas GravesRobert Joseph Evans
MAPREDUCE-3687If AM dies before it returns new tracking URL, proxy redirects to http://N/A/ and doesn't return error codeMajormrv2David CapwellRavi Prakash
MAPREDUCE-3686history server web ui - job counter values for map/reduce not shown properlyCriticalmrv2Thomas GravesBhallamudi Venkata Siva Kamesh
MAPREDUCE-3680FifoScheduler web service rest API can print out invalid JSONMajormrv2Thomas Graves
MAPREDUCE-3634All daemons should crash instead of hanging around when their EventHandlers get exceptionsMajormrv2Vinod Kumar VavilapalliVinod Kumar Vavilapalli
MAPREDUCE-3614finalState UNDEFINED if AM is killed by handMajormrv2Ravi PrakashRavi Prakash
MAPREDUCE-3583ProcfsBasedProcessTree#constructProcessInfo() may throw NumberFormatExceptionCritical.Ted YuTed Yu
MAPREDUCE-3497missing documentation for yarn cli and subcommands - similar to commands_manual.htmlMajordocumentation, mrv2Thomas GravesThomas Graves
MAPREDUCE-3034NM should act on a REBOOT command from RMCriticalmrv2, nodemanagerVinod Kumar VavilapalliDevaraj K
MAPREDUCE-3009RM UI -> Applications -> Application(Job History) -> Map Tasks -> Task ID -> Node link is not workingMajorjobhistoryserver, mrv2chackaravarthychackaravarthy
MAPREDUCE-2855ResourceBundle lookup during counter name resolution takes a lot of timeMajor.Todd LipconSiddharth Seth
MAPREDUCE-2793[MR-279] Maintain consistency in naming appIDs, jobIDs and attemptIDsCriticalmrv2Ramya SunilBikas Saha

TESTS:

JIRASummaryPriorityComponentReporterContributor
HDFS-3104Add tests for mkdir -pMajortestDaryn SharpDaryn Sharp
HDFS-3098Update FsShell tests for quoted metacharsMajortestDaryn SharpDaryn Sharp
HDFS-3060Bump TestDistributedUpgrade#testDistributedUpgrade timeoutMinortestEli CollinsEli Collins
HDFS-2038Update test to handle relative paths with globsCriticaltestDaryn SharpKihwal Lee
MAPREDUCE-3877Add a test to formalise the current state transitions of the yarn lifecycleMinormrv2Steve LoughranSteve Loughran
MAPREDUCE-3798TestJobCleanup testCustomCleanup is failingMajortestRavi PrakashRavi Prakash

SUB-TASKS:

JIRASummaryPriorityComponentReporterContributor
HADOOP-8175Add mkdir -p flagMajorfsDaryn SharpDaryn Sharp
HADOOP-8173FsShell needs to handle quoted metacharsMajorfsDaryn SharpDaryn Sharp
MAPREDUCE-3944JobHistory web services are slower then the UI and can easly overload the JHBlockermrv2Robert Joseph EvansRobert Joseph Evans

OTHER:

JIRASummaryPriorityComponentReporterContributor
HADOOP-8032mvn site:stage-deploy should be able to use the scp protocol to stage documentsMajorbuild, documentationRavi PrakashRavi Prakash
HDFS-2931Switch the DataNode's BlockVolumeChoosingPolicy to be a private-audience interfaceMinordatanodeHarsh JHarsh J