Apache Hadoop Changelog

Release 0.23.5 - 2012-11-29

INCOMPATIBLE CHANGES:

JIRASummaryPriorityComponentReporterContributor
HDFS-4080Add a separate logger for block state change logs to enable turning off those logsMajornamenodeKihwal LeeKihwal Lee

IMPORTANT ISSUES:

JIRASummaryPriorityComponentReporterContributor

NEW FEATURES:

JIRASummaryPriorityComponentReporterContributor

IMPROVEMENTS:

JIRASummaryPriorityComponentReporterContributor
HADOOP-8932JNI-based user-group mapping modules can be too chatty on lookup failuresMajorsecurityKihwal LeeKihwal Lee
HADOOP-8930Cumulative code coverage calculationMajortestAndrey KlochkovAndrey Klochkov
HADOOP-8926hadoop.util.PureJavaCrc32 cache hit-ratio is low for static dataMajorutilGopal VGopal V
HADOOP-8889Upgrade to Surefire 2.12.3Majorbuild, testTodd LipconTodd Lipcon
HADOOP-8851Use -XX:+HeapDumpOnOutOfMemoryError JVM option in the forked testsMinortestIvan A. VeselovskyIvan A. Veselovsky
HADOOP-8789Tests setLevel(Level.OFF) should be Level.ERRORMinortestAndy IsaacsonAndy Isaacson
HADOOP-8755Print thread dump when tests fail due to timeoutMajortestAndrey KlochkovAndrey Klochkov
HDFS-3483Better error message when hdfs fsck is run against a ViewFS configMajor.Stephen ChuStephen Fritz
MAPREDUCE-4802Takes a long time to load the task list on the AM for large jobsMajormr-am, mrv2, webappsRavi PrakashRavi Prakash
MAPREDUCE-4763repair test org.apache.hadoop.mapreduce.security.TestUmbilicalProtocolWithJobTokenMinor.Ivan A. Veselovsky
MAPREDUCE-4752Reduce MR AM memory usage through String InterningMajormrv2Robert Joseph EvansRobert Joseph Evans
MAPREDUCE-4666JVM metrics for history serverMinorjobhistoryserverJason LoweJason Lowe
MAPREDUCE-4517Too many INFO messages written out during AM to RM heartbeatMinorapplicationmasterJames KinleyJason Lowe
MAPREDUCE-4229Counter names' memory usage can be decreased by interningMajorjobtrackerTodd LipconMiomir Boljanovic
YARN-216Remove jquery theming supportMajor.Todd LipconRobert Joseph Evans
YARN-165RM should point tracking URL to RM web page for app when AM failsBlockerresourcemanagerJason LoweJason Lowe

BUG FIXES:

JIRASummaryPriorityComponentReporterContributor
HADOOP-9025org.apache.hadoop.tools.TestCopyListing failingMajor.Robert Joseph EvansJonathan Eagles
HADOOP-9022Hadoop distcp tool fails to copy file if -m 0 specifiedMajor.Haiyang JiangJonathan Eagles
HADOOP-8986Server$Call object is never released after it is sentCriticalipcRobert Joseph EvansRobert Joseph Evans
HADOOP-8962RawLocalFileSystem.listStatus fails when a child filename contains a colonCriticalfsJason LoweJason Lowe
HADOOP-8906paths with multiple globs are unreliableCriticalfsDaryn SharpDaryn Sharp
HADOOP-8819Should use && instead of & in a few places in FTPFileSystem,FTPInputStream,S3InputStream,ViewFileSystem,ViewFsMajorfsBrandon LiBrandon Li
HADOOP-8791rm “Only deletes non empty directory and files.”MajordocumentationBertrand DechouxJing Zhao
HADOOP-8775MR2 distcp permits non-positive value to -bandwidth option which causes job never to completeMajor.Sandy RyzaSandy Ryza
HADOOP-8386hadoop script doesn't work if ‘cd’ prints to stdout (default behavior in Ubuntu)MajorscriptsChristopher BernerChristopher Berner
HDFS-4186logSync() is called with the write lock held while releasing leaseCriticalnamenodeKihwal LeeKihwal Lee
HDFS-4182SecondaryNameNode leaks NameCache entriesCriticalnamenodeTodd LipconRobert Joseph Evans
HDFS-4181LeaseManager tries to double remove and prints extra messagesCriticalnamenodeKihwal LeeKihwal Lee
HDFS-4172namenode does not URI-encode parameters when building URI for datanode requestMinornamenodeDerek DagitDerek Dagit
HDFS-4162Some malformed and unquoted HTML strings are returned from datanode web uiMinordatanodeDerek DagitDerek Dagit
HDFS-4090getFileChecksum() result incompatible when called against zero-byte files.Criticalhdfs-clientKihwal LeeKihwal Lee
HDFS-4075Reduce recommissioning overheadCriticalnamenodeKihwal LeeKihwal Lee
HDFS-4016back-port HDFS-3582 to branch-0.23Minor.Ivan A. VeselovskyIvan A. Veselovsky
HDFS-3996Add debug log removed in HDFS-3873 backMinor.Eli CollinsEli Collins
HDFS-3990NN's health report has severe performance problemsCriticalnamenodeDaryn SharpDaryn Sharp
HDFS-3919MiniDFSCluster:waitClusterUp can hang foreverMinortestAndy IsaacsonAndy Isaacson
HDFS-3905Secure cluster cannot use hftp to an insecure clusterCriticalhdfs-client, securityDaryn SharpDaryn Sharp
HDFS-3829TestHftpURLTimeouts fails intermittently with JDK7MajortestTrevor RobinsonTrevor Robinson
HDFS-3824TestHftpDelegationToken fails intermittently with JDK7MajortestTrevor RobinsonTrevor Robinson
HDFS-3224Bug in check for DN re-registration with different storage IDMinor.Eli CollinsJason Lowe
MAPREDUCE-4801ShuffleHandler can generate large logs due to prematurely closed channelsCritical.Jason LoweJason Lowe
MAPREDUCE-4797LocalContainerAllocator can loop forever trying to contact the RMMajorapplicationmasterJason LoweJason Lowe
MAPREDUCE-4787TestJobMonitorAndPrint is brokenMajortestRavi PrakashRobert Parker
MAPREDUCE-4786Job End Notification retry interval is 5 milliseconds by defaultMajormrv2Ravi PrakashRavi Prakash
MAPREDUCE-4782NLineInputFormat skips first line of last InputSplitBlockerclientMark FuhsMark Fuhs
MAPREDUCE-4774JobImpl does not handle asynchronous task events in FAILED stateMajorapplicationmaster, mrv2Ivan A. VeselovskyJason Lowe
MAPREDUCE-4772Fetch failures can take way too long for a map to be restartedCriticalmrv2Robert Joseph EvansRobert Joseph Evans
MAPREDUCE-4771KeyFieldBasedPartitioner not partitioning properly when configuredMajormrv2Jason LoweJason Lowe
MAPREDUCE-4751AM stuck in KILL_WAIT for daysMajor.Ravi PrakashVinod Kumar Vavilapalli
MAPREDUCE-4748Invalid event: T_ATTEMPT_SUCCEEDED at SUCCEEDEDBlockermrv2Robert Joseph EvansJason Lowe
MAPREDUCE-4746The MR Application Master does not have a config to set environment variablesMajorapplicationmasterRobert ParkerRobert Parker
MAPREDUCE-4741WARN and ERROR messages logged during normal AM shutdownMinorapplicationmaster, mrv2Jason LoweVinod Kumar Vavilapalli
MAPREDUCE-4740only .jars can be added to the Distributed Cache classpathBlockermrv2Robert Joseph EvansRobert Joseph Evans
MAPREDUCE-4733Reducer can fail to make progress during shuffle if too many reducers complete consecutivelyMajorapplicationmaster, mrv2Jason LoweJason Lowe
MAPREDUCE-4730AM crashes due to OOM while serving up map task completion eventsBlockerapplicationmaster, mrv2Jason LoweJason Lowe
MAPREDUCE-4729job history UI not showing all job attemptsMajorjobhistoryserverThomas GravesVinod Kumar Vavilapalli
MAPREDUCE-4724job history web ui applications page should be sorted to display last app firstMajorjobhistoryserverThomas GravesThomas Graves
MAPREDUCE-4721Task startup time in JHS is same as job startup time.MajorjobhistoryserverRavi PrakashRavi Prakash
MAPREDUCE-4720Browser thinks History Server main page JS is taking too longMajor.Robert Joseph EvansRavi Prakash
MAPREDUCE-4705Historyserver links expire before the history data doesCriticaljobhistoryserver, mrv2Jason LoweJason Lowe
MAPREDUCE-4674Hadoop examples secondarysort has a typo “secondarysrot” in the usageMinor.Robert JusticeRobert Justice
MAPREDUCE-4554Job Credentials are not transmitted if security is turned offMajorjob submission, securityBenoy AntonyBenoy Antony
MAPREDUCE-4549Distributed cache conflicts breaks backwards compatabilityBlockermrv2Robert Joseph EvansRobert Joseph Evans
MAPREDUCE-4521mapreduce.user.classpath.first incompatibility with 0.20/1.xMajormrv2Jason LoweRavi Prakash
MAPREDUCE-4479Fix parameter order in assertEquals() in TestCombineInputFileFormat.javaMajortestMariappan AsokanMariappan Asokan
MAPREDUCE-4425Speculation + Fetch failures can lead to a hung jobCriticalmrv2Siddharth SethJason Lowe
MAPREDUCE-1806CombineFileInputFormat does not work with paths not on default FSMajorharchivePaul YangGera Shegalov
YARN-214RMContainerImpl does not handle event EXPIRE at state RUNNINGMajorresourcemanagerJason LoweJonathan Eagles
YARN-212NM state machine ignores an APPLICATION_CONTAINER_FINISHED event when it shouldn'tBlockernodemanagerNathan RobertsNathan Roberts
YARN-206TestApplicationCleanup.testContainerCleanup occasionally failsMajorresourcemanagerJason LoweJason Lowe
YARN-201CapacityScheduler can take a very long time to schedule containers if requests are off clusterCriticalcapacityschedulerJason LoweJason Lowe
YARN-189deadlock in RM - AMResponse objectBlockerresourcemanagerThomas GravesThomas Graves
YARN-180Capacity scheduler - containers that get reserved create container token to earlyCriticalcapacityschedulerThomas GravesArun C Murthy
YARN-178Fix custom ProcessTree instance creationCritical.Radim KolarRadim Kolar
YARN-177CapacityScheduler - adding a queue while the RM is running has wacky resultsCriticalcapacityschedulerThomas GravesArun C Murthy
YARN-174TestNodeStatusUpdater is failing in trunkMajornodemanagerRobert Joseph EvansVinod Kumar Vavilapalli
YARN-166capacity scheduler doesn't allow capacity < 1.0MajorcapacityschedulerThomas GravesThomas Graves
YARN-163Retrieving container log via NM webapp can hang with multibyte characters in logMajornodemanagerJason LoweJason Lowe
YARN-161Yarn Common has multiple compiler warnings for unchecked operationsMajorapiChris NaurothChris Nauroth
YARN-159RM web ui applications page should be sorted to display last app firstMajorresourcemanagerThomas GravesThomas Graves
YARN-151Browser thinks RM main page JS is taking too longMajor.Robert Joseph EvansRavi Prakash
YARN-144MiniMRYarnCluster launches RM and JHS on default portsMajor.Robert ParkerRobert Parker
YARN-139Interrupted Exception within AsyncDispatcher leads to user confusionMajorapiNathan RobertsVinod Kumar Vavilapalli
YARN-131Incorrect ACL properties in capacity scheduler documentationMajorcapacityschedulerAhmed RadwanAhmed Radwan
YARN-116RM is missing ability to add include/exclude files without a restartMajorresourcemanagerxieguimingxieguiming
YARN-102Move the apache licence header to the top of the file in MemStore.javaTrivialresourcemanagerDevaraj KDevaraj K
YARN-43TestResourceTrackerService fail intermittently on jdk7Major.Thomas GravesThomas Graves
YARN-32TestApplicationTokens fails intermintently on jdk7Major.Thomas GravesVinod Kumar Vavilapalli
YARN-30TestNMWebServicesApps, TestRMWebServicesApps and TestRMWebServicesNodes fail on jdk7Major.Thomas GravesThomas Graves
YARN-28TestCompositeService fails on jdk7Major.Thomas GravesThomas Graves

TESTS:

JIRASummaryPriorityComponentReporterContributor

SUB-TASKS:

JIRASummaryPriorityComponentReporterContributor
MAPREDUCE-4107Fix tests in org.apache.hadoop.ipc.TestSocketFactoryMajormrv2Devaraj KDevaraj K
YARN-219NM should aggregate logs when application finishes.CriticalnodemanagerRobert Joseph EvansRobert Joseph Evans
YARN-202Log Aggregation generates a storm of fsync() for namenodeCritical.Kihwal LeeKihwal Lee
YARN-188Coverage fixing for CapacitySchedulerMajorcapacityschedulerAleksey GorshkovAleksey Gorshkov
YARN-186Coverage fixing LinuxContainerExecutorMajorresourcemanager, schedulerAleksey GorshkovAleksey Gorshkov

OTHER:

JIRASummaryPriorityComponentReporterContributor
HADOOP-8860Split MapReduce and YARN sections in documentation navigationMajordocumentationTom WhiteTom White
MAPREDUCE-4596Split StateMachine state from states seen by MRClientProtocol (for Job, Task, TaskAttempt)Majorapplicationmaster, mrv2Siddharth SethSiddharth Seth
MAPREDUCE-4266remove Ant remnants from MRMajorbuildAlejandro AbdelnurAlejandro Abdelnur
YARN-154Create Yarn trunk and commit jobsMajor.Eli CollinsRobert Joseph Evans