Apache Hadoop Changelog

Release 1.1.0 - 2012-10-13

INCOMPATIBLE CHANGES:

JIRASummaryPriorityComponentReporterContributor
HADOOP-8552Conflict: Same security.log.file for multiple users.Majorconf, securityKarthik KambatlaKarthik Kambatla
HADOOP-8365Add flag to disable durable syncBlocker.Eli CollinsEli Collins
HADOOP-8314HttpServer#hasAdminAccess should return false if authorization is enabled but user is not authenticatedMajorsecurityAlejandro AbdelnurAlejandro Abdelnur
HADOOP-8230Enable sync by default and disable appendMajor.Eli CollinsEli Collins
HADOOP-8154DNS#getIPs shouldn't silently return the local host IP for bogus interface namesMajorconfEli CollinsEli Collins
HADOOP-5464DFSClient does not treat write timeout of 0 properlyMajor.Raghu AngadiRaghu Angadi
HDFS-3522If NN is in safemode, it should throw SafeModeException when getBlockLocations has zero locationsMajornamenodeBrandon LiBrandon Li
HDFS-3044fsck move should be non-destructive by defaultMajornamenodeEli CollinsColin Patrick McCabe
HDFS-2617Replaced Kerberized SSL for image transfer and fsck with SPNEGO-based solutionMajorsecurityJakob HomanJakob Homan

IMPORTANT ISSUES:

JIRASummaryPriorityComponentReporterContributor

NEW FEATURES:

JIRASummaryPriorityComponentReporterContributor
HADOOP-7823port HADOOP-4012 to branch-1 (splitting support for bzip2)Major.Tim BrobergAndrew Purtell
HADOOP-7806Support binding to sub-interfacesMajorutilHarsh JHarsh J
HDFS-3150Add option for clients to contact DNs via hostnameMajordatanode, hdfs-clientEli CollinsEli Collins
HDFS-3148The client should be able to use multiple local interfaces for data transferMajorhdfs-client, performanceEli CollinsEli Collins
HDFS-3055Implement recovery mode for branch-1Minor.Colin Patrick McCabeColin Patrick McCabe
MAPREDUCE-3837Job tracker is not able to recover job in case of crash and after that no user can submit job.Major.Mayank BansalMayank Bansal
MAPREDUCE-3118Backport Gridmix and Rumen features from trunk to Hadoop 0.20 security branchMajorcontrib/gridmix, tools/rumenRavi GummadiRavi Gummadi

IMPROVEMENTS:

JIRASummaryPriorityComponentReporterContributor
HADOOP-8748Move dfsclient retry to a util classMinorioArun C MurthyArun C Murthy
HADOOP-8656backport forced daemon shutdown of HADOOP-8353 into branch-1MinorbinSteve LoughranRoman Shaposhnik
HADOOP-8430Backport new FileSystem methods introduced by HADOOP-8014 to branch-1Major.Eli CollinsEli Collins
HADOOP-8209Add option to relax build-version check for branch-1Major.Eli CollinsEli Collins
HADOOP-7509Improve message when Authentication is requiredTrivial.Ravi PrakashRavi Prakash
HADOOP-6995Allow wildcards to be used in ProxyUsers configurationsMinorsecurityTodd LipconTodd Lipcon
HDFS-3871Change NameNodeProxies to use HADOOP-8748Minorhdfs-clientArun C MurthyArun C Murthy
HDFS-3814Make the replication monitor multipliers configurable in 1.xMajornamenodeSuresh SrinivasJing Zhao
HDFS-3703Decrease the datanode failure detection timeMajordatanode, namenodeNicolas LiochonJing Zhao
HDFS-3667Add retry support to WebHdfsFileSystemMajorwebhdfsTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HDFS-3617Port HDFS-96 to branch-1 (support blocks greater than 2GB)Major.Matt FoleyHarsh J
HDFS-3596Improve FSEditLog pre-allocation in branch-1Minor.Colin Patrick McCabeColin Patrick McCabe
HDFS-3516Check content-type in WebHdfsFileSystemMajorhdfs-client, webhdfsTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HDFS-3504Configurable retry in DFSClientMajorhdfs-clientSiddharth SethTsz Wo Nicholas Sze
HDFS-3131Improve TestStorageRestoreMinor.Tsz Wo Nicholas SzeBrandon Li
HDFS-3094add -nonInteractive and -force option to namenode -format commandMajor.Arpit GuptaArpit Gupta
HDFS-2872Add sanity checks during edits loading that generation stamps are non-decreasingMajornamenodeTodd LipconColin Patrick McCabe
HDFS-2654Make BlockReaderLocal not extend RemoteBlockReader2MajordatanodeEli CollinsEli Collins
HDFS-2653DFSClient should cache whether addrs are non-local when short-circuiting is enabledMajordatanodeEli CollinsEli Collins
HDFS-2638Improve a block recovery logMinornamenodeEli CollinsEli Collins
HDFS-2465Add HDFS support for fadvise readahead and drop-behindMajordatanode, performanceTodd LipconTodd Lipcon
HDFS-1378Edit log replay should track and report file offsets in case of errorsMajornamenodeTodd LipconColin Patrick McCabe
HDFS-496Use PureJavaCrc32 in HDFSMinordatanode, hdfs-client, performanceTodd LipconTodd Lipcon
MAPREDUCE-4603Allow JobClient to retry job-submission when JT is in safemodeMajor.Arun C MurthyArun C Murthy
MAPREDUCE-4511Add IFile readaheadMajormrv1, mrv2, performanceAhmed RadwanAhmed Radwan
MAPREDUCE-4328Add the option to quiesce the JobTrackerMajormrv1Arun C MurthyArun C Murthy
MAPREDUCE-4001Improve MAPREDUCE-3789‘s fix logic by looking at job’s slot demands insteadMinorcapacity-schedHarsh JHarsh J
MAPREDUCE-3597Provide a way to access other info of history file from RumentoolMajortools/rumenRavi GummadiRavi Gummadi
MAPREDUCE-3395Add mapred.disk.healthChecker.interval to mapred-default.xmlTrivialdocumentationEli CollinsEli Collins
MAPREDUCE-3394Add log guard for a debug message in ReduceTaskTrivialtaskTodd LipconTodd Lipcon
MAPREDUCE-3365Uncomment eventlog settings from the documentationTrivialcontrib/fair-shareSho ShimauchiSho Shimauchi
MAPREDUCE-3289Make use of fadvise in the NM's shuffle handlerMajormrv2, nodemanager, performanceTodd LipconTodd Lipcon
MAPREDUCE-32780.20: avoid a busy-loop in ReduceTask schedulingMajormrv1, performance, taskTodd LipconTodd Lipcon
MAPREDUCE-2919The JT web UI should show job start timesMinorjobtrackerEli CollinsHarsh J
MAPREDUCE-2836Provide option to fail jobs when submitted to non-existent pools.Minorcontrib/fair-shareJeff BeanAhmed Radwan
MAPREDUCE-2835Make per-job counter limits configurableMajor.Tom WhiteTom White
MAPREDUCE-2103task-controller shouldn't require o-r permissionsTrivialtask-controllerTodd LipconTodd Lipcon
MAPREDUCE-1906Lower default minimum heartbeat interval for tasktracker > JobtrackerMajorjobtracker, performance, tasktrackerScott CareyTodd Lipcon
MAPREDUCE-782Use PureJavaCrc32 in mapreduce spillsMinorperformanceTodd LipconTodd Lipcon

BUG FIXES:

JIRASummaryPriorityComponentReporterContributor
HADOOP-8617backport pure Java CRC32 calculator changes to branch-1MajorperformanceBrandon LiBrandon Li
HADOOP-8445Token should not print the password in toStringMajorsecurityRavi PrakashRavi Prakash
HADOOP-8417HADOOP-6963 didn't update hadoop-core-pom-template.xmlMajor.Ted YuTed Yu
HADOOP-8399Remove JDK5 dependency from Hadoop 1.0+ lineMajorbuildKonstantin BoudnikKonstantin Boudnik
HADOOP-8329Build fails with Java 7MajorbuildKumar RaviEli Collins
HADOOP-8269Fix some javadoc warnings on branch-1TrivialdocumentationEli CollinsEli Collins
HADOOP-8159NetworkTopology: getLeaf should check for invalid topologiesMajor.Colin Patrick McCabeColin Patrick McCabe
HADOOP-7988Upper case in hostname part of the principals doesn't work with kerberos.Major.Jitendra Nath PandeyJitendra Nath Pandey
HADOOP-7982UserGroupInformation fails to login if thread‘s context classloader can’t load HadoopLoginModuleMajorsecurityTodd LipconTodd Lipcon
HADOOP-7908Fix three javadoc warnings on branch-1TrivialdocumentationEli CollinsEli Collins
HADOOP-7898Fix javadoc warnings in AuthenticationToken.javaMinorsecuritySuresh SrinivasSuresh Srinivas
HADOOP-7879DistributedFileSystem#createNonRecursive should also incrementWriteOps statistics.Trivial.Jonathan HsiehJonathan Hsieh
HADOOP-7870fix SequenceFile#createWriter with boolean createParent arg to respect createParent.Major.Jonathan HsiehJonathan Hsieh
HADOOP-7745I switched variable names in HADOOP-7509Major.Ravi PrakashRavi Prakash
HADOOP-7666branch-0.20-security doesn't include o.a.h.security.TestAuthenticationFilterMajorsecurityAaron T. MyersAaron T. Myers
HADOOP-7665branch-0.20-security doesn't include SPNEGO settings in core-default.xmlMajorsecurityAaron T. MyersAaron T. Myers
HADOOP-7653tarball doesn't include .eclipse.templatesMinorbuildJonathan NatkinsJonathan Natkins
HADOOP-7634Cluster setup docs specify wrong owner for task-controller.cfgMinordocumentation, securityEli CollinsEli Collins
HADOOP-7629regression with MAPREDUCE-2289 - setPermission passed immutable FsPermission (rpc failure)Major.Patrick HuntTodd Lipcon
HADOOP-7621alfredo config should be in a file not readable by usersCriticalsecurityAlejandro AbdelnurAaron T. Myers
HADOOP-7297Error in the documentation regarding Checkpoint/Backup NodeTrivialdocumentationarnaud pHarsh J
HADOOP-6947Kerberos relogin should set refreshKrb5Config to trueMajorsecurityTodd LipconTodd Lipcon
HADOOP-6546BloomMapFile can return false negativesMajorioClark JefcoatClark Jefcoat
HADOOP-6527UserGroupInformation::createUserForTesting clobbers already defined group mappingsMajorsecurityJakob HomanIvan Mitic
HADOOP-5836Bug in S3N handling of directory markers using an object with a trailing “/” causes jobs to failMajorfs/s3Ian NowlandIan Nowland
HDFS-3966For branch-1, TestFileCreation should use JUnit4 to make assumeTrue workMinor.Jing ZhaoJing Zhao
HDFS-3701HDFS may miss the final block when reading a file opened for writing if one of the datanode is deadCriticalhdfs-clientNicolas LiochonNicolas Liochon
HDFS-3698TestHftpFileSystem is failing in branch-1 due to changed default secure portMajorsecurityAaron T. MyersAaron T. Myers
HDFS-3696Create files with WebHdfsFileSystem goes OOM when file size is bigCriticalwebhdfsKihwal LeeTsz Wo Nicholas Sze
HDFS-3551WebHDFS CREATE does not use client location for redirectionMajorwebhdfsTsz Wo Nicholas SzeTsz Wo Nicholas Sze
HDFS-3518Provide API to check HDFS operational stateMajorhdfs-clientBikas SahaTsz Wo Nicholas Sze
HDFS-3466The SPNEGO filter for the NameNode should come out of the web keytab fileMajornamenode, securityOwen O'MalleyOwen O'Malley
HDFS-3461HFTP should use the same port & protocol for getting the delegation tokenMajor.Owen O'MalleyOwen O'Malley
HDFS-3453HDFS does not use ClientProtocol in a backward-compatible wayMajorhdfs-clientKihwal LeeKihwal Lee
HDFS-3330If GetImageServlet throws an Error or RTE, response has HTTP “OK” statusCriticalnamenodeTodd LipconTodd Lipcon
HDFS-3176JsonUtil should not parse the MD5MD5CRC32FileChecksum bytes on its own.Majorhdfs-clientKihwal LeeKihwal Lee
HDFS-30782NN https port setting is brokenMajor.Eli CollinsEli Collins
HDFS-3008Negative caching of local addrs doesn't workMajorhdfs-clientEli CollinsEli Collins
HDFS-2877If locking of a storage dir fails, it will remove the other NN's lock file on exitMajornamenodeTodd LipconTodd Lipcon
HDFS-2869Error in Webhdfs documentation for mkdirMinorwebhdfsHarsh JHarsh J
HDFS-2790FSNamesystem.setTimes throws exception with wrong configuration name in the messageMinor.Arpit GuptaArpit Gupta
HDFS-2751Datanode drops OS cache behind reads even for short readsMajordatanodeTodd LipconTodd Lipcon
HDFS-2741dfs.datanode.max.xcievers missing in 0.20.205.0Minor.Markus Jelsma
HDFS-2728Remove dfsadmin -printTopology from branch-1 docs since it does not existMinornamenodeHarsh JHarsh J
HDFS-2637The rpc timeout for block recovery is too lowMajorhdfs-clientEli CollinsEli Collins
HDFS-2547ReplicationTargetChooser has incorrect block placement commentsTrivialnamenodeHarsh JHarsh J
HDFS-2541For a sufficiently large value of blocks, the DN Scanner may request a random number with a negative seed value.MajordatanodeHarsh JHarsh J
HDFS-2305Running multiple 2NNs can result in corrupt file systemMajornamenodeAaron T. MyersAaron T. Myers
HDFS-1910when dfs.name.dir and dfs.name.edits.dir are same fsimage will be saved twice every timeMinornamenodeGokul
MAPREDUCE-4698TestJobHistoryConfig throws Exception in testJobHistoryLoggingMinor.Gopal VGopal V
MAPREDUCE-4675TestKillSubProcesses fails as the process is still alive after the job is doneMajortestArpit GuptaBikas Saha
MAPREDUCE-4673make TestRawHistoryFile and TestJobHistoryServer more robustMajortestArpit GuptaArpit Gupta
MAPREDUCE-4558TestJobTrackerSafeMode is failingMajor.Siddharth SethSiddharth Seth
MAPREDUCE-4400Fix performance regression for small jobs/workflowsMajorperformance, taskLuke LuLuke Lu
MAPREDUCE-4241Pipes examples do not compile on Ubuntu 12.04Majorbuild, examplesAndrew BayerAndrew Bayer
MAPREDUCE-4095TestJobInProgress#testLocality uses a bogus topologyMajor.Eli CollinsColin Patrick McCabe
MAPREDUCE-4088Task stuck in JobLocalizer prevented other tasks on the same node from committingCriticalmrv1Ravi PrakashRavi Prakash
MAPREDUCE-4087[Gridmix] GenerateDistCacheData job of Gridmix can become slow in some casesMajor.Ravi GummadiRavi Gummadi
MAPREDUCE-3789CapacityTaskScheduler may perform unnecessary reservations in heterogenous tracker environmentsCriticalcapacity-sched, schedulerHarsh JHarsh J
MAPREDUCE-3674If invoked with no queueName request param, jobqueue_details.jsp injects a null queue name into schedulers.CriticaljobtrackerHarsh JHarsh J
MAPREDUCE-3419Don't mark exited TT threads as dead in MiniMRClusterMajortasktracker, testEli CollinsEli Collins
MAPREDUCE-3405MAPREDUCE-3015 broke compilation of contrib scheduler testsCriticalcapacity-sched, contrib/fair-shareTodd LipconTodd Lipcon
MAPREDUCE-2932Missing instrumentation plugin class shouldn't crash the TT startup per designTrivialtasktrackerHarsh JHarsh J
MAPREDUCE-2905CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)Majorcontrib/fair-shareJeff BeanJeff Bean
MAPREDUCE-2903Map Tasks graph is throwing XML Parse error when Job is executed with 0 mapsMajorjobtrackerDevaraj KDevaraj K
MAPREDUCE-2806[Gridmix] Load job fails with timeout errors when resource emulation is turned onMajorcontrib/gridmixAmar KamatAmar Kamat
MAPREDUCE-2377task-controller fails to parse configuration if it doesn't end in \nMajortask-controllerTodd LipconBenoy Antony
MAPREDUCE-2129Job may hang if mapreduce.job.committer.setup.cleanup.needed=false and mapreduce.map/reduce.failures.maxpercent>0MajorjobtrackerKang XiaoSubroto Sanyal
MAPREDUCE-1740NPE in getMatchingLevelForNodes when node locations are variable depthMajorjobtrackerTodd LipconAhmed Radwan

TESTS:

JIRASummaryPriorityComponentReporterContributor
HADOOP-7942enabling clover coverage reports fails hadoop unit test compilationMajor.Giridharan KesavanJitendra Nath Pandey
HDFS-3129NetworkTopology: add test that getLeaf should check for invalid topologiesMinor.Colin Patrick McCabeColin Patrick McCabe
HDFS-2332Add test for HADOOP-7629: using an immutable FsPermission as an IPC parameterMajortestTodd LipconTodd Lipcon
MAPREDUCE-2073TestTrackerDistributedCacheManager should be up-front about requirements on build environmentTrivialdistributed-cache, testTodd LipconTodd Lipcon

SUB-TASKS:

JIRASummaryPriorityComponentReporterContributor
HADOOP-7753Support fadvise and sync_data_range in NativeIO, add ReadaheadPool classMajorio, native, performanceTodd LipconTodd Lipcon
MAPREDUCE-3424Some LinuxTaskController cleanupMinortasktrackerEli CollinsEli Collins
MAPREDUCE-3015Add local dir failure info to metrics and the web UIMajortasktrackerEli CollinsEli Collins
MAPREDUCE-3008[Gridmix] Improve cumulative CPU usage emulation for short running tasksMajorcontrib/gridmixAmar KamatAmar Kamat
MAPREDUCE-2957The TT should not re-init if it has no good local dirsMajortasktrackerEli CollinsEli Collins
MAPREDUCE-2850Add test for TaskTracker disk failure handling (MR-2413)MajortasktrackerEli CollinsRavi Gummadi

OTHER:

JIRASummaryPriorityComponentReporterContributor
MAPREDUCE-2517Porting Gridmix v3 system tests into trunk branch.Majorcontrib/gridmixVinay Kumar ThotaVinay Kumar Thota