blob: ddf21ed838b20b419a9ac155c4287b25a3233b59 [file] [log] [blame]
Hadoop MapReduce Change Log
Trunk (Unreleased)
MAPREDUCE-778. Rumen Anonymizer. (Amar Kamat and Chris Douglas via amarrk)
MAPREDUCE-2669. Add new examples for Mean, Median, and Standard Deviation.
(Plamen Jeliazkov via shv)
MAPREDUCE-4887. Add RehashPartitioner, to smooth distributions
with poor implementations of Object#hashCode(). (Radim Kolar via cutting)
MAPREDUCE-5232. Add a configuration to be able to log classpath and other
system properties on mapreduce JVMs startup. (Sangjin Lee via vinodkv)
MAPREDUCE-5910. Make MR AM resync with RM in case of work-preserving
RM-restart. (Rohith via jianhe)
MAPREDUCE-3481. [Gridmix] Improve Gridmix STRESS mode. (amarrk)
MAPREDUCE-3597. [Rumen] Rumen should provide APIs to access all the
job-history related information.
MAPREDUCE-3375. [Gridmix] Memory Emulation system tests.
(Vinay Thota via amarrk)
MAPREDUCE-2733. [Gridmix] Gridmix3 cpu emulation system tests.
(Vinay Thota via amarrk)
MAPREDUCE-2836. Provide option to fail jobs when submitted to non-existent
fair scheduler pools. (Ahmed Radwan via todd)
MAPREDUCE-3171. normalize nodemanager native code compilation with common/hdfs
native. (tucu)
MAPREDUCE-3149. Add a test to verify that TokenCache handles file system
uri with no authority. (John George via jitendra)
MAPREDUCE-3169. Create a new MiniMRCluster equivalent which only provides
client APIs cross MR1 and MR2 (Ahmed via tucu)
MAPREDUCE-2944. Improve checking of input for JobClient.displayTasks()
(XieXianshan via harsh)
MAPREDUCE-3956. Remove the use of the deprecated Syncable.sync() method from
TeraOutputFormat in the terasort example. (szetszwo)
MAPREDUCE-3935. Annotate Counters.Counter and Counters.Group as @Public.
HADOOP-8285 MR changes for Use ProtoBuf for RpcPayLoadHeader (sanjay radia)
MAPREDUCE-3302. Remove the last dependency call from
org.apache.hadoop.record package in MR. (harsh)
MAPREDUCE-2384. The job submitter should make sure to validate
jobs before creation of necessary files. (harsh)
MAPREDUCE-4371. Check for cyclic dependencies in Jobcontrol job DAG
(madhukara phatak via bobby)
MAPREDUCE-4686. hadoop-mapreduce-client-core fails compilation in Eclipse
due to missing Avro-generated classes (Chris Nauroth via harsh)
MAPREDUCE-4735. Make arguments in TestDFSIO case insensitive.
(Brandon Li via suresh)
MAPREDUCE-5197. Add a service for checkpointing task state.
(Carlo Curino via cdouglas)
MAPREDUCE-5189. Add policies and wiring to respond to preemption requests
from YARN. (Carlo Curino via cdouglas)
MAPREDUCE-5196. Add bookkeeping for managing checkpoints of task state.
(Carlo Curino via cdouglas)
MAPREDUCE-5912. Task.calculateOutputSize does not handle Windows files after
MAPREDUCE-5196. (Remus Rusanu via cnauroth)
MAPREDUCE-6019. MapReduce changes for exposing YARN/MR endpoints on multiple
interfaces. (Craig Welch, Milan Potocnik, Arpit Agarwal via xgong)
MAPREDUCE-6013. [post-HADOOP-9902] mapred version is missing (Akira AJISAKA
via aw)
MAPREDUCE-5714. Removed forceful JVM exit in shutDownJob.
(Jinghui Wang via Eric Yang)
MAPREDUCE-3194. "mapred mradmin" command is broken in mrv2
(Jason Lowe via bobby)
MAPREDUCE-3462. Fix Gridmix JUnit testcase failures.
(Ravi Prakash and Ravi Gummadi via amarrk)
MAPREDUCE-3349. Log rack-name in JobHistory for unsuccessful tasks.
(Devaraj K and Amar Kamat via amarrk)
MAPREDUCE-3412. Fix 'ant docs'. (amarrk)
MAPREDUCE-3346. [Rumen] LoggedTaskAttempt#getHostName() returns null.
MAPREDUCE-2950. [Gridmix] TestUserResolve fails in trunk.
(Ravi Gummadi via amarrk)
MAPREDUCE-2784. [Gridmix] Bug fixes in ExecutionSummarizer and
ResourceUsageMatcher. (amarrk)
MAPREDUCE-2978. Fixed test-patch to make Jenkins report correct number of
findBugs, correct links to findBugs artifacts and no links to the
artifacts when there are no warnings. (Tom White via vinodkv).
MAPREDUCE-3664. Federation Documentation has incorrect configuration example.
(Brandon Li via jitendra)
MAPREDUCE-1740. NPE in getMatchingLevelForNodes when node locations are
variable depth (ahmed via tucu) [IMPORTANT: this is dead code in trunk]
MAPREDUCE-3990. MRBench allows Long-sized input-lines value
but parses CLI argument as an Integer. (harsh)
MAPREDUCE-3868. Make Raid Compile. (Weiyan Wang via schen)
MAPREDUCE-4685. DBCount should not use ACCESS. (Viji via harsh)
MAPREDUCE-3223. Remove MR1 configs from mapred-default.xml (tlipcon via harsh)
MAPREDUCE-4695. Fix LocalRunner on trunk after MAPREDUCE-3223 broke it
MAPREDUCE-4574. Fix TotalOrderParitioner to work with
non-WritableComparable key types. (harsh)
MAPREDUCE-5012. Typo in javadoc for IdentityMapper class. (Adam Monsen
via suresh)
MAPREDUCE-4987. TestMRJobs#testDistributedCache fails on Windows due to
classpath problems and unexpected behavior of symlinks (Chris Nauroth via
MAPREDUCE-5191. TestQueue#testQueue fails with timeout on Windows. (Ivan
Mitic via hitesh)
MAPREDUCE-5717. Task pings are interpreted as task progress (jlowe)
MAPREDUCE-5867. Fix NPE in KillAMPreemptionPolicy related to
ProportionalCapacityPreemptionPolicy (Sunil G via devaraj)
MAPREDUCE-5890. Support for encrypting Intermediate
data and spills in local filesystem. (asuresh via tucu)
MAPREDUCE-6007. Add support to distcp to preserve raw.* namespace
extended attributes. (clamb)
MAPREDUCE-6041. Fix TestOptionsParser. (clamb)
Release 2.6.0 - UNRELEASED
MAPREDUCE-5971. Move the default options for distcp -p to
DistCpOptionSwitch. (clamb via wang)
MAPREDUCE-5963. ShuffleHandler DB schema should be versioned with
compatible/incompatible changes (Junping Du via jlowe)
MAPREDUCE-883. harchive: Document how to unarchive (Akira AJISAKA and
Koji Noguchi via aw)
MAPREDUCE-4791. Javadoc for KeyValueTextInputFormat should include default
separator and how to change it (Akira AJISAKA via aw)
MAPREDUCE-5906. Inconsistent configuration in property
"mapreduce.reduce.shuffle.input.buffer.percent" (Akira AJISAKA via aw)
MAPREDUCE-5974. Allow specifying multiple MapOutputCollectors with
fallback. (Todd Lipcon via kasha)
MAPREDUCE-5130. Add missing job config options to mapred-default.xml
(Ray Chiang via Sandy Ryza)
MAPREDUCE-5866. TestFixedLengthInputFormat fails in windows.
(Varun Vasudev via cnauroth)
MAPREDUCE-5956. Made MR AM not use maxAttempts to determine if the current
attempt is the last retry. (Wangda Tan via zjshen)
MAPREDUCE-5957. AM throws ClassNotFoundException with job classloader
enabled if custom output format/committer is used (Sangjin Lee via jlowe)
MAPREDUCE-5756. CombineFileInputFormat.getSplits() including directories
in its results (Jason Dere via jlowe)
MAPREDUCE-6014. New task status field in task attempts table can lead to
an empty web page (Mit Desai via jlowe)
MAPREDUCE-6021. MR AM should have working directory in LD_LIBRARY_PATH
MAPREDUCE-6010. HistoryServerFileSystemStateStore fails to update tokens
MAPREDUCE-5878. some standard JDK APIs are not part of system classes
defaults (Sangjin Lee via jlowe)
MAPREDUCE-5944. Remove MRv1 commands from CommandsManual.apt.vm
(Akira AJISAKA via aw)
MAPREDUCE-5943. Separate mapred commands from CommandManual.apt.vm
(Akira AJISAKA via aw)
MAPREDUCE-5363. Fix doc and spelling for TaskCompletionEvent#getTaskStatus
and getStatus (Akira AJISAKA via aw)
MAPREDUCE-5595. Typo in (Akira AJISAKA via aw)
MAPREDUCE-5597. Missing alternatives in javadocs for deprecated constructors
in mapreduce.Job (Akira AJISAKA via aw)
MAPREDUCE-5950. incorrect description in distcp2 document (Akira AJISAKA
via aw)
MAPREDUCE-5998. CompositeInputFormat javadoc is broken (Akira AJISAKA via
MAPREDUCE-5999. Fix dead link in InputFormat javadoc (Akira AJISAKA via aw)
MAPREDUCE-6032. Made MR jobs write job history files on the default FS when
the current context's FS is different. (Benjamin Zhitomirsky via zjshen)
MAPREDUCE-6024. Shortened the time when Fetcher is stuck in retrying before
concluding the failure by configuration. (Yunjiong Zhao via zjshen)
MAPREDUCE-6036. TestJobEndNotifier fails intermittently in branch-2 (chang
li via jlowe)
MAPREDUCE-6012. DBInputSplit creates invalid ranges on Oracle.
(Wei Yan via kasha)
MAPREDUCE-6044. Fully qualified intermediate done dir path breaks per-user dir
creation on Windows. (zjshen)
Release 2.5.0 - 2014-08-11
MAPREDUCE-5671. NaN can be created by client and assign to Progress (Chen
He via jeagles)
MAPREDUCE-5665. Add audience annotations to MiniMRYarnCluster and
MiniMRCluster. (Anubhav Dhoot via kasha)
MAPREDUCE-5765. Update hadoop-pipes examples README (Mit Desai via jeagles)
MAPREDUCE-5713. InputFormat and JobConf JavaDoc Fixes (Chen He via jeagles)
MAPREDUCE-5456. TestFetcher.testCopyFromHostExtraBytes is missing (Jason
Lowe via jeagles)
MAPREDUCE-5804. TestMRJobsWithProfiler#testProfiler timesout (Mit Desai
via kihwal)
MAPREDUCE-5825. Provide diagnostics for reducers killed during ramp down
(Gera Shegalov via jlowe)
MAPREDUCE-5836. Fix typo in RandomTextWriter (Akira AJISAKA via jeagles)
MAPREDUCE-5852. Prepare MapReduce codebase for JUnit 4.11. (cnauroth)
MAPREDUCE-5639. Port DistCp2 document to trunk (Akira AJISAKA via jeagles)
MAPREDUCE-5812. Make job context available to
OutputCommitter.isRecoverySupported() (Mohammad Kamrul Islam via jlowe)
MAPREDUCE-5638. Port Hadoop Archives document to trunk (Akira AJISAKA via
MAPREDUCE-5402. In DynamicInputFormat, change MAX_CHUNKS_TOLERABLE,
(Tsuyoshi OZAWA via szetszwo)
MAPREDUCE-5637. Convert Hadoop Streaming document to APT (Akira AJISAKA via
MAPREDUCE-5636. Convert MapReduce Tutorial document to APT (Akira AJISAKA
via jeagles)
MAPREDUCE-5774. Job overview in History UI should list reducer phases in
chronological order. (Gera Shegalov via kasha)
MAPREDUCE-5652. NM Recovery. ShuffleHandler should handle NM restarts.
(Jason Lowe via kasha)
MAPREDUCE-5861. finishedSubMaps field in LocalContainerLauncher does not
need to be volatile. (Tsuyoshi OZAWA via junping_du)
MAPREDUCE-5809. Enhance distcp to support preserving HDFS ACLs. (cnauroth)
MAPREDUCE-5899. Support incremental data copy in DistCp. (jing9)
MAPREDUCE-5886. Allow wordcount example job to accept multiple input paths.
MAPREDUCE-5834. Increased test-timeouts in TestGridMixClasses to avoid
occassional failures. (Mit Desai via vinodkv)
MAPREDUCE-5896. InputSplits should indicate which locations have the block
cached in memory. (Sandy Ryza via kasha)
MAPREDUCE-5844. Add a configurable delay to reducer-preemption.
(Maysam Yabandeh via kasha)
MAPREDUCE-5790. Made it easier to enable hprof profile options by default.
(Gera Shegalov via vinodkv)
MAPREDUCE-6033. Updated access check for displaying job information
(Yu Gao via Eric Yang)
MAPREDUCE-5759. Remove unnecessary conf load in Limits (Sandy Ryza)
MAPREDUCE-5014. Extend Distcp to accept a custom CopyListing.
(Srikanth Sundarrajan via amareshwari)
MAPREDUCE-5775. Remove unnecessary job.setNumReduceTasks in SleepJob.createJob
(jhanver chand sharma via devaraj)
MAPREDUCE-4937. MR AM handles an oversized split metainfo file poorly
(Eric Payne via jlowe)
MAPREDUCE-5642. TestMiniMRChildTask fails on Windows.
(Chuan Liu via cnauroth)
MAPREDUCE-5846. Rumen doesn't understand JobQueueChangedEvent (Nathan Roberts via raviprak)
MAPREDUCE-5837. MRAppMaster fails when checking on uber mode. (wheat9)
MAPREDUCE-5749. TestRMContainerAllocator#testReportedAppProgress Failed
MAPREDUCE-5884. History server uses short user name when canceling tokens
(Mohammad Kamrul Islam via jlowe)
MAPREDUCE-5888. Failed job leaves hung AM after it unregisters (Jason Lowe
via jeagles)
MAPREDUCE-5814. fat jar with *-default.xml may fail when
mapreduce.job.classloader=true. (Gera Shegalov via jlowe)
MAPREDUCE-5309. 2.0.4 JobHistoryParser can't parse certain failed job
history files generated by 2.0.3 history server (Rushabh S Shah via jlowe)
MAPREDUCE-5862. Line records longer than 2x split size aren't handled
correctly (bc Wong via jlowe)
MAPREDUCE-5895. Close streams properly to avoid leakage in TaskLog.
(Kousuke Saruta via devaraj)
MAPREDUCE-5777. Support utf-8 text with Byte Order Marker.
(Zhihai Xu via kasha)
MAPREDUCE-5898. distcp to support preserving HDFS extended attributes(XAttrs)
(Yi Liu via umamahesh)
MAPREDUCE-5920. Add Xattr option in DistCp docs. (Yi Liu via cnauroth)
MAPREDUCE-5924. Changed TaskAttemptImpl to ignore TA_COMMIT_PENDING event
at COMMIT_PENDING state. (Zhijie Shen via jianhe)
MAPREDUCE-5939. StartTime showing up as the epoch time in JHS UI after
upgrade (Chen He via jlowe)
MAPREDUCE-5900. Changed to the interpret container preemption exit code as a
task attempt killing event. (Mayank Bansal via zjshen)
MAPREDUCE-5868. Fixed an issue with TestPipeApplication that was causing the
nightly builds to fail. (Akira Ajisaka via vinodkv)
MAPREDUCE-5517. Fixed MapReduce ApplicationMaster to not validate reduce side
resource configuration for deciding uber-mode on map-only jobs. (Siqi Li via
MAPREDUCE-5952. LocalContainerLauncher#renameMapOutputForReduce incorrectly
assumes a single dir for mapOutIndex. (Gera Shegalov via kasha)
MAPREDUCE-6002. Made MR task avoid reporting error to AM when the task process
is shutting down. (Wangda Tan via zjshen)
Release 2.4.1 - 2014-06-23
MAPREDUCE-5830. Added back the private API HostUtil.getTaskLogUrl(..) for
binary compatibility with older clients like Hive 0.13. (Akira Ajisaka via
MAPREDUCE-5818. Added "hsadmin" command into mapred.cmd. (Jian He via zjshen)
MAPREDUCE-5824. Fixed test-failure of TestPipesNonJavaInputFormat in
Windows. (Xuan Gong via vinodkv)
MAPREDUCE-5815. Fixed test-failure of TestMRAppMaster by making MRAppMaster
gracefully handle empty-queue names. (Akira Ajisaka via vinodkv)
MAPREDUCE-5826. Fixed HistoryServerFileSystemStore to use right permissions
on Windows for temporary files and thus also fix the test-issue with
TestHistoryServerFileSystemStateStoreService. (Varun Vasudev via vinodkv)
MAPREDUCE-5828. Fixed a test issue with TestMapReduceJobControl that was
causing it to fail on Windows. (vinodkv)
MAPREDUCE-5827. TestSpeculativeExecutionWithMRApp fails.
(Zhijie Shen via cnauroth)
MAPREDUCE-5833. TestRMContainerAllocator fails ocassionally.
(Zhijie Shen via cnauroth)
MAPREDUCE-5832. Fixed TestJobClient to not fail on JDK7 or on Windows. (Jian
He and Vinod Kumar Vavilapalli via vinodkv)
MAPREDUCE-5841. uber job doesn't terminate on getting mapred job kill
(Sangjin Lee via jlowe)
MAPREDUCE-5843. Fixed TestMRKeyValueTextInputFormat to not leak files and
thus avoid failing on Windows. (Varun Vasudev via vinodkv)
MAPREDUCE-5835. Killing Task might cause the job to go to ERROR state
(Ming Ma via jlowe)
MAPREDUCE-5821. Avoid unintentional reallocation of byte arrays in segments
during merge. (Todd Lipcon via cdouglas)
Release 2.4.0 - 2014-04-07
MAPREDUCE-5787. Added the ability to keep alive shuffle connections in the
MapReduce shuffle-handler. (Rajesh Balamohan via vinodkv)
MAPREDUCE-5464. Add analogs of the SLOTS_MILLIS counters that jive with the
YARN resource model (Sandy Ryza)
MAPREDUCE-5732. Report proper queue when job has been automatically placed
(Sandy Ryza)
MAPREDUCE-5699. Allow setting tags on MR jobs (kasha)
MAPREDUCE-5761. Added a simple log message to denote when encrypted shuffle
is on in the shuffle-handler. (Jian He via vinodkv)
MAPREDUCE-5754. Preserve Job diagnostics in history (Gera Shegalov via
MAPREDUCE-5766. Moved ping messages from TaskAttempts to be at DEBUG level
inside the ApplicationMaster log. (Jian He via vinodkv)
MAPREDUCE-5773. Provide dedicated MRAppMaster syslog length limit (Gera
Shegalov via jlowe)
MAPREDUCE-5553. Allow users to easily access
completed/pending/successful/failed tasks on MR AM web-ui. (Paul Han via
MAPREDUCE-4052. Improved MapReduce clients to use NodeManagers' ability to
handle cross platform application submissions. (Jian He via vinodkv)
MAPREDUCE-2349. Modified FileInputFormat to be able to issue file and block
location calls in parallel. (Siddharth Seth via vinodkv)
MAPREDUCE-5746. Job diagnostics can implicate wrong task for a failed job.
(Jason Lowe via kasha)
MAPREDUCE-5670. CombineFileRecordReader should report progress when moving
to the next file (Chen He via jlowe)
MAPREDUCE-5757. ConcurrentModificationException in JobControl.toList
MAPREDUCE-5770. Fixed MapReduce ApplicationMaster to correctly redirect
to the YARN's web-app proxy with the correct scheme prefix. (Jian He via
MAPREDUCE-5768. TestMRJobs.testContainerRollingLog fails on trunk (Gera
Shegalov via jlowe)
MAPREDUCE-5780. SliveTest should use the specified path to get the
particular FileSystem instead of using the default FileSystem. (szetszwo)
MAPREDUCE-5028. Fixed a bug in MapTask that was causing mappers to fail
when a large value of io.sort.mb is set. (Karthik Kambatla via vinodkv)
MAPREDUCE-5778. JobSummary does not escape newlines in the job name (Akira
AJISAKA via jlowe)
MAPREDUCE-5789. Average Reduce time is incorrect on Job Overview page
(Rushabh S Shah via jlowe)
MAPREDUCE-5794. SliveMapper always uses default FileSystem. (szetszwo via
Arpit Agarwal)
MAPREDUCE-5751. MR app master fails to start in some cases if
mapreduce.job.classloader is true (Sangjin Lee via jlowe)
MAPREDUCE-5688. TestStagingCleanup fails intermittently with JDK7 (Mit
Desai via jeagles)
MAPREDUCE-5769. Unregistration to RM should not be called if AM is crashed
before registering with RM (Rohith via jlowe)
MAPREDUCE-5570. Map task attempt with fetch failure has incorrect attempt
finish time (Rushabh S Shah via jlowe)
MAPREDUCE-5806. Fixed a bug in MRAppMaster so as to enable users to properly
override HADOOP_ROOT_LOGGER or HADOOP_CLIENT_OPTS. (Varun Vasudev via
MAPREDUCE-5791. Shuffle phase is slow in Windows -
FadviseFileRegion::transferTo does not read disks efficiently.
(Nikola Vujic via cnauroth)
MAPREDUCE-5795. Fixed MRAppMaster to record the correct job-state after it
recovers from a commit during a previous attempt. (Xuan Gong via vinodkv)
MAPREDUCE-5805. Fixed MapReduce JobHistory encoding of queue-name to escape
hyphens and thus avoid parsing errors. (Akira AJISAKA via vinodkv)
MAPREDUCE-5810. Removed the faulty and failing streaming test
TestStreamingTaskLog. (Akira Ajisaka via vinodkv)
MAPREDUCE-5813. Fix YarnChild to explicitly load job.xml from the
local-filesystem, rather than rely on the classpath. (Gera Shegalov via
Release 2.3.1 - UNRELEASED
Release 2.3.0 - 2014-02-18
MAPREDUCE-5265. History server admin service to refresh user and superuser
group mappings (Ashwin Shankar via jlowe)
MAPREDUCE-5356. Ability to refresh aggregated log retention period and
check interval (Ashwin Shankar via jlowe)
MAPREDUCE-5386. Ability to refresh history server job retention and job
cleaner settings (Ashwin Shankar via jlowe)
MAPREDUCE-5411. Refresh size of loaded job cache on history server (Ashwin
Shankar via jlowe)
MAPREDUCE-5332. Support token-preserving restart of history server (jlowe)
MAPREDUCE-5329. Allow MR applications to use additional AuxServices,
which are compatible with the default MapReduce shuffle.
(Avner BenHanoch via sseth)
MAPREDUCE-5463. Deprecate SLOTS_MILLIS counters (Tzuyoshi Ozawa via Sandy
MAPREDUCE-5457. Add a KeyOnlyTextOutputReader to enable streaming to write
out text files without separators (Sandy Ryza)
MAPREDUCE-5596. Allow configuring the number of threads used to serve
shuffle connections (Sandy Ryza via jlowe)
MAPREDUCE-434. LocalJobRunner limited to single reducer (Sandy Ryza and
Aaron Kimball via Sandy Ryza)
MAPREDUCE-4421. Run MapReduce framework via the distributed cache (jlowe)
MAPREDUCE-1176. FixedLengthInputFormat and FixedLengthRecordReader
(Mariappan Asokan and BitsOfInfo via Sandy Ryza)
MAPREDUCE-5613. DefaultSpeculator holds and checks hashmap that is always
empty (Gera Shegalov via Sandy Ryza)
MAPREDUCE-5431. Missing pom dependency in MR-client (Timothy St. Clair
via stevel)
MAPREDUCE-5624 Move grizzly-test and junit dependencies to test scope
(Ted Yu via stevel)
MAPREDUCE-5481. Enable uber jobs to have multiple reducers (Sandy Ryza)
MAPREDUCE-5052. Job History UI and web services confusing job start time and
job submit time (Chen He via jeagles)
MAPREDUCE-5692. Add explicit diagnostics when a task attempt is killed due
to speculative execution (Gera Shegalov via Sandy Ryza)
MAPREDUCE-5550. Task Status message (reporter.setStatus) not shown in UI
with Hadoop 2.0 (Gera Shegalov via Sandy Ryza)
MAPREDUCE-3310. Custom grouping comparator cannot be set for Combiners (tucu)
MAPREDUCE-5672. Provide optional RollingFileAppender for container log4j
(syslog) (Gera Shegalov via jlowe)
MAPREDUCE-5725. Make explicit that TestNetworkedJob relies on the Capacity
Scheduler (Sandy Ryza)
MAPREDUCE-5744. Job hangs because
RMContainerAllocator$AssignedRequests.preemptReduce() violates the
comparator contract (Gera Shegalov via kasha)
MAPREDUCE-4680. Job history cleaner should only check timestamps of files in
old enough directories (Robert Kanter via Sandy Ryza)
MAPREDUCE-5484. YarnChild unnecessarily loads job conf twice (Sandy Ryza)
MAPREDUCE-5487. In task processes, JobConf is unnecessarily loaded again
in Limits (Sandy Ryza)
MAPREDUCE-5601. ShuffleHandler fadvises file regions as DONTNEED even when
fetch fails (Sandy Ryza)
MAPREDUCE-5569. FloatSplitter is not generating correct splits (Nathan
Roberts via jlowe)
MAPREDUCE-5546. mapred.cmd on Windows set HADOOP_OPTS incorrectly (Chuan Liu
via cnauroth)
MAPREDUCE-5518. Fixed typo "can't read paritions file". (Albert Chu
via devaraj)
testcase failing on trunk (Karthik Kambatla via jlowe)
MAPREDUCE-5598. TestUserDefinedCounters.testMapReduceJob is flakey
(Robert Kanter via jlowe)
MAPREDUCE-5604. TestMRAMWithNonNormalizedCapabilities fails on Windows due to
exceeding max path length. (cnauroth)
MAPREDUCE-5451. MR uses LD_LIBRARY_PATH which doesn't mean anything in
Windows. (Yingda Chen via cnauroth)
MAPREDUCE-5409. MRAppMaster throws InvalidStateTransitonException: Invalid
event: TA_TOO_MANY_FETCH_FAILURE at KILLED for TaskAttemptImpl (Gera
Shegalov via jlowe)
MAPREDUCE-5674. Missing start and finish time in mapred.JobStatus.
(Chuan Liu via cnauroth)
MAPREDUCE-5650. Job fails when hprof
is specified (Gera Shegalov via Sandy Ryza)
MAPREDUCE-5316. job -list-attempt-ids command does not handle illegal
task-state (Ashwin Shankar via jlowe)
MAPREDUCE-5380. Invalid mapred command should return non-zero exit code
(Stephen Chu via jlowe)
MAPREDUCE-5404. HSAdminServer does not use ephemeral ports in minicluster
mode (Ted Yu via jlowe)
MAPREDUCE-5522. Incorrect oreder expected from JobQueueInfo (Jinghui Wang
via bobby)
MAPREDUCE-5514. Fix TestRMContainerAllocator. (Zhijie Shen via acmurthy)
MAPREDUCE-5102. fix coverage org.apache.hadoop.mapreduce.lib.db and
org.apache.hadoop.mapred.lib.db (Aleksey Gorshkov, Andrey Klochkov, and
Nathan Roberts via jlowe)
MAPREDUCE-5585. TestCopyCommitter#testNoCommitAction Fails on JDK7
MAPREDUCE-5186. mapreduce.job.max.split.locations causes some splits
created by CombineFileInputFormat to fail (Robert Parker and Jason Lowe
via jlowe)
MAPREDUCE-5610. TestSleepJob fails in jdk7 (Jonathan Eagles via jlowe)
MAPREDUCE-5616. MR Client-AppMaster RPC max retries on socket timeout is too
high. (cnauroth)
MAPREDUCE-5625. TestFixedLengthInputFormat fails in jdk7 environment
(Mariappan Asokan via jeagles)
MAPREDUCE-5631. TestJobEndNotifier.testNotifyRetries fails with Should
have taken more than 5 seconds in jdk7 (Jonathan Eagles via jlowe)
MAPREDUCE-5645. TestFixedLengthInputFormat fails with native libs (Mit
Desai via jeagles)
MAPREDUCE-5632. TestRMContainerAllocator#testUpdatedNodes fails (jeagles)
MAPREDUCE-5656. bzip2 codec can drop records when reading data in splits
MAPREDUCE-5623. TestJobCleanup fails because of RejectedExecutionException
and NPE. (jlowe)
MAPREDUCE-5679. TestJobHistoryParsing has race condition (Liyin Liang via
MAPREDUCE-5687. Fixed failure in TestYARNRunner caused by YARN-1446. (Jian He
via vinodkv)
MAPREDUCE-5694. Fixed MR AppMaster to shutdown the LogManager so as to avoid
losing syslog in some conditions. (Mohammad Kamrul Islam via vinodkv)
MAPREDUCE-5685. Fixed a bug with JobContext getCacheFiles API inside the
WrappedReducer class. (Yi Song via vinodkv)
MAPREDUCE-5689. MRAppMaster does not preempt reducers when scheduled maps
cannot be fulfilled. (lohit via kasha)
MAPREDUCE-5724. JobHistoryServer does not start if HDFS is not running.
MAPREDUCE-5729. mapred job -list throws NPE (kasha)
MAPREDUCE-5693. Restore MRv1 behavior for log flush (Gera Shegalov via
MAPREDUCE-5723. MR AM container log can be truncated or empty.
(Mohammad Kamrul Islam via kasha)
MAPREDUCE-5743. Fixed the test failure in TestRMContainerAllocator.
(Ted Yu and Vinod Kumar Vavilapalli via zjshen)
Release 2.2.0 - 2013-10-13
MAPREDUCE-5504. mapred queue -info inconsistent with types (Kousuke Saruta
via tgraves)
MAPREDUCE-5488. Changed MR client to keep trying to reach the application
when it sees that on attempt's AM is down. (Jian He via vinodkv)
MAPREDUCE-5515. Fixed MR AM's webapp to depend on a new config
mapreduce.ssl.enabled to enable https and disabling it by default as MR AM
needs to set up its own certificates etc and not depend on clusters'.
(Omkar Vinit Joshi via vinodkv)
MAPREDUCE-5505. Clients should be notified job finished only after job
successfully unregistered (Zhijie Shen via bikas)
MAPREDUCE-5503. Fixed a test issue in TestMRJobClient. (Jian He via vinodkv)
MAPREDUCE-5170. Fixed a wrong log message in CombineFileInputFormat class.
(Sangjin Lee via vinodkv)
MAPREDUCE-5525. Increase timeout of TestDFSIO.testAppend and
TestMRJobsWithHistoryService.testJobHistoryData. (Chuan Liu via cnauroth)
MAPREDUCE-5513. ConcurrentModificationException in JobControl (Robert
Parker via jlowe)
MAPREDUCE-5531. Fix compat with hadoop-1 in mapreduce.(TaskID,
TaskAttemptID) by re-introducing missing constructors. (Robert Kanter via
MAPREDUCE-5545. org.apache.hadoop.mapred.TestTaskAttemptListenerImpl.testCommitWindow
times out (Robert Kanter via jlowe)
MAPREDUCE-5529. Fix compat with hadoop-1 in mapred.TotalOrderPartitioner
by re-introducing (get,set)PartitionFile which takes in JobConf. (Robert
Kanter via acmurthy)
MAPREDUCE-5538. Fixed MR AppMaster to send job-notification URL only after
the job is really done - a bug caused by MAPREDUCE-5505. (Zhijie Shen via
MAPREDUCE-5551. Fix compat with hadoop-1 in
SequenceFileAsBinaryOutputFormat.WritableValueBytes by re-introducing
missing constructors. (Zhijie Shen via acmurthy)
MAPREDUCE-5544. JobClient#getJob loads job conf twice. (Sandy Ryza)
MAPREDUCE-5536. Fixed MR AM and JHS to respect
mapreduce.jobhistory.webapp.https.address. (Omkar Vinit Joshi via vinodkv)
MAPREDUCE-5530. Fix compat with hadoop-1 in
mapred.lib.CombinFileInputFormat by re-introducing
isSplittable(FileSystem, Path) api and ensuring semantic compatibility.
(Robert Kanter via acmurthy)
MAPREDUCE-5459. Update documentation on how to run MRv1 examples on YARN.
(Zhijie Shen via acmurthy)
MAPREDUCE-5554. hdfs-site.xml included in hadoop-mapreduce-client-jobclient
tests jar is breaking tests for downstream components (Robert Kanter via
Sandy Ryza)
MAPREDUCE-5489. MR jobs hangs as it does not use the node-blacklisting
feature in RM requests (Zhijie Shen via bikas)
Windows. (Yingda Chen via cnauroth)
MAPREDUCE-5533. Fixed MR speculation code to track any TaskAttempts that
aren't heart-beating for a while, so that we can aggressively speculate
instead of waiting for task-timeout (Xuan Gong via vinodkv)
MAPREDUCE-5562. Fixed MR App Master to perform pending tasks like staging-dir
cleanup, sending job-end notification correctly when unregister with RM
fails. (Zhijie Shen via vinodkv)
Release 2.1.1-beta - 2013-09-23
MAPREDUCE-5478. TeraInputFormat unnecessarily defines its own FileSplit
subclass (Sandy Ryza)
MAPREDUCE-5497. Changed MRAppMaster to sleep only after doing everything else
but just before ClientService to avoid race conditions during RM restart.
(Jian He via vinodkv)
MAPREDUCE-5379. Include token tracking ids in jobconf. (kkambatl via tucu)
MAPREDUCE-5523. Added separate configuration properties for https for JHS
without which even when https is enabled, it starts on http port itself.
(Omkar Vinit Joshi via vinodkv)
MAPREDUCE-5446. TestJobHistoryEvents and TestJobHistoryParsing have race
conditions (jlowe via kihwal)
MAPREDUCE-5462. In map-side sort, swap entire meta entries instead of
indexes for better cache performance. (Sandy Ryza)
MAPREDUCE-1981. Improve getSplits performance by using listLocatedStatus
(Hairong Kuang and Jason Lowe via jlowe)
MAPREDUCE-5385. Fixed a bug with JobContext getCacheFiles API. (Omkar Vinit
Joshi via vinodkv)
MAPREDUCE-5428. HistoryFileManager doesn't stop threads when service is
stopped (Karthik Kambatla via jlowe)
MAPREDUCE-5251. Reducer should not implicate map attempt if it has
insufficient space to fetch map output (Ashwin Shankar via jlowe)
MAPREDUCE-5317. Stale files left behind for failed jobs (Ravi Prakash via
MAPREDUCE-5358. MRAppMaster throws invalid transitions for JobImpl
(Devaraj K via jlowe)
MAPREDUCE-3193. FileInputFormat doesn't read files recursively in the
input path dir (Devaraj K via jlowe)
MAPREDUCE-5440. TestCopyCommitter Fails on JDK7 (Robert Parker via jlowe)
MAPREDUCE-5367. Local jobs all use same local working directory
(Sandy Ryza)
MAPREDUCE-5425. Junit in TestJobHistoryServer failing in jdk 7 (Robert
Parker via jlowe)
MAPREDUCE-5454. TestDFSIO fails intermittently on JDK7 (Karthik Kambatla
via Sandy Ryza)
MAPREDUCE-5001. LocalJobRunner has race condition resulting in job
failures (Sandy Ryza via jlowe)
MAPREDUCE-5466. Changed MR AM to not promote history files of intermediate
AMs in case they are exiting because of errors and thus help history-server
pick up the right history file for the last successful AM. (Jian He via
MAPREDUCE-5468. Fix MR AM recovery for map-only jobs. (vinodkv via
MAPREDUCE-5470. LocalJobRunner does not work on Windows. (Sandy Ryza via
MAPREDUCE-5476. Changed MR AM recovery code to cleanup staging-directory
only after unregistering from the RM. (Jian He via vinodkv)
MAPREDUCE-5483. revert MAPREDUCE-5357. (rkanter via tucu)
MAPREDUCE-5441. Changed MR AM to return RUNNING state if exiting when RM
commands to reboot, so that client can continue to track the overall job.
(Jian He via vinodkv)
MAPREDUCE-5475. MRClientService does not verify ACLs properly (jlowe)
MAPREDUCE-5414. TestTaskAttempt fails in JDK7 with NPE (Nemon Lou via
MAPREDUCE-5020. Compile failure with JDK8 (Trevor Robinson via tgraves)
MAPREDUCE-5164. mapred job and queue commands omit HADOOP_CLIENT_OPTS
(Nemon Lou via devaraj)
MAPREDUCE-5493. Cleanup in-memory & on-disk segments to prevent leak on
shuffle completion. (jlowe via acmurthy)
Release 2.1.0-beta - 2013-08-22
MAPREDUCE-4067. Changed MRClientProtocol api to throw IOException only (Xuan
Gong via vinodkv)
MAPREDUCE-5234. Change mapred.TaskReport and mapreduce.TaskReport for binary
compatibility with mapred in 1.x but incompatible with 0.23.x. (Mayank Bansal
via vinodkv)
MAPREDUCE-5156. Change hadoop examples ProgramDriver to be able to run
1.x examples jar on top of YARN. This change breaks 0.23.x direct usages of
ProgramDriver. (Zhijie Shen via vinodkv)
MAPREDUCE-5233. Add methods that are changed or removed from JobControl.Job
when compared to 1.x. This breaks 0.23.x users of one API in Job. (Mayank
Bansal via vinodkv)
MAPREDUCE-5237. Add methods that were removed from ClusterStatus back into
2.x so as to be compatible with 1.x. Incompatible as
ClusterStatus.UNINITIALIZED_MEMORY_VALUE is a long now and so breaks 0.23.x
but it shouldn't be a big deal in reality. (Zhijie Shen via vinodkv)
HADOOP-8562. Enhancements to support Hadoop on Windows Server and Windows
Azure environments. (See breakdown of tasks below for subtasks and
MAPREDUCE-3008. Improvements to cumulative CPU emulation for short running
tasks in Gridmix. (amarrk via tgraves)
MAPREDUCE-5033. mapred shell script should respect usage flags
(--help -help -h). (Andrew Wang via atm)
MAPREDUCE-4892. Modify CombineFileInputFormat to not skew input slits'
allocation on small clusters. (Bikas Saha via vinodkv)
MAPREDUCE-4990. Construct debug strings conditionally in
ShuffleHandler.Shuffle#sendMapOutput(). (kkambatl via tucu)
MAPREDUCE-4875. coverage fixing for org.apache.hadoop.mapred
(Aleksey Gorshkov via bobby)
MAPREDUCE-5129. Allow tags to JobHistory for deeper analytics. (billie via
MAPREDUCE-3787. [Gridmix] Optimize job monitoring and STRESS mode for
faster job submission. (amarrk via tgraves)
MAPREDUCE-5079. Changes job recovery to restore state directly from job
history, instaed of simulating state machine events.
(Jason Lowe and Robert Parker via sseth)
MAPREDUCE-4981. Add WordMean, WordMedian, WordStandardDeviation
to ExamplesDriver. (Plamen Jeliazkov via shv)
MAPREUDUCE-5059. Change average merge time on Job overview page to be the
time delta between the end of the shuffle and the start of the reduce.
(Omkar Vinit Joshi via vinodkv)
MAPREDUCE-4985. Add compression option to TestDFSIO usage.
(Plamen Jeliazkov via shv)
MAPREDUCE-5152. Make MR App to simply pass through the container from RM
instead of extracting and populating information itself to start any
container. (vinodkv)
MAPREDUCE-5175. Updated MR App to not set envs that will be set by NMs
anyways after YARN-561. (Xuan Gong via vinodkv)
MAPREDUCE-5069. add concrete common implementations of
CombineFileInputFormat (Sangjin Lee via bobby)
MAPREDUCE-5145. Changed default max-attempts to be more than one for MR jobs
inline with YARN. (Zhijie Shen via vinodkv)
MAPREDUCE-5036. Default shuffle handler port should not be 8080.
(Sandy Ryza via tomwhite)
MAPREDUCE-5159. Change ValueAggregatorJob to add APIs which can support
binary compatibility with hadoop-1 examples. (Zhijie Shen via vinodkv)
MAPREDUCE-5157. Bring back old sampler related code so that we can support
binary compatibility with hadoop-1 sorter example. (Zhijie Shen via vinodkv)
MAPREDUCE-5222. Bring back some methods and constants in Jobclient for
binary compatibility with mapred in 1.x. (Karthik Kambatla via vinodkv)
MAPREDUCE-5235. Bring back old fields and exceptions in Counters for
binary compatibility with mapred in 1.x. (Mayank Bansal via vinodkv)
MAPREDUCE-5246. Specify application-type at the time of job submission after
YARN-563. (Mayank Bansal via vinodkv)
MAPREDUCE-5230. Bring back NLineInputFormat.createFileSplit for binary
compatibility with mapred in 1.x (Mayank Bansal via vinodkv)
MAPREDUCE-5270. Migrated MR app from using BuilderUtil factory methods to
individual record factory methods. (Jian He via vinodkv)
MAPREDUCE-5263. Bring back old methods and fields in
filecache.DistributedCache for binary compatibility with mapred in 1.x.
(Zhijie Shen via vinodkv)
MAPREDUCE-5228. Bring back FileInputFormat.Counter and
FileOuputFormat.Counter for binary compatibility with 1.x mapred APIs.
(Mayank Bansal via vinodkv)
MAPREDUCE-5176. Add annotation for tagging tasks as responsive to
preemption. (Carlo Curino, cdouglas)
MAPREDUCE-5275. Bring back a couple of APIs in
for binary compatibility with 1.x mapreduce APIs. (Mayank Bansal via vinodkv)
MAPREDUCE-5231. Bring back a constructor in mapred's
DBInputFormat.DBRecordReader for binary compatibility with 1.x mapred APIs.
(Zhijie Shen via vinodkv)
MAPREDUCE-5273. Bring back a couple of protected variables in mapred and
mapreduce CombineFileRecordReader for binary compatibility with 1.x APIs.
(Mayank Bansal via vinodkv)
MAPREDUCE-5280. Bring back removed constructor and a method in mapreduce
ClusterMetrics for binary compatibility with 1.x APIs. (Mayank Bansal via
MAPREDUCE-5289. Updated MR App to use Token directly after YARN-717. (Jian He
via vinodkv)
MAPREDUCE-5229. Put back FileOutputCommitter.TEMP_DIR_NAME in mapreduce for
binary compatibility with 1.x APIs. (Zhijie Shen via vinodkv)
MAPREDUCE-5274. Bring back SecureShuffleUtils.toHex in mapreduce for binary
compatibility with 1.x APIs. (Mayank Bansal via vinodkv)
MAPREDUCE-5300. Fix backward incompatibility for
o.a.h.mapreduce.filecache.DistributedCache. (Zhijie Shen via acmurthy)
MAPREDUCE-5283. Over 10 different tests have near identical
implementations of AppContext (Sandy Ryza via jlowe)
MAPREDUCE-5199. Removing ApplicationTokens file as it is no longer needed.
(Daryn Sharp via vinodkv)
MAPREDUCE-5192. Allow for alternate resolutions of TaskCompletionEvents.
(cdouglas via acmurthy)
MAPREDUCE-5184. Document compatibility for MapReduce applications in
hadoop-2 vis-a-vis hadoop-1. (Zhijie Shen via acmurthy)
MAPREDUCE-5194. Heed interrupts during Fetcher shutdown. (cdouglas)
MAPREDUCE-5326. Added version to shuffle header. (Zhijie Shen via
MAPREDUCE-5333. Add test that verifies MRAM works correctly when sending
requests with non-normalized capabilities. (ywskycn via tucu)
MAPREDUCE-5398. MR changes for YARN-513 (Jian He via bikas)
MAPREDUCE-4974. Optimising the LineRecordReader initialize() method
(Gelesh via bobby)
MAPREDUCE-5268. Improve history server startup performance (Karthik
Kambatla via jlowe)
MAPREDUCE-5352. Optimize node local splits generated by
CombineFileInputFormat. (sseth)
MAPREDUCE-4671. AM does not tell the RM about container requests which are
no longer needed. (Bikas Saha via sseth)
MAPREDUCE-4994. -jt generic command line option does not work. (sandyr via tucu)
MAPREDUCE-5000. Fixes getCounters when speculating by fixing the selection
of the best attempt for a task. (Jason Lowe via sseth)
MAPREDUCE-4994. Addendum fixing testcases failures. (sandyr via tucu)
MAPREDUCE-4846. Some JobQueueInfo methods are public in MR1 but protected
in MR2. (Sandy Ryza via tomwhite)
MAPREDUCE-5013. mapred.JobStatus compatibility: MR2 missing constructors
from MR1. (Sandy Ryza via tomwhite)
MAPREDUCE-4951. Container preemption interpreted as task failure.
(Sandy Ryza via tomwhite)
MAPREDUCE-5008. Merger progress miscounts with respect to EOF_MARKER.
(Sandy Ryza via tomwhite)
MAPREDUCE-4693. History server should include counters for failed tasks.
(Xuan Gong via sseth)
MAPREDUCE-4896. mapred queue -info spits out ugly exception when queue does
not exist. (sandyr via tucu)
MAPREDUCE-3685. Fix bugs in MergeManager to ensure compression codec is
appropriately used and that on-disk segments are correctly sorted on
file-size. (Anty Rao and Ravi Prakash via acmurthy)
MAPREDUCE-4571. TestHsWebServicesJobs fails on jdk7. (tgraves via tucu)
MAPREDUCE-4716. TestHsWebServicesJobsQuery.testJobsQueryStateInvalid
fails with jdk7. (tgraves via tucu)
MAPREDUCE-5075. DistCp leaks input file handles since ThrottledInputStream
does not close the wrapped InputStream. (Chris Nauroth via szetszwo)
MAPREDUCE-3872. Fix an event handling races in ContainerLauncherImpl.
(Robert Kanter via sseth)
MAPREDUCE-5062. Fix MR AM to read max-retries from the RM. (Zhijie Shen via
MAPREDUCE-3829. [Gridmix] Gridmix should give better error message when
input data directory already exists and -generate opton is
given.(ravigummadi via tgraves)
MAPREDUCE-2722. [Gridmix] Gridmix simulated job's map's hdfsBytesRead
counter is wrong when compressed input is used.(ravigummadi via tgraves)
MAPREDUCE-3953. [Gridmix] Gridmix throws NPE and does not simulate a
job if the trace contains null taskStatus for a task. (ravigummadi via
MAPREDUCE-4087. [Gridmix] GenerateDistCacheData job of Gridmix can
become slow in some cases (ravigummadi via tgraves).
MAPREDUCE-5077. Remove mapreduce.util.ResourceCalculatorPlugin and related
code. (Karthik Kambatla via sseth)
MAPREDUCE-4083. [Gridmix] NPE in cpu emulation. (amarrk via tgraves)
MAPREDUCE-4100. [Gridmix] Bug fixed in compression emulation feature for
map only jobs. (amarrk via tgraves)
MAPREDUCE-4356. [Rumen] Provide access to the method
ParsedTask.obtainTaskAttempts(). (ravigummadi via tgraves)
MAPREDUCE-4149. [Rumen] Rumen fails to parse certain counter
strings. (ravigummadi via tgraves)
MAPREDUCE-3757. [Rumen] Fixed Rumen Folder to adjust shuffleFinished and
sortFinished times when needed. (Ravi Gummadi via tgraves)
MAPREDUCE-5138. Fix LocalDistributedCacheManager after YARN-112. (Omkar Vinit
Joshi via vinodkv)
MAPREDUCE-5086. MR app master deletes staging dir when sent a reboot
command from the RM. (Jian He via jlowe)
MAPREDUCE-5113. Streaming input/output types are ignored with java
mapper/reducer. (sandyr via tucu)
MAPREDUCE-5098. Fix findbugs warnings in gridmix. (kkambatl via tucu)
MAPREDUCE-5137. AM web UI: clicking on Map Task results in 500 error
(Thomas Graves via jlowe)
MAPREDUCE-5136. TestJobImpl->testJobNoTasks fails with IBM JAVA (Amir
Sanjar via jlowe)
MAPREDUCE-5139. Update MR AM to use the modified startContainer API after
YARN-486. (Xuan Gong via vinodkv)
MAPREDUCE-5151. Update MR AM to use standard exit codes from the API after
YARN-444. (Sandy Ryza via vinodkv)
MAPREDUCE-5140. MR part of YARN-514 (Zhijie Shen via bikas)
MAPREDUCE-5128. mapred-default.xml is missing a bunch of history server
configs. (sandyr via tucu)
MAPREDUCE-4898. FileOutputFormat.checkOutputSpecs and
FileOutputFormat.setOutputPath incompatible with MR1. (rkanter via tucu)
MAPREDUCE-5078. TestMRAppMaster fails on Windows due to mismatched path
separators. (Chris Nauroth via sseth)
MAPREDUCE-4932. mapreduce.job#getTaskCompletionEvents incompatible with
Hadoop 1. (rkanter via tucu)
MAPREDUCE-5163. Update MR App to not use API utility methods for collections
after YARN-441. (Xuan Gong via vinodkv)
MAPREDUCE-5066. Added a timeout for the job.end.notification.url. (Ivan
Mitic via acmurthy)
MAPREDUCE-5146. application classloader may be used too early to load
classes. (Sangjin Lee via tomwhite)
MAPREDUCE-4737. Ensure that mapreduce APIs are semantically consistent
with mapred API w.r.t Mapper.cleanup and Reducer.cleanup; in the sense that
cleanup is now called even if there is an error. The old mapred API
already ensures that Mapper.close and Reducer.close are invoked during
error handling. Note that it is an incompatible change, however end-users
can override and to get the old (inconsistent)
behaviour. (acmurthy)
MAPREDUCE-5166. Fix ConcurrentModificationException due to insufficient
synchronization on updates to task Counters. (Sandy Ryza via acmurthy)
MAPREDUCE-5181. RMCommunicator should not use AMToken from the env.
(Vinod Kumar Vavilapalli via sseth)
MAPREDUCE-5178. Update MR App to set progress in ApplicationReport after
YARN-577. (Hitesh Shah via vinodkv)
MAPREDUCE-5167. Update MR App after YARN-562 to use the new builder API
for the container. (Jian He via vinodkv)
MAPREDUCE-5179. Fix unit test in TestHSWebServices which fails when
versionInfo has parantheses like when running on a git checkout. (Hitesh Shah
via vinodkv)
MAPREDUCE-5193. A few MR tests use block sizes which are smaller than the
default minimum block size. (Andrew Wang via atm)
MAPREDUCE-5205. Fixed MR App to load tokens correctly. (vinodkv)
MAPREDUCE-5204. Handling YarnRemoteException separately from IOException in
MR app after YARN-629. (Xuan Gong via vinodkv)
MAPREDUCE-5209. Fix units in a ShuffleScheduler log message.
(Tsuyoshi OZAWA via cdouglas)
MAPREDUCE-5212. Handling YarnRemoteException separately from IOException in
MR App's use of ClientRMProtocol after YARN-631. (Xuan Gong via vinodkv)
MAPREDUCE-5226. Handling YarnRemoteException separately from IOException in
MR App's use of AMRMProtocol after YARN-630. (Xuan Gong via vinodkv)
MAPREDUCE-4942. mapreduce.Job has a bunch of methods that throw
InterruptedException so its incompatible with MR1. (rkanter via tucu)
MAPREDUCE-5239. Updated MR App to reflect YarnRemoteException changes after
YARN-634. (Siddharth Seth via vinodkv)
MAPREDUCE-5208. Modified ShuffleHandler to use SecureIOUtils for reading
local files. (Omkar Vinit Joshi via vinodkv)
MAPREDUCE-5220. Setter methods in TaskCompletionEvent are public in MR1 and
protected in MR2. (sandyr via tucu)
MAPREDUCE-5240. Fix a bug in MRAppMaster because of which OutputCommitter
could not access credentials set by the user. (vinodkv)
MAPREDUCE-5244. Two functions changed their visibility in JobStatus.
(zjshen via tucu)
MAPREDUCE-4927. Historyserver 500 error due to NPE when accessing specific
counters page for failed job. (Ashwin Shankar via jlowe)
MAPREDUCE-5257. Fix issues in TestContainerLauncherImpl after YARN-617.
(Omkar Vinit Joshi via vinodkv)
MAPREDUCE-5282. Updating MR App to use immutable ApplicationID after
YARN-716. (Siddharth Seth via vinodkv)
MAPREDUCE-5286. Change MapReduce to use ContainerTokenIdentifier instead
of the entire Container in the startContainer call - YARN-684.
(Vinod Kumar Vavilapalli via sseth)
MAPREDUCE-5299. Fix backward incompatibility for TaskCompletionEvent by
adding back setTaskID. (Zhijie Shen via acmurthy)
MAPREDUCE-5296. Fix backward incompatibility for JobControl by adding the
omitted addJob. (Zhijie Shen via acmurthy)
MAPREDUCE-5245. Added back constants to JobConf to fix incompatibilities.
(Zhijie Shen via acmurthy)
MAPREDUCE-5297. Updated MR App since BuilderUtils is no longer public
after YARN-748. (Jian He via vinodkv)
MAPREDUCE-5301. Updated MR code to work with YARN-635 changes of renaming
YarnRemoteException to YarnException. (Siddharth Seth via vinodkv)
MAPREDUCE-5308. Shuffling to memory can get out-of-sync when fetching
multiple compressed map outputs (Nathan Roberts via jlowe)
MAPREDUCE-5315. DistCp reports success even on failure. (mithun and jlowe
via daryn)
MAPREDUCE-5259. TestTaskLog fails on Windows because of path separators
missmatch. (Ivan Mitic via cnauroth)
MAPREDUCE-4019. -list-attempt-ids is not working (Ashwin Shankar,
Devaraj K, and B Anil Kumar via jlowe)
MAPREDUCE-5334. Fix failing unit tests - TestContainerLauncher,
TestContainerLauncherImpl. (Vinod Kumar Vavilapalli via sseth)
MAPREDUCE-5325. MR changes related to YARN-727. ClientRMProtocol.getAllApplications
should accept ApplicationType as a parameter. (Xuan Gong via hitesh)
MAPREDUCE-5291. Change MR App to use updated property names in (Zhijie Shen via sseth)
MAPREDUCE-5303. Changed MR app after moving ProtoBase to package impl.pb via
YARN-724. (Jian He via vinodkv)
MAPREDUCE-5312. TestRMNMInfo is failing. (sandyr via tucu)
MAPREDUCE-5304. mapreduce.Job killTask/failTask/getTaskCompletionEvents
methods have incompatible signature changes. (kkambatl via tucu)
MAPREDUCE-5298. Moved MapReduce services to YARN-530 stricter lifecycle.
(Steve Loughran via vinodkv)
MAPREDUCE-5319. Set in job.xml. (Xuan Gong via acmurthy)
MAPREDUCE-5310. MRAM should not normalize allocation request capabilities.
MAPREDUCE-5213. Re-assess TokenCache methods marked @Private.
(kkambatl via tucu)
MAPREDUCE-5412. Update MR app to use multiple containers API of
ContainerManager after YARN-926. (Jian He via vinodkv)
MAPREDUCE-5421. Fixed TestNonExistentJob failure after YARN-873. (Junping Du
via vinodkv)
MAPREDUCE-5419. TestSlive is getting FileNotFound Exception (Robert Parker
via jlowe)
MAPREDUCE-5399. Unnecessary Configuration instantiation in IFileInputStream
slows down merge. (Stanislav Barton via Sandy Ryza)
MAPREDUCE-4739. Some MapReduce tests fail to find winutils.
(Chris Nauroth via suresh)
MAPREDUCE-4780. MapReduce distribution build fails on Windows.
(Chris Nauroth via suresh)
MAPREDUCE-4790. MapReduce build script would be more readable using abspath.
(Chris Nauroth via suresh)
MAPREDUCE-4869. Fix TestMapReduceChildJVM. (Chris Nauroth via acmurthy)
MAPREDUCE-4870. Fix TestMRJobsWithHistoryService. (Chris Nauroth via acmurthy)
MAPREDUCE-4983. Fixed various platform specific assumptions in various tests,
so that they can pass on Windows too. (Chris Nauroth via vinodkv)
HADOOP-9372. Fix bad timeout annotations on tests.
(Arpit Agarwal via suresh)
MAPREDUCE-4885. Streaming tests have multiple failures on Windows. (Chris
Nauroth via bikas)
MAPREDUCE-5177. Use common utils FileUtil#setReadable/Writable/Executable &
FileUtil#canRead/Write/Execute. (Ivan Mitic via suresh)
MAPREDUCE-5349. TestClusterMapReduceTestCase and TestJobName fail on Windows
in branch-2. (Chuan Liu via cnauroth)
MAPREDUCE-5355. MiniMRYarnCluster with localFs does not work on Windows.
(Chuan Liu via cnauroth)
MAPREDUCE-5359. JobHistory should not use File.separator to match timestamp
in path. (Chuan Liu via cnauroth)
MAPREDUCE-5357. Job staging directory owner checking could fail on Windows.
(Chuan Liu via cnauroth)
MAPREDUCE-5360. TestMRJobClient fails on Windows due to path format.
(Chuan Liu via cnauroth)
MAPREDUCE-5366. TestMRAsyncDiskService fails on Windows. (Chuan Liu via
MAPREDUCE-5187. Create mapreduce command scripts on Windows. (Chuan Liu via
MAPREDUCE-4374. Fix child task environment variable config and add support
for Windows. (Chuan Liu via cnauroth)
Release 2.0.6-alpha - 08/22/2013
Release 2.0.5-alpha - 06/06/2013
MAPREDUCE-5240 inside of FileOutputCommitter the initialized Credentials cache
appears to be empty. (vinodkv)
Release 2.0.4-alpha - 2013-04-25
MAPREDUCE-5006. Fix failing streaming tests due to MAPREDUCE-4994.
(Sandy Ryza via tomwhite)
MAPREDUCE-5088. MR Client gets an renewer token exception while Oozie is
submitting a job (Daryn Sharp via cos)
MAPREDUCE-5117. Changed MRClientProtocolPBClientImpl to be closeable and thus
fix failures in renewal of HistoryServer's delegations tokens. (Siddharth
Seth via vinodkv)
MAPREDUCE-5083. MiniMRCluster should use a random component when creating an
actual cluster (Siddharth Seth via hitesh)
MAPREDUCE-5094. Disabled memory monitoring by default in MiniMRYarnCluster
to avoid some downstream tests failing. (Siddharth Seth via vinodkv)
Release 2.0.3-alpha - 2013-02-06
MAPREDUCE-4123. Remove the 'mapred groups' command, which is no longer
supported. (Devaraj K via sseth)
MAPREDUCE-4938. Use token request messages defined in hadoop common.
MAPREDUCE-4520. Added support for MapReduce applications to request for
CPU cores along-with memory post YARN-2. (acmurthy)
MAPREDUCE-4810. Added new admin command options for MR AM. (Jerry Chen via
MAPREDUCE-4049. Experimental api to allow for alternate shuffle plugins.
(Avner BenHanoch via acmurthy)
MAPREDUCE-4807. Allow MapOutputBuffer to be pluggable. (masokan via tucu)
MAPREDUCE-4808. Refactor MapOutput and MergeManager to facilitate reuse
by Shuffle implementations. (masokan via tucu)
MAPREDUCE-3678. The Map tasks logs should have the value of input
split it processed. (harsh)
MAPREDUCE-4616. Improve javadoc for MultipleOutputs. (Tony Burton via
HADOOP-8911. CRLF characters in source and text files.
(Raja Aluri via suresh)
MAPREDUCE-4723. Fix warnings found by findbugs 2. (Sandy Ryza via eli)
MAPREDUCE-4703. Add the ability to start the MiniMRClientCluster using
the configurations used before it is being stopped. (ahmed.radwan via tucu)
MAPREDUCE-4845. ClusterStatus.getMaxMemory() and getUsedMemory() exist in
MR1 but not MR2. (Sandy Ryza via tomwhite)
MAPREDUCE-4899. Implemented a MR specific plugin for tracking finished
applications that YARN's ResourceManager doesn't keep track off anymore
(Derek Dagit via vinodkv)
MAPREDUCE-4920. Use security token protobuf definition from hadoop common.
(Suresh Srinivas via vinodkv)
MAPREDUCE-4907. TrackerDistributedCacheManager issues too many getFileStatus
calls. (sandyr via tucu)
MAPREDUCE-4949. Enable multiple pi jobs to run in parallel. (sandyr via tucu)
MAPREDUCE-4809. Change visibility of classes for pluggable sort changes.
(masokan via tucu)
MAPREDUCE-4838. Add additional fields like Locality, Avataar to the
JobHistory logs. (Zhijie Shen via sseth)
MAPREDUCE-4971. Minor extensibility enhancements to Counters &
FileOutputFormat. (Arun C Murthy via sseth)
MAPREDUCE-4977. Documentation for pluggable shuffle and pluggable sort.
MAPREDUCE-4893. Fixed MR ApplicationMaster to do optimal assignment of
containers to get maximum locality. (Bikas Saha via vinodkv)
MAPREDUCE-4272. SortedRanges.Range#compareTo is not spec compliant.
(Yu Gao via llu)
MAPREDUCE-4607. Race condition in ReduceTask completion can result in Task
being incorrectly failed. (Bikas Saha via tomwhite)
MAPREDUCE-4646. Fixed MR framework to send diagnostic information correctly
to clients in case of failed jobs also. (Jason Lowe via vinodkv)
MAPREDUCE-4674. Hadoop examples secondarysort has a typo
"secondarysrot" in the usage. (Robert Justice via eli)
MAPREDUCE-4681. Fix unit tests broken by HDFS-3910. (acmurthy)
MAPREDUCE-4712. doesn't accept --config
(Vinod Kumar Vavilapalli via tgraves)
MAPREDUCE-4654. TestDistCp is ignored. (Sandy Ryza via tomwhite)
MAPREDUCE-4736. Remove obsolete option [-rootDir] from TestDFSIO.
(Brandon Li via suresh)
MAPREDUCE-4637. Handle TaskAttempt diagnostic updates while in the NEW and
UNASSIGNED states. (Mayank Bansal via sseth)
MAPREDUCE-1806. CombineFileInputFormat does not work with paths not on default FS. (Gera Shegalov via tucu)
MAPREDUCE-4777. In TestIFile, testIFileReaderWithCodec relies on
testIFileWriterWithCodec. (Sandy Ryza via tomwhite)
MAPREDUCE-4800. Cleanup o.a.h.mapred.MapTaskStatus - remove unused
code. (kkambatl via tucu)
MAPREDUCE-4861. Cleanup: Remove unused
(kkambatl via tucu)
MAPREDUCE-4856. TestJobOutputCommitter uses same directory as
TestJobCleanup. (Sandy Ryza via tomwhite)
MAPREDUCE-4278. Cannot run two local jobs in parallel from the same
gateway. (Sandy Ryza via tomwhite)
MAPREDUCE-1700. User supplied dependencies may conflict with MapReduce
system JARs. (tomwhite)
MAPREDUCE-4936. JobImpl uber checks for cpu are wrong (Arun C Murthy via
MAPREDUCE-4924. flakey test: org.apache.hadoop.mapred.TestClusterMRNotification.testMR.
(rkanter via tucu)
MAPREDUCE-4923. Add toString method to TaggedInputSplit. (sandyr via tucu)
MAPREDUCE-4948. Fix a failing unit test TestYARNRunner.testHistoryServerToken.
(Junping Du via sseth)
MAPREDUCE-4803. Remove duplicate copy of TestIndexCache. (Mariappan Asokan
via sseth)
MAPREDUCE-2264. Job status exceeds 100% in some cases.
(devaraj.k and sandyr via tucu)
MAPREDUCE-4969. TestKeyValueTextInputFormat test fails with Open JDK 7.
(Arpit Agarwal via suresh)
MAPREDUCE-4884. Streaming tests fail to start MiniMRCluster due to missing
queue configuration. (Chris Nauroth via suresh)
MAPREDUCE-4953. HadoopPipes misuses fprintf. (Andy Isaacson via atm)
Release 2.0.2-alpha - 2012-09-07
MAPREDUCE-987. Exposing MiniDFS and MiniMR clusters as a single process
command-line. (ahmed via tucu)
MAPREDUCE-4417. add support for encrypted shuffle (tucu)
MAPREDUCE-4355. Add RunningJob.getJobStatus() (kkambatl via tucu)
MAPREDUCE-3451. Port Fair Scheduler to MR2 (pwendell via tucu)
MAPREDUCE-4438. Add a simple, generic client to run 'easy' AMs in YARN.
(Bikas Saha via acmurthy)
MAPREDUCE-4157. ResourceManager should not kill apps that are well behaved
(Jason Lowe via bobby)
MAPREDUCE-4511. Add IFile readahead (ahmed via tucu)
MAPREDUCE-4408. allow jobs to set a JAR that is in the distributed cached
(rkanter via tucu)
MAPREDUCE-4440. Changed SchedulerApp and SchedulerNode to be a minimal
interface to allow schedulers to maintain their own. (acmurthy)
MAPREDUCE-4146. Support limits on task status string length and number of
block locations in branch-2. (Ahmed Radwan via tomwhite)
MAPREDUCE-3871. Allow symlinking in LocalJobRunner DistributedCache.
MAPREDUCE-3921. MR AM should act on node health status changes.
(Bikas Saha via sseth)
MAPREDUCE-2220. Fix new API FileOutputFormat-related typos in
mapred-default.xml (Rui Kubo via harsh)
MAPREDUCE-3907. Document entries mapred-default.xml for the
jobhistory server. (Eugene Koontz via harsh)
MAPREDUCE-3906. Fix inconsistency in documentation regarding
mapreduce.jobhistory.principal. (Eugene Koontz via harsh)
MAPREDUCE-4432. Confusing warning message when GenericOptionsParser
is not used. (Gabriel Reid via harsh)
MAPREDUCE-4427. Added an 'unmanaged' mode for AMs so as to ease
development of new applications. (Bikas Saha via acmurthy)
MAPREDUCE-4447. Remove aop from cruft from the ant build. (eli)
MAPREDUCE-3289. Make use of fadvise in the NM's shuffle handler.
(Todd Lipcon and Siddharth Seth via sseth)
MAPREDUCE-4580. Change MapReduce to use the yarn-client module.
(Vinod Kumar Vavilapalli via sseth)
MAPREDUCE-4579. Split TestTaskAttempt into two so as to pass tests on
jdk7. (Thomas Graves via vinodkv)
MAPREDUCE-4638. MR AM supplies MapReduce jars in classpath rather than
rely on YARN. (acmurthy)
MAPREDUCE-4422. YARN_APPLICATION_CLASSPATH needs a documented default value in
YarnConfiguration. (ahmed via tucu)
MAPREDUCE-4406. Users should be able to specify the MiniCluster ResourceManager
and JobHistoryServer ports. (ahmed via tucu)
MAPREDUCE-4407. Add hadoop-yarn-server-tests-<version>-tests.jar to hadoop dist
package. (ahmed via tucu)
MAPREDUCE-4465. Update description of yarn.nodemanager.address property.
(bowang via tucu)
MAPREDUCE-4342. Distributed Cache gives inconsistent result if cache files
get deleted from tasktracker. (mayank_bansal via tucu)
MAPREDUCE-4498. Remove hsqldb jar from Hadoop runtime classpath. (rkanter via tucu)
MAPREDUCE-4494. TestFifoScheduler failing with Metrics source QueueMetrics,q0=default
already exists!. (ahmed.radwan via tucu)
MAPREDUCE-4484. Incorrect IS_MINI_YARN_CLUSTER property name in YarnConfiguration.
(ahmed.radwan via tucu)
MAPREDUCE-4562. Support for "FileSystemCounter" legacy counter group name
for compatibility reasons is creating incorrect counter name.
(Jarek Jarcec Cecho via tomwhite)
MAPREDUCE-4068. Jars in lib subdirectory of the submittable JAR are not added to the
classpath (rkanter via tucu)
MAPREDUCE-4577. HDFS-3672 broke
TestCombineFileInputFormat.testMissingBlocks() test. (atm)
MAPREDUCE-4470. Fix TestCombineFileInputFormat.testForEmptyFile (ikatsov via tucu)
MAPREDUCE-4608. hadoop-mapreduce-client is missing some dependencies.
(tucu via tomwhite)
MAPREDUCE-4610. Support deprecated mapreduce.job.counters.limit property in
MR2. (tomwhite)
MAPREDUCE-4629. Remove JobHistory.DEBUG_MODE (Karthik Kambatla via bobby)
MAPREDUCE-4642. MiniMRClientClusterFactory should not use job.setJar() (rkanter via tucu)
MAPREDUCE-4148. MapReduce should not have a compile-time dependency on
HDFS. (tomwhite)
MAPREDUCE-4250. missing variable exports, causes Yarn
jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu)
MAPREDUCE-4002. MultiFileWordCount job fails if the input path is not
from default file system. (Bhallamudi Venkata Siva Kamesh via todd)
MAPREDUCE-4274 MapOutputBuffer should use native byte order for kvmeta.
(todd via bobby)
MAPREDUCE-4262. NM gives wrong log message saying "Connected to
ResourceManager" before trying to connect. (Devaraj K via tgraves)
MAPREDUCE-4276. Allow setting yarn.nodemanager.delete.debug-delay-sec
property to "-1" for easier container debugging. (ahmed via tucu)
MAPREDUCE-4224. TestFifoScheduler throws
org.apache.hadoop.metrics2.MetricsException (Devaraj K via tgraves)
MAPREDUCE-3493. Add the default mapreduce.shuffle.port property
to mapred-default.xml (Madhukara Phatak via harsh)
MAPREDUCE-4307. TeraInputFormat calls FileSystem.getDefaultBlockSize()
without a Path - Failure when using ViewFileSystem. (Ahmed Radwan via eli)
MAPREDUCE-4313. TestTokenCache doesn't compile due
TokenCache.getDelegationToken compilation error (bobby)
MAPREDUCE-3873. Fixed NodeManagers' decommissioning at RM to accept IP
addresses also. (xieguiming via vinodkv)
MAPREDUCE-4306. Fix distributed shell to work with users other than the one
running the daemons. (Ahmed Radwan via sseth)
MAPREDUCE-4031. Prevent a Node Manager hang during shutdown.
(Devaraj K via sseth)
MAPREDUCE-4336. Distributed Shell fails when used with the CapacityScheduler
(ahmed via tucu)
MAPREDUCE-4290. Fix Yarn Applicaiton Status to MR JobState conversion.
(Devaraj K via sseth)
MAPREDUCE-2289. Permissions race can make getStagingDir fail on local filesystem
(ahmed via tucu)
MAPREDUCE-4372. Deadlock in Resource Manager (Devaraj K via bobby)
MAPREDUCE-4376. TestClusterMRNotification times out (Kihwal Lee via bobby)
MAPREDUCE-4383. needs to include unistd.h.
(Andy Isaacson via eli)
MAPREDUCE-2739. Update installation docs (remove YarnClientFactory) (bowang via tucu)
MAPREDUCE-3993. Graceful handling of codec errors during decompression
(kkambatl via tucu)
MAPREDUCE-4416. Some tests fail if Clover is enabled (Kihwal Lee via bobby)
MAPREDUCE-4441. Fix build issue caused by MR-3451 (kkambatl via tucu)
HADOOP-8499. Lower to 500 for the tests.
(Colin Patrick McCabe via eli)
MAPREDUCE-4395. Possible NPE at ClientDistributedCacheManager
#determineTimestamps (Bhallamudi via bobby)
MAPREDUCE-4380. Empty Userlogs directory is getting created under logs
directory (Devaraj K via bobby)
MAPREDUCE-4649. Ensure MapReduce JobHistory Daemon doens't assume
HADOOP_YARN_HOME and HADOOP_MAPRED_HOME are the same. (vinodkv via
Release 2.0.0-alpha - 05-23-2012
MAPREDUCE-3545. Remove Avro RPC. (suresh)
MAPREDUCE-2887. Due to HADOOP-7524, change RPC to allow multiple protocols
including multuple versions of the same protocol (Sanjay Radia)
MAPREDUCE-2934. MR portion of HADOOP-7607 - Simplify the RPC proxy cleanup
process (atm)
HADOOP-7862. MR changes to work with HADOOP 7862: Move the support for
multiple protocols to lower layer so that Writable, PB and Avro can all
use it (Sanjay Radia)
MAPREDUCE-3909 Javadoc the Service interfaces (stevel)
MAPREDUCE-3885. Avoid an unnecessary copy for all requests/responses in
MRs ProtoOverHadoopRpcEngine. (Devaraj Das via sseth)
MAPREDUCE-3991. Streaming FAQ has some wrong instructions about input files
splitting. (harsh)
MAPREDUCE-3773. Add queue metrics with buckets for job run times. (omalley
via acmurthy)
MAPREDUCE-3970. Add ServiceOperations class to aid working with Services
MAPREDUCE-3353. Add a channel between RM and AM to get information on
nodes. (Bikas Saha via acmurthy)
MAPREDUCE-3955. Change MR to use ProtobufRpcEngine from hadoop-common
instead of ProtoOverHadoopRpcEngine. (Jitendra Nath Pandey via sseth)
MAPREDUCE-4103. Fix HA docs for changes to shell command fencer args (todd)
MAPREDUCE-4093. Improve RM WebApp start up when proxy address is not set
(Devaraj K vai bobby)
MAPREDUCE-4138. Reduce memory usage of counters due to non-static nested
classes. (tomwhite)
MAPREDUCE-3883. Document yarn.nodemanager.delete.debug-delay-sec
configuration property (Eugene Koontz via tgraves)
MAPREDUCE-4219. make default container-executor.conf.dir be a path
relative to the container-executor binary. (rvs via tucu)
MAPREDUCE-4205. retrofit all JVM shutdown hooks to use ShutdownHookManager
MAPREDUCE-3740. Fixed broken mapreduce compilation after the patch for
HADOOP-7965. (Devaraj K via vinodkv)
MAPREDUCE-3818. Fixed broken compilation in TestSubmitJob after the patch
for HDFS-2895. (Suresh Srinivas via vinodkv)
MAPREDUCE-2942. TestNMAuditLogger.testNMAuditLoggerWithIP failing (Thomas
Graves via mahadev)
MAPREDUCE-3933. Failures because MALLOC_ARENA_MAX is not set (ahmed via tucu)
MAPREDUCE-3728. ShuffleHandler can't access results when configured in a
secure mode (ahmed via tucu)
MAPREDUCE-3952. In MR2, when Total input paths to process == 1,
CombinefileInputFormat.getSplits() returns 0 split. (zhenxiao via tucu)
MAPREDUCE-3578. Starting nodemanager as root gives "Unknown -jvm option"
MAPREDUCE-3348. Fixed a bug in MR client to redirect to JobHistoryServer
correctly when RM forgets the app. (Devaraj K via vinodkv)
MAPREDUCE-3974. TestSubmitJob in MR1 tests doesn't compile after HDFS-162
merge. (atm)
MAPREDUCE-4007. JobClient getJob(JobID) should return NULL if the job
does not exist (for backwards compatibility) (tucu)
MAPREDUCE-3431 NPE in Resource Manager shutdown. (stevel)
MAPREDUCE-4010. TestWritableJobConf fails on trunk (tucu via bobby)
MAPREDUCE-3992. Reduce fetcher doesn't verify HTTP status code of response
MAPREDUCE-4066. Use default value when fetching MR_AM_STAGING_DIR
(xieguiming via harsh)
MAPREDUCE-3377. Added a unit test to ensure OutputCommitter.checkOutputSpecs
is called prior to copying job.xml. (Jane Chen via acmurthy)
MAPREDUCE-4081. does not compile (Jason Lowe via
MAPREDUCE-4082. hadoop-mapreduce-client-app's mrapp-generated-classpath
file should not be in the module JAR (tucu)
MAPREDUCE-3916. various issues with running yarn proxyserver (devaraj via tucu)
MAPREDUCE-4091. tools testcases failing because of MAPREDUCE-4082 (tucu)
MAPREDUCE-4095. TestJobInProgress#testLocality uses a bogus topology.
(Colin Patrick McCabe via eli)
MAPREDUCE-4098. TestMRApps testSetClasspath fails (tucu)
MAPREDUCE-4097. tools testcases fail because missing mrapp-generated-classpath
file in classpath (rvs via tucu)
MAPREDUCE-4113. Fix tests org.apache.hadoop.mapred.TestClusterMRNotification
(Devaraj K via bobby)
MAPREDUCE-4112. Fix tests org.apache.hadoop.mapred.TestClusterMapReduceTestCase
(Devaraj K via bobby)
MAPREDUCE-4111. Fix tests in org.apache.hadoop.mapred.TestJobName (Devaraj
K via bobby)
MAPREDUCE-4110. Fix tests in org.apache.hadoop.mapred.TestMiniMRClasspath &
org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers (Devaraj K via
MAPREDUCE-4105. Yarn RackResolver ignores rack configurations.
(Ahmed Radwan via tomwhite)
MAPREDUCE-3869. Fix classpath for DistributedShell application. (Devaraj K
via sseth)
MAPREDUCE-4057. Update RAID for the HA and fsdataset changes. (Devaraj K
via szetszwo)
MAPREDUCE-4076. Stream job fails with ZipException when use yarn jar
command (Devaraj K via bobby)
MAPREDUCE-4108. Fix tests in org.apache.hadoop.util.TestRunJar
(Devaraj K via tgraves)
MAPREDUCE-4107. Fix tests in org.apache.hadoop.ipc.TestSocketFactory
(Devaraj K via tgraves)
MAPREDUCE-4147. YARN should not have a compile-time dependency on HDFS.
MAPREDUCE-4008. ResourceManager throws MetricsException on start up
saying QueueMetrics MBean already exists (Devaraj K via tgraves)
MAPREDUCE-3867. MiniMRYarn/MiniYarn uses fixed ports (tucu)
MAPREDUCE-4141. clover integration broken, also mapreduce poms are
pulling in clover as a dependency. (phunt via tucu)
MAPREDUCE-4193. broken doc link for yarn-default.xml in site.xml.
(phunt via tomwhite)
MAPREDUCE-4202. TestYarnClientProtocolProvider is broken (Daryn Sharp via
MAPREDUCE-3173. MRV2 UI doesn't work properly without internet (Devaraj K
via bobby)
MAPREDUCE-3958. RM: Remove RMNodeState and replace it with NodeState
(Bikas Saha via bobby)
MAPREDUCE-4231. Update RAID to use the new BlockCollection interface.
MAPREDUCE-4483. 2.0 build does not work (John George via bobby)
MAPREDUCE-4444. nodemanager fails to start when one of the local-dirs is
bad (Jason Lowe via bobby)
Release 0.23.10 - UNRELEASED
MAPREDUCE-5640. Rename TestLineRecordReader in jobclient module (Jason Lowe
via jeagles)
MAPREDUCE-3191. docs for map output compression incorrectly reference
SequenceFile (Chen He via jeagles)
MAPREDUCE-1981. Improve getSplits performance by using listLocatedStatus
(Hairong Kuang and Jason Lowe via jlowe)
MAPREDUCE-3193. FileInputFormat doesn't read files recursively in the
input path dir (Devaraj K via jlowe)
MAPREDUCE-5380. Invalid mapred command should return non-zero exit code
(Stephen Chu via jlowe)
MAPREDUCE-5317. Stale files left behind for failed jobs (Ravi Prakash via
MAPREDUCE-5251. Reducer should not implicate map attempt if it has
insufficient space to fetch map output (Ashwin Shankar via jlowe)
MAPREDUCE-5419. TestSlive is getting FileNotFound Exception (Robert Parker
via jlowe)
MAPREDUCE-5440. TestCopyCommitter Fails on JDK7 (Robert Parker via jlowe)
MAPREDUCE-5001. LocalJobRunner has race condition resulting in job
failures (Sandy Ryza via jlowe)
MAPREDUCE-5475. MRClientService does not verify ACLs properly (jlowe)
MAPREDUCE-5504. mapred queue -info inconsistent with types (Kousuke Saruta
via tgraves)
MAPREDUCE-5513. ConcurrentModificationException in JobControl (Robert
Parker via jlowe)
MAPREDUCE-5586. TestCopyMapper#testCopyFailOnBlockSizeDifference fails when
run from hadoop-tools/hadoop-distcp directory (jeagles)
MAPREDUCE-5587. TestTextOutputFormat fails on JDK7 (jeagles)
MAPREDUCE-5373. TestFetchFailure.testFetchFailureMultipleReduces could fail
intermittently (jeagles)
Release 0.23.9 - 2013-07-08
MAPREDUCE-5084. fix coverage and
org.apache.hadoop.mapreduce.v2.hs.webapp (Aleksey Gorshkov via jeagles)
MAPREDUCE-5268. Improve history server startup performance (Karthik
Kambatla via jlowe)
MAPREDUCE-5308. Shuffling to memory can get out-of-sync when fetching
multiple compressed map outputs (Nathan Roberts via jlowe)
MAPREDUCE-5315. DistCp reports success even on failure. (mithun and jlowe
via daryn)
MAPREDUCE-4019. -list-attempt-ids is not working (Ashwin Shankar,
Devaraj K, and B Anil Kumar via jlowe)
MAPREDUCE-5316. job -list-attempt-ids command does not handle illegal
task-state (Ashwin Shankar via jlowe)
Release 0.23.8 - 2013-06-05
MAPREDUCE-5065. DistCp should skip checksum comparisons if block-sizes
are different on source/target (Mithun Radhakrishnan via kihwal)
MAPREDUCE-5015. Coverage fix for
(Aleksey Gorshkov via tgraves)
MAPREDUCE-5147. Maven build should create
hadoop-mapreduce-client-app-VERSION.jar directly (Robert Parker via tgraves)
MAPREDUCE-4927. Historyserver 500 error due to NPE when accessing specific
counters page for failed job. (Ashwin Shankar via jlowe)
Release 0.23.7 - 2013-04-18
MAPREDUCE-4905. test org.apache.hadoop.mapred.pipes
(Aleksey Gorshkov via bobby)
MAPREDUCE-4989. JSONify DataTables input data for Attempts page (Ravi
Prakash via jlowe)
MAPREDUCE-5027. Shuffle does not limit number of outstanding connections
(Robert Parker via jeagles)
MAPREDUCE-4972. Coverage fixing for org.apache.hadoop.mapreduce.jobhistory
(Aleksey Gorshkov via bobby)
MAPREDUCE-4946. Fix a performance problem for large jobs by reducing the
number of map completion event type conversions. (Jason Lowe via sseth)
MAPREDUCE-4822. Unnecessary conversions in History Events. (Chu Tong via
MAPREDUCE-4458. Warn if java.library.path is used for AM or Task
(Robert Parker via jeagles)
MAPREDUCE-4992. AM hangs in RecoveryService when recovering tasks with
speculative attempts (Robert Parker via jlowe)
MAPREDUCE-5009. Killing the Task Attempt slated for commit does not clear
the value from the Task commitAttempt member (Robert Parker via jeagles)
MAPREDUCE-4871. AM uses mapreduce.jobtracker.split.metainfo.maxsize but
mapred-default has mapreduce.job.split.metainfo.maxsize (Jason Lowe via
MAPREDUCE-4794. DefaultSpeculator generates error messages on normal
shutdown (Jason Lowe via jeagles)
MAPREDUCE-5043. Fetch failure processing can cause AM event queue to
backup and eventually OOM (Jason Lowe via bobby)
MAPREDUCE-5023. History Server Web Services missing Job Counters (Ravi
Prakash via tgraves)
MAPREDUCE-5060. Fetch failures that time out only count against the first
map task (Robert Joseph Evans via jlowe)
MAPREDUCE-5042. Reducer unable to fetch for a map task that was recovered
(Jason Lowe via bobby)
MAPREDUCE-5053. java.lang.InternalError from decompression codec cause
reducer to fail (Robert Parker via jeagles)
MAPREDUCE-4991. coverage for gridmix (Aleksey Gorshkov via tgraves)
MAPREDUCE-5007. fix coverage org.apache.hadoop.mapreduce.v2.hs (Aleksey
Gorshkov via tgraves)
MAPREDUCE-5137. AM web UI: clicking on Map Task results in 500 error
(Thomas Graves via jlowe)
Release 0.23.6 - 2013-02-06
MAPREDUCE-4811. JobHistoryServer should show when it was started in WebUI
About page (Ravi Prakash via jlowe)
MAPREDUCE-4802. Takes a long time to load the task list on the AM for
large jobs (Ravi Prakash via bobby)
MAPREDUCE-4764. repair TestBinaryTokenFile (Ivan A. Veselovsky via bobby)
MAPREDUCE-4825. JobImpl.finished doesn't expect ERROR as a final job state
(jlowe via bobby)
MAPREDUCE-4817. Hardcoded task ping timeout kills tasks localizing large
amounts of data (tgraves)
MAPREDUCE-4836. Elapsed time for running tasks on AM web UI tasks page is 0
(Ravi Prakash via jeagles)
MAPREDUCE-4842. Shuffle race can hang reducer (Mariappan Asokan via jlowe)
MAPREDUCE-4833. Task can get stuck in FAIL_CONTAINER_CLEANUP (Robert
Parker via jlowe)
MAPREDUCE-4793. Problem with adding resources when using both -files and
-file to hadoop streaming (jlowe)
MAPREDUCE-4890. Invalid TaskImpl state transitions when task fails while
speculating (jlowe)
MAPREDUCE-4902. Fix typo "receievd" should be "received" in log output
(Albert Chu via jlowe)
MAPREDUCE-4813. AM timing out during job commit (jlowe via bobby)
MAPREDUCE-4279. getClusterStatus() fails with null pointer exception when
running jobs in local mode (Devaraj K via bobby)
MAPREDUCE-4832. MR AM can get in a split brain situation (jlowe)
MAPREDUCE-4894. Renewal / cancellation of JobHistory tokens (Siddharth
Seth via tgraves)
MAPREDUCE-4819. AM can rerun job after reporting final job status to the
client (bobby and Bikas Saha via bobby)
MAPREDUCE-4913. TestMRAppMaster#testMRAppMasterMissingStaging occasionally
exits (Jason Lowe via tgraves)
MAPREDUCE-4848. TaskAttemptContext cast error during AM recovery (Jerry
Chen via jlowe)
MAPREDUCE-4921. JobClient should acquire HS token with RM principal
(daryn via bobby)
MAPREDUCE-4934. Maven RAT plugin is not checking all source files (tgraves)
MAPREDUCE-4678. Running the Pentomino example with defaults throws
java.lang.NegativeArraySizeException (Chris McConnell via harsh)
MAPREDUCE-4925. The pentomino option parser may be buggy.
(Karthik Kambatla via harsh)
Release 0.23.5 - 2012-11-28
MAPREDUCE-4596. Split StateMachine state from states seen by MRClientProtocol
for Job, Task and TaskAttempt. (Siddarth Seth via vinodkv)
MAPREDUCE-4752. Reduce MR AM memory usage through String Interning (Robert
Evans via tgraves)
MAPREDUCE-4266. remove Ant remnants from MR (tgraves via bobby)
MAPREDUCE-4666. JVM metrics for history server (jlowe via jeagles)
MAPREDUCE-4720. Browser thinks History Server main page JS is taking too
long (Ravi Prakash via bobby)
MAPREDUCE-4554. Job Credentials are not transmitted if security is turned
off (Benoy Antony via bobby)
MAPREDUCE-4705. Fix a bug in job history lookup, which makes older jobs
inaccessible despite the presence of a valid history file. (Jason Lowe
via sseth)
MAPREDUCE-4521. mapreduce.user.classpath.first incompatibility with 0.20/1.x
(Ravi Prakash via bobby)
MAPREDUCE-4721. Task startup time in JHS is same as job startup time.
(Ravi Prakash via bobby)
MAPREDUCE-4479. Fix parameter order in assertEquals() in (Mariappan Asokan via bobby)
MAPREDUCE-4733. Reducer can fail to make progress during shuffle if too many
reducers complete consecutively. (Jason Lowe via vinodkv)
MAPREDUCE-4740. only .jars can be added to the Distributed Cache
classpath. (Robert Joseph Evans via jlowe)
MAPREDUCE-4229. Intern counter names in the JT (Miomir Boljanovic and bobby via daryn)
MAPREDUCE-4741. WARN and ERROR messages logged during normal AM shutdown.
(Vinod Kumar Vavilapalli via jlowe)
MAPREDUCE-4730. Fix Reducer's EventFetcher to scale the map-completion
requests slowly to avoid HADOOP-8942. (Jason Lowe via vinodkv)
MAPREDUCE-4748. Invalid event: T_ATTEMPT_SUCCEEDED at SUCCEEDED. (jlowe)
MAPREDUCE-4724. job history web ui applications page should be sorted to
display last app first (tgraves via bobby)
MAPREDUCE-4746. The MR Application Master does not have a config to set
environment variables (Rob Parker via bobby)
MAPREDUCE-4729. job history UI not showing all job attempts. (Vinod
Kumar Vavilapalli via jlowe)
MAPREDUCE-4763 repair test TestUmbilicalProtocolWithJobToken (Ivan A.
Veselovsky via bobby)
MAPREDUCE-4771. KeyFieldBasedPartitioner not partitioning properly when
configured (jlowe via bobby)
MAPREDUCE-4772. Fetch failures can take way too long for a map to be
restarted (bobby)
MAPREDUCE-4782. NLineInputFormat skips first line of last InputSplit
(Mark Fuhs via bobby)
MAPREDUCE-4774. JobImpl does not handle asynchronous task events in FAILED
state (jlowe via bobby)
MAPREDUCE-4751. AM stuck in KILL_WAIT for days (vinodkv via bobby)
MAPREDUCE-4787. TestJobMonitorAndPrint is broken (Rob Parker via bobby)
MAPREDUCE-4425. Speculation + Fetch failures can lead to a hung job (jlowe
via bobby)
MAPREDUCE-4786. Job End Notification retry interval is 5 milliseconds by
default (Ravi Prakash via bobby)
MAPREDUCE-4517. Too many INFO messages written out during AM to RM heartbeat
(Jason Lowe via tgraves)
MAPREDUCE-4797. LocalContainerAllocator can loop forever trying to contact
the RM (jlowe via bobby)
MAPREDUCE-4801. ShuffleHandler can generate large logs due to prematurely
closed channels (jlowe via bobby)
Release 0.23.4
MAPREDUCE-2786. Add compression option for TestDFSIO.
(Plamen Jeliazkov via shv)
MAPREDUCE-4645. Provide a random seed to Slive to make the sequence
of file names deterministic. (Ravi Prakash via shv)
MAPREDUCE-4651. Benchmarking random reads with DFSIO. (shv)
MAPREDUCE-4647. We should only unjar jobjar if there is a lib directory
in it. (Robert Evans via tgraves)
MAPREDUCE-4691. Historyserver can report "Unknown job" after RM says job
has completed (Robert Joseph Evans via jlowe)
MAPREDUCE-4689. JobClient.getMapTaskReports on failed job results in NPE
(jlowe via bobby)
Release 0.23.3
MAPREDUCE-4072. User set java.library.path seems to overwrite default
creating problems native lib loading (Anupam Seth via bobby)
MAPREDUCE-3812. Lower default allocation sizes, fix allocation
configurations and document them (Harsh J via bobby)
MAPREDUCE-4059. The history server should have a separate pluggable
storage/query interface. (Robert Evans via tgraves)
MAPREDUCE-3942. Randomize master key generation for
ApplicationTokenSecretManager and roll it every so often. (Vinod Kumar
Vavilapalli via sseth)
MAPREDUCE-4151. RM scheduler web page should filter apps to those that
are relevant to scheduling (Jason Lowe via tgraves)
MAPREDUCE-4134. Remove references of mapred.child.ulimit etc. since they
are not being used any more (Ravi Prakash via bobby)
MAPREDUCE-3972. Fix locking and exception issues in JobHistory server.
(Robert Joseph Evans via sseth)
MAPREDUCE-4161. create sockets consistently (Daryn Sharp via bobby)
MAPREDUCE-4079. Allow MR AppMaster to limit ephemeral port range.
(bobby via tgraves)
MAPREDUCE-4210. Expose listener address for WebApp (Daryn Sharp via bobby)
MAPREDUCE-4162. Correctly set token service (Daryn Sharp via bobby)
MAPREDUCE-4301. Dedupe some strings in MRAM for memory savings
(bobby via tgraves)
MAPREDUCE-4267. mavenize pipes (tgraves via bobby)
MAPREDUCE-4375. Show Configuration Tracability in MR UI (bobby
via tgraves)
MAPREDUCE-4569. Fixed TestHsWebServicesJobsQuery to pass on JDK7 by not
depending on test order. (Thomas Graves via vinodkv)
MAPREDUCE-3850. Avoid redundant calls for tokens in TokenCache (Daryn
Sharp via bobby)
MAPREDUCE-4092. commitJob Exception does not fail job (Jon Eagles via
MAPREDUCE-4089. Hung Tasks never time out. (Robert Evans via tgraves)
MAPREDUCE-4024. RM webservices can't query on finalStatus (Tom Graves
via bobby)
MAPREDUCE-4060. Multiple SLF4J binding warning (Jason Lowe via bobby)
MAPREDUCE-3983. TestTTResourceReporting can fail, and should just be
deleted (Ravi Prakash via bobby)
MAPREDUCE-4012 Hadoop Job setup error leaves no useful info to users
(when LinuxTaskController is used). (tgraves)
MAPREDUCE-4062. AM Launcher thread can hang forever (tgraves via bobby)
MAPREDUCE-3988. mapreduce.job.local.dir doesn't point to a single directory
on a node. (Eric Payne via bobby)
MAPREDUCE-3999. Tracking link gives an error if the AppMaster hasn't
started yet (Ravi Prakash via bobby)
MAPREDUCE-4020. Web services returns incorrect JSON for deep queue tree
(Anupam Seth via tgraves)
MAPREDUCE-3672. Killed maps shouldn't be counted towards
JobCounter.NUM_FAILED_MAPS. (Anupam Seth via tgraves)
MAPREDUCE-3682 Tracker URL says AM tasks run on localhost.
(Ravi Prakash via tgraves)
MAPREDUCE-3082. Archive command take wrong path for input file with current
directory (John George via bobby)
MAPREDUCE-3650. testGetTokensForHftpFS() fails (Ravi Prakash via bobby)
MAPREDUCE-3621. TestDBJob and TestDataDrivenDBInputFormat ant tests fail
(Ravi Prakash via tgraves)
MAPREDUCE-4073. CS assigns multiple off-switch containers when using
multi-level-queues (Siddharth Seth via bobby)
MAPREDUCE-4051. Remove the empty hadoop-mapreduce-project/assembly/all.xml
file (Ravi Prakash via bobby)
MAPREDUCE-4117. mapred job -status throws NullPointerException (Devaraj K
via bobby)
MAPREDUCE-4099. ApplicationMaster may fail to remove staging directory
(Jason Lowe via bobby)
MAPREDUCE-4017. Add jobname to jobsummary log (tgraves and Koji Noguchi
via bobby)
MAPREDUCE-4040. History links should use hostname rather than IP address.
(Bhallamudi Venkata Siva Kamesh via sseth)
MAPREDUCE-4099 amendment. ApplicationMaster will remove staging directory
after the history service is stopped. (Jason Lowe via sseth)
MAPREDUCE-3932. Fix the TaskAttempt state machine to handle
states. (Robert Joseph Evans via sseth)
MAPREDUCE-4140. mapreduce classes incorrectly importing
"*" classes. (Patrick Hunt via tomwhite)
MAPREDUCE-4050. For tasks without assigned containers, changes the node
text on the UI to N/A instead of a link to null. (Bhallamudi Venkata Siva
Kamesh via sseth)
MAPREDUCE-4128. AM Recovery expects all attempts of a completed task to
also be completed. (Bikas Saha via bobby)
MAPREDUCE-4144. Fix a NPE in the ResourceManager when handling node
updates. (Jason Lowe via sseth)
MAPREDUCE-4156. ant build fails compiling JobInProgress (tgraves)
MAPREDUCE-4160. some mrv1 ant tests fail with timeout - due to 4156
MAPREDUCE-4074. Client continuously retries to RM When RM goes down
before launching Application Master (xieguiming via tgraves)
MAPREDUCE-4159. Job is running in Uber mode after setting
"mapreduce.job.ubertask.maxreduces" to zero (Devaraj K via bobby)
MAPREDUCE-4165. Committing is misspelled as commiting in task logs
(John Eagles via bobby)
MAPREDUCE-4129. Lots of unneeded counters log messages (Ahmed Radwan via
MAPREDUCE-3947. not documented
(Devaraj K via bobby)
MAPREDUCE-4190. Improve web UI for task attempts userlog link (Tom Graves
via bobby)
MAPREDUCE-4133. MR over viewfs is broken (John George via bobby)
MAPREDUCE-4194. ConcurrentModificationError in DirectoryCollection
(Jonathan Eagles via bobby)
MAPREDUCE-3613. web service calls header contains 2 content types
MAPREDUCE-4169. Container Logs appear in unsorted order (Jonathan Eagles
via bobby)
MAPREDUCE-4189. TestContainerManagerSecurity is failing (Devaraj K via
MAPREDUCE-4209. junit dependency in hadoop-mapreduce-client is missing
scope test (Radim Kolar via bobby)
MAPREDUCE-4206. Sorting by Last Health-Update on the RM nodes page sorts
does not work correctly (Jonathon Eagles via tgraves)
MAPREDUCE-4212. TestJobClientGetJob sometimes fails
(Daryn Sharp via tgraves)
MAPREDUCE-4211. Error conditions (missing appid, appid not found) are
masked in the RM app page (Jonathan Eagles via bobby)
MAPREDUCE-4163. consistently set the bind address (Daryn Sharp via bobby)
MAPREDUCE-4048. NullPointerException exception while accessing the
Application Master UI (Devaraj K via bobby)
MAPREDUCE-4220. RM apps page starttime/endtime sorts are incorrect
(Jonathan Eagles via bobby)
MAPREDUCE-4226. ConcurrentModificationException in FileSystemCounterGroup.
MAPREDUCE-4215. RM app page shows 500 error on appid parse error
(Jonathon Eagles via tgraves)
MAPREDUCE-4237. TestNodeStatusUpdater can fail if localhost has a domain
associated with it (bobby)
MAPREDUCE-4233. NPE can happen in RMNMNodeInfo. (bobby)
MAPREDUCE-4238. mavenize data_join. (tgraves)
MAPREDUCE-4102. job counters not available in Jobhistory webui for
killed jobs (Bhallamudi Venkata Siva Kamesh via tgraves)
MAPREDUCE-3543. Mavenize Gridmix. (tgraves)
MAPREDUCE-4197. Include the hsqldb jar in the hadoop-mapreduce tar
file (Ravi Prakash via tgraves)
MAPREDUCE-4269. documentation: Gridmix has javadoc warnings in
StressJobFactory (Jonathon Eagles via tgraves).
MAPREDUCE-3870. Invalid App Metrics
(Bhallamudi Venkata Siva Kamesh via tgraves).
MAPREDUCE-4152. map task left hanging after AM dies trying to connect to RM
(Tom Graves via bobby)
MAPREDUCE-4297. Usersmap file in gridmix should not fail on empty lines
(Ravi Prakash via bobby)
MAPREDUCE-4302. NM goes down if error encountered during log aggregation
(Daryn Sharp via bobby)
MAPREDUCE-3350. Per-app RM page should have the list of application-attempts
like on the app JHS page (Jonathon Eagles via tgraves)
MAPREDUCE-3842. Stop webpages from automatic refreshing (tgraves)
MAPREDUCE-3927. Shuffle hang when set map.failures.percent
(Bhallamudi Venkata Siva Kamesh via tgraves)
MAPREDUCE-4311. Capacity scheduler.xml does not accept decimal values for
capacity and maximum-capacity settings (Karthik Kambatla via tgraves)
MAPREDUCE-4341. add types to capacity scheduler properties documentation
(Karthik Kambatla via tgraves)
MAPREDUCE-4270. Move the data_join test classes to the correct path.
(Thomas Graves via sseth)
MAPREDUCE-3889. job client tries to use /tasklog interface, but that
doesn't exist anymore (Devaraj K via bobby)
MAPREDUCE-4320. gridmix mainClass wrong in pom.xml (tgraves)
MAPREDUCE-4295. RM crashes due to DNS issue (tgraves)
MAPREDUCE-4228. mapreduce.job.reduce.slowstart.completedmaps is not working
properly (Jason Lowe via bobby)
MAPREDUCE-4392. Counters.makeCompactString() changed behavior from 0.20
(Jason Lowe via bobby)
MAPREDUCE-4384. Race conditions in IndexCache (Kihwal Lee via tgraves)
MAPREDUCE-4387. RM gets fatal error and exits during TestRM
(Kihwal Lee via tgraves)
MAPREDUCE-4379. Node Manager throws java.lang.OutOfMemoryError: Java heap
space due to org.apache.hadoop.fs.LocalDirAllocator.contexts (Devaraj K
via bobby)
MAPREDUCE-4402. TestFileInputFormat fails intermittently (Jason Lowe via
MAPREDUCE-4300. OOM in AM can turn it into a zombie. (Robert Evans via
MAPREDUCE-4252. MR2 job never completes with 1 pending task (Tom White via
MAPREDUCE-3940. ContainerTokens should have an expiry interval. (Siddharth
Seth and Vinod Kumar Vavilapalli via vinodkv)
MAPREDUCE-4419. ./mapred queue -info <queuename> -showJobs displays all
the jobs irrespective of <queuename> (Devaraj K via bobby)
MAPREDUCE-4299. Terasort hangs with MR2 FifoScheduler (Tom White via
MAPREDUCE-4437. Race in MR ApplicationMaster can cause reducers to never be
scheduled (Jason Lowe via bobby)
MAPREDUCE-4449. Incorrect MR_HISTORY_STORAGE property name in JHAdminConfig
(Ahmed Radwan via bobby)
MAPREDUCE-4283. Display tail of aggregated logs by default (Jason Lowe via
MAPREDUCE-4448. Fix NM crash during app cleanup if aggregation didn't
init. (Jason Lowe via daryn)
MAPREDUCE-3893. allow capacity scheduler configs maximum-applications and
maximum-am-resource-percent configurable on a per queue basis (tgraves via
MAPREDUCE-4467. IndexCache failures due to missing synchronization
(Kihwal Lee via tgraves)
MAPREDUCE-4423. Potential infinite fetching of map output (Robert Evans
via tgraves)
MAPREDUCE-4456. LocalDistributedCacheManager can get an
ArrayIndexOutOfBounds when creating symlinks (Robert Evans via tgraves)
MAPREDUCE-4496. AM logs link is missing user name (Jason Lowe via bobby)
MAPREDUCE-4493. Distibuted Cache Compatability Issues (Robert Evans
via tgraves)
MAPREDUCE-4492. Configuring total queue capacity between 100.5 and 99.5 at
perticular level is sucessfull (Mayank Bansal via bobby)
MAPREDUCE-4457. mr job invalid transition TA_TOO_MANY_FETCH_FAILURE at
FAILED (Robert Evans via tgraves)
MAPREDUCE-4234. is incompatible with multi-user or
parallel use (due to a /tmp file with static name) (Robert Evans via
MAPREDUCE-4504. SortValidator writes to wrong directory (Robert Evans
via tgraves)
MAPREDUCE-4503. Should throw InvalidJobConfException if duplicates found in
cacheArchives or cacheFiles (Robert Evans via jeagles)
MAPREDUCE-3782. teragen terasort jobs fail when using webhdfs:// (Jason
Lowe via bobby)
MAPREDUCE-4053. Counters group names deprecation is wrong, iterating over
group names deprecated names don't show up (Robert Evans via tgraves)
MAPREDUCE-3506. Calling getPriority on JobInfo after parsing a history log
with JobHistoryParser throws a NullPointerException (Jason Lowe via bobby)
MAPREDUCE-4570. ProcfsBasedProcessTree#constructProcessInfo() prints a
warning if procfsDir/<pid>/stat is not found. (Ahmed Radwan via bobby)
MAPREDUCE-4600. from MRV1 no longer compiles (daryn
via bobby)
MAPREDUCE-4612. job summary file permissions not set when its created
(tgraves via bobby)
MAPREDUCE-4614. Simplify debugging a job's tokens (daryn via bobby)
MAPREDUCE-4611. MR AM dies badly when Node is decommissioned (Robert
Evans via tgraves)
MAPREDUCE-4604. In mapred-default, &
mapreduce.reduce.maxattempts defaults are set to 4 as well as
mapreduce.job.maxtaskfailures.per.tracker. (Ravi Prakash via jeagles)
MAPREDUCE-4633. history server doesn't set permissions on all subdirs
(tgraves via bobby)
MAPREDUCE-4641. Exception in commitJob marks job as successful in job
history (Jason Lowe via bobby)
Release 0.23.2 - UNRELEASED
MAPREDUCE-3849. Change TokenCache's reading of the binary token file
(Daryn Sharp via bobby)
MAPREDUCE-3854. Fixed and reenabled tests related to MR child JVM's
environmental variables in TestMiniMRChildTask. (Tom White via vinodkv)
MAPREDUCE-3877 Add a test to formalise the current state transitions
of the yarn lifecycle. (stevel)
MAPREDUCE-3866. Fixed the bin/yarn script to not print the command line
unnecessarily. (vinodkv)
MAPREDUCE-3730. Modified RM to allow restarted NMs to be able to join the
cluster without waiting for expiry. (Jason Lowe via vinodkv)
MAPREDUCE-2793. Corrected AppIDs, JobIDs, TaskAttemptIDs to be of correct
format on the web pages. (Bikas Saha via vinodkv)
MAPREDUCE-3614. Fixed MR AM to close history file quickly and send a correct
final state to the RM when it is killed. (Ravi Prakash via vinodkv)
MAPREDUCE-3497. Added docs for YARN CLI. (tgraves via acmurthy)
MAPREDUCE-3954. Added new envs to separate heap size for different daemons
started via bin scripts. (Robert Joseph Evans via vinodkv)
MAPREDUCE-4025. AM can crash if task attempt reports bogus progress value
(Jason Lowe via bobby)
MAPREDUCE-4034. Unable to view task logs on history server with
mapreduce.job.acl-view-job=* (Jason Lowe and Siddarth Seth via bobby)
MAPREDUCE-4043. Secret keys set in Credentials are not seen by tasks
(Jason Lowe via bobby)
MAPREDUCE-3989. Cap space usage of default log4j rolling policy.
(Patrick Hunt via eli)
MAPREDUCE-3901. Modified JobHistory records in YARN to lazily load job and
task reports so as to improve UI response times. (Siddarth Seth via vinodkv)
MAPREDUCE-2855. Passing a cached class-loader to ResourceBundle creator to
minimize counter names lookup time. (Siddarth Seth via vinodkv)
MAPREDUCE-3944. Change the history jobs/ webservice to return partial job
info for a significant performance improvement. (Robert Joseph Evans via
MAPREDUCE-3918 proc_historyserver no longer in command line arguments for
HistoryServer (Jon Eagles via bobby)
MAPREDUCE-3862. Nodemanager can appear to hang on shutdown due to lingering
DeletionService threads (Jason Lowe via bobby)
MAPREDUCE-3680. FifoScheduler web service rest API can print out invalid
JSON. (B Anil Kumar via tgraves)
MAPREDUCE-3852. Test TestLinuxResourceCalculatorPlugin failing. (Thomas
Graves via mahadev)
MAPREDUCE-3736. Variable substitution depth too large for
causes jobs to fail (ahmed via tucu).
MAPREDUCE-3864. Fix cluster setup docs for correct SecondaryNameNode
HTTPS parameters. (todd)
MAPREDUCE-3583. Change pid to String and stime to BigInteger in order to
avoid NumberFormatException caused by overflow. (Zhihong Yu via szetszwo)
MAPREDUCE-3634. Fixed all daemons to crash instead of hanging around when
their EventHandlers get exceptions. (vinodkv)
MAPREDUCE-3798. Fixed failing TestJobCleanup.testCusomCleanup() and moved it
to the maven build. (Ravi Prakash via vinodkv)
MAPREDUCE-3884. PWD should be first in the classpath of MR tasks (tucu)
MAPREDUCE-3878. Null user on filtered jobhistory job page (Jonathon Eagles
via tgraves)
MAPREDUCE-3738. MM can hang during shutdown if AppLogAggregatorImpl thread
dies unexpectedly (Jason Lowe via sseth)
MAPREDUCE-3904 Job history produced with mapreduce.cluster.acls.enabled
false can not be viewed with mapreduce.cluster.acls.enabled true
(Jonathon Eagles via tgraves)
MAPREDUCE-3910. Fixed a bug in CapacityScheduler LeafQueue which was causing
app-submission to fail. (John George via vinodkv)
MAPREDUCE-3686. Fixed two bugs in Counters because of which web app displays
zero counter values for framework counters. (Bhallamudi Venkata Siva Kamesh
via vinodkv)
MAPREDUCE-3913. RM application webpage is unresponsive after 2000 jobs
(Jason Lowe via tgraves)
MAPREDUCE-3922. Fixed build to not compile 32bit container-executor binary
by default on all platforms. (Hitesh Shah via vinodkv)
MAPREDUCE-3790. Broken pipe on streaming job can lead to truncated output for
a successful job (Jason Lowe via bobby)
MAPREDUCE-3816. capacity scheduler web ui bar graphs for used capacity wrong
(tgraves via bobby)
MAPREDUCE-3930. Fixed an NPE while accessing the AM page/webservice for a
task attempt without an assigned container. (Robert Joseph Evans via
MAPREDUCE-3931. Changed PB implementation of LocalResource to take locks
so that race conditions don't fail tasks by inadvertantly changing the
timestamps. (Siddarth Seth via vinodkv)
MAPREDUCE-3687. If AM dies before it returns new tracking URL, proxy
redirects to http://N/A/ and doesn't return error code (Ravi Prakash via
MAPREDUCE-3920. Revise yarn default port number selection
(Dave Thompson via tgraves)
MAPREDUCE-3903. Add support for mapreduce admin users. (Thomas Graves via
MAPREDUCE-3706. Fix circular redirect error in job-attempts page. (bobby
via acmurthy)
MAPREDUCE-3896. Add user information to the delegation token issued by the
history server. (Vinod Kumar Vavilapalli via sseth)
MAPREDUCE-3792. Fix "bin/mapred job -list" to display all jobs instead of
only the jobs owned by the user. (Jason Lowe via vinodkv)
MAPREDUCE-3929. Fixed output of 'bin/mapred queue -showacl' command to
clarify ACLs for users. (John George via acmurthy)
MAPREDUCE-3960. Fix web-proxy to forward request to AM with configured
hostname or IP. (tgraves via acmurthy)
MAPREDUCE-3897. Fixed computation of maxActiveAppsPerUser for queues by
using capacity and not max-capacity since we are already scaling it by
userLimitFactor. (Eric Payne via acmurthy)
MAPREDUCE-3009. Fixed node link on JobHistory webapp. (chackaravarthy via
MAPREDUCE-3964. ResourceManager does not have JVM metrics (Jason Lowe via
MAPREDUCE-3034. Ensure NodeManager reboots itself on direction from
ResourceManager. (Devaraj K & Eric Payne via acmurthy)
MAPREDUCE-3976. TestRMContainerAllocator failing (Jason Lowe via bobby)
MAPREDUCE-3961. Map/ReduceSlotMillis computation incorrect (Siddharth Seth
via bobby)
MAPREDUCE-3977. LogAggregationService leaks log aggregator objects
(Jason Lowe via bobby)
MAPREDUCE-3975. Default value not set for Configuration parameter
mapreduce.job.local.dir (Eric Payne via bobby)
MAPREDUCE-3982. Fixed FileOutputCommitter to not err out for an 'empty-job'
whose tasks don't write any outputs. (Robert Joseph Evans via vinodkv)
MAPREDUCE-4005. Fixes broken AM container logs URL on ResourceManager
Application Info page. (Jason Lowe via sseth)
MAPREDUCE-4006. History server container log web UI sometimes combines
stderr/stdout/syslog contents together (Siddharth Seth via tgraves)
MAPREDUCE-4061. RM only has 1 AM launcher thread (tgraves via bobby)
Release 0.23.1 - 2012-02-17
MAPREDUCE-3121. NodeManager should handle disk-failures (Ravi Gummadi via mahadev)
MAPREDUCE-2863. Support web services for YARN and MR components. (Thomas
Graves via vinodkv)
MAPREDUCE-3251. Network ACLs can prevent some clients to talk to MR ApplicationMaster
(Anupam Seth via mahadev)
MAPREDUCE-3297. Moved log related components into yarn-common so that
HistoryServer and clients can use them without depending on the
yarn-server-nodemanager module. (Siddharth Seth via vinodkv)
MAPREDUCE-3336. Replaced guice internal.Preconditions api usage with the
public Preconditions API. (Thomas Graves via vinodkv)
MAPREDUCE-3280. Removed the unnecessary job user-name configuration in
mapred-site.xml. (vinodkv)
MAPREDUCE-3370. Fixed MiniMRYarnCluster and related tests to not use
a hard-coded path for the mr-app jar. (Ahmed Radwan via vinodkv)
MAPREDUCE-3325. Improvements to CapacityScheduler doc. (Thomas Graves
via mahadev)
MAPREDUCE-3341. Enhance logging of initalized queue limit values.
(Anupam Seth via mahadev)
MAPREDUCE-3243. Invalid tracking URL for streaming jobs (Jonathan Eagles
via mahadev)
MAPREDUCE-3331. Improvement to single node cluster setup documentation for
0.23 (Anupam Seth via mahadev)
MAPREDUCE-3102. Changed NodeManager to fail fast when LinuxContainerExecutor
has wrong configuration or permissions. (Hitesh Shah via vinodkv)
MAPREDUCE-3415. improve MiniMRYarnCluster & DistributedShell JAR resolution.
MAPREDUCE-3373. Hadoop scripts unconditionally source
"$bin"/../libexec/ (Bruno Mahé via tomwhite)
MAPREDUCE-3372. HADOOP_PREFIX cannot be overridden.
(Bruno Mahé via tomwhite)
MAPREDUCE-3411. Performance Upgrade for jQuery (Jonathan Eagles via
MAPREDUCE-3371. Review and improve the yarn-api javadocs. (Ravi Prakash
via mahadev)
MAPREDUCE-3238. Small cleanup in SchedulerApp. (Todd Lipcon via mahadev)
MAPREDUCE-3413. RM web ui applications not sorted in any order by default.
(Jonathan Eagles via mahadev)
MAPREDUCE-3045. Fixed UI filters to not filter on hidden title-numeric
sort fields. (Jonathan Eagles via sseth)
MAPREDUCE-3448. TestCombineOutputCollector javac unchecked warning on mocked
generics (Jonathan Eagles via mahadev)
MAPREDUCE-3169 amendment. Deprecate MiniMRCluster. (Ahmed Radwan via
MAPREDUCE-3369. Migrate MR1 tests to run on MR2 using the new interfaces
introduced in MAPREDUCE-3169. (Ahmed Radwan via tomwhite)
MAPREDUCE-3518. mapred queue -info <queue> -showJobs throws NPE.
(Jonathan Eagles via mahadev)
MAPREDUCE-3391. Making a trivial change to correct a log message in
DistributedShell app's AM. (Subroto Sanyal via vinodkv)
MAPREDUCE-3547. Added a bunch of unit tests for the the RM/NM webservices.
(Thomas Graves via acmurthy)
MAPREDUCE-3610. Remove use of the 'dfs.block.size' config for default block
size fetching. Use FS#getDefaultBlocksize instead. (Sho Shimauchi via harsh)
MAPREDUCE-3478. Cannot build against ZooKeeper 3.4.0. (Tom White via mahadev)
MAPREDUCE-3528. Fixed TaskHeartBeatHandler to use a new configuration
for the thread loop interval separate from task-timeout configuration
property. (Siddharth Seth via vinodkv)
MAPREDUCE-3312. Modified MR AM to not send a stop-container request for
a container that isn't launched at all. (Robert Joseph Evans via vinodkv)
MAPREDUCE-3382. Enhanced MR AM to use a proxy to ping the job-end
notification URL. (Ravi Prakash via vinodkv)
MAPREDUCE-3299. Added AMInfo table to the MR AM job pages to list all the
job-attempts when AM restarts and recovers. (Jonathan Eagles via vinodkv)
MAPREDUCE-3251. Network ACLs can prevent some clients to talk to MR AM.
Improved the earlier patch to not to JobHistoryServer repeatedly.
(Anupam Seth via vinodkv)
MAPREDUCE-3553. Add support for data returned when exceptions thrown from web
service apis to be in either xml or in JSON. (Thomas Graves via mahadev)
MAPREDUCE-3641. Making CapacityScheduler more conservative so as to
assign only one off-switch container in a single scheduling
iteration. (Arun C Murthy via vinodkv)
MAPREDUCE-3692. yarn-resourcemanager out and log files can get big. (eli)
MAPREDUCE-3710. Improved FileInputFormat to return better locality for the
last split. (Siddarth Seth via vinodkv)
MAPREDUCE-2765. DistCp Rewrite. (Mithun Radhakrishnan via mahadev)
MAPREDUCE-3737. The Web Application Proxy's is not documented very well.
(Robert Evans via mahadev)
MAPREDUCE-3699. Increased RPC handlers for all YARN servers to reasonable
values for working at scale. (Hitesh Shah via vinodkv)
MAPREDUCE-3693. Added mapreduce.admin.user.env to mapred-default.xml.
(Roman Shapshonik via acmurthy)
MAPREDUCE-3732. Modified CapacityScheduler to use only users with pending
requests for computing user-limits. (Arun C Murthy via vinodkv)
MAPREDUCE-3679. AM logs and others should not automatically refresh after every 1
second. (Vinod KV via mahadev)
MAPREDUCE-3754. Modified RM UI to filter applications based on state of the
applications. (vinodkv)
MAPREDUCE-3774. Moved yarn-default.xml to hadoop-yarn-common from
hadoop-server-common. (Mahadev Konar via vinodkv)
MAPREDUCE-3771. Un-deprecated the old mapred apis, port of MAPREDUCE-1735.
MAPREDUCE-3784. Fixed CapacityScheduler so that maxActiveApplications and
maxActiveApplicationsPerUser per queue are not too low for small
clusters. (Arun C Murthy via vinodkv)
MAPREDUCE-3567. Extraneous JobConf objects in AM heap. (Vinod Kumar
Vavilapalli via sseth)
MAPREDUCE-3399. Modifying ContainerLocalizer to send a heartbeat to NM
immediately after downloading a resource instead of always waiting for a
second. (Siddarth Seth via vinodkv)
MAPREDUCE-3568. Optimized Job's progress calculations in MR AM. (vinodkv)
MAPREDUCE-3569. TaskAttemptListener holds a global lock for all
task-updates. (Vinod Kumar Vavilapalli via sseth)
MAPREDUCE-3511. Removed a multitude of cloned/duplicate counters in the AM
thereby reducing the AM heap size and preventing full GCs. (vinodkv)
MAPREDUCE-3618. Fixed TaskHeartbeatHandler to not hold a global lock for all
task-updates. (Siddarth Seth via vinodkv)
MAPREDUCE-3512. Batching JobHistory flushing to DFS so that we don't flush
for every event slowing down AM. (Siddarth Seth via vinodkv)
MAPREDUCE-3718. Change default AM heartbeat interval to 1 second. (Hitesh
Shah via sseth)
MAPREDUCE-3360. Added information about lost/rebooted/decommissioned nodes
on the webapps. (Bhallamudi Venkata Siva Kamesh and Jason Lowe via vinodkv)
MAPREDUCE-3756. Made single shuffle limit configurable. (Hitesh Shah via
MAPREDUCE-3811. Made jobclient-to-AM retries configurable. (sseth via
MAPREDUCE-3221. Reenabled the previously ignored test in TestSubmitJob
and fixed bugs in it. (Devaraj K via vinodkv)
MAPREDUCE-3215. Reenabled and fixed bugs in the failing test
TestNoJobSetupCleanup. (Hitesh Shah via vinodkv)
MAPREDUCE-3219. Reenabled and fixed bugs in the failing test
TestDelegationToken. (Hitesh Shah via vinodkv)
MAPREDUCE-3217. Reenabled and fixed bugs in the failing ant test
TestAuditLogger. (Devaraj K via vinodkv)
MAPREDUCE-3291. App fail to launch due to delegation token not
found in cache (Robert Evans via mahadev)
MAPREDUCE-3344. o.a.h.mapreduce.Reducer since 0.21 blindly casts to
ReduceContext.ValueIterator. (Brock Noland via tomwhite)
MAPREDUCE-3342. Fixed JobHistoryServer to also show the job's queue
name. (Jonathan Eagles via vinodkv)
MAPREDUCE-3345. Fixed a race condition in ResourceManager that was causing
TestContainerManagerSecurity to fail sometimes. (Hitesh Shah via vinodkv)
MAPREDUCE-3333. Fixed bugs in ContainerLauncher of MR AppMaster due to
which per-container connections to NodeManager were lingering long enough
to hit the ulimits on number of processes. (vinodkv)
MAPREDUCE-3392. Fixed Cluster's getDelegationToken's API to return null
when there isn't a supported token. (John George via vinodkv)
MAPREDUCE-3379. Fixed LocalResourceTracker in NodeManager to remove deleted
cache entries correctly. (Siddharth Seth via vinodkv)
MAPREDUCE-3324. Not All HttpServer tools links (stacks,logs,config,metrics) are
accessible through all UI servers (Jonathan Eagles via mahadev)
MAPREDUCE-3355. Fixed MR AM's ContainerLauncher to handle node-command
timeouts correctly. (vinodkv)
MAPREDUCE-3407. Fixed pom files to refer to the correct MR app-jar needed
by the integration tests. (Hitesh Shah via vinodkv)
MAPREDUCE-3434. Nightly build broken (Hitesh Shah via mahadev)
MAPREDUCE-3447. mapreduce examples not working (mahadev)
MAPREDUCE-3444. trunk/0.23 builds broken (Hitesh Shah via mahadev)
MAPREDUCE-3454. [Gridmix] TestDistCacheEmulation is broken (Hitesh Shah
via mahadev)
MAPREDUCE-3408. unconditionnaly sets yarn.root.logger
(Bruno Mahe via mahadev)
MAPREDUCE-3329. Fixed CapacityScheduler to ensure maximum-capacity cannot
be lesser than capacity for any queue. (acmurthy)
MAPREDUCE-3464. mapreduce jsp pages missing DOCTYPE. (Dave Vronay via mattf)
MAPREDUCE-3265. Removed debug logs during job submission to LOG.debug to
cut down noise. (acmurthy)
MAPREDUCE-3433. Finding counters by legacy group name returns empty
counters. (tomwhite)
MAPREDUCE-3450. NM port info no longer available in JobHistory.
(Siddharth Seth via mahadev)
MAPREDUCE-3488. Streaming jobs are failing because the main class
isnt set in the pom files. (mahadev)
MAPREDUCE-3463. Second AM fails to recover properly when first AM is killed with
java.lang.IllegalArgumentException causing lost job. (Siddharth Seth via mahadev)
MAPREDUCE-3452. fifoscheduler web ui page always shows 0% used for the queue.
(Jonathan Eagles via mahadev)
MAPREDUCE-3443. JobClient and Job should function in the context of the
UGI which created them. (Mahadev Konar via sseth)
MAPREDUCE-3460. MR AM can hang if containers are allocated on a node
blacklisted by the AM. (Hitesh Shah and Robert Joseph Evans via sseth)
MAPREDUCE-3453. RM web ui application details page shows RM cluster about
information. (Jonathan Eagles via sseth)
MAPREDUCE-3479. JobClient#getJob cannot find local jobs. (tomwhite)
MAPREDUCE-3456. $HADOOP_PREFIX/bin/yarn should set defaults for
$HADOOP_*_HOME (Eric Payne via mahadev)
MAPREDUCE-3458. Fix findbugs warnings in hadoop-examples. (Devaraj K
via mahadev)
MAPREDUCE-3485. DISKS_FAILED -101 error code should be defined in same location as
ABORTED_CONTAINER_EXIT_STATUS. (Ravi Gummadi via mahadev)
MAPREDUCE-3496. Fixed client to print queue acls in consistent order.
(Jonathan Eagles via acmurthy)
MAPREDUCE-3147. Handle leaf queues with the same name properly. (Ravi Prakash via
MAPREDUCE-3327. RM web ui scheduler link doesn't show correct max value
for queues (Anupam Seth via mahadev)
MAPREDUCE-3513. Capacity Scheduler web UI has a spelling mistake for Memory.
(chackaravarthy via mahadev)
MAPREDUCE-3519. Fixed a deadlock in NodeManager LocalDirectories's handling
service. (Ravi Gummadi via vinodkv)
MAPREDUCE-3527. Fix minor API incompatibilities between 1.0 and 0.23.
MAPREDUCE-3328. mapred queue -list output inconsistent and missing child
queues. (Ravi Prakash via mahadev)
MAPREDUCE-3510. Capacity Scheduler inherited ACLs not displayed by mapred queue
-showacls (Jonathan Eagles via mahadev)
MAPREDUCE-3537. Fix race condition in DefaultContainerExecutor which led
to container localization occuring in wrong directories. (acmurthy)
MAPREDUCE-3542. Support "FileSystemCounter" legacy counter group name for
compatibility. (tomwhite)
MAPREDUCE-3426. Fixed MR AM in uber mode to write map intermediate outputs
in the correct directory to work properly in secure mode. (Hitesh Shah via
MAPREDUCE-3541. Fix broken TestJobQueueClient test. (Ravi Prakash via
MAPREDUCE-3398. Fixed log aggregation to work correctly in secure mode.
(Siddharth Seth via vinodkv)
MAPREDUCE-3530. Fixed an NPE occuring during scheduling in the
ResourceManager. (Arun C Murthy via vinodkv)
MAPREDUCE-3484. Fixed JobEndNotifier to not get interrupted before completing
all its retries. (Ravi Prakash via vinodkv)
MAPREDUCE-3531. Fixed a race in ContainerTokenSecretManager. (Robert Joseph
Evans via sseth)
MAPREDUCE-3560. TestRMNodeTransitions is failing on trunk.
(Siddharth Seth via mahadev)
MAPREDUCE-3487. Fixed JobHistory web-UI to display links to single task's
counters' page. (Jason Lowe via vinodkv)
MAPREDUCE-3564. Fixed failures in TestStagingCleanup and TestJobEndNotifier
tests. (Siddharth Seth via vinodkv)
MAPREDUCE-3422. Counter display names are not being picked up. (Jonathan
Eagles via sseth)
MAPREDUCE-3366. Mapreduce component should use consistent directory structure
layout as HDFS/common (Eric Yang via mahadev)
MAPREDUCE-3387. Fixed AM's tracking URL to always go through the proxy, even
before the job started, so that it works properly with oozie throughout
the job execution. (Robert Joseph Evans via vinodkv)
MAPREDUCE-3579. ConverterUtils shouldn't include a port in a path from a url
without a port. (atm via harsh)
MAPREDUCE-3563. Fixed LocalJobRunner to work correctly with new mapreduce
apis. (acmurthy)
MAPREDUCE-3376. Fixed Task to ensure it passes reporter to combiners using
old MR api. (Subroto Sanyal via acmurthy)
MAPREDUCE-3339. Fixed MR AM to stop considering node blacklisting after the
number of nodes blacklisted crosses a threshold. (Siddharth Seth via vinodkv)
MAPREDUCE-3588. Fixed bin/yarn which was broken by MAPREDUCE-3366 so that
yarn daemons can start. (Arun C Murthy via vinodkv)
MAPREDUCE-3586. Modified CompositeService to avoid duplicate stop operations
thereby solving race conditions in MR AM shutdown. (vinodkv)
MAPREDUCE-3604. Fixed streaming to use new to
check for local mode. (acmurthy)
MAPREDUCE-3521. Fixed streaming to ensure it doesn't silently ignore
unknown arguments. (Robert Evans via acmurthy)
MAPREDUCE-3522. Ensure queues inherit ACLs from parent if they aren't
explicitly specified. (Jonathan Eagles via acmurthy)
MAPREDUCE-3608. Fixed compile issue with MAPREDUCE-3522. (mahadev via
MAPREDUCE-3490. Fixed MapReduce AM to count failed maps also towards Reduce
ramp up. (Sharad Agarwal and Arun C Murthy via vinodkv)
MAPREDUCE-1744. DistributedCache creates its own FileSytem instance when
adding a file/archive to the path. (Dick King via tucu)
MAPREDUCE-3529. TokenCache does not cache viewfs credentials correctly
MAPREDUCE-3595. Add missing TestCounters#testCounterValue test from branch
1 to 0.23 (Tom White via sseth)
MAPREDUCE-3566. Fixed MR AM to construct CLC only once across all tasks.
(vinodkv via acmurthy)
MAPREDUCE-3572. Moved AM event dispatcher to a separate thread for
performance reasons. (vinodkv via acmurthy)
MAPREDUCE-3615. Fix some ant test failures. (Thomas Graves via sseth)
MAPREDUCE-3326. Added detailed information about queue's to the
CapacityScheduler web-ui. (Jason Lowe via acmurthy)
MAPREDUCE-3548. Added more unit tests for MR AM & JHS web-services.
(Thomas Graves via acmurthy)
MAPREDUCE-3617. Removed wrong default value for
yarn.resourcemanager.principal and yarn.nodemanager.principal. (Jonathan
Eagles via acmurthy)
MAPREDUCE-3183. hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
missing license header. (Hitesh Shah via tucu).
MAPREDUCE-3003. Publish MR JARs to Maven snapshot repository. (tucu)
MAPREDUCE-3204. mvn site:site fails on MapReduce. (tucu)
MAPREDUCE-3014. Rename and invert logic of '-cbuild' profile to 'native' and off
by default. (tucu)
MAPREDUCE-3477. Hadoop site documentation cannot be built anymore. (jeagles via tucu)
MAPREDUCE-3500. MRJobConfig creates an LD_LIBRARY_PATH using the platform ARCH. (tucu)
MAPREDUCE-3389. MRApps loads the 'mrapp-generated-classpath' file with
classpath from the build machine. (tucu)
MAPREDUCE-3544. gridmix build is broken, requires hadoop-archives to be added as
ivy dependency. (tucu)
MAPREDUCE-3557. MR1 test fail to compile because of missing hadoop-archives dependency.
MAPREDUCE-3624. Remove unnecessary dependency on JDK's tools.jar. (mahadev
via acmurthy)
MAPREDUCE-3616. Thread pool for launching containers in MR AM not
expanding as expected. (vinodkv via sseth)
MAPREDUCE-3639. Fixed TokenCache to work with absent FileSystem canonical
service-names. (Siddharth Seth via vinodkv)
MAPREDUCE-3380. Token infrastructure for running clients which are not kerberos
authenticated. (mahadev)
MAPREDUCE-3648. TestJobConf failing. (Thomas Graves via mahadev)
MAPREDUCE-3651. TestQueueManagerRefresh fails. (Thomas Graves via mahadev)
MAPREDUCE-3645. TestJobHistory fails. (Thomas Graves via mahadev)
MAPREDUCE-3652. org.apache.hadoop.mapred.TestWebUIAuthorization.testWebUIAuthorization
fails. (Thomas Graves via mahadev)
MAPREDUCE-3625. CapacityScheduler web-ui display of queue's used capacity is broken.
(Jason Lowe via mahadev)
MAPREDUCE-3596. Fix scheduler to handle cleaned up containers, which NMs
may subsequently report as running. (Vinod Kumar Vavilapalli via sseth)
MAPREDUCE-3656. Fixed a race condition in MR AM which is failing the sort
benchmark consistently. (Siddarth Seth via vinodkv)
MAPREDUCE-3532. Modified NM to report correct http address when an ephemeral
web port is configured. (Bhallamudi Venkata Siva Kamesh via vinodkv)
MAPREDUCE-3404. Corrected MR AM to honor speculative configuration and enable
speculating either maps or reduces. (Eric Payne via vinodkv)
MAPREDUCE-3649. Job End notification gives an error on calling back.
(Ravi Prakash via mahadev)
MAPREDUCE-3657. State machine visualize build fails. (Jason Lowe
via mahadev)
MAPREDUCE-2450. Fixed a corner case with interrupted communication threads
leading to a long timeout in Task. (Rajesh Balamohan via acmurthy)
MAPREDUCE-3669. Allow clients to talk to MR HistoryServer using both
delegation tokens and kerberos. (mahadev via acmurthy)
MAPREDUCE-3684. LocalDistributedCacheManager does not shut down its thread
pool (tomwhite)
MAPREDUCE-3582. Move successfully passing MR1 tests to MR2 maven tree.
(ahmed via tucu)
MAPREDUCE-3698. Client cannot talk to the history server in secure mode.
MAPREDUCE-3689. RM web UI doesn't handle newline in job name.
(Thomas Graves via mahadev)
MAPREDUCE-3549. write api documentation for web service apis for RM, NM,
mapreduce app master, and job history server (Thomas Graves via mahadev)
MAPREDUCE-3705. ant build fails on 0.23 branch. (Thomas Graves via
MAPREDUCE-3691. webservices add support to compress response.
(Thomas Graves via mahadev)
MAPREDUCE-3702. internal server error trying access application master
via proxy with filter enabled (Thomas Graves via mahadev)
MAPREDUCE-3646. Remove redundant URL info from "mapred job" output.
(Jonathan Eagles via mahadev)
MAPREDUCE-3681. Fixed computation of queue's usedCapacity. (acmurthy)
MAPREDUCE-3505. yarn APPLICATION_CLASSPATH needs to be overridable.
(ahmed via tucu)
MAPREDUCE-3714. Fixed EventFetcher and Fetcher threads to shut-down properly
so that reducers don't hang in corner cases. (vinodkv)
MAPREDUCE-3712. The mapreduce tar does not contain the hadoop-mapreduce-client-
jobclient-tests.jar. (mahadev)
MAPREDUCE-3717. JobClient test jar has missing files to run all the test programs.
MAPREDUCE-3630. Fixes a NullPointer exception while running TeraGen - if a
map is asked to generate 0 records. (Mahadev Konar via sseth)
MAPREDUCE-3683. Fixed maxCapacity of queues to be product of parent
maxCapacities. (acmurthy)
MAPREDUCE-3713. Fixed the way head-room is allocated to applications by
CapacityScheduler so that it deducts current-usage per user and not
per-application. (Arun C Murthy via vinodkv)
MAPREDUCE-3721. Fixed a race in shuffle which caused reduces to hang.
(sseth via acmurthy)
MAPREDUCE-3733. Add Apache License Header to hadoop-distcp/pom.xml.
MAPREDUCE-3735. Add distcp jar to the distribution (tar).
MAPREDUCE-3720. Changed bin/mapred job -list to not print job-specific
information not available at RM. (vinodkv via acmurthy)
MAPREDUCE-3742. "yarn logs" command fails with ClassNotFoundException.
(Jason Lowe via mahadev)
MAPREDUCE-3703. ResourceManager should provide node lists in JMX output.
(Eric Payne via mahadev)
MAPREDUCE-3716. Fixing YARN+MR to allow MR jobs to be able to use to create temporary files as part of their
tasks. (Jonathan Eagles via vinodkv)
MAPREDUCE-3748. Changed a log in CapacityScheduler.nodeUpdate to debug.
(ramya via acmurthy)
MAPREDUCE-3764. Fixed resource usage metrics for queues and users.
MAPREDUCE-3749. ConcurrentModificationException in counter groups.
MAPREDUCE-3762. Fixed default CapacityScheduler configs. (mahadev via
MAPREDUCE-3499. New MiniMR does not setup proxyuser configuration
correctly, thus tests using doAs do not work. (johnvijoe via tucu)
MAPREDUCE-3696. MR job via oozie does not work on hadoop 23.
(John George via mahadev)
MAPREDUCE-3427. Fix streaming unit tests broken after mavenization.
(Hitesh Shah via acmurthy)
MAPREDUCE-3640. Allow AMRecovery to work with partial JobHistory files.
(Arun C Murthy via sseth)
MAPREDUCE-3752. Modified application limits to include queue max-capacities
besides the usual user limits. (Arun C Murthy via vinodkv)
MAPREDUCE-3744. Fix the yarn logs command line. Improve error messages for
mapred job -logs. (Jason Lowe via sseth)
MAPREDUCE-3780. Fixed a bug where applications killed before getting
activated were not getting cleaned up properly. (Hitesh Shah via acmurthy)
MAPREDUCE-3708. Metrics: Incorrect Apps Submitted Count (Bhallamudi via
MAPREDUCE-3727. jobtoken location property in jobconf refers to wrong
jobtoken file (tucu)
MAPREDUCE-3711. Fixed MR AM recovery so that only single selected task
output is recovered and thus reduce the unnecessarily bloated recovery
time. (Robert Joseph Evans via vinodkv)
MAPREDUCE-3760. Changed active nodes list to not contain unhealthy nodes
on the webUI and metrics. (vinodkv)
MAPREDUCE-3417. Fixed job-access-controls to work with MR AM and
JobHistoryServer web-apps. (Jonathan Eagles via vinodkv)
MAPREDUCE-3803. Fix broken build of raid contrib due to HDFS-2864.
(Ravi Prakash via suresh)
MAPREDUCE-3791. can't build site in hadoop-yarn-server-common.
MAPREDUCE-3723. TestAMWebServicesJobs & TestHSWebServicesJobs
incorrectly asserting tests (Bhallamudi Venkata Siva Kamesh
via mahadev)
MAPREDUCE-3795. "job -status" command line output is malformed.
(vinodkv via mahadev)
MAPREDUCE-3759. ClassCastException thrown in -list-active-trackers when
there are a few unhealthy nodes (vinodkv via mahadev)
MAPREDUCE-3775. Change MiniYarnCluster to escape special chars in testname.
(Hitesh Shah via mahadev)
MAPREDUCE-3765. FifoScheduler does not respect yarn.scheduler.fifo.minimum-
allocation-mb setting (Hitesh Shah via mahadev)
MAPREDUCE-3747. Initialize queue metrics upfront and added start/finish
time to RM Web-UI. (acmurthy)
MAPREDUCE-3814. Fixed MRV1 compilation. (Arun C Murthy via vinodkv)
MAPREDUCE-3810. Performance tweaks - reduced logging in AM and defined
hascode/equals for ResourceRequest & Priority. (vinodkv via acmurthy)
MAPREDUCE-3813. Added a cache for resolved racks. (vinodkv via acmurthy)
MAPREDUCE-3808. Fixed an NPE in FileOutputCommitter for jobs with maps
but no reduces. (Robert Joseph Evans via vinodkv)
MAPREDUCE-3804. yarn webapp interface vulnerable to cross scripting attacks
(Dave Thompson via bobby)
MAPREDUCE-3354. Changed scripts so that jobhistory server is started by
bin/mapred instead of bin/yarn. (Jonathan Eagles via acmurthy)
MAPREDUCE-3809. Ensure that there is no needless sleep in Task at the end
of the task. (sseth via acmurthy)
MAPREDUCE-3794. Support mapred.Task.Counter and mapred.JobInProgress.Counter
enums for compatibility (Tom White via mahadev)
MAPREDUCE-3697. Support binary compatibility for Counters after
MAPREDUCE-901. (mahadev via acmurthy)
MAPREDUCE-3709. TestDistributedShell is failing. (Hitesh Shah via
MAPREDUCE-3817. Fixed bin/mapred to allow running of distcp and archive
jobs. (Arpit Gupta via acmurthy)
MAPREDUCE-3709. TestDistributedShell is failing. (Hitesh Shah via
MAPREDUCE-3436. JobHistory webapp address should use the host configured
in the jobhistory address. (Ahmed Radwan via sseth)
MAPREDUCE-3815. Fixed MR AM to always use hostnames and never IPs when
requesting containers so that scheduler can give off data local containers
correctly. (Siddarth Seth via vinodkv)
MAPREDUCE-3833. Fixed a bug in reinitiaziling of queues. (Jason Lowe via
MAPREDUCE-3826. Fixed a bug in RM web-ui which broke sorting. (Jonathan
Eagles via acmurthy)
MAPREDUCE-3823. Ensure counters are calculated only once after a job
finishes. (Vinod Kumar Vavilapalli via sseth)
MAPREDUCE-3827. Changed Counters to use ConcurrentSkipListMap for
performance. (vinodkv via acmurthy)
MAPREDUCE-3822. Changed FS counter computation to use all occurences of
the same FS scheme, instead of randomly using one. (Mahadev Konar via
MAPREDUCE-3834. Changed MR AM to not add the same rack entry multiple times
into the container request table when multiple hosts for a split happen to
be on the same rack. (Siddarth Seth via vinodkv)
MAPREDUCE-3828. Ensure that urls in single-node mode are correct. (sseth
via acmurthy)
MAPREDUCE-3770. Zombie.getJobConf() results into NPE. (amarrk)
MAPREDUCE-3840. JobEndNotifier doesn't use the proxyToUse during connecting
(Ravi Prakash via bobby)
MAPREDUCE-3843. Job summary log file found missing on the RM host
(Anupam Seth via tgraves)
MAPREDUCE-3846. Addressed MR AM hanging issues during AM restart and then
the recovery. (vinodkv)
MAPREDUCE-3802. Added test to validate that AM can crash multiple times and
still can recover successfully after MAPREDUCE-3846. (vinodkv)
MAPREDUCE-3858. Task attempt failure during commit results in task never completing.
(Tom White via mahadev)
MAPREDUCE-3856. Instances of RunningJob class givs incorrect job tracking
urls when mutiple jobs are submitted from same client jvm. (Eric Payne via
MAPREDUCE-3880. Changed LCE binary to be 32-bit. (acmurthy)
Release 0.23.0 - 2011-11-01
MAPREDUCE-2455. Remove deprecated JobTracker.State in favour of
JobTrackerStatus. (tomwhite)
MAPREDUCE-2430. Remove mrunit contrib. (nigel via eli)
MAPREDUCE-2606. Remove IsolationRunner. (Alejandro Abdelnur via eli)
MAPREDUCE-2682. Add "mapred classpath" command to print classpath
for MR applications. (vinodkv via acmurthy)
MAPREDUCE-2107. [Gridmix] Total heap usage emulation in Gridmix.
(Amar Kamat and Ravi Gummadi via amarrk)
MAPREDUCE-2106. [Gridmix] Cumulative CPU usage emulation in Gridmix.
MAPREDUCE-2543. [Gridmix] High-Ram feature emulation in Gridmix. (amarrk)
MAPREDUCE-2408. [Gridmix] Compression emulation in Gridmix. (amarrk)
MAPREDUCE-2473. Add "mapred groups" command to query the server-side groups
resolved for a user. (Aaron T. Myers via todd)
MAPREDUCE-461. Enable ServicePlugins for the JobTracker.
(Fredrik Hedberg via tomwhite)
MAPREDUCE-2521. Create RPM and Debian packages for MapReduce. Changes
deployment layout to be consistent across the binary tgz, rpm, and deb.
(Eric Yang via omalley)
MAPREDUCE-2323. Add metrics to the fair scheduler. (todd)
MAPREDUCE-2037. Capture intermediate progress, CPU and memory usage for
tasks. (Dick King via acmurthy)
MAPREDUCE-279. MapReduce 2.0. Merging MR-279 branch into trunk. Contributed by
Arun C Murthy, Christopher Douglas, Devaraj Das, Greg Roelofs, Jeffrey
Naisbitt, Josh Wills, Jonathan Eagles, Krishna Ramachandran, Luke Lu, Mahadev
Konar, Robert Evans, Sharad Agarwal, Siddharth Seth, Thomas Graves, and Vinod
Kumar Vavilapalli.
MAPREDUCE-2930. Added the ability to be able to generate graphs from the
state-machine definitions. (Binglin Chang via vinodkv)
MAPREDUCE-2719. Add a simple, DistributedShell, application to illustrate
alternate frameworks on YARN. (Hitesh Shah via acmurthy)
MAPREDUCE-3104. Implemented Application-acls. (vinodkv)
MAPREDUCE-2708. Designed and implemented MR Application Master recovery to
make MR AMs resume their progress after restart. (Sharad Agarwal via vinodkv)
MAPREDUCE-2858. Added a WebApp Proxy for applications. (Robert Evans via