Google Git
Sign in
apache / hadoop / 92164c86c7e997542fb756d81f081156cdfb380f / . / branch-2.0.4-alpha / hadoop-mapreduce-project / CHANGES.txt
blob: 195e757e870836fe1176f076bf6b4eda5fa8164a [file] [log] [blame]
Hadoop MapReduce Change Log
Release 2.0.4-alpha - UNRELEASED
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
OPTIMIZATIONS
BUG FIXES
MAPREDUCE-5088. MR Client gets an renewer token exception while Oozie is
submitting a job (Daryn Sharp via cos)
MAPREDUCE-5117. Changed MRClientProtocolPBClientImpl to be closeable and thus
fix failures in renewal of HistoryServer's delegations tokens. (Siddharth
Seth via vinodkv)
MAPREDUCE-5083. MiniMRCluster should use a random component when creating an
actual cluster (Siddharth Seth via hitesh)
MAPREDUCE-5006. Fix failing streaming tests due to MAPREDUCE-4994.
(Sandy Ryza via tomwhite)
MAPREDUCE-5094. Disabled memory monitoring by default in MiniMRYarnCluster
to avoid some downstream tests failing. (Siddharth Seth via vinodkv)
Release 2.0.3-alpha - 2013-02-06
INCOMPATIBLE CHANGES
MAPREDUCE-4123. Remove the 'mapred groups' command, which is no longer
supported. (Devaraj K via sseth)
MAPREDUCE-4938. Use token request messages defined in hadoop common.
(suresh)
NEW FEATURES
MAPREDUCE-4520. Added support for MapReduce applications to request for
CPU cores along-with memory post YARN-2. (acmurthy)
MAPREDUCE-4810. Added new admin command options for MR AM. (Jerry Chen via
vinodkv)
MAPREDUCE-4049. Experimental api to allow for alternate shuffle plugins.
(Avner BenHanoch via acmurthy)
MAPREDUCE-4807. Allow MapOutputBuffer to be pluggable. (masokan via tucu)
MAPREDUCE-4808. Refactor MapOutput and MergeManager to facilitate reuse
by Shuffle implementations. (masokan via tucu)
IMPROVEMENTS
MAPREDUCE-3678. The Map tasks logs should have the value of input
split it processed. (harsh)
MAPREDUCE-4616. Improve javadoc for MultipleOutputs. (Tony Burton via
acmurthy)
HADOOP-8911. CRLF characters in source and text files.
(Raja Aluri via suresh)
MAPREDUCE-4703. Add the ability to start the MiniMRClientCluster using
the configurations used before it is being stopped. (ahmed.radwan via tucu)
MAPREDUCE-4845. ClusterStatus.getMaxMemory() and getUsedMemory() exist in
MR1 but not MR2. (Sandy Ryza via tomwhite)
MAPREDUCE-4899. Implemented a MR specific plugin for tracking finished
applications that YARN's ResourceManager doesn't keep track off anymore
(Derek Dagit via vinodkv)
MAPREDUCE-4920. Use security token protobuf definition from hadoop common.
(Suresh Srinivas via vinodkv)
MAPREDUCE-4907. TrackerDistributedCacheManager issues too many getFileStatus
calls. (sandyr via tucu)
MAPREDUCE-4949. Enable multiple pi jobs to run in parallel. (sandyr via tucu)
MAPREDUCE-4809. Change visibility of classes for pluggable sort changes.
(masokan via tucu)
MAPREDUCE-4838. Add additional fields like Locality, Avataar to the
JobHistory logs. (Zhijie Shen via sseth)
MAPREDUCE-4971. Minor extensibility enhancements to Counters &
FileOutputFormat. (Arun C Murthy via sseth)
MAPREDUCE-4977. Documentation for pluggable shuffle and pluggable sort.
(tucu)
OPTIMIZATIONS
MAPREDUCE-4893. Fixed MR ApplicationMaster to do optimal assignment of
containers to get maximum locality. (Bikas Saha via vinodkv)
BUG FIXES
MAPREDUCE-4272. SortedRanges.Range#compareTo is not spec compliant.
(Yu Gao via llu)
MAPREDUCE-4607. Race condition in ReduceTask completion can result in Task
being incorrectly failed. (Bikas Saha via tomwhite)
MAPREDUCE-4646. Fixed MR framework to send diagnostic information correctly
to clients in case of failed jobs also. (Jason Lowe via vinodkv)
MAPREDUCE-4674. Hadoop examples secondarysort has a typo
"secondarysrot" in the usage. (Robert Justice via eli)
MAPREDUCE-4681. Fix unit tests broken by HDFS-3910. (acmurthy)
MAPREDUCE-4712. mr-jobhistory-daemon.sh doesn't accept --config
(Vinod Kumar Vavilapalli via tgraves)
MAPREDUCE-4654. TestDistCp is ignored. (Sandy Ryza via tomwhite)
MAPREDUCE-4637. Handle TaskAttempt diagnostic updates while in the NEW and
UNASSIGNED states. (Mayank Bansal via sseth)
MAPREDUCE-1806. CombineFileInputFormat does not work with paths not on default FS. (Gera Shegalov via tucu)
MAPREDUCE-4777. In TestIFile, testIFileReaderWithCodec relies on
testIFileWriterWithCodec. (Sandy Ryza via tomwhite)
MAPREDUCE-4800. Cleanup o.a.h.mapred.MapTaskStatus - remove unused
code. (kkambatl via tucu)
MAPREDUCE-4861. Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal.
(kkambatl via tucu)
MAPREDUCE-4856. TestJobOutputCommitter uses same directory as
TestJobCleanup. (Sandy Ryza via tomwhite)
MAPREDUCE-4895. Fix compilation failure of org.apache.hadoop.mapred.
gridmix.TestResourceUsageEmulators (Dennis Y via tgraves)
MAPREDUCE-4278. Cannot run two local jobs in parallel from the same
gateway. (Sandy Ryza via tomwhite)
MAPREDUCE-1700. User supplied dependencies may conflict with MapReduce
system JARs. (tomwhite)
MAPREDUCE-4936. JobImpl uber checks for cpu are wrong (Arun C Murthy via
jlowe)
MAPREDUCE-4924. flakey test: org.apache.hadoop.mapred.TestClusterMRNotification.testMR.
(rkanter via tucu)
MAPREDUCE-4923. Add toString method to TaggedInputSplit. (sandyr via tucu)
MAPREDUCE-4948. Fix a failing unit test TestYARNRunner.testHistoryServerToken.
(Junping Du via sseth)
MAPREDUCE-4803. Remove duplicate copy of TestIndexCache. (Mariappan Asokan
via sseth)
MAPREDUCE-2264. Job status exceeds 100% in some cases.
(devaraj.k and sandyr via tucu)
MAPREDUCE-4969. TestKeyValueTextInputFormat test fails with Open JDK 7.
(Arpit Agarwal via suresh)
MAPREDUCE-4884. Streaming tests fail to start MiniMRCluster due to missing
queue configuration. (Chris Nauroth via suresh)
MAPREDUCE-4953. HadoopPipes misuses fprintf. (Andy Isaacson via atm)
Release 2.0.2-alpha - 2012-09-07
INCOMPATIBLE CHANGES
NEW FEATURES
MAPREDUCE-987. Exposing MiniDFS and MiniMR clusters as a single process
command-line. (ahmed via tucu)
MAPREDUCE-4417. add support for encrypted shuffle (tucu)
MAPREDUCE-4355. Add RunningJob.getJobStatus() (kkambatl via tucu)
MAPREDUCE-3451. Port Fair Scheduler to MR2 (pwendell via tucu)
MAPREDUCE-4438. Add a simple, generic client to run 'easy' AMs in YARN.
(Bikas Saha via acmurthy)
IMPROVEMENTS
MAPREDUCE-4157. ResourceManager should not kill apps that are well behaved
(Jason Lowe via bobby)
MAPREDUCE-4511. Add IFile readahead (ahmed via tucu)
MAPREDUCE-4408. allow jobs to set a JAR that is in the distributed cached
(rkanter via tucu)
MAPREDUCE-4440. Changed SchedulerApp and SchedulerNode to be a minimal
interface to allow schedulers to maintain their own. (acmurthy)
MAPREDUCE-4146. Support limits on task status string length and number of
block locations in branch-2. (Ahmed Radwan via tomwhite)
MAPREDUCE-3871. Allow symlinking in LocalJobRunner DistributedCache.
(tomwhite)
MAPREDUCE-3921. MR AM should act on node health status changes.
(Bikas Saha via sseth)
MAPREDUCE-4355. Add RunningJob.getJobStatus() (kkambatl via tucu)
MAPREDUCE-4427. Added an 'unmanaged' mode for AMs so as to ease
development of new applications. (Bikas Saha via acmurthy)
MAPREDUCE-3289. Make use of fadvise in the NM's shuffle handler.
(Todd Lipcon and Siddharth Seth via sseth)
MAPREDUCE-4580. Change MapReduce to use the yarn-client module.
(Vinod Kumar Vavilapalli via sseth)
MAPREDUCE-4579. Split TestTaskAttempt into two so as to pass tests on
jdk7. (Thomas Graves via vinodkv)
MAPREDUCE-4638. MR AM supplies MapReduce jars in classpath rather than
rely on YARN. (acmurthy)
BUG FIXES
MAPREDUCE-4422. YARN_APPLICATION_CLASSPATH needs a documented default value in
YarnConfiguration. (ahmed via tucu)
MAPREDUCE-4406. Users should be able to specify the MiniCluster ResourceManager
and JobHistoryServer ports. (ahmed via tucu)
MAPREDUCE-4407. Add hadoop-yarn-server-tests-<version>-tests.jar to hadoop dist
package. (ahmed via tucu)
MAPREDUCE-4465. Update description of yarn.nodemanager.address property.
(bowang via tucu)
MAPREDUCE-4342. Distributed Cache gives inconsistent result if cache files
get deleted from tasktracker. (mayank_bansal via tucu)
MAPREDUCE-4498. Remove hsqldb jar from Hadoop runtime classpath. (rkanter via tucu)
MAPREDUCE-4494. TestFifoScheduler failing with Metrics source QueueMetrics,q0=default
already exists!. (ahmed.radwan via tucu)
MAPREDUCE-4484. Incorrect IS_MINI_YARN_CLUSTER property name in YarnConfiguration.
(ahmed.radwan via tucu)
MAPREDUCE-4562. Support for "FileSystemCounter" legacy counter group name
for compatibility reasons is creating incorrect counter name.
(Jarek Jarcec Cecho via tomwhite)
MAPREDUCE-4068. Jars in lib subdirectory of the submittable JAR are not added to the
classpath (rkanter via tucu)
MAPREDUCE-4577. HDFS-3672 broke
TestCombineFileInputFormat.testMissingBlocks() test (atm)
MAPREDUCE-4470. Fix TestCombineFileInputFormat.testForEmptyFile (ikatsov via tucu)
MAPREDUCE-4608. hadoop-mapreduce-client is missing some dependencies.
(tucu via tomwhite)
MAPREDUCE-4610. Support deprecated mapreduce.job.counters.limit property in
MR2. (tomwhite)
MAPREDUCE-4629. Remove JobHistory.DEBUG_MODE (Karthik Kambatla via bobby)
MAPREDUCE-4642. MiniMRClientClusterFactory should not use job.setJar() (rkanter via tucu)
MAPREDUCE-4148. MapReduce should not have a compile-time dependency on
HDFS. (tomwhite)
MAPREDUCE-4250. hadoop-config.sh missing variable exports, causes Yarn
jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu)
MAPREDUCE-4002. MultiFileWordCount job fails if the input path is not
from default file system. (Bhallamudi Venkata Siva Kamesh via todd)
MAPREDUCE-4274 MapOutputBuffer should use native byte order for kvmeta.
(todd via bobby)
MAPREDUCE-4262. NM gives wrong log message saying "Connected to
ResourceManager" before trying to connect. (Devaraj K via tgraves)
MAPREDUCE-4276. Allow setting yarn.nodemanager.delete.debug-delay-sec
property to "-1" for easier container debugging. (ahmed via tucu)
MAPREDUCE-4224. TestFifoScheduler throws
org.apache.hadoop.metrics2.MetricsException (Devaraj K via tgraves)
MAPREDUCE-3493. Add the default mapreduce.shuffle.port property
to mapred-default.xml (Madhukara Phatak via harsh)
MAPREDUCE-4307. TeraInputFormat calls FileSystem.getDefaultBlockSize()
without a Path - Failure when using ViewFileSystem. (Ahmed Radwan via eli)
MAPREDUCE-4313. TestTokenCache doesn't compile due
TokenCache.getDelegationToken compilation error (bobby)
MAPREDUCE-3873. Fixed NodeManagers' decommissioning at RM to accept IP
addresses also. (xieguiming via vinodkv)
MAPREDUCE-4306. Fix distributed shell to work with users other than the one
running the daemons. (Ahmed Radwan via sseth)
MAPREDUCE-4031. Prevent a Node Manager hang during shutdown.
(Devaraj K via sseth)
MAPREDUCE-4336. Distributed Shell fails when used with the CapacityScheduler
(ahmed via tucu)
MAPREDUCE-4290. Fix Yarn Applicaiton Status to MR JobState conversion.
(Devaraj K via sseth)
MAPREDUCE-2289. Permissions race can make getStagingDir fail on local filesystem
(ahmed via tucu)
MAPREDUCE-4372. Deadlock in Resource Manager (Devaraj K via bobby)
MAPREDUCE-4376. TestClusterMRNotification times out (Kihwal Lee via bobby)
MAPREDUCE-4383. HadoopPipes.cc needs to include unistd.h.
(Andy Isaacson via eli)
MAPREDUCE-2739. Update installation docs (remove YarnClientFactory) (bowang via tucu)
MAPREDUCE-3993. Graceful handling of codec errors during decompression
(kkambatl via tucu)
MAPREDUCE-4416. Some tests fail if Clover is enabled (Kihwal Lee via bobby)
MAPREDUCE-4441. Fix build issue caused by MR-3451 (kkambatl via tucu)
HADOOP-8499. Lower min.user.id to 500 for the tests.
(Colin Patrick McCabe via eli)
MAPREDUCE-4395. Possible NPE at ClientDistributedCacheManager
#determineTimestamps (Bhallamudi via bobby)
MAPREDUCE-4380. Empty Userlogs directory is getting created under logs
directory (Devaraj K via bobby)
MAPREDUCE-4649. Ensure MapReduce JobHistory Daemon doens't assume
HADOOP_YARN_HOME and HADOOP_MAPRED_HOME are the same. (vinodkv via
acmurthy)
Release 2.0.0-alpha - 05-23-2012
INCOMPATIBLE CHANGES
MAPREDUCE-3545. Remove Avro RPC. (suresh)
NEW FEATURES
IMPROVEMENTS
MAPREDUCE-2887. Due to HADOOP-7524, change RPC to allow multiple protocols
including multuple versions of the same protocol (Sanjay Radia)
MAPREDUCE-2934. MR portion of HADOOP-7607 - Simplify the RPC proxy cleanup
process (atm)
HADOOP-7862. MR changes to work with HADOOP 7862: Move the support for
multiple protocols to lower layer so that Writable, PB and Avro can all
use it (Sanjay Radia)
MAPREDUCE-3909 Javadoc the Service interfaces (stevel)
MAPREDUCE-3885. Avoid an unnecessary copy for all requests/responses in
MRs ProtoOverHadoopRpcEngine. (Devaraj Das via sseth)
MAPREDUCE-3935. Annotate Counters.Counter and Counters.Group as @Public.
(tomwhite)
MAPREDUCE-3991. Streaming FAQ has some wrong instructions about input files
splitting. (harsh)
MAPREDUCE-3773. Add queue metrics with buckets for job run times. (omalley
via acmurthy)
MAPREDUCE-3970. Add ServiceOperations class to aid working with Services
(stevel)
MAPREDUCE-3353. Add a channel between RM and AM to get information on
nodes. (Bikas Saha via acmurthy)
MAPREDUCE-3955. Change MR to use ProtobufRpcEngine from hadoop-common
instead of ProtoOverHadoopRpcEngine. (Jitendra Nath Pandey via sseth)
MAPREDUCE-4103. Fix HA docs for changes to shell command fencer args (todd)
MAPREDUCE-4093. Improve RM WebApp start up when proxy address is not set
(Devaraj K vai bobby)
MAPREDUCE-4138. Reduce memory usage of counters due to non-static nested
classes. (tomwhite)
MAPREDUCE-3883. Document yarn.nodemanager.delete.debug-delay-sec
configuration property (Eugene Koontz via tgraves)
MAPREDUCE-4219. make default container-executor.conf.dir be a path
relative to the container-executor binary. (rvs via tucu)
MAPREDUCE-4205. retrofit all JVM shutdown hooks to use ShutdownHookManager
(tucu)
HADOOP-8285 MR changes for Use ProtoBuf for RpcPayLoadHeader (sanjay radia)
MAPREDUCE-2220. Fix new API FileOutputFormat-related typos in
mapred-default.xml (Rui Kubo via harsh)
MAPREDUCE-3907. Document entries mapred-default.xml for the
jobhistory server. (Eugene Koontz via harsh)
MAPREDUCE-3906. Fix inconsistency in documentation regarding
mapreduce.jobhistory.principal. (Eugene Koontz via harsh)
MAPREDUCE-4432. Confusing warning message when GenericOptionsParser
is not used. (Gabriel Reid via harsh)
OPTIMIZATIONS
BUG FIXES
MAPREDUCE-3740. Fixed broken mapreduce compilation after the patch for
HADOOP-7965. (Devaraj K via vinodkv)
MAPREDUCE-3818. Fixed broken compilation in TestSubmitJob after the patch
for HDFS-2895. (Suresh Srinivas via vinodkv)
MAPREDUCE-2942. TestNMAuditLogger.testNMAuditLoggerWithIP failing (Thomas
Graves via mahadev)
MAPREDUCE-3933. Failures because MALLOC_ARENA_MAX is not set (ahmed via tucu)
MAPREDUCE-3728. ShuffleHandler can't access results when configured in a
secure mode (ahmed via tucu)
MAPREDUCE-3952. In MR2, when Total input paths to process == 1,
CombinefileInputFormat.getSplits() returns 0 split. (zhenxiao via tucu)
MAPREDUCE-3578. Starting nodemanager as root gives "Unknown -jvm option"
(tomwhite)
MAPREDUCE-3348. Fixed a bug in MR client to redirect to JobHistoryServer
correctly when RM forgets the app. (Devaraj K via vinodkv)
MAPREDUCE-3974. TestSubmitJob in MR1 tests doesn't compile after HDFS-162
merge. (atm)
MAPREDUCE-4007. JobClient getJob(JobID) should return NULL if the job
does not exist (for backwards compatibility) (tucu)
MAPREDUCE-3431 NPE in Resource Manager shutdown. (stevel)
MAPREDUCE-4010. TestWritableJobConf fails on trunk (tucu via bobby)
MAPREDUCE-3992. Reduce fetcher doesn't verify HTTP status code of response
(todd)
MAPREDUCE-4066. Use default value when fetching MR_AM_STAGING_DIR
(xieguiming via harsh)
MAPREDUCE-3377. Added a unit test to ensure OutputCommitter.checkOutputSpecs
is called prior to copying job.xml. (Jane Chen via acmurthy)
MAPREDUCE-4081. TestMROutputFormat.java does not compile (Jason Lowe via
bobby)
MAPREDUCE-4082. hadoop-mapreduce-client-app's mrapp-generated-classpath
file should not be in the module JAR (tucu)
MAPREDUCE-3916. various issues with running yarn proxyserver (devaraj via tucu)
MAPREDUCE-4091. tools testcases failing because of MAPREDUCE-4082 (tucu)
MAPREDUCE-4098. TestMRApps testSetClasspath fails (tucu)
MAPREDUCE-4097. tools testcases fail because missing mrapp-generated-classpath
file in classpath (rvs via tucu)
MAPREDUCE-4113. Fix tests org.apache.hadoop.mapred.TestClusterMRNotification
(Devaraj K via bobby)
MAPREDUCE-4112. Fix tests org.apache.hadoop.mapred.TestClusterMapReduceTestCase
(Devaraj K via bobby)
MAPREDUCE-4111. Fix tests in org.apache.hadoop.mapred.TestJobName (Devaraj
K via bobby)
MAPREDUCE-4110. Fix tests in org.apache.hadoop.mapred.TestMiniMRClasspath &
org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers (Devaraj K via
bobby)
MAPREDUCE-4105. Yarn RackResolver ignores rack configurations.
(Ahmed Radwan via tomwhite)
MAPREDUCE-3869. Fix classpath for DistributedShell application. (Devaraj K
via sseth)
MAPREDUCE-4057. Update RAID for the HA and fsdataset changes. (Devaraj K
via szetszwo)
MAPREDUCE-4076. Stream job fails with ZipException when use yarn jar
command (Devaraj K via bobby)
MAPREDUCE-4108. Fix tests in org.apache.hadoop.util.TestRunJar
(Devaraj K via tgraves)
MAPREDUCE-4107. Fix tests in org.apache.hadoop.ipc.TestSocketFactory
(Devaraj K via tgraves)
MAPREDUCE-4147. YARN should not have a compile-time dependency on HDFS.
(tomwhite)
MAPREDUCE-4008. ResourceManager throws MetricsException on start up
saying QueueMetrics MBean already exists (Devaraj K via tgraves)
MAPREDUCE-3867. MiniMRYarn/MiniYarn uses fixed ports (tucu)
MAPREDUCE-4141. clover integration broken, also mapreduce poms are
pulling in clover as a dependency. (phunt via tucu)
MAPREDUCE-4193. broken doc link for yarn-default.xml in site.xml.
(phunt via tomwhite)
MAPREDUCE-4202. TestYarnClientProtocolProvider is broken (Daryn Sharp via
bobby)
MAPREDUCE-3173. MRV2 UI doesn't work properly without internet (Devaraj K
via bobby)
MAPREDUCE-3958. RM: Remove RMNodeState and replace it with NodeState
(Bikas Saha via bobby)
MAPREDUCE-4231. Update RAID to use the new BlockCollection interface.
(szetszwo)
MAPREDUCE-4483. 2.0 build does not work (John George via bobby)
MAPREDUCE-4444. nodemanager fails to start when one of the local-dirs is
bad (Jason Lowe via bobby)
Release 0.23.7 - UNRELEASED
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
MAPREDUCE-4905. test org.apache.hadoop.mapred.pipes
(Aleksey Gorshkov via bobby)
MAPREDUCE-4989. JSONify DataTables input data for Attempts page (Ravi
Prakash via jlowe)
MAPREDUCE-5027. Shuffle does not limit number of outstanding connections
(Robert Parker via jeagles)
OPTIMIZATIONS
MAPREDUCE-4946. Fix a performance problem for large jobs by reducing the
number of map completion event type conversions. (Jason Lowe via sseth)
MAPREDUCE-4822. Unnecessary conversions in History Events. (Chu Tong via
jlowe)
BUG FIXES
MAPREDUCE-4458. Warn if java.library.path is used for AM or Task
(Robert Parker via jeagles)
MAPREDUCE-4992. AM hangs in RecoveryService when recovering tasks with
speculative attempts (Robert Parker via jlowe)
MAPREDUCE-5009. Killing the Task Attempt slated for commit does not clear
the value from the Task commitAttempt member (Robert Parker via jeagles)
MAPREDUCE-4871. AM uses mapreduce.jobtracker.split.metainfo.maxsize but
mapred-default has mapreduce.job.split.metainfo.maxsize (Jason Lowe via
jeagles)
MAPREDUCE-4794. DefaultSpeculator generates error messages on normal
shutdown (Jason Lowe via jeagles)
MAPREDUCE-5043. Fetch failure processing can cause AM event queue to
backup and eventually OOM (Jason Lowe via bobby)
Release 0.23.6 - UNRELEASED
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
MAPREDUCE-4811. JobHistoryServer should show when it was started in WebUI
About page (Ravi Prakash via jlowe)
OPTIMIZATIONS
BUG FIXES
MAPREDUCE-4802. Takes a long time to load the task list on the AM for
large jobs (Ravi Prakash via bobby)
MAPREDUCE-4764. repair TestBinaryTokenFile (Ivan A. Veselovsky via bobby)
MAPREDUCE-4825. JobImpl.finished doesn't expect ERROR as a final job state
(jlowe via bobby)
MAPREDUCE-4817. Hardcoded task ping timeout kills tasks localizing large
amounts of data (tgraves)
MAPREDUCE-4836. Elapsed time for running tasks on AM web UI tasks page is 0
(Ravi Prakash via jeagles)
MAPREDUCE-4842. Shuffle race can hang reducer (Mariappan Asokan via jlowe)
MAPREDUCE-4833. Task can get stuck in FAIL_CONTAINER_CLEANUP (Robert
Parker via jlowe)
MAPREDUCE-4793. Problem with adding resources when using both -files and
-file to hadoop streaming (jlowe)
MAPREDUCE-4890. Invalid TaskImpl state transitions when task fails while
speculating (jlowe)
MAPREDUCE-4902. Fix typo "receievd" should be "received" in log output
(Albert Chu via jlowe)
MAPREDUCE-4813. AM timing out during job commit (jlowe via bobby)
MAPREDUCE-4279. getClusterStatus() fails with null pointer exception when
running jobs in local mode (Devaraj K via bobby)
MAPREDUCE-4832. MR AM can get in a split brain situation (jlowe)
MAPREDUCE-4894. Renewal / cancellation of JobHistory tokens (Siddharth
Seth via tgraves)
MAPREDUCE-4819. AM can rerun job after reporting final job status to the
client (bobby and Bikas Saha via bobby)
MAPREDUCE-4913. TestMRAppMaster#testMRAppMasterMissingStaging occasionally
exits (Jason Lowe via tgraves)
MAPREDUCE-4848. TaskAttemptContext cast error during AM recovery (Jerry
Chen via jlowe)
MAPREDUCE-4921. JobClient should acquire HS token with RM principal
(daryn via bobby)
MAPREDUCE-4934. Maven RAT plugin is not checking all source files (tgraves)
MAPREDUCE-4678. Running the Pentomino example with defaults throws
java.lang.NegativeArraySizeException (Chris McConnell via harsh)
MAPREDUCE-4925. The pentomino option parser may be buggy.
(Karthik Kambatla via harsh)
Release 0.23.5 - 2012-11-28
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
MAPREDUCE-4596. Split StateMachine state from states seen by MRClientProtocol
for Job, Task and TaskAttempt. (Siddarth Seth via vinodkv)
MAPREDUCE-4752. Reduce MR AM memory usage through String Interning (Robert
Evans via tgraves)
MAPREDUCE-4266. remove Ant remnants from MR (tgraves via bobby)
MAPREDUCE-4666. JVM metrics for history server (jlowe via jeagles)
OPTIMIZATIONS
MAPREDUCE-4720. Browser thinks History Server main page JS is taking too
long (Ravi Prakash via bobby)
BUG FIXES
MAPREDUCE-4554. Job Credentials are not transmitted if security is turned
off (Benoy Antony via bobby)
MAPREDUCE-4705. Fix a bug in job history lookup, which makes older jobs
inaccessible despite the presence of a valid history file. (Jason Lowe
via sseth)
MAPREDUCE-4521. mapreduce.user.classpath.first incompatibility with 0.20/1.x
(Ravi Prakash via bobby)
MAPREDUCE-4721. Task startup time in JHS is same as job startup time.
(Ravi Prakash via bobby)
MAPREDUCE-4479. Fix parameter order in assertEquals() in
TestCombineInputFileFormat.java (Mariappan Asokan via bobby)
MAPREDUCE-4733. Reducer can fail to make progress during shuffle if too many
reducers complete consecutively. (Jason Lowe via vinodkv)
MAPREDUCE-4740. only .jars can be added to the Distributed Cache
classpath. (Robert Joseph Evans via jlowe)
MAPREDUCE-4229. Intern counter names in the JT (Miomir Boljanovic and bobby via daryn)
MAPREDUCE-4741. WARN and ERROR messages logged during normal AM shutdown.
(Vinod Kumar Vavilapalli via jlowe)
MAPREDUCE-4730. Fix Reducer's EventFetcher to scale the map-completion
requests slowly to avoid HADOOP-8942. (Jason Lowe via vinodkv)
MAPREDUCE-4748. Invalid event: T_ATTEMPT_SUCCEEDED at SUCCEEDED. (jlowe)
MAPREDUCE-4724. job history web ui applications page should be sorted to
display last app first (tgraves via bobby)
MAPREDUCE-4746. The MR Application Master does not have a config to set
environment variables (Rob Parker via bobby)
MAPREDUCE-4729. job history UI not showing all job attempts. (Vinod
Kumar Vavilapalli via jlowe)
MAPREDUCE-4763 repair test TestUmbilicalProtocolWithJobToken (Ivan A.
Veselovsky via bobby)
MAPREDUCE-4771. KeyFieldBasedPartitioner not partitioning properly when
configured (jlowe via bobby)
MAPREDUCE-4772. Fetch failures can take way too long for a map to be
restarted (bobby)
MAPREDUCE-4782. NLineInputFormat skips first line of last InputSplit
(Mark Fuhs via bobby)
MAPREDUCE-4774. JobImpl does not handle asynchronous task events in FAILED
state (jlowe via bobby)
MAPREDUCE-4751. AM stuck in KILL_WAIT for days (vinodkv via bobby)
MAPREDUCE-4787. TestJobMonitorAndPrint is broken (Rob Parker via bobby)
MAPREDUCE-4425. Speculation + Fetch failures can lead to a hung job (jlowe
via bobby)
MAPREDUCE-4786. Job End Notification retry interval is 5 milliseconds by
default (Ravi Prakash via bobby)
MAPREDUCE-4517. Too many INFO messages written out during AM to RM heartbeat
(Jason Lowe via tgraves)
MAPREDUCE-4797. LocalContainerAllocator can loop forever trying to contact
the RM (jlowe via bobby)
MAPREDUCE-4801. ShuffleHandler can generate large logs due to prematurely
closed channels (jlowe via bobby)
Release 0.23.4
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
MAPREDUCE-2786. Add compression option for TestDFSIO.
(Plamen Jeliazkov via shv)
MAPREDUCE-4645. Provide a random seed to Slive to make the sequence
of file names deterministic. (Ravi Prakash via shv)
MAPREDUCE-4651. Benchmarking random reads with DFSIO. (shv)
OPTIMIZATIONS
BUG FIXES
MAPREDUCE-4647. We should only unjar jobjar if there is a lib directory
in it. (Robert Evans via tgraves)
MAPREDUCE-4691. Historyserver can report "Unknown job" after RM says job
has completed (Robert Joseph Evans via jlowe)
MAPREDUCE-4689. JobClient.getMapTaskReports on failed job results in NPE
(jlowe via bobby)
Release 0.23.3
INCOMPATIBLE CHANGES
MAPREDUCE-4072. User set java.library.path seems to overwrite default
creating problems native lib loading (Anupam Seth via bobby)
MAPREDUCE-3812. Lower default allocation sizes, fix allocation
configurations and document them (Harsh J via bobby)
NEW FEATURES
IMPROVEMENTS
MAPREDUCE-4059. The history server should have a separate pluggable
storage/query interface. (Robert Evans via tgraves)
MAPREDUCE-3942. Randomize master key generation for
ApplicationTokenSecretManager and roll it every so often. (Vinod Kumar
Vavilapalli via sseth)
MAPREDUCE-4151. RM scheduler web page should filter apps to those that
are relevant to scheduling (Jason Lowe via tgraves)
MAPREDUCE-4134. Remove references of mapred.child.ulimit etc. since they
are not being used any more (Ravi Prakash via bobby)
MAPREDUCE-3972. Fix locking and exception issues in JobHistory server.
(Robert Joseph Evans via sseth)
MAPREDUCE-4161. create sockets consistently (Daryn Sharp via bobby)
MAPREDUCE-4079. Allow MR AppMaster to limit ephemeral port range.
(bobby via tgraves)
MAPREDUCE-4210. Expose listener address for WebApp (Daryn Sharp via bobby)
MAPREDUCE-4162. Correctly set token service (Daryn Sharp via bobby)
MAPREDUCE-4301. Dedupe some strings in MRAM for memory savings
(bobby via tgraves)
MAPREDUCE-4267. mavenize pipes (tgraves via bobby)
MAPREDUCE-4375. Show Configuration Tracability in MR UI (bobby
via tgraves)
MAPREDUCE-4569. Fixed TestHsWebServicesJobsQuery to pass on JDK7 by not
depending on test order. (Thomas Graves via vinodkv)
OPTIMIZATIONS
MAPREDUCE-3850. Avoid redundant calls for tokens in TokenCache (Daryn
Sharp via bobby)
BUG FIXES
MAPREDUCE-4092. commitJob Exception does not fail job (Jon Eagles via
bobby)
MAPREDUCE-4089. Hung Tasks never time out. (Robert Evans via tgraves)
MAPREDUCE-4024. RM webservices can't query on finalStatus (Tom Graves
via bobby)
MAPREDUCE-4060. Multiple SLF4J binding warning (Jason Lowe via bobby)
MAPREDUCE-3983. TestTTResourceReporting can fail, and should just be
deleted (Ravi Prakash via bobby)
MAPREDUCE-4012 Hadoop Job setup error leaves no useful info to users
(when LinuxTaskController is used). (tgraves)
MAPREDUCE-4062. AM Launcher thread can hang forever (tgraves via bobby)
MAPREDUCE-3988. mapreduce.job.local.dir doesn't point to a single directory
on a node. (Eric Payne via bobby)
MAPREDUCE-3999. Tracking link gives an error if the AppMaster hasn't
started yet (Ravi Prakash via bobby)
MAPREDUCE-4020. Web services returns incorrect JSON for deep queue tree
(Anupam Seth via tgraves)
MAPREDUCE-3672. Killed maps shouldn't be counted towards
JobCounter.NUM_FAILED_MAPS. (Anupam Seth via tgraves)
MAPREDUCE-3682 Tracker URL says AM tasks run on localhost.
(Ravi Prakash via tgraves)
MAPREDUCE-3082. Archive command take wrong path for input file with current
directory (John George via bobby)
MAPREDUCE-3650. testGetTokensForHftpFS() fails (Ravi Prakash via bobby)
MAPREDUCE-3621. TestDBJob and TestDataDrivenDBInputFormat ant tests fail
(Ravi Prakash via tgraves)
MAPREDUCE-4073. CS assigns multiple off-switch containers when using
multi-level-queues (Siddharth Seth via bobby)
MAPREDUCE-4051. Remove the empty hadoop-mapreduce-project/assembly/all.xml
file (Ravi Prakash via bobby)
MAPREDUCE-4117. mapred job -status throws NullPointerException (Devaraj K
via bobby)
MAPREDUCE-4099. ApplicationMaster may fail to remove staging directory
(Jason Lowe via bobby)
MAPREDUCE-4017. Add jobname to jobsummary log (tgraves and Koji Noguchi
via bobby)
MAPREDUCE-4040. History links should use hostname rather than IP address.
(Bhallamudi Venkata Siva Kamesh via sseth)
MAPREDUCE-4099 amendment. ApplicationMaster will remove staging directory
after the history service is stopped. (Jason Lowe via sseth)
MAPREDUCE-3932. Fix the TaskAttempt state machine to handle
CONTIANER_LAUNCHED and CONTIANER_LAUNCH_FAILED events in additional
states. (Robert Joseph Evans via sseth)
MAPREDUCE-4140. mapreduce classes incorrectly importing
"clover.org.apache.*" classes. (Patrick Hunt via tomwhite)
MAPREDUCE-4050. For tasks without assigned containers, changes the node
text on the UI to N/A instead of a link to null. (Bhallamudi Venkata Siva
Kamesh via sseth)
MAPREDUCE-4128. AM Recovery expects all attempts of a completed task to
also be completed. (Bikas Saha via bobby)
MAPREDUCE-4144. Fix a NPE in the ResourceManager when handling node
updates. (Jason Lowe via sseth)
MAPREDUCE-4156. ant build fails compiling JobInProgress (tgraves)
MAPREDUCE-4160. some mrv1 ant tests fail with timeout - due to 4156
(tgraves)
MAPREDUCE-4074. Client continuously retries to RM When RM goes down
before launching Application Master (xieguiming via tgraves)
MAPREDUCE-4159. Job is running in Uber mode after setting
"mapreduce.job.ubertask.maxreduces" to zero (Devaraj K via bobby)
MAPREDUCE-4165. Committing is misspelled as commiting in task logs
(John Eagles via bobby)
MAPREDUCE-4129. Lots of unneeded counters log messages (Ahmed Radwan via
bobby)
MAPREDUCE-3947. yarn.app.mapreduce.am.resource.mb not documented
(Devaraj K via bobby)
MAPREDUCE-4190. Improve web UI for task attempts userlog link (Tom Graves
via bobby)
MAPREDUCE-4133. MR over viewfs is broken (John George via bobby)
MAPREDUCE-4194. ConcurrentModificationError in DirectoryCollection
(Jonathan Eagles via bobby)
MAPREDUCE-3613. web service calls header contains 2 content types
(tgraves)
MAPREDUCE-4169. Container Logs appear in unsorted order (Jonathan Eagles
via bobby)
MAPREDUCE-4189. TestContainerManagerSecurity is failing (Devaraj K via
bobby)
MAPREDUCE-4209. junit dependency in hadoop-mapreduce-client is missing
scope test (Radim Kolar via bobby)
MAPREDUCE-4206. Sorting by Last Health-Update on the RM nodes page sorts
does not work correctly (Jonathon Eagles via tgraves)
MAPREDUCE-4212. TestJobClientGetJob sometimes fails
(Daryn Sharp via tgraves)
MAPREDUCE-4211. Error conditions (missing appid, appid not found) are
masked in the RM app page (Jonathan Eagles via bobby)
MAPREDUCE-4163. consistently set the bind address (Daryn Sharp via bobby)
MAPREDUCE-4048. NullPointerException exception while accessing the
Application Master UI (Devaraj K via bobby)
MAPREDUCE-4220. RM apps page starttime/endtime sorts are incorrect
(Jonathan Eagles via bobby)
MAPREDUCE-4226. ConcurrentModificationException in FileSystemCounterGroup.
(tomwhite)
MAPREDUCE-4215. RM app page shows 500 error on appid parse error
(Jonathon Eagles via tgraves)
MAPREDUCE-4237. TestNodeStatusUpdater can fail if localhost has a domain
associated with it (bobby)
MAPREDUCE-4233. NPE can happen in RMNMNodeInfo. (bobby)
MAPREDUCE-4238. mavenize data_join. (tgraves)
MAPREDUCE-4102. job counters not available in Jobhistory webui for
killed jobs (Bhallamudi Venkata Siva Kamesh via tgraves)
MAPREDUCE-3543. Mavenize Gridmix. (tgraves)
MAPREDUCE-4197. Include the hsqldb jar in the hadoop-mapreduce tar
file (Ravi Prakash via tgraves)
MAPREDUCE-4269. documentation: Gridmix has javadoc warnings in
StressJobFactory (Jonathon Eagles via tgraves).
MAPREDUCE-3870. Invalid App Metrics
(Bhallamudi Venkata Siva Kamesh via tgraves).
MAPREDUCE-4152. map task left hanging after AM dies trying to connect to RM
(Tom Graves via bobby)
MAPREDUCE-4297. Usersmap file in gridmix should not fail on empty lines
(Ravi Prakash via bobby)
MAPREDUCE-4302. NM goes down if error encountered during log aggregation
(Daryn Sharp via bobby)
MAPREDUCE-3350. Per-app RM page should have the list of application-attempts
like on the app JHS page (Jonathon Eagles via tgraves)
MAPREDUCE-3842. Stop webpages from automatic refreshing (tgraves)
MAPREDUCE-3927. Shuffle hang when set map.failures.percent
(Bhallamudi Venkata Siva Kamesh via tgraves)
MAPREDUCE-4311. Capacity scheduler.xml does not accept decimal values for
capacity and maximum-capacity settings (Karthik Kambatla via tgraves)
MAPREDUCE-4341. add types to capacity scheduler properties documentation
(Karthik Kambatla via tgraves)
MAPREDUCE-4270. Move the data_join test classes to the correct path.
(Thomas Graves via sseth)
MAPREDUCE-3889. job client tries to use /tasklog interface, but that
doesn't exist anymore (Devaraj K via bobby)
MAPREDUCE-4320. gridmix mainClass wrong in pom.xml (tgraves)
MAPREDUCE-4295. RM crashes due to DNS issue (tgraves)
MAPREDUCE-4228. mapreduce.job.reduce.slowstart.completedmaps is not working
properly (Jason Lowe via bobby)
MAPREDUCE-4392. Counters.makeCompactString() changed behavior from 0.20
(Jason Lowe via bobby)
MAPREDUCE-4384. Race conditions in IndexCache (Kihwal Lee via tgraves)
MAPREDUCE-4387. RM gets fatal error and exits during TestRM
(Kihwal Lee via tgraves)
MAPREDUCE-4379. Node Manager throws java.lang.OutOfMemoryError: Java heap
space due to org.apache.hadoop.fs.LocalDirAllocator.contexts (Devaraj K
via bobby)
MAPREDUCE-4402. TestFileInputFormat fails intermittently (Jason Lowe via
bobby)
MAPREDUCE-4300. OOM in AM can turn it into a zombie. (Robert Evans via
tgraves)
MAPREDUCE-4252. MR2 job never completes with 1 pending task (Tom White via
bobby)
MAPREDUCE-3940. ContainerTokens should have an expiry interval. (Siddharth
Seth and Vinod Kumar Vavilapalli via vinodkv)
MAPREDUCE-4419. ./mapred queue -info <queuename> -showJobs displays all
the jobs irrespective of <queuename> (Devaraj K via bobby)
MAPREDUCE-4299. Terasort hangs with MR2 FifoScheduler (Tom White via
bobby)
MAPREDUCE-4437. Race in MR ApplicationMaster can cause reducers to never be
scheduled (Jason Lowe via bobby)
MAPREDUCE-4449. Incorrect MR_HISTORY_STORAGE property name in JHAdminConfig
(Ahmed Radwan via bobby)
MAPREDUCE-4283. Display tail of aggregated logs by default (Jason Lowe via
bobby)
MAPREDUCE-4448. Fix NM crash during app cleanup if aggregation didn't
init. (Jason Lowe via daryn)
MAPREDUCE-3893. allow capacity scheduler configs maximum-applications and
maximum-am-resource-percent configurable on a per queue basis (tgraves via
bobby)
MAPREDUCE-4467. IndexCache failures due to missing synchronization
(Kihwal Lee via tgraves)
MAPREDUCE-4423. Potential infinite fetching of map output (Robert Evans
via tgraves)
MAPREDUCE-4456. LocalDistributedCacheManager can get an
ArrayIndexOutOfBounds when creating symlinks (Robert Evans via tgraves)
MAPREDUCE-4496. AM logs link is missing user name (Jason Lowe via bobby)
MAPREDUCE-4493. Distibuted Cache Compatability Issues (Robert Evans
via tgraves)
MAPREDUCE-4492. Configuring total queue capacity between 100.5 and 99.5 at
perticular level is sucessfull (Mayank Bansal via bobby)
MAPREDUCE-4457. mr job invalid transition TA_TOO_MANY_FETCH_FAILURE at
FAILED (Robert Evans via tgraves)
MAPREDUCE-4234. SortValidator.java is incompatible with multi-user or
parallel use (due to a /tmp file with static name) (Robert Evans via
jeagles)
MAPREDUCE-4504. SortValidator writes to wrong directory (Robert Evans
via tgraves)
MAPREDUCE-4503. Should throw InvalidJobConfException if duplicates found in
cacheArchives or cacheFiles (Robert Evans via jeagles)
MAPREDUCE-3782. teragen terasort jobs fail when using webhdfs:// (Jason
Lowe via bobby)
MAPREDUCE-4053. Counters group names deprecation is wrong, iterating over
group names deprecated names don't show up (Robert Evans via tgraves)
MAPREDUCE-3506. Calling getPriority on JobInfo after parsing a history log
with JobHistoryParser throws a NullPointerException (Jason Lowe via bobby)
MAPREDUCE-4570. ProcfsBasedProcessTree#constructProcessInfo() prints a
warning if procfsDir/<pid>/stat is not found. (Ahmed Radwan via bobby)
MAPREDUCE-4600. TestTokenCache.java from MRV1 no longer compiles (daryn
via bobby)
MAPREDUCE-4612. job summary file permissions not set when its created
(tgraves via bobby)
MAPREDUCE-4614. Simplify debugging a job's tokens (daryn via bobby)
MAPREDUCE-4611. MR AM dies badly when Node is decommissioned (Robert
Evans via tgraves)
MAPREDUCE-4604. In mapred-default, mapreduce.map.maxattempts &
mapreduce.reduce.maxattempts defaults are set to 4 as well as
mapreduce.job.maxtaskfailures.per.tracker. (Ravi Prakash via jeagles)
MAPREDUCE-4633. history server doesn't set permissions on all subdirs
(tgraves via bobby)
MAPREDUCE-4641. Exception in commitJob marks job as successful in job
history (Jason Lowe via bobby)
MAPREDUCE-4549. Distributed cache conflicts breaks backwards compatability
(Robert Evans via tucu)
Release 0.23.2 - UNRELEASED
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
MAPREDUCE-3849. Change TokenCache's reading of the binary token file
(Daryn Sharp via bobby)
MAPREDUCE-3854. Fixed and reenabled tests related to MR child JVM's
environmental variables in TestMiniMRChildTask. (Tom White via vinodkv)
MAPREDUCE-3877 Add a test to formalise the current state transitions
of the yarn lifecycle. (stevel)
MAPREDUCE-3866. Fixed the bin/yarn script to not print the command line
unnecessarily. (vinodkv)
MAPREDUCE-3730. Modified RM to allow restarted NMs to be able to join the
cluster without waiting for expiry. (Jason Lowe via vinodkv)
MAPREDUCE-2793. Corrected AppIDs, JobIDs, TaskAttemptIDs to be of correct
format on the web pages. (Bikas Saha via vinodkv)
MAPREDUCE-3614. Fixed MR AM to close history file quickly and send a correct
final state to the RM when it is killed. (Ravi Prakash via vinodkv)
MAPREDUCE-3497. Added docs for YARN CLI. (tgraves via acmurthy)
MAPREDUCE-3954. Added new envs to separate heap size for different daemons
started via bin scripts. (Robert Joseph Evans via vinodkv)
MAPREDUCE-4025. AM can crash if task attempt reports bogus progress value
(Jason Lowe via bobby)
MAPREDUCE-4034. Unable to view task logs on history server with
mapreduce.job.acl-view-job=* (Jason Lowe and Siddarth Seth via bobby)
MAPREDUCE-4043. Secret keys set in Credentials are not seen by tasks
(Jason Lowe via bobby)
MAPREDUCE-3989. Cap space usage of default log4j rolling policy.
(Patrick Hunt via eli)
OPTIMIZATIONS
MAPREDUCE-3901. Modified JobHistory records in YARN to lazily load job and
task reports so as to improve UI response times. (Siddarth Seth via vinodkv)
MAPREDUCE-2855. Passing a cached class-loader to ResourceBundle creator to
minimize counter names lookup time. (Siddarth Seth via vinodkv)
MAPREDUCE-3944. Change the history jobs/ webservice to return partial job
info for a significant performance improvement. (Robert Joseph Evans via
sseth)
BUG FIXES
MAPREDUCE-3918 proc_historyserver no longer in command line arguments for
HistoryServer (Jon Eagles via bobby)
MAPREDUCE-3862. Nodemanager can appear to hang on shutdown due to lingering
DeletionService threads (Jason Lowe via bobby)
MAPREDUCE-3680. FifoScheduler web service rest API can print out invalid
JSON. (B Anil Kumar via tgraves)
MAPREDUCE-3852. Test TestLinuxResourceCalculatorPlugin failing. (Thomas
Graves via mahadev)
MAPREDUCE-3864. Fix cluster setup docs for correct SecondaryNameNode
HTTPS parameters. (todd)
MAPREDUCE-3583. Change pid to String and stime to BigInteger in order to
avoid NumberFormatException caused by overflow. (Zhihong Yu via szetszwo)
MAPREDUCE-3634. Fixed all daemons to crash instead of hanging around when
their EventHandlers get exceptions. (vinodkv)
MAPREDUCE-3798. Fixed failing TestJobCleanup.testCusomCleanup() and moved it
to the maven build. (Ravi Prakash via vinodkv)
MAPREDUCE-3884. PWD should be first in the classpath of MR tasks (tucu)
MAPREDUCE-3878. Null user on filtered jobhistory job page (Jonathon Eagles
via tgraves)
MAPREDUCE-3738. MM can hang during shutdown if AppLogAggregatorImpl thread
dies unexpectedly (Jason Lowe via sseth)
MAPREDUCE-3904 Job history produced with mapreduce.cluster.acls.enabled
false can not be viewed with mapreduce.cluster.acls.enabled true
(Jonathon Eagles via tgraves)
MAPREDUCE-3910. Fixed a bug in CapacityScheduler LeafQueue which was causing
app-submission to fail. (John George via vinodkv)
MAPREDUCE-3686. Fixed two bugs in Counters because of which web app displays
zero counter values for framework counters. (Bhallamudi Venkata Siva Kamesh
via vinodkv)
MAPREDUCE-3913. RM application webpage is unresponsive after 2000 jobs
(Jason Lowe via tgraves)
MAPREDUCE-3922. Fixed build to not compile 32bit container-executor binary
by default on all platforms. (Hitesh Shah via vinodkv)
MAPREDUCE-3790. Broken pipe on streaming job can lead to truncated output for
a successful job (Jason Lowe via bobby)
MAPREDUCE-3816. capacity scheduler web ui bar graphs for used capacity wrong
(tgraves via bobby)
MAPREDUCE-3930. Fixed an NPE while accessing the AM page/webservice for a
task attempt without an assigned container. (Robert Joseph Evans via
sseth)
MAPREDUCE-3931. Changed PB implementation of LocalResource to take locks
so that race conditions don't fail tasks by inadvertantly changing the
timestamps. (Siddarth Seth via vinodkv)
MAPREDUCE-3687. If AM dies before it returns new tracking URL, proxy
redirects to http://N/A/ and doesn't return error code (Ravi Prakash via
bobby)
MAPREDUCE-3920. Revise yarn default port number selection
(Dave Thompson via tgraves)
MAPREDUCE-3903. Add support for mapreduce admin users. (Thomas Graves via
sseth)
MAPREDUCE-3706. Fix circular redirect error in job-attempts page. (bobby
via acmurthy)
MAPREDUCE-3896. Add user information to the delegation token issued by the
history server. (Vinod Kumar Vavilapalli via sseth)
MAPREDUCE-3792. Fix "bin/mapred job -list" to display all jobs instead of
only the jobs owned by the user. (Jason Lowe via vinodkv)
MAPREDUCE-3929. Fixed output of 'bin/mapred queue -showacl' command to
clarify ACLs for users. (John George via acmurthy)
MAPREDUCE-3960. Fix web-proxy to forward request to AM with configured
hostname or IP. (tgraves via acmurthy)
MAPREDUCE-3897. Fixed computation of maxActiveAppsPerUser for queues by
using capacity and not max-capacity since we are already scaling it by
userLimitFactor. (Eric Payne via acmurthy)
MAPREDUCE-3009. Fixed node link on JobHistory webapp. (chackaravarthy via
vinodkv)
MAPREDUCE-3964. ResourceManager does not have JVM metrics (Jason Lowe via
bobby)
MAPREDUCE-3034. Ensure NodeManager reboots itself on direction from
ResourceManager. (Devaraj K & Eric Payne via acmurthy)
MAPREDUCE-3976. TestRMContainerAllocator failing (Jason Lowe via bobby)
MAPREDUCE-3961. Map/ReduceSlotMillis computation incorrect (Siddharth Seth
via bobby)
MAPREDUCE-3977. LogAggregationService leaks log aggregator objects
(Jason Lowe via bobby)
MAPREDUCE-3975. Default value not set for Configuration parameter
mapreduce.job.local.dir (Eric Payne via bobby)
MAPREDUCE-3982. Fixed FileOutputCommitter to not err out for an 'empty-job'
whose tasks don't write any outputs. (Robert Joseph Evans via vinodkv)
MAPREDUCE-4005. Fixes broken AM container logs URL on ResourceManager
Application Info page. (Jason Lowe via sseth)
MAPREDUCE-4006. History server container log web UI sometimes combines
stderr/stdout/syslog contents together (Siddharth Seth via tgraves)
MAPREDUCE-4061. RM only has 1 AM launcher thread (tgraves via bobby)
Release 0.23.1 - 2012-02-17
NEW FEATURES
MAPREDUCE-778. Rumen Anonymizer. (Amar Kamat and Chris Douglas via amarrk)
MAPREDUCE-3121. NodeManager should handle disk-failures (Ravi Gummadi via mahadev)
MAPREDUCE-2863. Support web services for YARN and MR components. (Thomas
Graves via vinodkv)
MAPREDUCE-3251. Network ACLs can prevent some clients to talk to MR ApplicationMaster.
(Anupam Seth via mahadev)
MAPREDUCE-778. Rumen Anonymizer. (Amar Kamat and Chris Douglas via amarrk)
IMPROVEMENTS
MAPREDUCE-3481. [Gridmix] Improve Gridmix STRESS mode. (amarrk)
MAPREDUCE-3597. [Rumen] Rumen should provide APIs to access all the
job-history related information.
MAPREDUCE-3375. [Gridmix] Memory Emulation system tests.
(Vinay Thota via amarrk)
MAPREDUCE-3840. JobEndNotifier doesn't use the proxyToUse during connecting
(Ravi Prakash via bobby)
MAPREDUCE-3736. Variable substitution depth too large for fs.default.name
causes jobs to fail (ahmed via tucu).
MAPREDUCE-2733. [Gridmix] Gridmix3 cpu emulation system tests.
(Vinay Thota via amarrk)
MAPREDUCE-3297. Moved log related components into yarn-common so that
HistoryServer and clients can use them without depending on the
yarn-server-nodemanager module. (Siddharth Seth via vinodkv)
MAPREDUCE-3336. Replaced guice internal.Preconditions api usage with the
public Preconditions API. (Thomas Graves via vinodkv)
MAPREDUCE-3280. Removed the unnecessary job user-name configuration in
mapred-site.xml. (vinodkv)
MAPREDUCE-3370. Fixed MiniMRYarnCluster and related tests to not use
a hard-coded path for the mr-app jar. (Ahmed Radwan via vinodkv)
MAPREDUCE-3325. Improvements to CapacityScheduler doc. (Thomas Graves
via mahadev)
MAPREDUCE-3341. Enhance logging of initalized queue limit values.
(Anupam Seth via mahadev)
MAPREDUCE-3243. Invalid tracking URL for streaming jobs (Jonathan Eagles
via mahadev)
MAPREDUCE-3331. Improvement to single node cluster setup documentation for
0.23 (Anupam Seth via mahadev)
MAPREDUCE-3102. Changed NodeManager to fail fast when LinuxContainerExecutor
has wrong configuration or permissions. (Hitesh Shah via vinodkv)
MAPREDUCE-3415. improve MiniMRYarnCluster & DistributedShell JAR
resolution. (tucu)
MAPREDUCE-3169. Create a new MiniMRCluster equivalent which only provides
client APIs cross MR1 and MR2. (Ahmed via tucu)
MAPREDUCE-3373. Hadoop scripts unconditionally source
"$bin"/../libexec/hadoop-config.sh. (Bruno Mahé via tomwhite)
MAPREDUCE-3372. HADOOP_PREFIX cannot be overridden.
(Bruno Mahé via tomwhite)
MAPREDUCE-3411. Performance Upgrade for jQuery (Jonathan Eagles via
mahadev)
MAPREDUCE-3371. Review and improve the yarn-api javadocs. (Ravi Prakash
via mahadev)
MAPREDUCE-3238. Small cleanup in SchedulerApp. (Todd Lipcon via mahadev)
MAPREDUCE-3413. RM web ui applications not sorted in any order by default.
(Jonathan Eagles via mahadev)
MAPREDUCE-3045. Fixed UI filters to not filter on hidden title-numeric
sort fields. (Jonathan Eagles via sseth)
MAPREDUCE-3448. TestCombineOutputCollector javac unchecked warning on mocked
generics (Jonathan Eagles via mahadev)
MAPREDUCE-3169 amendment. Deprecate MiniMRCluster. (Ahmed Radwan via
sseth)
MAPREDUCE-3369. Migrate MR1 tests to run on MR2 using the new interfaces
introduced in MAPREDUCE-3169. (Ahmed Radwan via tomwhite)
MAPREDUCE-3518. mapred queue -info <queue> -showJobs throws NPE.
(Jonathan Eagles via mahadev)
MAPREDUCE-3391. Making a trivial change to correct a log message in
DistributedShell app's AM. (Subroto Sanyal via vinodkv)
MAPREDUCE-3547. Added a bunch of unit tests for the the RM/NM webservices.
(Thomas Graves via acmurthy)
MAPREDUCE-3610. Remove use of the 'dfs.block.size' config for default block
size fetching. Use FS#getDefaultBlocksize instead. (Sho Shimauchi via harsh)
MAPREDUCE-3478. Cannot build against ZooKeeper 3.4.0. (Tom White via mahadev)
MAPREDUCE-3528. Fixed TaskHeartBeatHandler to use a new configuration
for the thread loop interval separate from task-timeout configuration
property. (Siddharth Seth via vinodkv)
MAPREDUCE-3312. Modified MR AM to not send a stop-container request for
a container that isn't launched at all. (Robert Joseph Evans via vinodkv)
MAPREDUCE-3382. Enhanced MR AM to use a proxy to ping the job-end
notification URL. (Ravi Prakash via vinodkv)
MAPREDUCE-3299. Added AMInfo table to the MR AM job pages to list all the
job-attempts when AM restarts and recovers. (Jonathan Eagles via vinodkv)
MAPREDUCE-3251. Network ACLs can prevent some clients to talk to MR AM.
Improved the earlier patch to not to JobHistoryServer repeatedly.
(Anupam Seth via vinodkv)
MAPREDUCE-3553. Add support for data returned when exceptions thrown from web
service apis to be in either xml or in JSON. (Thomas Graves via mahadev)
MAPREDUCE-3641. Making CapacityScheduler more conservative so as to
assign only one off-switch container in a single scheduling
iteration. (Arun C Murthy via vinodkv)
MAPREDUCE-3692. yarn-resourcemanager out and log files can get big. (eli)
MAPREDUCE-3710. Improved FileInputFormat to return better locality for the
last split. (Siddarth Seth via vinodkv)
MAPREDUCE-2765. DistCp Rewrite. (Mithun Radhakrishnan via mahadev)
MAPREDUCE-3737. The Web Application Proxy's is not documented very well.
(Robert Evans via mahadev)
MAPREDUCE-3699. Increased RPC handlers for all YARN servers to reasonable
values for working at scale. (Hitesh Shah via vinodkv)
MAPREDUCE-3693. Added mapreduce.admin.user.env to mapred-default.xml.
(Roman Shapshonik via acmurthy)
MAPREDUCE-3732. Modified CapacityScheduler to use only users with pending
requests for computing user-limits. (Arun C Murthy via vinodkv)
MAPREDUCE-3679. AM logs and others should not automatically refresh after every 1
second. (Vinod KV via mahadev)
MAPREDUCE-3754. Modified RM UI to filter applications based on state of the
applications. (vinodkv)
MAPREDUCE-3774. Moved yarn-default.xml to hadoop-yarn-common from
hadoop-server-common. (Mahadev Konar via vinodkv)
MAPREDUCE-3771. Un-deprecated the old mapred apis, port of MAPREDUCE-1735.
(acmurthy)
MAPREDUCE-3784. Fixed CapacityScheduler so that maxActiveApplications and
maxActiveApplicationsPerUser per queue are not too low for small
clusters. (Arun C Murthy via vinodkv)
OPTIMIZATIONS
MAPREDUCE-3567. Extraneous JobConf objects in AM heap. (Vinod Kumar
Vavilapalli via sseth)
MAPREDUCE-3399. Modifying ContainerLocalizer to send a heartbeat to NM
immediately after downloading a resource instead of always waiting for a
second. (Siddarth Seth via vinodkv)
MAPREDUCE-3568. Optimized Job's progress calculations in MR AM. (vinodkv)
MAPREDUCE-3569. TaskAttemptListener holds a global lock for all
task-updates. (Vinod Kumar Vavilapalli via sseth)
MAPREDUCE-3511. Removed a multitude of cloned/duplicate counters in the AM
thereby reducing the AM heap size and preventing full GCs. (vinodkv)
MAPREDUCE-3618. Fixed TaskHeartbeatHandler to not hold a global lock for all
task-updates. (Siddarth Seth via vinodkv)
MAPREDUCE-3512. Batching JobHistory flushing to DFS so that we don't flush
for every event slowing down AM. (Siddarth Seth via vinodkv)
MAPREDUCE-3718. Change default AM heartbeat interval to 1 second. (Hitesh
Shah via sseth)
MAPREDUCE-3360. Added information about lost/rebooted/decommissioned nodes
on the webapps. (Bhallamudi Venkata Siva Kamesh and Jason Lowe via vinodkv)
MAPREDUCE-3756. Made single shuffle limit configurable. (Hitesh Shah via
acmurthy)
MAPREDUCE-3811. Made jobclient-to-AM retries configurable. (sseth via
acmurthy)
BUG FIXES
MAPREDUCE-2784. [Gridmix] Bug fixes in ExecutionSummarizer and
ResourceUsageMatcher. (amarrk)
MAPREDUCE-3194. "mapred mradmin" command is broken in mrv2
(Jason Lowe via bobby)
MAPREDUCE-3462. Fix Gridmix JUnit testcase failures.
(Ravi Prakash and Ravi Gummadi via amarrk)
MAPREDUCE-2950. [Rumen] Fixed TestUserResolve. (Ravi Gummadi via amarrk)
MAPREDUCE-3412. Fix 'ant docs'. (amarrk)
MAPREDUCE-3346 [Rumen] LoggedTaskAttempt#getHostName() returns null.
(amarrk)
MAPREDUCE-3221. Reenabled the previously ignored test in TestSubmitJob
and fixed bugs in it. (Devaraj K via vinodkv)
MAPREDUCE-3215. Reenabled and fixed bugs in the failing test
TestNoJobSetupCleanup. (Hitesh Shah via vinodkv)
MAPREDUCE-3219. Reenabled and fixed bugs in the failing test
TestDelegationToken. (Hitesh Shah via vinodkv)
MAPREDUCE-3217. Reenabled and fixed bugs in the failing ant test
TestAuditLogger. (Devaraj K via vinodkv)
MAPREDUCE-3291. App fail to launch due to delegation token not
found in cache (Robert Evans via mahadev)
MAPREDUCE-3344. o.a.h.mapreduce.Reducer since 0.21 blindly casts to
ReduceContext.ValueIterator. (Brock Noland via tomwhite)
MAPREDUCE-3342. Fixed JobHistoryServer to also show the job's queue
name. (Jonathan Eagles via vinodkv)
MAPREDUCE-3345. Fixed a race condition in ResourceManager that was causing
TestContainerManagerSecurity to fail sometimes. (Hitesh Shah via vinodkv)
MAPREDUCE-3368. Fixed test compilation. (Hitesh Shah via vinodkv)
MAPREDUCE-3333. Fixed bugs in ContainerLauncher of MR AppMaster due to
which per-container connections to NodeManager were lingering long enough
to hit the ulimits on number of processes. (vinodkv)
MAPREDUCE-3392. Fixed Cluster's getDelegationToken's API to return null
when there isn't a supported token. (John George via vinodkv)
MAPREDUCE-3379. Fixed LocalResourceTracker in NodeManager to remove deleted
cache entries correctly. (Siddharth Seth via vinodkv)
MAPREDUCE-3324. Not All HttpServer tools links (stacks,logs,config,metrics) are
accessible through all UI servers (Jonathan Eagles via mahadev)
MAPREDUCE-3355. Fixed MR AM's ContainerLauncher to handle node-command
timeouts correctly. (vinodkv)
MAPREDUCE-3407. Fixed pom files to refer to the correct MR app-jar needed
by the integration tests. (Hitesh Shah via vinodkv)
MAPREDUCE-3437. Fix examples pom to refer to the correct 0.23 snapshot
version. (Jonathan Eagles via todd)
MAPREDUCE-3434. Nightly build broken (Hitesh Shah via mahadev)
MAPREDUCE-3447. mapreduce examples not working (mahadev)
MAPREDUCE-3444. trunk/0.23 builds broken (Hitesh Shah via mahadev)
MAPREDUCE-3454. [Gridmix] TestDistCacheEmulation is broken (Hitesh Shah
via mahadev)
MAPREDUCE-3408. yarn-daemon.sh unconditionnaly sets yarn.root.logger
(Bruno Mahe via mahadev)
MAPREDUCE-3329. Fixed CapacityScheduler to ensure maximum-capacity cannot
be lesser than capacity for any queue. (acmurthy)
MAPREDUCE-3464. mapreduce jsp pages missing DOCTYPE. (Dave Vronay via mattf)
MAPREDUCE-3265. Removed debug logs during job submission to LOG.debug to
cut down noise. (acmurthy)
MAPREDUCE-3468. Changed ant based infrastructure to use 0.23.1 version.
(sseth via acmurthy)
MAPREDUCE-3433. Finding counters by legacy group name returns empty
counters. (tomwhite)
MAPREDUCE-3450. NM port info no longer available in JobHistory.
(Siddharth Seth via mahadev)
MAPREDUCE-3477. Hadoop site documentation cannot be built anymore.
(jeagles via tucu)
MAPREDUCE-3488. Streaming jobs are failing because the main class
isnt set in the pom files. (mahadev)
MAPREDUCE-3463. Second AM fails to recover properly when first AM is killed with
java.lang.IllegalArgumentException causing lost job. (Siddharth Seth via mahadev)
MAPREDUCE-3452. fifoscheduler web ui page always shows 0% used for the queue.
(Jonathan Eagles via mahadev)
MAPREDUCE-3443. JobClient and Job should function in the context of the
UGI which created them. (Mahadev Konar via sseth)
MAPREDUCE-3460. MR AM can hang if containers are allocated on a node
blacklisted by the AM. (Hitesh Shah and Robert Joseph Evans via sseth)
MAPREDUCE-3453. RM web ui application details page shows RM cluster about
information. (Jonathan Eagles via sseth)
MAPREDUCE-3479. JobClient#getJob cannot find local jobs. (tomwhite)
MAPREDUCE-3500. MRJobConfig creates an LD_LIBRARY_PATH using the platform ARCH. (tucu)
MAPREDUCE-3456. $HADOOP_PREFIX/bin/yarn should set defaults for
$HADOOP_*_HOME (Eric Payne via mahadev)
MAPREDUCE-3458. Fix findbugs warnings in hadoop-examples. (Devaraj K
via mahadev)
MAPREDUCE-3485. DISKS_FAILED -101 error code should be defined in same location as
ABORTED_CONTAINER_EXIT_STATUS. (Ravi Gummadi via mahadev)
MAPREDUCE-3389. MRApps loads the 'mrapp-generated-classpath' file with
classpath from the build machine. (tucu)
MAPREDUCE-3496. Fixed client to print queue acls in consistent order.
(Jonathan Eagles via acmurthy)
MAPREDUCE-3147. Handle leaf queues with the same name properly.
(Ravi Prakash via mahadev)
MAPREDUCE-3327. RM web ui scheduler link doesn't show correct max value
for queues (Anupam Seth via mahadev)
MAPREDUCE-3513. Capacity Scheduler web UI has a spelling mistake for Memory.
(chackaravarthy via mahadev)
MAPREDUCE-3519. Fixed a deadlock in NodeManager LocalDirectories's handling
service. (Ravi Gummadi via vinodkv)
MAPREDUCE-3527. Fix minor API incompatibilities between 1.0 and 0.23.
(tomwhite)
MAPREDUCE-3328. mapred queue -list output inconsistent and missing child
queues. (Ravi Prakash via mahadev)
MAPREDUCE-3510. Capacity Scheduler inherited ACLs not displayed by mapred queue
-showacls (Jonathan Eagles via mahadev)
MAPREDUCE-3537. Fix race condition in DefaultContainerExecutor which led
to container localization occuring in wrong directories. (acmurthy)
MAPREDUCE-3542. Support "FileSystemCounter" legacy counter group name for
compatibility. (tomwhite)
MAPREDUCE-3426. Fixed MR AM in uber mode to write map intermediate outputs
in the correct directory to work properly in secure mode. (Hitesh Shah via
vinodkv)
MAPREDUCE-3544. gridmix build is broken, requires hadoop-archives to be
added as ivy dependency. (tucu)
MAPREDUCE-3557. MR1 test fail to compile because of missing hadoop-archives
dependency. (tucu)
MAPREDUCE-3541. Fix broken TestJobQueueClient test. (Ravi Prakash via
mahadev)
MAPREDUCE-3398. Fixed log aggregation to work correctly in secure mode.
(Siddharth Seth via vinodkv)
MAPREDUCE-3530. Fixed an NPE occuring during scheduling in the
ResourceManager. (Arun C Murthy via vinodkv)
MAPREDUCE-3484. Fixed JobEndNotifier to not get interrupted before completing
all its retries. (Ravi Prakash via vinodkv)
MAPREDUCE-3531. Fixed a race in ContainerTokenSecretManager. (Robert Joseph
Evans via sseth)
MAPREDUCE-3560. TestRMNodeTransitions is failing on trunk.
(Siddharth Seth via mahadev)
MAPREDUCE-3487. Fixed JobHistory web-UI to display links to single task's
counters' page. (Jason Lowe via vinodkv)
MAPREDUCE-3564. Fixed failures in TestStagingCleanup and TestJobEndNotifier
tests. (Siddharth Seth via vinodkv)
MAPREDUCE-3422. Counter display names are not being picked up. (Jonathan
Eagles via sseth)
MAPREDUCE-3366. Mapreduce component should use consistent directory structure
layout as HDFS/common (Eric Yang via mahadev)
MAPREDUCE-3387. Fixed AM's tracking URL to always go through the proxy, even
before the job started, so that it works properly with oozie throughout
the job execution. (Robert Joseph Evans via vinodkv)
MAPREDUCE-3579. ConverterUtils shouldn't include a port in a path from a url
without a port. (atm via harsh)
MAPREDUCE-3563. Fixed LocalJobRunner to work correctly with new mapreduce
apis. (acmurthy)
MAPREDUCE-3376. Fixed Task to ensure it passes reporter to combiners using
old MR api. (Subroto Sanyal via acmurthy)
MAPREDUCE-3339. Fixed MR AM to stop considering node blacklisting after the
number of nodes blacklisted crosses a threshold. (Siddharth Seth via vinodkv)
MAPREDUCE-3588. Fixed bin/yarn which was broken by MAPREDUCE-3366 so that
yarn daemons can start. (Arun C Murthy via vinodkv)
MAPREDUCE-3349. Log rack-name in JobHistory for unsuccessful tasks. (Amar
Kamat and Devaraj K via sseth)
MAPREDUCE-3586. Modified CompositeService to avoid duplicate stop operations
thereby solving race conditions in MR AM shutdown. (vinodkv)
MAPREDUCE-3604. Fixed streaming to use new mapreduce.framework.name to
check for local mode. (acmurthy)
MAPREDUCE-3521. Fixed streaming to ensure it doesn't silently ignore
unknown arguments. (Robert Evans via acmurthy)
MAPREDUCE-3522. Ensure queues inherit ACLs from parent if they aren't
explicitly specified. (Jonathan Eagles via acmurthy)
MAPREDUCE-3608. Fixed compile issue with MAPREDUCE-3522. (mahadev via
acmurthy)
MAPREDUCE-3490. Fixed MapReduce AM to count failed maps also towards Reduce
ramp up. (Sharad Agarwal and Arun C Murthy via vinodkv)
MAPREDUCE-3529. TokenCache does not cache viewfs credentials correctly
(sseth)
MAPREDUCE-3595. Add missing TestCounters#testCounterValue test from branch
1 to 0.23 (Tom White via sseth)
MAPREDUCE-3566. Fixed MR AM to construct CLC only once across all tasks.
(vinodkv via acmurthy)
MAPREDUCE-3572. Moved AM event dispatcher to a separate thread for
performance reasons. (vinodkv via acmurthy)
MAPREDUCE-3615. Fix some ant test failures. (Thomas Graves via sseth)
MAPREDUCE-1744. DistributedCache creates its own FileSytem instance when
adding a file/archive to the path. (Dick King via tucu)
MAPREDUCE-3326. Added detailed information about queue's to the
CapacityScheduler web-ui. (Jason Lowe via acmurthy)
MAPREDUCE-3548. Added more unit tests for MR AM & JHS web-services.
(Thomas Graves via acmurthy)
MAPREDUCE-3617. Removed wrong default value for
yarn.resourcemanager.principal and yarn.nodemanager.principal. (Jonathan
Eagles via acmurthy)
MAPREDUCE-3624. Remove unnecessary dependency on JDK's tools.jar. (mahadev
via acmurthy)
MAPREDUCE-3616. Thread pool for launching containers in MR AM not
expanding as expected. (vinodkv via sseth)
MAPREDUCE-3639. Fixed TokenCache to work with absent FileSystem canonical
service-names. (Siddharth Seth via vinodkv)
MAPREDUCE-3380. Token infrastructure for running clients which are not kerberos
authenticated. (mahadev)
MAPREDUCE-3648. TestJobConf failing. (Thomas Graves via mahadev)
MAPREDUCE-3651. TestQueueManagerRefresh fails. (Thomas Graves via mahadev)
MAPREDUCE-3645. TestJobHistory fails. (Thomas Graves via mahadev)
MAPREDUCE-3652. org.apache.hadoop.mapred.TestWebUIAuthorization.testWebUIAuthorization
fails. (Thomas Graves via mahadev)
MAPREDUCE-3625. CapacityScheduler web-ui display of queue's used capacity is broken.
(Jason Lowe via mahadev)
MAPREDUCE-3596. Fix scheduler to handle cleaned up containers, which NMs
may subsequently report as running. (Vinod Kumar Vavilapalli via sseth)
MAPREDUCE-3656. Fixed a race condition in MR AM which is failing the sort
benchmark consistently. (Siddarth Seth via vinodkv)
MAPREDUCE-3532. Modified NM to report correct http address when an ephemeral
web port is configured. (Bhallamudi Venkata Siva Kamesh via vinodkv)
MAPREDUCE-3404. Corrected MR AM to honor speculative configuration and enable
speculating either maps or reduces. (Eric Payne via vinodkv)
MAPREDUCE-3664. Federation Documentation has incorrect configuration example.
(Brandon Li via jitendra)
MAPREDUCE-3649. Job End notification gives an error on calling back.
(Ravi Prakash via mahadev)
MAPREDUCE-3657. State machine visualize build fails. (Jason Lowe
via mahadev)
MAPREDUCE-2450. Fixed a corner case with interrupted communication threads
leading to a long timeout in Task. (Rajesh Balamohan via acmurthy)
MAPREDUCE-3669. Allow clients to talk to MR HistoryServer using both
delegation tokens and kerberos. (mahadev via acmurthy)
MAPREDUCE-3684. LocalDistributedCacheManager does not shut down its thread
pool (tomwhite)
MAPREDUCE-3582. Move successfully passing MR1 tests to MR2 maven tree.
(ahmed via tucu)
MAPREDUCE-3698. Client cannot talk to the history server in secure mode.
(mahadev)
MAPREDUCE-3689. RM web UI doesn't handle newline in job name.
(Thomas Graves via mahadev)
MAPREDUCE-3701. Delete HadoopYarnRPC from 0.23 branch.
(mahadev)
MAPREDUCE-3549. write api documentation for web service apis for RM, NM,
mapreduce app master, and job history server (Thomas Graves via mahadev)
MAPREDUCE-3705. ant build fails on 0.23 branch. (Thomas Graves via
mahadev)
MAPREDUCE-3691. webservices add support to compress response.
(Thomas Graves via mahadev)
MAPREDUCE-3702. internal server error trying access application master
via proxy with filter enabled (Thomas Graves via mahadev)
MAPREDUCE-3646. Remove redundant URL info from "mapred job" output.
(Jonathan Eagles via mahadev)
MAPREDUCE-3681. Fixed computation of queue's usedCapacity. (acmurthy)
MAPREDUCE-3505. yarn APPLICATION_CLASSPATH needs to be overridable.
(ahmed via tucu)
MAPREDUCE-3714. Fixed EventFetcher and Fetcher threads to shut-down properly
so that reducers don't hang in corner cases. (vinodkv)
MAPREDUCE-3712. The mapreduce tar does not contain the hadoop-mapreduce-client-
jobclient-tests.jar. (mahadev)
MAPREDUCE-3717. JobClient test jar has missing files to run all the test programs.
(mahadev)
MAPREDUCE-3630. Fixes a NullPointer exception while running TeraGen - if a
map is asked to generate 0 records. (Mahadev Konar via sseth)
MAPREDUCE-3683. Fixed maxCapacity of queues to be product of parent
maxCapacities. (acmurthy)
MAPREDUCE-3713. Fixed the way head-room is allocated to applications by
CapacityScheduler so that it deducts current-usage per user and not
per-application. (Arun C Murthy via vinodkv)
MAPREDUCE-3721. Fixed a race in shuffle which caused reduces to hang.
(sseth via acmurthy)
MAPREDUCE-3733. Add Apache License Header to hadoop-distcp/pom.xml.
(mahadev)
MAPREDUCE-3735. Add distcp jar to the distribution (tar).
(mahadev)
MAPREDUCE-3720. Changed bin/mapred job -list to not print job-specific
information not available at RM. (vinodkv via acmurthy)
MAPREDUCE-3742. "yarn logs" command fails with ClassNotFoundException.
(Jason Lowe via mahadev)
MAPREDUCE-3703. ResourceManager should provide node lists in JMX output.
(Eric Payne via mahadev)
MAPREDUCE-3716. Fixing YARN+MR to allow MR jobs to be able to use
java.io.File.createTempFile to create temporary files as part of their
tasks. (Jonathan Eagles via vinodkv)
MAPREDUCE-3748. Changed a log in CapacityScheduler.nodeUpdate to debug.
(ramya via acmurthy)
MAPREDUCE-3764. Fixed resource usage metrics for queues and users.
(acmurthy)
MAPREDUCE-3749. ConcurrentModificationException in counter groups.
(tomwhite)
MAPREDUCE-3762. Fixed default CapacityScheduler configs. (mahadev via
acmurthy)
MAPREDUCE-3499. New MiniMR does not setup proxyuser configuration
correctly, thus tests using doAs do not work. (johnvijoe via tucu)
MAPREDUCE-3696. MR job via oozie does not work on hadoop 23.
(John George via mahadev)
MAPREDUCE-3427. Fix streaming unit tests broken after mavenization.
(Hitesh Shah via acmurthy)
MAPREDUCE-3640. Allow AMRecovery to work with partial JobHistory files.
(Arun C Murthy via sseth)
MAPREDUCE-3752. Modified application limits to include queue max-capacities
besides the usual user limits. (Arun C Murthy via vinodkv)
MAPREDUCE-3744. Fix the yarn logs command line. Improve error messages for
mapred job -logs. (Jason Lowe via sseth)
MAPREDUCE-3780. Fixed a bug where applications killed before getting
activated were not getting cleaned up properly. (Hitesh Shah via acmurthy)
MAPREDUCE-3708. Metrics: Incorrect Apps Submitted Count (Bhallamudi via
mahadev)
MAPREDUCE-3727. jobtoken location property in jobconf refers to wrong
jobtoken file (tucu)
MAPREDUCE-3711. Fixed MR AM recovery so that only single selected task
output is recovered and thus reduce the unnecessarily bloated recovery
time. (Robert Joseph Evans via vinodkv)
MAPREDUCE-3760. Changed active nodes list to not contain unhealthy nodes
on the webUI and metrics. (vinodkv)
MAPREDUCE-3417. Fixed job-access-controls to work with MR AM and
JobHistoryServer web-apps. (Jonathan Eagles via vinodkv)
MAPREDUCE-3803. Fix broken build of raid contrib due to HDFS-2864.
(Ravi Prakash via suresh)
MAPREDUCE-3791. can't build site in hadoop-yarn-server-common.
(mahadev)
MAPREDUCE-3723. TestAMWebServicesJobs & TestHSWebServicesJobs
incorrectly asserting tests (Bhallamudi Venkata Siva Kamesh
via mahadev)
MAPREDUCE-3795. "job -status" command line output is malformed.
(vinodkv via mahadev)
MAPREDUCE-3759. ClassCastException thrown in -list-active-trackers when
there are a few unhealthy nodes (vinodkv via mahadev)
MAPREDUCE-3775. Change MiniYarnCluster to escape special chars in testname.
(Hitesh Shah via mahadev)
MAPREDUCE-3765. FifoScheduler does not respect yarn.scheduler.fifo.minimum-
allocation-mb setting (Hitesh Shah via mahadev)
MAPREDUCE-3747. Initialize queue metrics upfront and added start/finish
time to RM Web-UI. (acmurthy)
MAPREDUCE-3814. Fixed MRV1 compilation. (Arun C Murthy via vinodkv)
MAPREDUCE-3810. Performance tweaks - reduced logging in AM and defined
hascode/equals for ResourceRequest & Priority. (vinodkv via acmurthy)
MAPREDUCE-3813. Added a cache for resolved racks. (vinodkv via acmurthy)
MAPREDUCE-3808. Fixed an NPE in FileOutputCommitter for jobs with maps
but no reduces. (Robert Joseph Evans via vinodkv)
MAPREDUCE-3804. yarn webapp interface vulnerable to cross scripting attacks
(Dave Thompson via bobby)
MAPREDUCE-3354. Changed scripts so that jobhistory server is started by
bin/mapred instead of bin/yarn. (Jonathan Eagles via acmurthy)
MAPREDUCE-3809. Ensure that there is no needless sleep in Task at the end
of the task. (sseth via acmurthy)
MAPREDUCE-3794. Support mapred.Task.Counter and mapred.JobInProgress.Counter
enums for compatibility (Tom White via mahadev)
MAPREDUCE-3697. Support binary compatibility for Counters after
MAPREDUCE-901. (mahadev via acmurthy)
MAPREDUCE-3817. Fixed bin/mapred to allow running of distcp and archive
jobs. (Arpit Gupta via acmurthy)
MAPREDUCE-3709. TestDistributedShell is failing. (Hitesh Shah via
mahadev)
MAPREDUCE-3436. JobHistory webapp address should use the host configured
in the jobhistory address. (Ahmed Radwan via sseth)
MAPREDUCE-3815. Fixed MR AM to always use hostnames and never IPs when
requesting containers so that scheduler can give off data local containers
correctly. (Siddarth Seth via vinodkv)
MAPREDUCE-3833. Fixed a bug in reinitiaziling of queues. (Jason Lowe via
acmurthy)
MAPREDUCE-3826. Fixed a bug in RM web-ui which broke sorting. (Jonathan
Eagles via acmurthy)
MAPREDUCE-3823. Ensure counters are calculated only once after a job
finishes. (Vinod Kumar Vavilapalli via sseth)
MAPREDUCE-3827. Changed Counters to use ConcurrentSkipListMap for
performance. (vinodkv via acmurthy)
MAPREDUCE-3822. Changed FS counter computation to use all occurences of
the same FS scheme, instead of randomly using one. (Mahadev Konar via
sseth)
MAPREDUCE-3834. Changed MR AM to not add the same rack entry multiple times
into the container request table when multiple hosts for a split happen to
be on the same rack. (Siddarth Seth via vinodkv)
MAPREDUCE-3828. Ensure that urls in single-node mode are correct. (sseth
via acmurthy)
MAPREDUCE-3770. Zombie.getJobConf() results into NPE. (amarrk)
MAPREDUCE-3843. Job summary log file found missing on the RM host
(Anupam Seth via tgraves)
MAPREDUCE-3846. Addressed MR AM hanging issues during AM restart and then
the recovery. (vinodkv)
MAPREDUCE-3802. Added test to validate that AM can crash multiple times and
still can recover successfully after MAPREDUCE-3846. (vinodkv)
MAPREDUCE-3858. Task attempt failure during commit results in task never completing.
(Tom White via mahadev)
MAPREDUCE-3856. Instances of RunningJob class givs incorrect job tracking
urls when mutiple jobs are submitted from same client jvm. (Eric Payne via
sseth)
MAPREDUCE-3880. Changed LCE binary to be 32-bit. (acmurthy)
Release 0.23.0 - 2011-11-01
INCOMPATIBLE CHANGES
MAPREDUCE-2455. Remove deprecated JobTracker.State in favour of
JobTrackerStatus. (tomwhite)
MAPREDUCE-2430. Remove mrunit contrib. (nigel via eli)
MAPREDUCE-2606. Remove IsolationRunner. (Alejandro Abdelnur via eli)
NEW FEATURES
MAPREDUCE-2682. Add "mapred classpath" command to print classpath
for MR applications. (vinodkv via acmurthy)
MAPREDUCE-2107. [Gridmix] Total heap usage emulation in Gridmix.
(Amar Kamat and Ravi Gummadi via amarrk)
MAPREDUCE-2106. [Gridmix] Cumulative CPU usage emulation in Gridmix.
(amarrk)
MAPREDUCE-2543. [Gridmix] High-Ram feature emulation in Gridmix. (amarrk)
MAPREDUCE-2408. [Gridmix] Compression emulation in Gridmix. (amarrk)
MAPREDUCE-2473. Add "mapred groups" command to query the server-side groups
resolved for a user. (Aaron T. Myers via todd)
MAPREDUCE-461. Enable ServicePlugins for the JobTracker.
(Fredrik Hedberg via tomwhite)
MAPREDUCE-2521. Create RPM and Debian packages for MapReduce. Changes
deployment layout to be consistent across the binary tgz, rpm, and deb.
(Eric Yang via omalley)
MAPREDUCE-2323. Add metrics to the fair scheduler. (todd)
MAPREDUCE-2037. Capture intermediate progress, CPU and memory usage for
tasks. (Dick King via acmurthy)
MAPREDUCE-279. MapReduce 2.0. Merging MR-279 branch into trunk. Contributed by
Arun C Murthy, Christopher Douglas, Devaraj Das, Greg Roelofs, Jeffrey
Naisbitt, Josh Wills, Jonathan Eagles, Krishna Ramachandran, Luke Lu, Mahadev
Konar, Robert Evans, Sharad Agarwal, Siddharth Seth, Thomas Graves, and Vinod
Kumar Vavilapalli.
MAPREDUCE-2930. Added the ability to be able to generate graphs from the
state-machine definitions. (Binglin Chang via vinodkv)
MAPREDUCE-2719. Add a simple, DistributedShell, application to illustrate
alternate frameworks on YARN. (Hitesh Shah via acmurthy)
MAPREDUCE-3104. Implemented Application-acls. (vinodkv)
MAPREDUCE-2708. Designed and implemented MR Application Master recovery to
make MR AMs resume their progress after restart. (Sharad Agarwal via vinodkv)
MAPREDUCE-2858. Added a WebApp Proxy for applications. (Robert Evans via
acmurthy)
IMPROVEMENTS
MAPREDUCE-2187. Reporter sends progress during sort/merge. (Anupam Seth via
acmurthy)
MAPREDUCE-2365. Add counters to track bytes (read,written) via
File(Input,Output)Format. (Siddharth Seth via acmurthy)
MAPREDUCE-2680. Display queue name in job client CLI. (acmurthy)
MAPREDUCE-2679. Minor changes to sync trunk with MR-279 branch. (acmurthy)
MAPREDUCE-2400. Remove Cluster's dependency on JobTracker via a
ServiceProvider for the actual implementation. (tomwhite via acmurthy)
MAPREDUCE-2596. [Gridmix] Summarize Gridmix runs. (amarrk)
MAPREDUCE-2563. [Gridmix] Add High-Ram emulation system tests to
Gridmix. (Vinay Kumar Thota via amarrk)
MAPREDUCE-2104. [Rumen] Add Cpu, Memory and Heap usages to
TraceBuilder's output. (amarrk)
MAPREDUCE-2554. [Gridmix] Add distributed cache emulation system tests
to Gridmix. (Vinay Kumar Thota via amarrk)
MAPREDUCE-2543. [Gridmix] High-Ram feature emulation testcase. (amarrk)
MAPREDUCE-2469. Task counters should also report the total heap usage of
the task. (Ravi Gummadi and Amar Ramesh Kamat via amarrk)
MAPREDUCE-2544. [Gridmix] Add compression emulation system tests to
Gridmix. (Vinay Kumar Thota via amarrk)
MAPREDUCE-2517. [Gridmix] Add system tests to Gridmix.
(Vinay Kumar Thota via amarrk)
MAPREDUCE-2492. The new MapReduce API should make available task's
progress to the task. (amarrk)
MAPREDUCE-2153. Bring in more job configuration properties in to the trace
file. (Rajesh Balamohan via amarrk)
MAPREDUCE-1461. Feature to instruct rumen-folder utility to skip jobs worth
of specific duration. (Rajesh Balamohan via amarrk)
MAPREDUCE-2172. Added test-patch.properties required by test-patch.sh
(nigel)
MAPREDUCE-2156. Raid-aware FSCK. (Patrick Kling via dhruba)
MAPREDUCE-2215. A more elegant FileSystem#listCorruptFileBlocks API
(RAID changes) (Patrick Kling via hairong)
MAPREDUCE-1831. BlockPlacement policy for HDFS-RAID.
(Scott Chen via dhruba)
MAPREDUCE-1906. Lower minimum heartbeat interval for TaskTracker
(Scott Carey and Todd Lipcon via todd)
MAPREDUCE-1382. MRAsyncDiscService should tolerate missing local.dir.
(Zheng Shao and tomwhite via tomwhite)
MAPREDUCE-2263. MapReduce side of HADOOP-6904: RPC compatibility.
(hairong)
MAPREDUCE-1706. Log RAID recoveries on HDFS. (schen)
MAPREDUCE-2334. Update BlockPlacementPolicyRaid for the new method
in BlockPlacementPolicy. (szetszwo)
MAPREDUCE-2254. Allow setting of end-of-record delimiter for
TextInputFormat (Ahmed Radwan via todd)
MAPREDUCE-1927. Unit test for HADOOP-6835 (concatenated gzip support).
(Greg Roelofs via tomwhite)
MAPREDUCE-2206. The task-cleanup tasks should be optional. (schen)
MAPREDUCE-2225. MultipleOutputs should not require the use of 'Writable'.
(Harsh J Chouraria via tomwhite)
MAPREDUCE-1811. Job.monitorAndPrintJob() should print status of the job
at completion. (Harsh J Chouraria via tomwhite)
MAPREDUCE-993. bin/hadoop job -events <jobid> <from-event-#> <#-of-events>
help message is confusing. (Harsh J Chouraria via tomwhite)
MAPREDUCE-2302. Add static factory methods in GaloisField. (schen)
MAPREDUCE-2351. mapred.job.tracker.history.completed.location should
support an arbitrary filesystem URI. (tomwhite)
MAPREDUCE-2239. BlockPlacementPolicyRaid should call getBlockLocations
only when necessary. (schen)
MAPREDUCE-2331. Add coverage of task graph servlet to fair scheduler system
test. (todd)
MAPREDUCE-2367. Allow using a file to exclude certain tests from build.
(todd)
MAPREDUCE-2202. Generalize CLITest structure and interfaces to faciliate
upstream adoption (e.g. for web or system testing). (cos)
MAPREDUCE-2420. JobTracker should be able to renew delegation token over
HTTP (Boris Shkolnik via jitendra)
MAPREDUCE-2474. Add docs to the new API Partitioner on how to access the
Job Configuration. (Harsh J Chouraria via todd)
MAPREDUCE-2475. Disable IPV6 for junit tests. (suresh srinivas via mahadev)
MAPREDUCE-2422. Removed unused internal methods from DistributedCache.
(tomwhite)
MAPREDUCE-2456. Log the reduce taskID and associated TaskTrackers with
failed fetch notifications in the JobTracker log.
(Jeffrey Naisbitt via cdouglas)
MAPREDUCE-869. Documentation for config to set map/reduce task environment
(Alejandro Abdelnur via todd)
MAPREDUCE-2410. Add entry to streaming FAQ about how streaming reducers
receive keys. (Harsh J Chouraria via todd)
MAPREDUCE-2499. MR part of HADOOP-7291. (eli)
MAPREDUCE-2497. Missing spaces in error messages. (eli)
MAPREDUCE-2502. JobSubmitter should use mapreduce.job.maps instead of
its deprecated equivalent. (eli via todd)
MAPREDUCE-2381. JobTracker instrumentation not consistent about error
handling. (Philip Zeyliger via tomwhite)
MAPREDUCE-2449. Allow for command line arguments when performing
"Run on Hadoop" action in Eclipse plugin. (Jeff Zemerick via todd)
MAPREDUCE-2483. Remove duplication of jars between Hadoop subprojects
from build artifacts. (Eric Yang via omalley)
MAPREDUCE-2372. TaskLogAppender mechanism shouldn't be set up in
log4j.properties (todd)
MAPREDUCE-2516. Rename webinterface.private.actions to
mapreduce.jobtracker.webinterface.trusted (Ari Rabkin via todd)
MAPREDUCE-2459. Cache HAR filesystem metadata. (Mac Yang via mahadev)
HADOOP-7259. Contrib modules should include the build.properties from
the enclosing hadoop directory. (omalley)
MAPREDUCE-2494. Order distributed cache deletions by LRU. (Robert Joseph
Evans via cdouglas)
MAPREDUCE-2452. Makes the cancellation of delegation tokens happen in a
separate thread. (ddas)
HADOOP-7106. Reorganize project SVN layout to "unsplit" the projects.
(todd, nigel)
MAPREDUCE-2249. Check the reflexive property of Counters objects when
comparing equality. (Devaraj K via todd)
MAPREDUCE-2623. Update ClusterMapReduceTestCase to use
MiniDFSCluster.Builder (Harsh J Chouraria via eli)
MAPREDUCE-2602. Allow setting of end-of-record delimiter for
TextInputFormat for the old API. (Ahmed Radwan via todd)
MAPREDUCE-2705. Permits parallel multiple task launches.
(Thomas Graves via ddas)
MAPREDUCE-2489. Jobsplits with random hostnames can make the queue
unusable (jeffrey naisbit via mahadev)
MAPREDUCE-2854. update INSTALL with config necessary run mapred on yarn.
(thomas graves via mahadev)
MAPREDUCE-2701. app/Job.java needs UGI for the user that launched it.
(Robert Evans via mahadev)
MAPREDUCE-2652. Enabled multiple NMs to be runnable on a single node by
making shuffle service port to be truely configurable. (Robert Evans via
vinodkv)
MAPREDUCE-2735. Add an applications summary log to ResourceManager.
(Thomas Graves via acmurthy)
MAPREDUCE-2697. Enhance CapacityScheduler to cap concurrently running
applications per-queue & per-user. (acmurthy)
Configuration changes:
add yarn.capacity-scheduler.maximum-am-resource-percent
MAPREDUCE-2774. Add startup message to ResourceManager & NodeManager on
startup. (Venu Gopala Rao via acmurthy)
MAPREDUCE-2655. Add audit logs to ResourceManager and NodeManager. (Thomas
Graves via acmurthy)
MAPREDUCE-2864. Normalize configuration variable names for YARN. (Robert
Evans via acmurthy)
MAPREDUCE-2690. Web-page for FifoScheduler. (Eric Payne via acmurthy)
MAPREDUCE-2711. Update TestBlockPlacementPolicyRaid for the new namesystem
and block management APIs. (szetszwo)
MAPREDUCE-2933. Change allocate call to return ContainerStatus for
completed containers rather than Container. (acmurthy)
MAPREDUCE-2675. Reformat JobHistory Server main page to be more
useful. (Robert Joseph Evans via vinodkv).
MAPREDUCE-2896. Simplify all apis to in
org.apache.hadoop.yarn.api.records.* to be get/set only. Added javadocs to
all public records. (acmurthy)
MAPREDUCE-2676. MR-279: JobHistory Job page needs reformatted. (Robert Evans via
mahadev)
MAPREDUCE-2899. Replace major parts of ApplicationSubmissionContext with a
ContainerLaunchContext (Arun Murthy via mahadev)
MAPREDUCE-2966. Added ShutDown hooks for MRV2 processes so that they can
gracefully exit. (Abhijit Suresh Shingate via vinodkv)
MAPREDUCE-2672. MR-279: JobHistory Server needs Analysis this job.
(Robert Evans via mahadev)
MAPREDUCE-2965. Streamlined the methods hashCode(), equals(), compareTo()
and toString() for all IDs. (Siddharth Seth via vinodkv)
MAPREDUCE-2726. Added job-file to the AM and JobHistoryServer web
interfaces. (Jeffrey Naisbitt via vinodkv)
MAPREDUCE-2880. Improve classpath-construction for mapreduce AM and
containers. (Arun C Murthy via vinodkv)
MAPREDUCE-3055. Simplified ApplicationAttemptId passing to
ApplicationMaster via environment variable. (vinodkv)
MAPREDUCE-3092. Removed a special comparator for JobIDs in JobHistory as
JobIDs are already comparable. (Devaraj K via vinodkv)
MAPREDUCE-3099. Add docs for setting up a single node MRv2 cluster.
(mahadev)
MAPREDUCE-3001. Added task-specific counters to AppMaster and JobHistory
web-UIs. (Robert Joseph Evans via vinodkv)
MAPREDUCE-3098. Fixed RM and MR AM to report YarnApplicationState and
application's FinalStatus separately. (Hitesh Shah via vinodkv)
MAPREDUCE-2889. Added documentation for writing new YARN applications.
(Hitesh Shah via acmurthy)
MAPREDUCE-3134. Added documentation the CapacityScheduler. (acmurthy)
MAPREDUCE-3013. Removed YarnConfiguration.YARN_SECURITY_INFO and its usage
as it doesn't affect security any more. (vinodkv)
MAPREDUCE-2907. Changed log level for various messages in ResourceManager
from INFO to DEBUG. (Ravi Prakash via vinodkv)
MAPREDUCE-2702. Added a new API in OutputCommitter for recovering
the outputs of tasks from a crashed job so as to support MR Application
Master recovery. (Sharad Agarwal and Arun C Murthy via vinodkv)
MAPREDUCE-2738. Added the missing cluster level statistics on the RM web
UI. (Robert Joseph Evans via vinodkv)
MAPREDUCE-2988. Reenabled TestLinuxContainerExecutor reflecting the
current NodeManager code. (Robert Joseph Evans via vinodkv)
MAPREDUCE-3161. Improved some javadocs and fixed some typos in
YARN. (Todd Lipcon via vinodkv)
MAPREDUCE-3148. Ported MAPREDUCE-2702 to old mapred api for aiding task
recovery. (acmurthy)
MAPREDUCE-3133. Running a set of methods in a Single Test Class.
(Jonathan Eagles via mahadev)
MAPREDUCE-3059. QueueMetrics do not have metrics for aggregate
containers-allocated and aggregate containers-released.
(Devaraj K via mahadev)
MAPREDUCE-3187. Add names for various unnamed threads in MR2.
(Todd Lipcon and Siddharth Seth via mahadev)
MAPREDUCE-3136. Added documentation for setting up Hadoop clusters in both
non-secure and secure mode for both HDFS & YARN. (acmurthy)
MAPREDUCE-3068. Added a whitelist of environment variables for containers
from the NodeManager and set MALLOC_ARENA_MAX for all daemons and
containers. (Chris Riccomini via acmurthy)
MAPREDUCE-3144. Augmented JobHistory with the information needed for
serving aggregated logs. (Siddharth Seth via vinodkv)
MAPREDUCE-3163. JobClient spews errors when killing MR2 job.
(mahadev)
MAPREDUCE-3239. Use new createSocketAddr API in MRv2 to give better
error messages on misconfig (Todd Lipcon via mahadev)
MAPREDUCE-2747. Cleaned up LinuxContainerExecutor binary sources and changed
the configuration to use yarn names. (Robert Joseph Evans via vinodkv)
MAPREDUCE-3205. Fix memory specifications to be physical rather than
virtual, allowing for a ratio between the two to be configurable. (todd
via acmurthy)
MAPREDUCE-2986. Fixed MiniYARNCluster to support multiple NodeManagers.
(Anupam Seth via vinodkv)
MAPREDUCE-2736. Remove unused contrib components dependent on MR1. (eli)
MAPREDUCE-2989. Modified JobHistory to link to task and AM logs from the
JobHistoryServer. (Siddharth Seth via vinodkv)
MAPREDUCE-3014. Rename and invert logic of '-cbuild' profile to 'native' and off
by default. (tucu)
MAPREDUCE-3171. normalize nodemanager native code compilation with common/hdfs
native. (tucu)
MAPREDUCE-3146. Added a MR specific command line to dump logs for a
given TaskAttemptID. (Siddharth Seth via vinodkv)
MAPREDUCE-3275. Added documentation for AM WebApp Proxy. (Robert Evans via
acmurthy)
MAPREDUCE-3322. Added a better index.html and an brief overview of YARN
architecture. (acmurthy)
OPTIMIZATIONS
MAPREDUCE-2026. Make JobTracker.getJobCounters() and
JobInProgress.getCounters() aquire locks in a shorter time period.
(Joydeep Sen Sarma via schen)
MAPREDUCE-2740. MultipleOutputs in new API creates needless
TaskAttemptContexts. (todd)
MAPREDUCE-901. Efficient framework counters. (llu via acmurthy)
MAPREDUCE-2629. Workaround a JVM class loading quirk which prevents
JIT compilation of inner classes methods in ReduceContextImpl.
BUG FIXES
MAPREDUCE-2603. Disable High-Ram emulation in system tests.
(Vinay Kumar Thota via amarrk)
MAPREDUCE-2539. Fixed NPE in getMapTaskReports in JobClient. (Robert Evans via
acmurthy)
MAPREDUCE-1978. Rumen TraceBuilder should provide recursive
input folder scanning.
MAPREDUCE-2416. Remove the restriction of specifying group names in
users-list file for Gridmix in RoundRobinUserResolver mode.
MAPREDUCE-2417. Fix Gridmix in RoundRobinUserResolver mode to
map testing/proxy users to unique users in a trace.
MAPREDUCE-2307. Exception thrown in Jobtracker logs, when the Scheduler
configured is FairScheduler. (Devaraj K via matei)
MAPREDUCE-2199. build is broken 0.22 branch creation. (cos)
MAPREDUCE-1752. Implement getFileBlockLocations in HarFilesystem.
(Patrick Kling via dhruba)
MAPREDUCE-2155. RaidNode should optionally use the mapreduce jobs to
fix missing blocks. (Patrick Kling via dhruba)
MAPREDUCE-1334. Fix TestIndexUpdater by ignoring _SUCCESS file in HDFS.
(Kay Kay via yhemanth)
MAPREDUCE-2232. Add missing methods to TestMapredGroupMappingServiceRefresh.
(Todd Lipcon via eli)
MAPREDUCE-2271. Fix TestSetupTaskScheduling failure on trunk.
(Liyin Liang via todd)
MAPREDUCE-2290. Fix compilation error in TestTaskCommit. (eli)
MAPREDUCE-2294. Fix compilation error in mumak. (todd)
MAPREDUCE-2300. Fix TestUmbilicalProtocolWithJobToken on trunk after
HADOOP-6904. (todd)
MAPREDUCE-2296. Fix references to misspelled method name
getProtocolSigature (todd)
MAPREDUCE-2311. Fix TestFairScheduler failure (schen)
MAPREDUCE-1996. API: Reducer.reduce() method detail misstatement.
(Harsh J Chouraria via tomwhite)
MAPREDUCE-2203. Wrong javadoc for TaskRunner's appendJobJarClasspaths
method. (Jingguo Yao via tomwhite)
MAPREDUCE-2074. Task should fail when symlink creation fails.
(Priyo Mustafi via tomwhite)
MAPREDUCE-1242. Chain APIs error misleading.
(Harsh J Chouraria via tomwhite)
MAPREDUCE-2379. Adds missing DistributedCache configurations in
mapred-default.xml (Todd Lipcon via amareshwari)
MAPREDUCE-2348. Disable mumak tests on trunk since they currently time out
(todd)
MAPREDUCE-2395. TestBlockFixer timing out on trunk. (Ramkumar Vadali via
todd)
MAPREDUCE-2426. Make TestFairSchedulerSystem fail with more verbose output
(todd)
MAPREDUCE-2448. NoSuchMethodError:
org.apache.hadoop.hdfs.TestDatanodeBlockScanner.corruptReplica(..) (eli)
MAPREDUCE-2460. Fix flaky test TestFairSchedulerSystem. (todd)
MAPREDUCE-2451. Log the details from health check script at the
JobTracker. (Thomas Graves via cdouglas)
MAPREDUCE-2467. HDFS-1052 changes break the raid contrib module in
MapReduce. (suresh srinivas via mahadev)
MAPREDUCE-2258. IFile reader closes stream and compressor in wrong order.
(todd via tomwhite)
MAPREDUCE-2518. The t flag is missing in distcp help message. (Wei Yongjun
via szetszwo)
MAPREDUCE-2514. Fix typo in TaskTracker ReinitTrackerAction log message.
(Jonathan Eagles via cdouglas)
MAPREDUCE-2490. Add logging to graylist and blacklist activity to aid
diagnosis of related issues. (Jonathan Eagles via cdouglas)
MAPREDUCE-2495. exit() the TaskTracker when the distributed cache cleanup
thread dies. (Robert Joseph Evans via cdouglas)
MAPREDUCE-2470. Fix NPE in RunningJobs::getCounters. (Robert Joseph Evans
via cdouglas)
MAPREDUCE-2536. Update FsShell -mv command usage in TestMRCLI. (Daryn
Sharp via szetszwo)
MAPREDUCE-2529. Add support for regex-based shuffle metric counting
exceptions. (Thomas Graves via cdouglas)
MAPREDUCE-2559. ant binary fails due to missing c++ lib dir. (eli)
MAPREDUCE-2573. Fix new findbugs warning introduced by MAPREDUCE-2494.
(Robert Joseph Evans via todd)
MAPREDUCE-2581. Spelling errors in log messages. (Tim Sell via eli)
MAPREDUCE-2588. Change raid to the new DataTransferProtocol API. (szetszwo)
MAPREDUCE-2576. Typo in comment in SimulatorLaunchTaskAction.java.
(Tim Sell via jghoman)
MAPREDUCE-2550. Fix bin/mapred to work properly from within a source
checkout (Eric Yang via todd)
MAPREDUCE-2620. Update RAID for HDFS-2087. (szetszwo)
MAPREDUCE-2624. Update RAID for HDFS-2107. (szetszwo)
MAPREDUCE-2670. Fixing spelling mistake in FairSchedulerServlet.java. (eli)
MAPREDUCE-2710. Update JobSubmitter.printTokens(..) for HDFS-2161.
(szetszwo)
MAPREDUCE-2409. DistributedCache maps files and archives to the same path,
despite semantic incompatibility. (Siddharth Seth via cdouglas)
MAPREDUCE-2575. TestMiniMRDFSCaching fails if test.build.dir is set
to something other than build/test (Thomas Graves via mahadev)
MAPREDUCE-2622. Remove the last remaining reference to the deprecated
configuration "io.sort.mb". (Harsh J Chouraria via todd)
MAPREDUCE-2732. Remove directly accessing FSNamesystem.LOG from
TestCopyFiles and TestDistCh. (szetszwo)
MAPREDUCE-2463. Job history files are not moved to done folder when job
history location is hdfs. (Devaraj K via szetszwo)
MAPREDUCE-2243. Close streams propely in a finally-block to avoid leakage
in CompletedJobStatusStore, TaskLog, EventWriter and TotalOrderPartitioner.
(Devaraj K via szetszwo)
MAPREDUCE-2741. Make ant build system work with hadoop-common JAR
generated by Maven. (Alejandro Abdelnur via tomwhite)
MAPREDUCE-2760. mapreduce.jobtracker.split.metainfo.maxsize typoed
in mapred-default.xml. (todd via eli)
MAPREDUCE-2797. Update mapreduce tests and RAID for HDFS-2239. (szetszwo)
MAPREDUCE-2805. Update RAID for HDFS-2241. (szetszwo)
MAPREDUCE-2837. Ported bug fixes from y-merge to prepare for MAPREDUCE-279
merge. (acmurthy)
MAPREDUCE-2541. Fixed a race condition in IndexCache.removeMap. (Binglin
Chang via acmurthy)
MAPREDUCE-2458. Rename sanitized pom.xml in build directory to work around IDE
bug (Luke Lu via mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Clear application notification if sent once
to NodeManager (mahadev)
MAPREDUCE-2433. YARNApplicationConstants hard code app master jar version (Luke
Lu via mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Implement restart for resource manager
phase 1 - Helper classes to store and restore the data structures. (mahadev)
MAPREDUCE-2414. Change MRv2 to use generic interfaces. (Siddharth Seth via
acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Implement health-checks for the node -
server side(ResourceManager) changes. (vinodkv)
MAPREDUCE-2405: Implement uber-AppMaster (in-cluster LocalJobRunner for MRv2)
(Greg Roelofs via mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Implementing Containers' memory monitoring.
(vinodkv)
MAPREDUCE-2440. Name clashes in TypeConverter (luke via mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Add fail count to the command line of the
application master. (mahadev)
MAPREDUCE-2424. Polish uber-AppMaster: add uber-AM counters and GUI indicators.
(Greg Roelofs via mahadev)
MAPREDUCE-2405. Implement uber-AppMaster (in-cluster LocalJobRunner for MRv2).
(Greg Roelofs and Sharad Agarwal via mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fix YarnRemoteException to give more
details. (Siddharth Seth via mahadev)
MAPREDUCE-279. Fix in MR-279 branch. WebApp for Job History (Krishna
Ramachandran via mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Install sanitized poms for downstream
sanity (Luke Lu via mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Add HistoryCleanerService to Job History
server. (Krishna Ramachandran via sharad)
MAPREDUCE-279. Fix in MR-279 branch. Implement 'bin/mapred job -list' and
'bin/mapred job -list-active-trackers'. (acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Implement 'bin/mapred queue [-info
[-showJobs]] [-list] and enhanced 'bin/mapred job -list' to show queue and
ApplicationMaster information. (acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Fixed computation of user-limits at
runtime. (acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Added functionality to refresh queues at
runtime via the 'bin/yarn rmadmin' command. (acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Added functionality to stop/start queues.
(acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Introducing web-UI for NodeManager and
linking it from RM UI. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Fix race condition in TestJobHistoryEvents
and TestJobHistoryParsing. (sharad)
MAPREDUCE-279. Fix in MR-279 branch. Add Containers' logs' view to NM UI and
link it from AM UI. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Add ACLs for queues and command-line
utilities for viewing them. (acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Recovery of MR Application Master from
failures. (sharad)
MAPREDUCE-279. Fix in MR-279 branch. Added support High-RAM applications in
CapacityScheduler. (acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Completing the ZooKeeper Store for
ResourceManager state. (mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Reorient container localization to be
per-container rather than per-application. (cdouglas)
MAPREDUCE-279. Fix in MR-279 branch. Fix file creation in
JobHistoryEventHandler. (sharad)
MAPREDUCE-279. Fix in MR-279 branch. Disable ContainerMonitoring for non-linux
systems. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Fix container launch w/ inconsistent
credential file naming. (cdouglas)
MAPREDUCE-2434. Metrics for ResourceManager. (Luke Lu via acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. RM Restart Phase 2 - Completed the recovery
of components in the RM (mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fix to send finish application event only
when the application is finished (mahadev)
MAPREDUCE-2462. Write job conf along with JobHistory, other minor improvements.
(Siddharth Seth via sharad)
MAPREDUCE-279. Fix in MR-279 branch. Implement 'delay scheduling' for better
locality in CapacityScheduler and improved high-ram applications. (acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Implement Job Acls in MR Application
Master. (sharad)
MAPREDUCE-279. Fix in MR-279 branch. Moving userlogs out of container work-dir
into a separate directory structure. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Completing RM Restart. Completed Phase 3 of
making sure events are logged and restored (mahadev)
MAPREDUCE-2468. Add metrics for NM Shuffle. (Luke Lu via cdouglas)
MAPREDUCE-279. Fix in MR-279 branch. Adding user log handling for YARN. Making
NM put the user-logs on DFS and providing log-dump tools. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Fixing three tight-loops in RM that are
causing high cpu-usage. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Replacing FileContext usage with FileSystem
to work around security authentication issues with FileContext against a secure
DFS. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Client reconnect to restarted AM. (sharad)
MAPREDUCE-279. Fix in MR-279 branch. Fix refreshProxy in ClientServiceDelegate.
(sharad)
MAPREDUCE-279. Fix in MR-279 branch. Fix Null Pointer in TestUberAM. (sharad)
MAPREDUCE-2478. Improve history server. (Siddharth Seth via sharad)
MAPREDUCE-279. Fix in MR-279 branch. Improved TestJobHistoryEvents and
TestJobHistoryParsing. (sharad)
MAPREDUCE-279. Fix in MR-279 branch. Fix NM to use multiple disks for local
files and the userlogs. (vinodkv)
MAPREDUCE-2480: MR App should not depend on hard coded version of shuffle (luke
lu via mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Propagate error back to client in case of a
job submission failure (mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fix assembly to add mapreduce shell scripts
to the assembly package. (mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fix TestQueueMetrics. (Luke Lu via sharad)
MAPREDUCE-279. Fix in MR-279 branch. Removal of stale application-log dirs from
NM local disks. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Add license header and minor cleanup in
history server. (Siddharth Seth via sharad)
MAPREDUCE-279. Fix in MR-279 branch. Minor fix for install instructions.
(mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fix a race in MR task that was causing MR
containers to overwrite each other's job.xml. Also fix leaking attempt-dirs in
app-local-dir. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Adding valid state to ASM on a finish when
its already completed and also disble UberAM. (mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fixed CS user limits. (acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Fixed reservation's bad interaction with
delay scheduling in CS. (acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Support mapreduce old (0.20) APIs. (sharad)
MAPREDUCE-279. Fix in MR-279 branch. Support fail-fast for MR jobs. (ddas)
MAPREDUCE-279. Fix in MR-279 branch. Fix for clearing container requests on an
AM failure and add tostring methods to taskids and taskattemptids for better
grep support. (mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Speed up communication between MR AM and RM
by relying on a new config rather than AM_EXPIRY_INTERVAL which is too large.
(acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Fix calculation of maximum capacity to use
parent's absolute-capacity rather than the leaf queue's absolute-capacity.
(acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Fixing a bug in JobIDPbImpl that's causing
AM to crash randomly. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Fix calculation of maximum capacity in
ParentQueue to use its parent's absolute-capacity rather than its own
absolute-capacity. (acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Launching bin/yarn and bin/mapred only
*once* in AM for constructing classpaths to avoid multiple forks and huge vmem
usage by AM. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Fix CapacityScheduler to release unused
reservations on application completion. (acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Fix CapacityScheduler (LeafQueue) to not
allocate DATA_LOCAL containers when they are not required on the rack.
(acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Makes uber-task disabled by default (ddas)
MAPREDUCE-279. Fix in MR-279 branch. Make logging and memory for AM configurable
for the user via command line (mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fixing a bug in previous patch (r1103657).
Now bin/yarn truly shouldn't be launched multiple times in a single AM.
(vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Fixing a bug to do with setting the staging
dir. (ddas)
MAPREDUCE-279. Fix in MR-279 branch. Fixed Composite Service to shutdown
services if an error occurs on starting any one of those (mahadev & chris)
MAPREDUCE-279. Fix in MR-279 branch. Fix the tests to use jvm fork mode to avoid
errors in shutting down services (sidharth seth)
MAPREDUCE-2500. PB factories are not thread safe (Siddharth Seth via mahadev)
MAPREDUCE-2504. race in JobHistoryEventHandler stop (Siddharth Seth via mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fix job hang if the AM launch fails.
(mahadev)
MAPREDUCE-2509. Fix NPE in UI for pending attempts. (Luke Lu via mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Add junit jar to lib in assembly (mahadev
and luke)
MAPREDUCE-279. Fix in MR-279 branch. Distributed cache bug fix to pass Terasort.
(vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Fix null pointer exception in kill task
attempt (mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Refactored RMContainerAllocator to release
unused containers. (sharad)
MAPREDUCE-279. Fix in MR-279 branch. Changed Scheduler to return available limit
to AM in the allocate api. (acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Fix nodemanager expiry to not throw OOM.
(mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Use DefaultContainerExecutor for
integration tests. (cdouglas)
MAPREDUCE-279. Fix in MR-279 branch. Fix NPE in test case (mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fix for regression on the scheduling of
reduces before maps are done (ddas)
MAPREDUCE-279. Fix in MR-279 branch. Fix distributed-cache related bugs.
(vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Ensure unused containers released by AM are
correctly counted for queue-capacity. (acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Fix TestRuntimeEstimators (Siddharth Seth
via ddas)
MAPREDUCE-279. Fix in MR-279 branch. Fix queue refresh to correctly record newly
added queues in CapacityScheduler. (acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Added metrics for tracking reservations in
CapacityScheduler. (Luke Lu via acmurthy)
MAPREDUCE-2522. Security for JobHistory service. (Siddharth Seth via mahadev)
MAPREDUCE-2534. Fix CI breaking hard coded version in jobclient pom. (Luke Lu
via mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Add public cache. (cdouglas)
MAPREDUCE-279. Fix in MR-279 branch. Made number of RPC server threads
configurable. (acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Added acl check for RMAdmin. (acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Adding job kill for any state that the job
is in with access control. (mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Add debug statements for AM not launching
(mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fixing race condition leader to hung jobs
in scheduler negotiator (mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Add debug config for delaying delete of
local files. (cdouglas)
MAPREDUCE-2527. Metrics for MRAppMaster (Luke lu via mahadev)
MAPREDUCE-2532. Metrics for NodeManager (Luke Lu via mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fixed an NPE during handling of unnecessary
reservations in CS. (acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Fix for public dist cache to work with non
default hdfs (mahadev &ddas)
MAPREDUCE-279. Fix in MR-279 branch. Making streaming -file option work. Also
minor fixes for successful compilation of contrib tests. (vinodkv)
MAPREDUCE-2536. Backporting changes to MR-279.
MAPREDUCE-279. Fix in MR-279 branch. Bugfix for using user staging directory for
history files (Siddharth Seth via mahadev)
MAPREDUCE-279. Fix in MR-279 branch. First fix for making basic speculative
execution work (ddas)
MAPREDUCE-279. Fix in MR-279 branch. Fixes for TestFail/Kill (ddas)
MAPREDUCE-279. Fix in MR-279 branch. Set correct version of avro-maven-plugin
that is available in apache maven repositories. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Fixing and reneabling
TestContainerTokenSecretManager. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Cleaning up configuration constants in
mapreduce modules. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Fixing NPE on speculator in MRAppMaster and
making job-history optional in tests to make test goal succeed. (vinodk and
sharadag).
MAPREDUCE-279. Fix in MR-279 branch. Fixed NPE in CS by checking Application
state before scheduling and fixing synchronization in CS. (acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Making pipes work with YARN. Changed pipes
to get log-locations from an environmental variable. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Ensure 'lost' NodeManagers are dealt
appropriately, the containers are released correctly. (acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Adding some more logging for AM expiry logs
(mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Reduce ramp up and zero maps support.
(sharad)
MAPREDUCE-279. Fix in MR-279 branch. Allowing hdfs calls from streaming/pipes
tasks. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Added ability to decommission nodes and
completed RM administration tools to achieve parity with JobTracker. (acmurthy)
MAPREDUCE-2551. Added JobSummaryLog. (Siddharth Seth via acmurthy)
MAPREDUCE-2552. Fixed NPE in CompletedJob in JobHistoryServer. (Siddharth Seth
via acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Fix reduce slow start. (sharad)
MAPREDUCE-279. Fix in MR-279 branch. Fixed TestFifoScheduler. (acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Fix build issue for using yarn.version
instead of hadoop-mapred.version (mahadev and giri)
MAPREDUCE-279. Fix in MR-279 branch. Fixes in the handling of KILL events in the
SUCCEEDED state for tasks in the application master (ddas)
MAPREDUCE-279. Fix in MR-279 branch. Fix for NPE in TestRMNMRPCResponseId.
(mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fix a deadlock in the resourcemanager.
(mahadev)
MAPREDUCE-279. Fix in MR-279 branch. NodeStatus.getNodeHealthStatus().setBlah
broken (Siddharth Seth)
MAPREDUCE-279. Fix in MR-279 branch. Fix another NPE in TestRMNMRPCResponseId.
(mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fix for NPE in TestNMExpiry (siddharth
seth)
MAPREDUCE-279. Fix in MR-279 branch. Making each node aggregate all its
user-logs to a separate hdfs file. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Fix calculation of max-capacity for a
queue, also fixed a bug in registration of NodeManagers. (acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. More cleaning up constants, removing stale
code, and making conspicuous the envs that apps depend on to be provided by
YARN. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Fix container size rounding in AM and
headroom in RM. (acmurthy and sharad)
MAPREDUCE-279. Fix in MR-279 branch. Disable Job acls until fixed (mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fix to report job status if the application
is KILLED/FAILED. (mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fix a corner case in headroom computation -
now reservations are taken into account and headroom is computed much later to
account for allocations/reservations. (acmurthy)
MAPREDUCE-2537. The RM writes its log to
yarn-mapred-resourcemanager-<RM_Host>.out (Robert Evans via mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fix logging for showing the state of job
(FAILED/KILLED/SUCCEEDED) when it completes (mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Re-enabled TestCapacityScheduler.
(acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Support for min and max container capacity.
(acmurthy and sharad)
MAPREDUCE-2531. Fixed jobcontrol to downgrade JobID. (Robert Evans via acmurthy)
MAPREDUCE-2539. Fixed NPE in getMapTaskReports in JobClient. (Robert Evans via
acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Fixing the wrong config key used in
JobHistory that prevented configuring move-thread interval. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Fixed inconsistency in QueueACL enums.
(acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Fix various issues with Web UI's. (Luke Lu)
MAPREDUCE-279. Fix in MR-279 branch. Fix class cast exception in Task abort for
old mapreduce apis. (sharad)
MAPREDUCE-279. Fix in MR-279 branch. Add deletion of distributed cache
resources. (cdouglas)
MAPREDUCE-279. Fix in MR-279 branch. Disable aggregation of logs onto DFS till
JobHistoryServer starts serving logs. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Cleanup redundant code in TaskAttemptImpl.
(sharad)
MAPREDUCE-279. Fix in MR-279 branch. Work around broken signaling in public
cache. (cdouglas)
MAPREDUCE-2566. YarnConfiguration should reloadConfiguration if instantiated
with a non YarnConfiguration object. (Siddharth Seth)
MAPREDUCE-279. Fix in MR-279 branch. Fully resolve paths when launching
containers. (Siddharth Seth)
MAPREDUCE-279. Fix in MR-279 branch. Re-enabling Uber-AM feature. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Fixed deadlock during expiring NMs.
(acmurthy)
MAPREDUCE-279. Fix in MR-279 branch. Solving NPEs during
ContainerManager#StopContainer. Also removing the unused
ContainerManager#CleanupContainer api. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Remove retries in dist cache so that NM's
do not shutdown (mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fix classpath construction for Task.
(vinodkv via sharad)
MAPREDUCE-279. Fix in MR-279 branch. Sending Job diagnostics from AM to RM and
redirect to history-url on job completion. (vinodkv and sharadag)
MAPREDUCE-279. Fix in MR-279 branch. Added clover in pom dependency. (sharad)
MAPREDUCE-279. Fix in MR-279 branch. Intermittent TestMRApp failures on faster
Linux desktop. (Luke lu via mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Including source files in release
distribution (Luke Lu via mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Intermittent TestMRApp failures on faster
Linux desktop (part 2) (Luke lu via mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Disable Uber AM. (mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Added few job diagnostic messages. (sharad)
MAPREDUCE-279. Fix in MR-279 branch. Add ability to includes src files in
assembly target for maven (Luke Lu via mahadev)
MAPREDUCE-2582. Cleanup JobHistory event generation.(Siddharth Seth via sharad)
MAPREDUCE-279. Fix in MR-279 branch. Fix rounding off problem in reduce ramp up.
(sharad)
MAPREDUCE-279. Fix in MR-279 branch. Fix more rounding off problems in reduce
ramp up. Also fix a bug preventing the application of the cap on reduce ramp-up.
(Sharad Agarwal via vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Fix to exclude images dir into the tar
distribution (Luke Lu via gkesavan)
MAPREDUCE-279. Fix in MR-279 branch. Changes a couple of usages of FileContext
to FileSystem in YarnRunner to handle distributed cache path resolutions on
non-default filesystems. (ddas)
MAPREDUCE-279. Fix in MR-279 branch. Display failed/killed attempts of the task
on MR AM UI separately. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Miscellaneous UI fixes + source code
formatting for MR JobHistoryEventHandler. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Fixing broken link to logs for container on
NM web UI. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Fixing the bug which was causing FAILED
jobs to be displayed as COMPLETED on the RM UI. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Job level node blacklisting. (sharad)
MAPREDUCE-279. Fix in MR-279 branch. Fix NPE in history event handling
(Siddharth Seth via mahadev)
MAPREDUCE-2569. Ensure root queue allocated 100% capacity. (Jonathan Eagles via
cdouglas)
MAPREDUCE-279. Fix in MR-279 branch. Fix ClassCastException in JobHistoryServer
for certain jobs. (Siddharth Seth via llu)
MAPREDUCE-279. Fix in MR-279 branch. Changes for invoking rack resolution in the
RM and in the AM (ddas)
MAPREDUCE-279. Fix in MR-279 branch. Fix concurrent modification exception in
the Capacity Scheduler (mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fix stopContainer for setsid challenged
platforms. (llu)
MAPREDUCE-2587. Generate yarn version for UI. (Thomas Graves via lluts page to
the history server UI. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Bug fix to set correct state on containers
so as to avoid duplicate containers from RM to AM. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Hack until MAPREDUCE-2365 is fixed to make
PIG work with MRV2. (mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Changes a couple of usages of FileContext
to FileSystem in TaskAttemptImpl to handle distributed cache path resolutions on
non-default filesystems. (ddas)
MAPREDUCE-279. Fix in MR-279 branch. Fix NPE when requesting attempts for
completed jobs. (Siddharth Seth via llu)
MAPREDUCE-279. Fix in MR-279 branch. Improve logging for AM when requesting
containers to show the right ask and release fields (mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fix race condition between multiple
localizers on a single node. (cdouglas via mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fix RM app start/finish time and
diagnostics. (llu)
MAPREDUCE-279. Fix in MR-279 branch. Fix to schedule reduces irrespective of the
headroom when all maps are done so as to avoid stall in reduce-scheduling when
slow-start is disabled. (Sharad Agarwal via vinodkv).
MAPREDUCE-279. Fix in MR-279 branch. Disabling locality-wait in
CapacityScheduler for now to prevent uber-slow scheduling for apps with no
data-locality constraints (sleep-job like). (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Fixing scheduling deadlock in AM because of
incorrect headRoom values from RM. The bug happens when AM releases containers
and RM decrements current memory usage twice for all those containers. (vinodkv)
MAPREDUCE-2611. Fix counters, finish times etc. in job history. (Siddharth Seth
via llu)
MAPREDUCE-279. Fix in MR-279 branch. Fix for ConcurrentModification exception
while iterating through tokens in a UGI in ContainerLauncherImpl. (ddas)
MAPREDUCE-279. Fix in MR-279 branch. Fix for NPE in YarnChild that was causing
lots of tasks to fail. (vinodkv)
MAPREDUCE-2615. Make killJob go through AM and fix JobSummaryLog. (Siddharth
Seth via llu)
MAPREDUCE-279. Fix in MR-279 branch. Fix class cast exception in release
reserved containers in capacity scheduler (mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fix diagnostics display for more than 100
apps in RM. (llu)
MAPREDUCE-279. Fix in MR-279 branch. Fix some invalid transitions in the RM.
(vinodkv via ddas)
MAPREDUCE-2618. Fix NPE in 0 map 0 reduce jobs. (Jeffrey Naisbitt via llu)
MAPREDUCE-2625. Add version info to nodemanager info page. (Jonathan Eagles via
llu)
MAPREDUCE-279. Fix in MR-279 branch. (1) Faster retries from AM to HistoryServer
(2) Correct diagnostics for containers. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Major ASM cleanup. Streamlining classes,
interface and events. (vinodkv)
MAPREDUCE-279. Fix in MR-279 branch. Reinstate resolve path fixes for viewfs.
(Siddharth Seth via llu)
MAPREDUCE-2633. Add a getCounter(Enum) method to the Counters record. (Josh
Wills via sharad)
MAPREDUCE-2645. Updates to MRv2 INSTALL and README documentation. (Josh Wills
via vinodkv)
MAPREDUCE-2628. Add compiled on date to NM and RM info/about page.
MAPREDUCE-2400. Remove Cluster's dependency on JobTracker via a ServiceProvider
for the actual implementation. (tomwhite via acmurthy)
MAPREDUCE-2663. Refactoring StateMachineFactory inner classes. (ahmed radwan via
mahadev)
MAPREDUCE-2678. minimum-user-limit-percent no longer honored. (naisbitt via
mahadev)
MAPREDUCE-2630. refreshQueues leads to NPEs when used w/FifoScheduler. (Josh
Wills via mahadev)
MAPREDUCE-2644. NodeManager fails to create containers when NM_LOG_DIR is not
explicitly set in the Configuration. (Josh Wills via vinodkv)
MAPREDUCE-2661. Fix TaskImpl to not access MapTaskImpl. (Ahmed Radwan via
sharad)
HADOOP-6929. Backport changes to MR-279 (mahadev and owen)
HADOOP-6929. Making Security Info abstract and not an interface (mahadev)
MAPREDUCE-2667. mapred job -kill leaves application in RUNNING state (thomas
graves via mahadev)
MAPREDUCE-2664. Implement JobCounters for Mtions as asynchronous. (vinodkv,
sharad and acmurthy)
MAPREDUCE-2773. server.api.records.NodeHealthStatus renamed but not updated in
client NodeHealthStatus.java (Thomas Graves via mahadev)
MAPREDUCE-2772. Fix MR-279 build after common mavenization. (Robert Joseph Evans
via llu)
MAPREDUCE-2772. Fix MR-279 build after common mavenization, part 2. (Thomas
Graves via llu)
MAPREDUCE-279. Fix in MR-279 branch. Harmonize slf4j versions. (llu)
MAPREDUCE-279. Fix in MR-279 branch. Fix NPE in FifoScheduler. (mahadev)
MAPREDUCE-2776. Fix some of the yarn findbug warnings. (Siddharth Seth via
mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fix findbugs warnings in mr-client modules,
part 1 (mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fix findbugs warnings in mr-client modules
part 2 (mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fix findbugs warnings in mr-client modules
part 3 (mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fix the poms to enable 0.23 snapshots for
hdfs/common from apache nightly builds (gkesavan)
MAPREDUCE-279. Fix in MR-279 branch. Fix ivy conf to work with the hadoop common
trunk maven build changes. (Giridharan Kesavan)
MAPREDUCE-279. Fix in MR-279 branch. Patch for findbugs warnings in Resource
Manager (Siddharth Seth via mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fix for running ant targets to use the
right set of common/test jars (gkesavan via mahadev)
MAPREDUCE-2782. Unit tests for CapacityScheduler. (acmurthy)
MAPREDUCE-2706. Log job submission failures. (Jeffrey Naisbitt via acmurthy)
MAPREDUCE-2781. mr279 RM application finishtime not set (Thomas Graves via
mahadev)
MAPREDUCE-279. Fix in MR-279 branch. Fixed CS locality wait factor. (acmurthy)
MAPREDUCE-2808. pull MAPREDUCE-2797 into mr279 branch (Thomas Graves via
mahadev)
MAPREDUCE-2639. Bug fixes in speculate.DataStatistics. (Josh Wills via acmurthy)
MAPREDUCE-2839. Fixed TokenCache to get delegation tokens using both new and old
apis. (Siddharth Seth via acmurthy)
MAPREDUCE-2727. Fix divide-by-zero error in SleepJob for sleepCount equals 0.
(Jeffrey Naisbitt via acmurthy)
MAPREDUCE-2839. Fixed TokenCache to get delegation tokens using both new
and old apis. (Siddharth Seth via acmurthy)
MAPREDUCE-2727. Fix divide-by-zero error in SleepJob for sleepCount equals
0. (Jeffrey Naisbitt via acmurthy)
MAPREDUCE-2860. Fix log4j logging in the maven test cases. (mahadev)
MAPREDUCE-2867. Remove Unused TestApplicaitonCleanup in resourcemanager/applicationsmanager.
(mahadev)
MAPREDUCE-2868. ant build broken in hadoop-mapreduce dir (mahadev, giri and arun via mahadev)
MAPREDUCE-2649. Handling of finished applications in RM. (Thomas Graves
via acmurthy)
MAPREDUCE-2838. Fix MapReduce builds to use new hadoop-common test jars.
(gkesavan via acmurthy)
MAPREDUCE-2859. Fix eclipse plugin contrib module compilation (gkesavan)
MAPREDUCE-2846. Fix missing synchronization in the task log management.
(omalley)
MAPREDUCE-2807. Fix AM restart and client redirection. (sharad)
MAPREDUCE-2877. Add missing Apache license header in some files in MR
and also add the rat plugin to the poms. (mahadev)
MAPREDUCE-2796. Set start times for MR applications for clients to see.
(Devaraj K via acmurthy)
MAPREDUCE-2879. Fix version for MR-279 to 0.23.0. (acmurthy)
MAPREDUCE-2881. Fix to include log4j 1.2.16 depenency (gkesavan)
MAPREDUCE-2885. Fix mapred-config.sh to look for hadoop-config.sh in
HADOOP_COMMON_HOME/libexec. (acmurthy)
MAPREDUCE-2893. Remove duplicate entry of YarnClientProtocolProvider in
ClientProtocolProvider services file. (Liang-Chi Hsieh via acmurthy)
MAPREDUCE-2891. Javadoc for AMRMProtocol and related records. (acmurthy)
MAPREDUCE-2898. Javadoc for ContainerManager protocol and related records.
(acmurthy)
MAPREDUCE-2904. Fixed bin/yarn to correctly include HDFS jars and
clean up of stale refs to pre-mavenized Hadoop Common and HDFS.
(Sharad Agarwal and Arun C. Murthy via acmurthy)
MAPREDUCE-2737. Update the progress of jobs on client side. (Siddharth Seth
and Mahadev Konar via mahadev)
MAPREDUCE-2886. Fix Javadoc warnings in MapReduce. (mahadev)
MAPREDUCE-2897. Javadoc for ClientRMProtocol protocol and related records.
(acmurthy)
MAPREDUCE-2916. Ivy build for MRv1 fails with bad organization for
common daemon. (mahadev)
MAPREDUCE-2917. Fixed corner case in container reservation which led to
starvation and hung jobs. (acmurthy)
MAPREDUCE-2756. Better error handling in JobControl for failed jobs.
(Robert Evans via acmurthy)
MAPREDUCE-2716. MRReliabilityTest job fails because of missing
job-file. (Jeffrey Naisbitt via vinodkv)
MAPREDUCE-2882. TestLineRecordReader depends on ant jars. (todd)
MAPREDUCE-2687. Fix NodeManager to use the right version of
LocalDirAllocator.getLocalPathToWrite. (mahadev & acmurthy)
MAPREDUCE-2800. Set final progress for tasks to ensure all task information
is correctly logged to JobHistory. (Siddharth Seth via acmurthy)
MAPREDUCE-2938. Log application submission failure in CapacityScheduler.
(acmurthy)
MAPREDUCE-2948. Hadoop streaming test failure, post MR-2767 (mahadev)
MAPREDUCE-2908. Fix all findbugs warnings. (vinodkv via acmurthy)
MAPREDUCE-2947. Fixed race condition in AuxiliaryServices. (vinodkv via
acmurthy)
MAPREDUCE-2844. Fixed display of nodes in UI. (Ravi Teja Ch N V via
acmurthy)
MAPREDUCE-2677. Fixed 404 for some links from HistoryServer. (Robert Evans
via acmurthy)
MAPREDUCE-2937. Ensure reason for application failure is displayed to the
user. (mahadev via acmurthy)
MAPREDUCE-2953. Fix a race condition on submission which caused client to
incorrectly assume application was gone by making submission synchronous
for RMAppManager. (Thomas Graves via acmurthy)
MAPREDUCE-2963. Fix hang in TestMRJobs. (Siddharth Seth via acmurthy)
MAPREDUCE-2954. Fixed a deadlock in NM caused due to wrong synchronization
in protocol buffer records. (Siddharth Seth via vinodkv)
MAPREDUCE-2975. Fixed YARNRunner to use YarnConfiguration rather than
Configuration. (mahadev via acmurthy)
MAPREDUCE-2971. ant build mapreduce fails protected access jc.displayJobList
(jobs) (Thomas Graves via mahadev)
MAPREDUCE-2691. Finishing up the cleanup of distributed cache file resources
and related tests. (Siddharth Seth via vinodkv)
MAPREDUCE-2749. Ensure NM registers with RM after starting all its services
correctly. (Thomas Graves via acmurthy)
MAPREDUCE-2979. Removed the needless ClientProtocolProvider configuration
from the hadoop-mapreduce-client-core module. (Siddharth Seth via vinodkv)
MAPREDUCE-2985. Fixed findbugs warnings in ResourceLocalizationService.
(Thomas Graves via acmurthy)
MAPREDUCE-2874. Fix formatting of ApplicationId in web-ui. (Eric Payne via
acmurthy)
MAPREDUCE-2995. Better handling of expired containers in MapReduce
ApplicationMaster. (vinodkv via acmurthy)
MAPREDUCE-2995. Fixed race condition in ContainerLauncher. (vinodkv via
acmurthy)
MAPREDUCE-2949. Fixed NodeManager to shut-down correctly if a service
startup fails. (Ravi Teja via vinodkv)
MAPREDUCE-3005. Fix both FifoScheduler and CapacityScheduler to correctly
enforce locality constraints. (acmurthy)
MAPREDUCE-3007. Fixed Yarn Mapreduce client to be able to connect to
JobHistoryServer in secure mode. (vinodkv)
MAPREDUCE-2987. Fixed display of logged user on RM Web-UI. (Thomas Graves
via acmurthy)
MAPREDUCE-3006. Fixed MapReduce AM to exit only after properly writing out
history file. (vinodkv)
MAPREDUCE-2925. Fixed Yarn+MR client code to behave saner with completed
jobs. (Devaraj K via vinodkv)
MAPREDUCE-3030. Fixed a bug in NodeId.equals() that was causing RM to
reject all NMs. (Devaraj K via vinodkv)
MAPREDUCE-3042. Fixed default ResourceTracker address. (Chris Riccomini
via acmurthy)
MAPREDUCE-3038. job history server not starting because conf() missing
HsController (Jeffrey Naisbitt via mahadev)
MAPREDUCE-3004. Fix ReduceTask to not assume 'local' mode in YARN. (Hitesh
Shah via acmurthy)
MAPREDUCE-3017. The Web UI shows FINISHED for killed/successful/failed jobs.
(mahadev)
MAPREDUCE-3040. Fixed extra copy of Configuration in
YarnClientProtocolProvider and ensured MiniMRYarnCluster sets JobHistory
configuration for tests. (acmurthy)
MAPREDUCE-3018. Fixed -file option for streaming. (mahadev via acmurthy)
MAPREDUCE-3036. Fixed metrics for reserved resources in CS. (Robert Evans
via acmurthy)
MAPREDUCE-2998. Fixed a bug in TaskAttemptImpl which caused it to fork
bin/mapred too many times. (vinodkv via acmurthy)
MAPREDUCE-3023. Fixed clients to display queue state correctly. (Ravi
Prakash via acmurthy)
MAPREDUCE-2970. Fixed NPEs in corner cases with different configurations
for mapreduce.framework.name. (Venu Gopala Rao via vinodkv)
MAPREDUCE-3062. Fixed default RMAdmin address. (Chris Riccomini
via acmurthy)
MAPREDUCE-3066. Fixed default ResourceTracker address for the NodeManager.
(Chris Riccomini via acmurthy)
MAPREDUCE-3044. Pipes jobs stuck without making progress. (mahadev)
MAPREDUCE-2754. Fixed MR AM stdout, stderr and syslog to redirect to
correct log-files. (Ravi Teja Ch N V via vinodkv)
MAPREDUCE-3073. Fixed build issues in MR1. (mahadev via acmurthy)
MAPREDUCE-2691. Increase threadpool size for launching containers in
MapReduce ApplicationMaster. (vinodkv via acmurthy)
MAPREDUCE-2990. Fixed display of NodeHealthStatus. (Subroto Sanyal via
acmurthy)
MAPREDUCE-3053. Better diagnostic message for unknown methods in ProtoBuf
RPCs. (vinodkv via acmurthy)
MAPREDUCE-2952. Fixed ResourceManager/MR-client to consume diagnostics
for AM failures in a couple of corner cases. (Arun C Murthy via vinodkv)
MAPREDUCE-3064. 27 unit test failures with Invalid
"mapreduce.jobtracker.address" configuration value for
JobTracker: "local" (Venu Gopala Rao via mahadev)
MAPREDUCE-3090. Fix MR AM to use ApplicationAttemptId rather than
(ApplicationId, startCount) consistently. (acmurthy)
MAPREDUCE-2646. Fixed AMRMProtocol to return containers based on
priority. (Sharad Agarwal and Arun C Murthy via vinodkv)
MAPREDUCE-3031. Proper handling of killed containers to prevent stuck
containers/AMs on an external kill signal. (Siddharth Seth via vinodkv)
MAPREDUCE-2984. Better error message for displaying completed containers.
(Devaraj K via acmurthy)
MAPREDUCE-3071. app master configuration web UI link under the Job menu
opens up application menu. (thomas graves via mahadev)
MAPREDUCE-3067. Ensure exit-code is set correctly for containers. (Hitesh
Shah via acmurthy)
MAPREDUCE-2999. Fix YARN webapp framework to properly filter servlet
paths. (Thomas Graves via vinodkv)
MAPREDUCE-3095. fairscheduler ivy including wrong version for hdfs.
(John George via mahadev)
MAPREDUCE-3054. Unable to kill submitted jobs. (mahadev)
MAPREDUCE-3021. Change base urls for RM web-ui. (Thomas Graves via
acmurthy)
MAPREDUCE-3041. Fixed ClientRMProtocol to provide min/max resource
capabilities along-with new ApplicationId for application submission.
(Hitesh Shah via acmurthy)
MAPREDUCE-2843. Fixed the node-table to be completely displayed and making
node entries on RM UI to be sortable. (Abhijit Suresh Shingate via vinodkv)
MAPREDUCE-3110. Fixed TestRPC failure. (vinodkv)
MAPREDUCE-3078. Ensure MapReduce AM reports progress correctly for
displaying on the RM Web-UI. (vinodkv via acmurthy)
MAPREDUCE-3114. Fixed invalid ApplicationURL on RM WebUI. (Subroto Sanyal
via vinodkv)
MAPREDUCE-2791. Added missing info on 'job -status' output. (Devaraj K via
acmurthy)
MAPREDUCE-2996. Add uber-ness information to JobHistory. (Jonathan Eagles
via acmurthy)
MAPREDUCE-3050. Add ability to get resource usage information for
applications and nodes. (Robert Evans via acmurthy)
MAPREDUCE-3113. Ensure bin/yarn and bin/yarn-daemon.sh identify the root
of the install properly. (Xie Xianshan via acmurthy)
MAPREDUCE-3137. Fix broken merge of MAPREDUCE-2179. (Hitesh Shah via
acmurthy)
MAPREDUCE-2792. Replace usage of node ip-addresses with hostnames.
(vinodkv via acmurthy)
MAPREDUCE-3112. Fixed recursive sourcing of HADOOP_OPTS environment
variable. (Eric Yang)
MAPREDUCE-3056. Changed the default staging directory to not include
user.name to prevent issues with non-secure mode. (Devaraj K via vinodkv)
MAPREDUCE-2913. Fixed TestMRJobs.testFailingMapper to assert the correct
TaskCompletionEventStatus. (Jonathan Eagles via vinodkv)
MAPREDUCE-2794. [MR-279] Incorrect metrics value for AvailableGB per
queue per user. (John George via mahadev)
MAPREDUCE-2783. Fixing RM web-UI to show no tracking-URL when AM
crashes. (Eric Payne via vinodkv)
MAPREDUCE-3141. Fix the broken MRAppMaster to work over YARN in security
mode.(vinodkv)
MAPREDUCE-2751. Modified NodeManager to stop leaving around local files
after application finishes. (Siddharth Seth via vinodkv)
MAPREDUCE-3033. Ensure Master interface pays attention to classic v/s yarn
frameworks. (Hitesh Shah via acmurthy)
MAPREDUCE-2802. Ensure JobHistory filenames have jobId. (Jonathan Eagles
via acmurthy)
MAPREDUCE-2876. Use a different config for ContainerAllocationExpirer.
(Anupam Seth via acmurthy)
MAPREDUCE-3153. Fix TestFileOutputCommitter which was broken by
MAPREDUCE-2702. (mahadev via acmurthy)
MAPREDUCE-3123. Fix NM to quote symlink names to escape special
characters. (Hitesh Shah via acmurthy)
MAPREDUCE-3154. Fix JobSubmitter to check for output specs before copying
job submission files to fail fast. (Abhijit Suresh Shingate via acmurthy)
MAPREDUCE-3158. Fix test failures in MRv1 due to default framework being
set to yarn. (Hitesh Shah via acmurthy)
MAPREDUCE-3167. container-executor is not being packaged with the assembly
target. (mahadev)
MAPREDUCE-3020. Fixed TaskAttemptImpl to log the correct node-address for
a finished Reduce task. (Chackaravarthy via vinodkv)
MAPREDUCE-2668. Fixed AuxServices to send a signal on application-finish
to all the services. (Thomas Graves via vinodkv)
MAPREDUCE-3126. Fixed a corner case in CapacityScheduler where headroom
wasn't updated on changes to cluster size. (acmurthy)
MAPREDUCE-3140. Fixed the invalid JobHistory URL for failed
applications. (Subroto Sanyal via vinodkv)
MAPREDUCE-3125. Modified TaskImpl to consider only non-failed, non-killed
task-attempts for obtaining task's progress. (Hitesh Shah via vinodkv)
MAPREDUCE-2666. Retrieve shuffle port number from JobHistory on MR AM
restart. (Jonathan Eagles via acmurthy)
MAPREDUCE-2789. Complete schedulingInfo on CLI. (Eric Payne via acmurthy)
MAPREDUCE-3170. Fixed job output commit for deep hierarchies. (Hitesh Shah
via acmurthy)
MAPREDUCE-3124. Fixed location of native libs i.e. libhadoop.so for
containers. (John George via acmurthy)
MAPREDUCE-3057. Job History Server goes of OutOfMemory with 1200 Jobs
and Heap Size set to 10 GB. (Eric Payne via mahadev)
MAPREDUCE-2840. mr279 TestUberAM.testSleepJob test fails. (jonathan eagles
via mahadev)
MAPREDUCE-3190. Ensure bin/yarn fails early with a clear error message
when HADOOP_COMMON_HOME or HADOOP_HDFS_HOME are not set. (todd & acmurthy
via acmurthy)
MAPREDUCE-3189. Add link decoration back to MR2's CSS. (Todd Lipcon via
mahadev)
MAPREDUCE-3127. Changed default value of yarn.resourcemanager.acl.enable
to true and added some more documentation. (acmurthy)
MAPREDUCE-3032. Fixed TaskAttemptImpl so that JobHistory can have error
information about failed tasks. (Devaraj K via vinodkv)
MAPREDUCE-3196. TestLinuxContainerExecutorWithMocks fails on Mac OSX.
(Arun Murthy via mahadev)
MAPREDUCE-3197. TestMRClientService failing on building clean checkout of
branch 0.23 (mahadev)
MAPREDUCE-2762. Cleanup MR staging directory on completion. (mahadev via
acmurthy)
MAPREDUCE-3165. Ensure logging options are set correctly for MR AM and
tasks. (todd via acmurthy)
MAPREDUCE-3203. Fix some javac warnings in MRAppMaster. (mahadev)
MAPREDUCE-3162. Separated application-init and container-init event types
in NodeManager's Application state machine. (Todd Lipcon via vinodkv)
MAPREDUCE-3176. Fixed ant mapreduce tests that are timing out because
of wrong framework name. (Hitesh Shah via vinodkv)
MAPREDUCE-3181. Fixed MapReduce runtime to load yarn-default.xml and
yarn-site.xml. (acmurthy)
MAPREDUCE-2788. Normalize resource requests in FifoScheduler
appropriately. (Ahmed Radwan via acmurthy)
MAPREDUCE-2693. Fix NPE in job-blacklisting. (Hitesh Shah via acmurthy)
MAPREDUCE-3208. Fix NPE task/container log appenders. (liangzhwa via
acmurthy)
MAPREDUCE-3212. Fix usage/help message for bin/yarn. (Bhallamudi Venkata
Siva Kamesh via acmurthy)
MAPREDUCE-3179. Ensure failed tests exit with right error code. (Jonathan
Eagles via acmurthy)
MAPREDUCE-3188. Ensure correct shutdown in services. (todd via acmurthy)
MAPREDUCE-3226. Fix shutdown of fetcher threads. (vinodkv via acmurthy)
MAPREDUCE-3070. Fix NodeManager to use ephemeral ports by default.
(Devaraj K via acmurthy)
MAPREDUCE-3242. Trunk compilation broken with bad interaction from
MAPREDUCE-3070 and MAPREDUCE-3239. (mahadev)
MAPREDUCE-3058. Fixed MR YarnChild to report failure when task throws an
error and thus prevent a hanging task and job. (vinodkv)
MAPREDUCE-3087. Fixed the mapreduce classpath to correctly include the
generated-classpath file needed for tests. (Ravi Prakash via vinodkv)
MAPREDUCE-3233. Fixed a bug in MR Job so as to be able to restart the
application on AM crash. (Mahadev Konar via vinodkv)
MAPREDUCE-3028. Added job-end notification support. (Ravi Prakash via
acmurthy)
MAPREDUCE-3249. Ensure shuffle-port is correctly used duringMR AM recovery.
(vinodkv via acmurthy)
MAPREDUCE-3252. Fix map tasks to not rewrite data an extra time when
map output fits in spill buffer. (todd)
MAPREDUCE-3159. Ensure DefaultContainerExecutor doesn't delete application
directories during app-init. (todd via acmurthy)
MAPREDUCE-2746. Yarn servers can't communicate with each other with
hadoop.security.authorization set to true (acmurthy via mahadev)
MAPREDUCE-2821. Added missing fields (resourcePerMap & resourcePerReduce)
to JobSummary logs. (mahadev via acmurthy)
MAPREDUCE-3253. Fixed ContextFactory to clone JobContext correctly.
(acmurthy)
MAPREDUCE-3263. Fixed the MAPREDUCE-3028 commit which broke MR1. (Hitesh
Shah via acmurthy)
MAPREDUCE-3269. Fixed log4j properties to correctly set logging options
for JobHistoryServer vis-a-vis JobSummary logs. (mahadev via acmurthy)
MAPREDUCE-3250. When AM restarts, client keeps reconnecting to the new AM
and prints a lots of logs. (vinodkv via mahadev)
MAPREDUCE-3254. Fixed streaming to set the job.jar by using the right
JobConf ctor. (acmurthy)
MAPREDUCE-3264. mapreduce.job.user.name needs to be set automatically.
(acmurthy via mahadev)
MAPREDUCE-3175. Add authorization to admin web-pages such as /stacks, /jmx
etc. (Jonathan Eagles via acmurthy)
MAPREDUCE-3257. Added authorization checks for the protocol between
ResourceManager and ApplicationMaster. (vinodkv via acmurthy)
MAPREDUCE-3259. Added java.library.path of NodeManager to
ContainerLocalizer in LinuxContainerExecutor. (Kihwal Lee via acmurthy)
MAPREDUCE-3279. Fixed TestJobHistoryParsing which assumed user name to be
mapred all the time. (Siddharth Seth via acmurthy)
MAPREDUCE-3240. Fixed NodeManager to be able to forcefully cleanup its
containers (process-trees) irrespective of whether the container succeeded,
or killed. Contributed by Hitesh Shah.
MAPREDUCE-3281. Fixed a bug in TestLinuxContainerExecutorWithMocks. (vinodkv)
MAPREDUCE-3228. Fixed MR AM to timeout RPCs to bad NodeManagers. (vinodkv
via acmurthy)
MAPREDUCE-3284. Moved JobQueueClient to hadoop-mapreduce-client-core.
(acmurthy)
MAPREDUCE-3282. bin/mapred job -list throws exception. (acmurthy via
mahadev)
MAPREDUCE-3186. User jobs are getting hanged if the Resource manager
process goes down and comes up while job is getting executed.
(Eric Payne via mahadev)
MAPREDUCE-3285. Tests on branch-0.23 failing (Siddharth Seth via mahadev)
MAPREDUCE-3258. Fixed AM & JobHistory web-ui to display counters properly.
(Siddharth Seth via acmurthy)
MAPREDUCE-3290. Fixed a NPE in ClientRMService. (acmurthy)
MAPREDUCE-3185. RM Web UI does not sort the columns in some cases.
(Jonathan Eagles via mahadev)
MAPREDUCE-3292. In secure mode job submission fails with Provider
org.apache.hadoop.mapreduce.security.token.JobTokenIndentifier$Renewer
not found. (mahadev)
MAPREDUCE-3296. Fixed the remaining nine FindBugs warnings. (vinodkv)
MAPREDUCE-2775. Fixed ResourceManager and NodeManager to force a
decommissioned node to shutdown. (Devaraj K via vinodkv)
MAPREDUCE-3304. Fixed intermittent test failure due to a race in
TestRMContainerAllocator#testBlackListedNodes. (Ravi Prakash via acmurthy)
MAPREDUCE-3306. Fixed a bug in NodeManager ApplicationImpl that was causing
NodeManager to crash. (vinodkv)
MAPREDUCE-3295. TestAMAuthorization failing on branch 0.23. (vinodkv via mahadev)
MAPREDUCE-3183. hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
missing license header. (Hitesh Shah via tucu).
MAPREDUCE-3003. Publish MR JARs to Maven snapshot repository. (tucu)
MAPREDUCE-3199. Fixed pom files to include correct log4j configuration for
tests. (vinodkv)
MAPREDUCE-3204. mvn site:site fails on MapReduce. (tucu)
MAPREDUCE-3248. Fixed log4j properties. (vinodkv via acmurthy)
MAPREDUCE-3256. Added authorization checks for the protocol between
NodeManager and ApplicationMaster. (vinodkv via acmurthy)
MAPREDUCE-3274. Fixed a race condition in MRAppMaster that was causing a
task-scheduling deadlock. (Robert Joseph Evans via vinodkv)
MAPREDUCE-3171 merge from trunk reverted changes from MAPREDUCE-2747 MAPREDUCE-3240.
MAPREDUCE-3313. Fixed initialization of ClusterMetrics which was failing
TestResourceTrackerService sometimes. (Hitesh Shah via vinodkv)
MAPREDUCE-2766. Fixed NM to set secure permissions for files and directories
in distributed-cache. (Hitesh Shah via vinodkv)
MAPREDUCE-2696. Fixed NodeManager to cleanup logs in a thread when logs'
aggregation is not enabled. (Siddharth Seth via vinodkv)
MAPREDUCE-3262. Fixed Container's state-machine in NodeManager to handle
a couple of events in failure states correctly. (Hitesh Shah and Siddharth
Seth via vinodkv)
MAPREDUCE-3035. Fixed MR JobHistory to ensure rack information is present.
(chakravarthy via acmurthy)
MAPREDUCE-3321. Disabled a few MR tests for 0.23. (Hitesh Shah via
acmurthy)
MAPREDUCE-3220. Fixed TestCombineOutputCollector. (Devaraj K via acmurthy)
MAPREDUCE-3103. Implement Job ACLs for MRAppMaster.
(mahadev)
MAPREDUCE-3241. [Rumen] Fix Rumen to ignore the AMStartedEvent. (amarrk)
MAPREDUCE-3166. [Rumen] Make Rumen use job history api instead of relying
on current history file name format. (Ravi Gummadi via amarrk)
MAPREDUCE-3157. [Rumen] Fix TraceBuilder to handle 0.20 history file
names also. (Ravi Gummadi via amarrk)
MAPREDUCE-3081. Fix vaidya startup script. (gkesavan via suhas).
MAPREDUCE-2764. Fix renewal of dfs delegation tokens. (Owen via jitendra)
MAPREDUCE-3192. Fix Javadoc warning in JobClient.java and Cluster.java.
(jitendra)
MAPREDUCE-3237. Move LocalJobRunner to hadoop-mapreduce-client-core.
(tomwhite via acmurthy)
MAPREDUCE-3316. Rebooted link is not working properly.
(Bhallamudi Venkata Siva Kamesh via mahadev)
MAPREDUCE-3317. Rumen TraceBuilder is emiting null as hostname.
(Ravi Gummadi via mahadev)
MAPREDUCE-3332. contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/
Sender#writeBlock related to checksum handling (Hitesh Shah via mahadev)
MAPREDUCE-3337. Added missing license headers. (acmurthy)
Release 0.22.1 - Unreleased
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
OPTIMIZATIONS
BUG FIXES
MAPREDUCE-3837. Job tracker is not able to recover jobs after crash.
(Mayank Bansal via shv)
Release 0.22.0 - 2011-11-29
INCOMPATIBLE CHANGES
MAPREDUCE-1866. Removes deprecated class
org.apache.hadoop.streaming.UTF8ByteArrayUtils. (amareshwari)
MAPREDUCE-1664. Changes the behaviour of the combination of job-acls
when they function together with queue-acls. (Ravi Gummadi via vinodkv)
MAPREDUCE-2994. Fixed a bug in ApplicationID parsing that affects RM
UI. (Devaraj K via vinodkv)
MAPREDUCE-1788. o.a.h.mapreduce.Job shouldn't make a copy of the JobConf.
(Arun Murthy via mahadev)
NEW FEATURES
MAPREDUCE-1804. Stress-test tool for HDFS introduced in HDFS-708.
(Joshua Harlow via shv)
MAPREDUCE-220. Collect cpu and memory statistics per task. (Scott Chen via
acmurthy)
MAPREDUCE-1970. Reed-Solomon code implementation for HDFS RAID.
(Scott Chen via dhruba)
MAPREDUCE-2169. Integrated Reed-Solomon code with RaidNode. (Ramkumar
Vadali via schen)
MAPREDUCE-2936. Contrib Raid compilation broken after HDFS-1620. (vinodkv)
IMPROVEMENTS
MAPREDUCE-2141. Add an "extra data" field to Task for use by Mesos. (matei)
MAPREDUCE-2140. Regenerate fair scheduler design doc PDF. (matei)
MAPREDUCE-1546. Redirect all job pages to corresponding history page
if job is not in JT memory. (Scott Chen via sharad)
MAPREDUCE-1092. Enable assertions for unit tests. (Eli Collins via
cdouglas)
MAPREDUCE-1680. Add a metric recording JobTracker heartbeats processed.
(Dick King via cdouglas)
MAPREDUCE-1761. FairScheduler allows separate configuration of node
and rack locality wait time (Scott Chen via dhruba)
MAPREDUCE-1539. authorization checks for inter-server protocol
(based on HADOOP-6600) (Boris Shkolnik via shv)
MAPREDUCE-1798. Names the configuration keys for the Kerberos
principals better. (Boris Shkolnik via ddas)
MAPREDUCE-1773. streaming doesn't support jobclient.output.filter.
(Amareshwari Sriramadasu via vinodkv)
MAPREDUCE-1785. Add streaming config option for not emitting the key.
(Eli Collins via sharad)
MAPREDUCE-572. If #link is missing from uri format of -cacheArchive
then streaming does not throw error. (Amareshwari Sriramadasu via
vinodkv)
MAPREDUCE-1545. Add timestamps for first task type launched in job summary.
(Luke Lu via cdouglas)
MAPREDUCE-1543. Add an audit log for authentication events. (Amar Kamat and
Luke Lu via cdouglas)
MAPREDUCE-1762. Add ability to set values of task counters. (Scott Chen via
cdouglas)
MAPREDUCE-1533. Reduce overhead of logging and string manipulation during
heartbeat processing. (Amar Kamat and Dick King via cdouglas)
MAPREDUCE-1516. JobTracker issues delegation tokens only if the user's
authentication is Kerberos. (Jitendra Pandey via ddas)
MAPREDUCE-647. Update distcp forrest documentation to reflect the changes
of HADOOP-5472, MAPREDUCE-642 and HADOOP-5620. (Rodrigo Schmidt via
szetszwo)
MAPREDUCE-1851. Documents configuration parameters in streaming.
(amareshwari)
MAPREDUCE-1868. Add a read and connection timeout to JobClient while
pulling tasklogs. (Krishna Ramachandran via acmurthy)
MAPREDUCE-1778. Ensure failure to setup CompletedJobStatusStore is not
silently ignored by the JobTracker. (Krishna Ramachandran via acmurthy)
MAPREDUCE-1850. Includes job submit host information (name and ip) in
jobconf and jobdetails display (Krishna Ramachandran via amareshwari)
MAPREDUCE-1893. Slive with multiple reducers. (shv)
MAPREDUCE-1248. Fixes redudant memory copying in StreamKeyValUtil.
(Ruibang He via amareshwari)
MAPREDUCE-1840. Enhancements to Gridmix benchmark simulating user
diversity, queue replay, and task duration for JobTracker load testing.
Also includes compatibility with security enhancements, and scalability
improvements. (Amar Kamat, Rahul Singh, Hong Tang, and cdouglas)
MAPREDUCE-1848. Put number of speculative, data local, rack local
tasks in JobTracker metrics. (Scott Chen via dhruba)
MAPREDUCE-1935. Makes the Distcp to work in a secure environment.
(Boris Shkolnik via ddas)
MAPREDUCE-1945. The MapReduce component for HADOOP-6632.
(Kan Zhang & Jitendra Pandey via ddas)
MAPREDUCE-1936. Modify Gridmix3 to support more tunable parameters for
stress submission and sleep jobs. (Hong Tang via cdouglas)
MAPREDUCE-1733. Makes pipes applications secure. (Jitendra Pandey via ddas)
MAPREDUCE-1566. Adds a configuration attribute using which job clients can
specify a credentials file. The tokens from there will be passed to the job.
(Jitendra Pandey and Owen O'Malley via ddas)
MAPREDUCE-1624. Documents the job credentials and associated details to do
with delegation tokens (on the client side).
(Jitendra Pandey and Devaraj Das via ddas)
MAPREDUCE-1834. TestSimulatorDeterministicReplay timesout on trunk.
(Hong Tang via mahadev)
MAPREDUCE-1993. Fixes test failure
TestTrackerDistributedCacheManagerWithLinuxTaskController. (Devaraj Das
via amareshwari)
MAPREDUCE-1523. Making Mumak work with Capacity-Scheduler (Anirban Das
via mahadev)
MAPREDUCE-1920. Enables completed jobstatus store by default. (Tom White
via amareshwari)
MAPREDUCE-1881. Improve TaskTrackerInstrumentation to enable collection of
advanced metrics. (Matei Zaharia via acmurthy)
MAPREDUCE-1548. Hadoop archives preserve times and other properties from
original files. (Rodrigo Schmidt via dhruba)
MAPREDUCE-1517. Supports streaming job to run in the background. (Bochun Bai
via amareshwari)
MAPREDUCE-1819. RaidNode is now smarter in submitting Raid jobs. (Ramkumar
Vadali via schen)
MAPREDUCE-2132. A command line option in RaidShell to fix blocks using raid
MAPREDUCE-2147. Remove redundant lines in JobInProgress ctor.
(Harsh J Chouraria via cutting)
HADOOP-7007. Update the hudson-test-patch ant target to work with the
latest test-patch.sh script (gkesavan)
MAPREDUCE-1818. RaidNode can specify scheduling parameters. (Ramkumar
Vadali via schen)
MAPREDUCE-2051. Contribute a fair scheduler preemption system test.
(Todd Lipcon via tomwhite)
MAPREDUCE-1892. RaidNode can allow layered policies more efficiently.
(Ramkumar Vadali via schen)
MAPREDUCE-1592. Generate Eclipse's .classpath file from Ivy config.
(tomwhite via nigel)
MAPREDUCE-2073. TestTrackerDistributedCacheManager should be up-front
about requirements on build environment. (Todd Lipcon via tomwhite)
MAPREDUCE-2093. Herriot JT and TT clients should vend statistics. (cos)
MAPREDUCE-2167. Faster directory traversal for raid node. (Ramkumar Vadali
via schen)
MAPREDUCE-1931. Gridmix forrest documentation . (Ranjit Mathew via vinodkv).
MAPREDUCE-2184. Port DistRaid.java to new mapreduce API. (Ramkumar Vadali
via schen)
MAPREDUCE-1878. Add MRUnit documentation. (Aaron Kimball via tomwhite)
MAPREDUCE-2180. Add coverage of fair scheduler servlet to system test (todd)
MAPREDUCE-2250. Fix logging in raid code. (Ramkumar Vadali via schen)
MAPREDUCE-2260. Remove auto-generated native build files. (rvs via eli)
MAPREDUCE-2314. configure files that are generated as part of the released
tarball need to have executable bit set (rvs via cos)
MAPREDUCE-1159. Limit Job name in web UI to be 80 char long. (Harsh J
Chouraria via szetszwo)
MAPREDUCE-2337. Remove dependence of public MapReduce API on classes in
server package. (tomwhite)
MAPREDUCE-2383. Improve documentation of DistributedCache methods (Harsh J
Chouraria via todd)
MAPREDUCE-2222. Ivy resolve force mode should be turned off by default.
(Luke Lu via tomwhite)
MAPREDUCE-2103. task-controller shouldn't require o-r permissions.
(todd via eli)
MAPREDUCE-2505. Explain how to use ACLs in the fair scheduler.
(matei via eli)
MAPREDUCE-3138. Add a utility to help applications bridge changes in
Context Objects APIs due to MAPREDUCE-954. (omalley via acmurthy)
OPTIMIZATIONS
MAPREDUCE-1354. Enhancements to JobTracker for better performance and
scalability. (Arun C. Murthy & Richard King via acmurthy)
MAPREDUCE-1829. JobInProgress.findSpeculativeTask should use min() to
find the candidate instead of sort(). (Scott Chen via vinodkv)
BUG FIXES
MAPREDUCE-1845. FairScheduler.tasksToPreempt() can return negative number.
(Scott Chen via matei)
MAPREDUCE-1707. TaskRunner can get NPE in getting ugi from TaskTracker.
(Vinod Kumar Vavilapalli)
MAPREDUCE-1532. Ensures that delegation tokens is obtained as the
actual user when the proxy-user is used for submitting jobs. Also
refactors the DelegationTokenToRenew class. (ddas)
MAPREDUCE-1558. Fixes MRAdmin to look up the principal of the
JobTracker and use that in the RefreshUserToGroupsMapping protocol and
RefreshAuthorizationPolicyProtocol. (Boris Shkolnik via ddas)
MAPREDUCE-1662. Remove unused methods from TaskRunner. (Amareshwari
Sriramadasu via cdouglas)
MAPREDUCE-1617. Use IPv4 stack for unit tests. (Amar Kamat and Luke Lu via
cdouglas)
MAPREDUCE-1599. Fixes MRBench so that it reuses tokens across jobs
correctly. (Jitendra Nath Pandey via ddas)
MAPREDUCE-1836. Refresh for proxy superuser config (mr part for HDFS-1096).
(Boris Shkolnik via shv)
MAPREDUCE-1505. Create RPC client on job submission, not in cstr of Job
instance. (Dick King via cdouglas)
MAPREDUCE-1813. NPE in PipeMapred.MRErrorThread. (Ravi Gummadi via vinodkv)
MAPREDUCE-1225. Fixes DistributedCache to check if the file is fresh or not,
for the first localization also. (Zhong Wang via amareshwari)
MAPREDUCE-1559. Fixes the token renewer to use the JobTracker's
credentials for talking to the NameNode. (ddas)
MAPREDUCE-1492. Delete obsolete har files used on the parity files
of hdfs raid. (Rodrigo Schmidt via dhruba)
MAPREDUCE-1857. Removes unused configuration parameters in streaming.
(amareshwari)
MAPREDUCE-1887. MRAsyncDiskService now properly absolutizes volume root
paths. (Aaron Kimball via zshao)
MAPREDUCE-1863. Fix NPE in Rumen when processing null CDF for failed task
attempts. (Amar Kamat via cdouglas)
MAPREDUCE-1864. Removes uninitialized/unused variables in
org.apache.hadoop.streaming.PipeMapRed. (amareshwari)
MAPREDUCE-1888. Fixes Streaming to override output key and value types,
only if mapper/reducer is a command. (Ravi Gummadi via amareshwari)
MAPREDUCE-577. Fixes duplicate records in StreamXmlRecordReader.
(Ravi Gummadi via amareshwari)
MAPREDUCE-1894. Fixed a bug in DistributedRaidFileSystem.readFully()
that was causing it to loop infinitely. (Ramkumar Vadali via dhruba)
MAPREDUCE-1838. Reduce the time needed for raiding a bunch of files
by randomly assigning files to map tasks. (Ramkumar Vadali via dhruba)
MAPREDUCE-1820. Fix InputSampler to clone sampled keys. (Alex Kozlov via
cdouglas)
MAPREDUCE-1528. Incorporates the changes to the credentials API done in
HADOOP-6845. Also, introduces Credentials in JobConf, and in JobContext.
(Jitendra Pandey and Arun Murthy via ddas)
MAPREDUCE-1865. Rumen should also support jobhistory files generated using
trunk. (Amar Kamat via amareshwari)
MAPREDUCE-1621. Fixes NPE in TextOutputReader.getLastOutput if it has never
read any output. (amareshwari)
MAPREDUCE-1911. Fixes errors in -info message in streaming. (amareshwari)
MAPREDUCE-1772. Corrects errors in streaming documentation in forrest.
(amareshwari)
MAPREDUCE-1925. Fix failing TestRumenJobTraces. (Ravi Gummadi via cdouglas)
MAPREDUCE-1718. Fixes a bug in the construction of jobconf key for the
mapping that the tasks use at runtime for looking up delegation tokens.
(Boris Shkolnik via ddas)
MAPREDUCE-1701. Fixes a problem to do with exception handling in