| Hadoop MapReduce Change Log |
| |
| Trunk (unreleased changes) |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| MAPREDUCE-1017. Compression and output splitting for Sqoop. |
| (Aaron Kimball via tomwhite) |
| |
| IMPROVEMENTS |
| |
| MAPREDUCE-707. Provide a jobconf property for explicitly assigning a job to |
| a pool in the Fair Scheduler. (Alan Heirich via matei) |
| |
| MAPREDUCE-999. Improve Sqoop test speed and refactor tests. |
| (Aaron Kimball via tomwhite) |
| |
| MAPREDUCE-906. Update Sqoop documentation. (Aaron Kimball via cdouglas) |
| |
| MAPREDUCE-947. Added commitJob and abortJob apis to OutputCommitter. |
| Enhanced FileOutputCommitter to create a _SUCCESS file for successful |
| jobs. (Amar Kamat & Jothi Padmanabhan via acmurthy) |
| |
| MAPREDUCE-1103. Added more metrics to Jobtracker. (sharad) |
| |
| MAPREDUCE-1048. Add occupied/reserved slot usage summary on jobtracker UI. |
| (Amareshwari Sriramadasu and Hemanth Yamijala via sharad) |
| |
| MAPREDUCE-1090. Modified log statement in TaskMemoryManagerThread to |
| include task attempt id. (yhemanth) |
| |
| MAPREDUCE-1069. Implement Sqoop API refactoring. (Aaron Kimball via |
| tomwhite) |
| |
| MAPREDUCE-1036. Document Sqoop API. (Aaron Kimball via cdouglas) |
| |
| OPTIMIZATIONS |
| |
| MAPREDUCE-270. Fix the tasktracker to optionally send an out-of-band |
| heartbeat on task-completion for better job-latency. (acmurthy) |
| Configuration changes: |
| add mapreduce.tasktracker.outofband.heartbeat |
| |
| BUG FIXES |
| |
| MAPREDUCE-1089. Fix NPE in fair scheduler preemption when tasks are |
| scheduled but not running. (Todd Lipcon via matei) |
| |
| MAPREDUCE-1014. Fix the libraries for common and hdfs. (omalley) |
| |
| MAPREDUCE-1111. JT Jetty UI not working if we run mumak.sh |
| off packaged distribution directory. (hong tang via mahadev) |
| |
| MAPREDUCE-1133. Eclipse .classpath template has outdated jar files and is |
| missing some new ones. (cos) |
| |
| MAPREDUCE-1098. Fixed the distributed-cache to not do i/o while holding a |
| global lock. (Amareshwari Sriramadasu via acmurthy) |
| |
| MAPREDUCE-1158. Fix JT running maps and running reduces metrics. |
| (sharad) |
| |
| MAPREDUCE-1160. Reduce verbosity of log lines in some Map/Reduce classes |
| to avoid filling up jobtracker logs on a busy cluster. |
| (Ravi Gummadi and Hong Tang via yhemanth) |
| |
| MAPREDUCE-1153. Fix tasktracker metrics when trackers are decommissioned. |
| (sharad) |
| |
| MAPREDUCE-1128. Fix MRUnit to prohibit iterating over values twice. (Aaron |
| Kimball via cdouglas) |
| |
| MAPREDUCE-665. Move libhdfs to HDFS subproject. (Eli Collins via dhruba) |
| |
| MAPREDUCE-1196. Fix FileOutputCommitter to use the deprecated cleanupJob |
| api correctly. (acmurthy) |
| |
| Release 0.21.0 - Unreleased |
| |
| INCOMPATIBLE CHANGES |
| |
| MAPREDUCE-516. Fix the starvation problem in the Capacity Scheduler |
| when running High RAM Jobs. (Arun Murthy via yhemanth) |
| |
| MAPREDUCE-358. Change org.apache.hadoop.examples. AggregateWordCount |
| and org.apache.hadoop.examples.AggregateWordHistogram to use new |
| mapreduce api. (Amareshwari Sriramadasu via sharad) |
| |
| MAPREDUCE-245. Change Job and jobcontrol classes to use the List interface |
| rather than ArrayList in APIs. (Tom White via cdouglas) |
| |
| MAPREDUCE-766. Enhanced list-blacklisted-trackers to display reasons |
| for blacklisting a node. (Sreekanth Ramakrishnan via yhemanth) |
| |
| MAPREDUCE-817. Add a cache for retired jobs with minimal job info and |
| provide a way to access history file url. (sharad) |
| |
| MAPREDUCE-711. Moved Distributed Cache from Common to Map/Reduce |
| project. (Vinod Kumar Vavilapalli via yhemanth) |
| |
| MAPREDUCE-895. Per the contract elucidated in HADOOP-6201, throw |
| FileNotFoundException from FileSystem::listStatus rather than returning |
| null. (Jakob Homan via cdouglas) |
| |
| MAPREDUCE-479. Provide full task id to map output servlet rather than the |
| reduce id, only. (Jiaqi Tan via cdouglas) |
| |
| MAPREDUCE-873. Simplify job recovery. Incomplete jobs are resubmitted on |
| jobtracker restart. Removes a public constructor in JobInProgress. (sharad) |
| |
| HADOOP-6230. Moved process tree and memory calculator related classes from |
| Common to Map/Reduce. (Vinod Kumar Vavilapalli via yhemanth) |
| |
| MAPREDUCE-157. Refactor job history APIs and change the history format to |
| JSON. (Jothi Padmanabhan via sharad) |
| |
| MAPREDUCE-849. Rename configuration properties. (Amareshwari Sriramadasu |
| via sharad) |
| |
| NEW FEATURES |
| |
| MAPREDUCE-706. Support for FIFO pools in the fair scheduler. |
| (Matei Zaharia) |
| |
| MAPREDUCE-546. Provide sample fair scheduler config file in conf/ and use |
| it by default if no other config file is specified. (Matei Zaharia) |
| |
| MAPREDUCE-551. Preemption support in the Fair Scheduler. (Matei Zaharia) |
| |
| HADOOP-5887. Sqoop should create tables in Hive metastore after importing |
| to HDFS. (Aaron Kimball via tomwhite) |
| |
| MAPREDUCE-567. Add a new example MR that always fails. (Philip Zeyliger |
| via tomwhite) |
| |
| MAPREDUCE-211. Provides ability to run a health check script on the |
| tasktracker nodes and blacklist nodes if they are unhealthy. |
| (Sreekanth Ramakrishnan via yhemanth) |
| |
| MAPREDUCE-637. Add an example, distbbp, which able to compute the n th bit |
| of Pi for some large n. (szetszwo) |
| |
| MAPREDUCE-532. Provide a way to limit the number of used slots |
| per queue in the capacity scheduler. |
| (Rahul Kumar Singh via yhemanth) |
| |
| MAPREDUCE-467. Provide ability to collect statistics about total tasks |
| and succeeded tasks in different time windows. (sharad) |
| |
| MAPREDUCE-740. Log a job-summary at the end of a job, while allowing it |
| to be configured to use a custom appender if desired. (acmurthy) |
| |
| MAPREDUCE-705. User-configurable quote and delimiter characters for Sqoop |
| records and record reparsing. (Aaron Kimball via tomwhite) |
| |
| MAPREDUCE-814. Provide a way to configure completed job history files |
| to be on HDFS. (sharad) |
| |
| MAPREDUCE-800. MRUnit should support the new API. (Aaron Kimball via |
| tomwhite) |
| |
| MAPREDUCE-798. MRUnit should be able to test a succession of MapReduce |
| passes. (Aaron Kimball via tomwhite) |
| |
| MAPREDUCE-768. Provide an option to dump jobtracker configuration in JSON |
| format to standard output. (V.V.Chaitanya Krishna via yhemanth) |
| |
| MAPREDUCE-824. Add support for a hierarchy of queues in the capacity |
| scheduler. (Rahul Kumar Singh via yhemanth) |
| |
| MAPREDUCE-751. Add Rumen, a tool for extracting statistics from job tracker |
| logs and generating job traces for simulation and analysis. (Dick King via |
| cdouglas) |
| |
| MAPREDUCE-938. Postgresql support for Sqoop. (Aaron Kimball via tomwhite) |
| |
| MAPREDUCE-830. Add support for splittable compression to TextInputFormats. |
| (Abdul Qadeer via cdouglas) |
| |
| MAPREDUCE-861. Add support for hierarchical queues in the Map/Reduce |
| framework. (Rahul Kumar Singh via yhemanth) |
| |
| MAPREDUCE-776. Add Gridmix, a benchmark processing Rumen traces to simulate |
| a measured mix of jobs on a cluster. (cdouglas) |
| |
| MAPREDUCE-862. Enhance JobTracker UI to display hierarchical queues. |
| (V.V.Chaitanya Krishna via yhemanth) |
| |
| MAPREDUCE-777. Brand new apis to track and query jobs as a |
| replacement for JobClient. (Amareshwari Sriramadasu via acmurthy) |
| |
| MAPREDUCE-775. Add native and streaming support for Vertica as an input |
| or output format taking advantage of parallel read and write properties of |
| the DBMS. (Omer Trajman via ddas) |
| |
| MAPREDUCE-679. XML-based metrics as JSP servlet for JobTracker. |
| (Aaron Kimball via tomwhite) |
| |
| MAPREDUCE-980. Modify JobHistory to use Avro for serialization. (cutting) |
| |
| MAPREDUCE-728. Add Mumak, a Hadoop map/reduce simulator. (Arun C Murthy, |
| Tamas Sarlos, Anirban Dasgupta, Guanying Wang, and Hong Tang via cdouglas) |
| |
| IMPROVEMENTS |
| |
| MAPREDUCE-816. Rename "local" mysql import to "direct" in Sqoop. |
| (Aaron Kimball via matei) |
| |
| HADOOP-5967. Sqoop should only use a single map task. (Aaron Kimball via |
| tomwhite) |
| |
| HADOOP-5968. Sqoop should only print a warning about mysql import speed |
| once. (Aaron Kimball via tomwhite) |
| |
| MAPREDUCE-463. Makes job setup and cleanup tasks as optional. |
| (Amareshwari Sriramadasu via sharad) |
| |
| MAPREDUCE-502. Allow jobtracker to be configured with zero completed jobs |
| in memory. (Amar Kamat via sharad) |
| |
| MAPREDUCE-416. Moves the history file to a "done" folder whenever a job |
| completes. (Amar Kamat via ddas) |
| |
| MAPREDUCE-646. Increase srcfilelist replication number in dictcp job. |
| (Ravi Gummadi via szetszwo) |
| |
| HADOOP-6106. Updated hadoop-core and test jars from hudson trunk |
| build #12. (Giridharan Kesavan) |
| |
| MAPREDUCE-642. A option to distcp that allows preserving the full |
| source path of a file in the specified destination directory. |
| (Rodrigo Schmidt via dhruba) |
| |
| MAPREDUCE-686. Move TestSpeculativeExecution.Fake* into a separate class |
| so that it can be used by other tests. (Jothi Padmanabhan via sharad) |
| |
| MAPREDUCE-625. Modify TestTaskLimits to improve execution time. |
| (Jothi Padmanabhan via sharad) |
| |
| MAPREDUCE-465. Deprecate o.a.h.mapred.lib.MultithreadedMapRunner and add |
| test for o.a.h.mapreduce.lib.MultithreadedMapper. |
| (Amareshwari Sriramadasu via sharad) |
| |
| MAPREDUCE-692. Make Hudson run Sqoop unit tests. |
| (Aaron Kimball via tomwhite) |
| |
| MAPREDUCE-701. Improves the runtime of the TestRackAwareTaskPlacement |
| by making it a unit test. (Jothi Padmanabhan via ddas) |
| |
| MAPREDUCE-674. Sqoop should allow a "where" clause to avoid having to |
| export entire tables. (Kevin Weil via tomwhite) |
| |
| MAPREDUCE-675. Sqoop should allow user-defined class and package names. |
| (Aaron Kimball via tomwhite) |
| |
| MAPREDUCE-371. Change KeyFieldBasedComparator and KeyFieldBasedPartitioner |
| to use new api. (Amareshwari Sriramadasu via sharad) |
| |
| MAPREDUCE-713. Sqoop has some superfluous imports. (Aaron Kimball via |
| tomwhite) |
| |
| MAPREDUCE-623. Resolve javac warnings in mapreduce. (Jothi Padmanabhan |
| via sharad) |
| |
| MAPREDUCE-655. Change KeyValueLineRecordReader and KeyValueTextInputFormat |
| to use new mapreduce api. (Amareshwari Sriramadasu via sharad) |
| |
| MAPREDUCE-632. Merge TestCustomOutputCommitter with |
| TestCommandLineJobSubmission. (Jothi Padmanabhan via sharad) |
| |
| MAPREDUCE-710. Sqoop should read and transmit passwords in a more secure |
| manner. (Aaron Kimball via tomwhite) |
| |
| MAPREDUCE-627. Improves execution time of TestTrackerBlacklistAcrossJobs. |
| (Jothi Padmanabhan via ddas) |
| |
| MAPREDUCE-630. Improves execution time of TestKillCompletedJob. |
| (Jothi Padmanabhan via ddas) |
| |
| MAPREDUCE-626. Improves the execution time of TestLostTracker. |
| (Jothi Padmanabhan via ddas) |
| |
| MAPREDUCE-353. Makes the shuffle read and connection timeouts |
| configurable. (Ravi Gummadi via ddas) |
| |
| MAPREDUCE-739. Allow relative paths to be created in archives. (Mahadev |
| Konar via cdouglas) |
| |
| MAPREDUCE-772. Merge HADOOP-4010 changes to LineRecordReader into mapreduce |
| package. (Abdul Qadeer via cdouglas) |
| |
| MAPREDUCE-785. Separate sub-test of TestReduceFetch to be included in |
| MR-670. (Jothi Padmanabhan via cdouglas) |
| |
| MAPREDUCE-784. Modify TestUserDefinedCounters to use LocalJobRunner |
| instead of MiniMR. (Jothi Padmanabhan via sharad) |
| |
| HADOOP-6160. Fix releaseaudit target to run on specific directories. |
| (gkesavan) |
| |
| MAPREDUCE-782. Use PureJavaCrc32 in SpillRecord. (Todd Lipcon via |
| szetszwo) |
| |
| MAPREDUCE-369. Change org.apache.hadoop.mapred.lib.MultipleInputs to |
| use new api. (Amareshwari Sriramadasu via sharad) |
| |
| MAPREDUCE-373. Change org.apache.hadoop.mapred.lib.FieldSelectionMapReduce |
| to use new api. (Amareshwari Sriramadasu via sharad) |
| |
| MAPREDUCE-628. Improves the execution time of TestJobInProgress. |
| (Jothi Padmanabhan via ddas) |
| |
| MAPREDUCE-793. Creates a new test that consolidates a few tests to |
| include in the commit-test list. (Jothi Padmanabhan via ddas) |
| |
| MAPREDUCE-797. Adds combiner support to MRUnit MapReduceDriver. |
| (Aaron Kimball via johan) |
| |
| MAPREDUCE-656. Change org.apache.hadoop.mapred.SequenceFile* classes |
| to use new mapreduce api. (Amareshwari Sriramadasu via sharad) |
| |
| MAPREDUCE-670. Creates ant target for 10 mins patch test build. |
| (Jothi Padmanabhan via gkesavan) |
| |
| MAPREDUCE-375. Change org.apache.hadoop.mapred.lib.NLineInputFormat |
| and org.apache.hadoop.mapred.MapFileOutputFormat to use new api. |
| (Amareshwari Sriramadasu via ddas) |
| |
| MAPREDUCE-779. Added node health failure counts into |
| JobTrackerStatistics. (Sreekanth Ramakrishnan via yhemanth) |
| |
| MAPREDUCE-789. Oracle support for Sqoop. (Aaron Kimball via tomwhite) |
| |
| MAPREDUCE-842. Setup secure permissions for localized job files, |
| intermediate outputs and log files on tasktrackers. |
| (Vinod Kumar Vavilapalli via yhemanth) |
| |
| MAPREDUCE-478. Allow map and reduce jvm parameters, environment variables |
| and ulimit to be set separately. |
| Configuration changes: |
| add mapred.map.child.java.opts |
| add mapred.reduce.child.java.opts |
| add mapred.map.child.env |
| add mapred.reduce.child.ulimit |
| add mapred.map.child.env |
| add mapred.reduce.child.ulimit |
| deprecated mapred.child.java.opts |
| deprecated mapred.child.env |
| deprecated mapred.child.ulimit |
| (acmurthy) |
| |
| MAPREDUCE-767. Remove the dependence on the CLI 2.0 snapshot. |
| (Amar Kamat via omalley) |
| |
| MAPREDUCE-712. Minor efficiency tweaks to RandomTextWriter. (cdouglas) |
| |
| MAPREDUCE-870. Remove the job retire thread and the associated |
| config parameters. (sharad) |
| |
| MAPREDUCE-749. Make Sqoop unit tests more Hudson-friendly. |
| (Aaron Kimball via tomwhite) |
| |
| MAPREDUCE-874. Rename the PiEstimator example to QuasiMonteCarlo. |
| (szetszwo) |
| |
| MAPREDUCE-336. Allow logging level of map/reduce tasks to be configurable. |
| Configuration changes: |
| add mapred.map.child.log.level |
| add mapred.reduce.child.log.level |
| (acmurthy) |
| |
| MAPREDUCE-355. Update mapred.join package to use the new API. (Amareshwari |
| Sriramadasu via cdouglas) |
| |
| HADOOP-6184. Updated hadoop common and test jars to get the new API |
| in Configuration for dumping in JSON format from Hudson trunk build #68. |
| (yhemanth) |
| |
| MAPREDUCE-476. Extend DistributedCache to work locally (LocalJobRunner). |
| (Philip Zeyliger via tomwhite) |
| |
| MAPREDUCE-750. Extensible ConnManager factory API. (Aaron Kimball via |
| tomwhite) |
| |
| MAPREDUCE-825. JobClient completion poll interval of 5s causes slow tests |
| in local mode. (Aaron Kimball via tomwhite) |
| |
| MAPREDUCE-910. Support counters in MRUnit. (Aaron Kimball via cdouglas) |
| |
| MAPREDUCE-788. Update gridmix2 to use the new API (Amareshwari Sriramadasu |
| via cdouglas) |
| |
| MAPREDUCE-875. Make DBRecordReader execute queries lazily. (Aaron Kimball |
| via enis) |
| |
| MAPREDUCE-318. Modularizes the shuffle code. (Jothi Padmanabhan and |
| Arun Murthy via ddas) |
| |
| MAPREDUCE-936. Allow a load difference for fairshare scheduler. |
| (Zheng Shao via dhruba) |
| |
| MAPREDUCE-370. Update MultipleOutputs to use the API, merge funcitonality |
| of MultipleOutputFormat. (Amareshwari Sriramadasu via cdouglas) |
| |
| MAPREDUCE-898. Changes DistributedCache to use the new API. |
| (Amareshwari Sriramadasu via ddas) |
| |
| MAPREDUCE-876. Sqoop import of large tables can time out. |
| (Aaron Kimball via tomwhite) |
| |
| MAPREDUCE-918. Test hsqldb server should be memory-only. |
| (Aaron Kimball via tomwhite) |
| |
| MAPREDUCE-144. Includes dump of the process tree in task diagnostics when |
| a task is killed due to exceeding memory limits. |
| (Vinod Kumar Vavilapalli via yhemanth) |
| |
| MAPREDUCE-945. Modifies MRBench and TestMapRed to use ToolRunner so that |
| options such as queue name can be passed via command line. |
| (Sreekanth Ramakrishnan via yhemanth) |
| |
| MAPREDUCE-963. Deprecate o.a.h.mapred.FileAlreadyExistsException and |
| replace it with o.a.h.fs.FileAlreadyExistsException. (Boris Shkolnik |
| via szetszwo) |
| |
| MAPREDUCE-960. Remove an unnecessary intermediate copy and obsolete API |
| from KeyValueLineRecordReader. (cdouglas) |
| |
| MAPREDUCE-930. Modify Rumen to resolve paths in the canonical way, rather |
| than defaulting to the local filesystem. (cdouglas) |
| |
| MAPREDUCE-944. Extend the LoadManager API of the fair-share scheduler |
| to support regulating tasks for a job based on resources currently in use |
| by that job. (dhruba) |
| |
| MAPREDUCE-973. Move FailJob and SleepJob from examples to test. (cdouglas |
| via omalley) |
| |
| MAPREDUCE-966. Modify Rumen to clean up interfaces and simplify integration |
| with other tools. (Hong Tang via cdouglas) |
| |
| MAPREDUCE-856. Setup secure permissions for distributed cache files. |
| (Vinod Kumar Vavilapalli via yhemanth) |
| |
| MAPREDUCE-885. More efficient SQL queries for DBInputFormat. (Aaron Kimball |
| via enis) |
| |
| MAPREDUCE-284. Enables ipc.client.tcpnodelay in Tasktracker's Child. |
| (Ravi Gummadi via sharad) |
| |
| MAPREDUCE-907. Sqoop should use more intelligent splits. (Aaron Kimball |
| via tomwhite) |
| |
| MAPREDUCE-916. Split the documentation to match the project split. |
| (Corinne Chandel via omalley) |
| |
| MAPREDUCE-649. Validate a copy by comparing the source and destination |
| checksums in distcp. Also adds an intra-task retry mechanism for errors |
| detected during the copy. (Ravi Gummadi via cdouglas) |
| |
| MAPREDUCE-654. Add a -dryrun option to distcp printing a summary of the |
| file data to be copied, without actually performing the copy. (Ravi Gummadi |
| via cdouglas) |
| |
| MAPREDUCE-664. Display the number of files deleted by distcp when the |
| -delete option is specified. (Ravi Gummadi via cdouglas) |
| |
| MAPREDUCE-781. Let the name of distcp jobs be configurable. (Venkatesh S |
| via cdouglas) |
| |
| MAPREDUCE-975. Add an API in job client to get the history file url for |
| a given job id. (sharad) |
| |
| MAPREDUCE-905. Add Eclipse launch tasks for MapReduce. (Philip Zeyliger |
| via tomwhite) |
| |
| MAPREDUCE-277. Makes job history counters available on the job history |
| viewers. (Jothi Padmanabhan via ddas) |
| |
| MAPREDUCE-893. Provides an ability to refresh queue configuration |
| without restarting the JobTracker. |
| (Vinod Kumar Vavilapalli and Rahul Kumar Singh via yhemanth) |
| |
| MAPREDUCE-1011. Add build.properties to svn and git ignore. (omalley) |
| |
| MAPREDUCE-954. Change Map-Reduce context objects to be interfaces. |
| (acmurthy) |
| |
| MAPREDUCE-639. Change Terasort example to reflect the 2009 updates. |
| (omalley) |
| |
| HDFS-663. DFSIO for append. (shv) |
| |
| MAPREDUCE-1063. Document gridmix benchmark. (cdouglas) |
| |
| MAPREDUCE-931. Use built-in interpolation classes for making up task |
| runtimes in Rumen. (Dick King via cdouglas) |
| |
| MAPREDUCE-1012. Mark Context interfaces as public evolving. (Tom White via |
| cdouglas) |
| |
| MAPREDUCE-971. Document use of distcp when copying to s3, managing timeouts |
| in particular. (Aaron Kimball via cdouglas) |
| |
| BUG FIXES |
| |
| MAPREDUCE-878. Rename fair scheduler design doc to |
| fair-scheduler-design-doc.tex and add Apache license header (matei) |
| |
| MAPREDUCE-703. Sqoop requires dependency on hsqldb in ivy. |
| (Aaron Kimball via matei) |
| |
| HADOOP-4687. MapReduce is split from Hadoop Core. It is a subproject under |
| Hadoop (Owen O'Malley) |
| |
| HADOOP-6096. Fix Eclipse project and classpath files following project |
| split. (tomwhite) |
| |
| MAPREDUCE-419. Reconcile mapred.userlog.limit.kb defaults in configuration |
| and code. (Philip Zeyliger via cdouglas) |
| |
| MAPREDUCE-2. Fixes a bug in KeyFieldBasedPartitioner in handling empty |
| keys. (Amar Kamat via sharad) |
| |
| MAPREDUCE-130. Delete the jobconf copy from the log directory of the |
| JobTracker when the job is retired. (Amar Kamat via sharad) |
| |
| MAPREDUCE-657. Fix hardcoded filesystem problem in CompletedJobStatusStore. |
| (Amar Kamat via sharad) |
| |
| MAPREDUCE-179. Update progress in new RecordReaders. (cdouglas) |
| |
| MAPREDUCE-658. Replace NPE in distcp with a meaningful error message when |
| the source path does not exist. (Ravi Gummadi via cdouglas) |
| |
| MAPREDUCE-671. Update ignore list to include untracked, generated |
| build artifacts and config files. (cdouglas) |
| |
| MAPREDUCE-433. Use more reliable counters in TestReduceFetch. (cdouglas) |
| |
| MAPREDUCE-124. Fix a bug in failure handling of abort task of |
| OutputCommiter. (Amareshwari Sriramadasu via sharad) |
| |
| MAPREDUCE-673. Sqoop depends on commons-cli, which is not in its ivy.xml. |
| (Kevin Weil via tomwhite) |
| |
| MAPREDUCE-694. Fix to add jsp-api jars to capacity-scheduler classpath. |
| (Giridharan Kesavan) |
| |
| MAPREDUCE-690. Sqoop's test "hive" script needs to be executable. |
| (Aaron Kimball via tomwhite) |
| |
| MAPREDUCE-702. Fix eclipse-plugin jar target (Giridharan Kesavan) |
| |
| MAPREDUCE-522. Replace TestQueueCapacities with simpler test case to |
| test integration between capacity scheduler and MR framework. |
| (Sreekanth Ramakrishnan via yhemanth) |
| |
| MAPREDUCE-683. Fixes an initialization problem in the JobHistory. |
| The initialization of JobHistoryFilesManager is now done in the |
| JobHistory.init call. (Amar Kamat via ddas) |
| |
| MAPREDUCE-708. Fixes a bug to allow updating the reason for |
| blacklisting a node on the JobTracker UI. |
| (Sreekanth Ramakrishnan via yhemanth) |
| |
| MAPREDUCE-709. Fixes message displayed for a blacklisted node where |
| the reason for blacklisting is due to the health check script |
| timing out. (Sreekanth Ramakrishnan via yhemanth) |
| |
| MAPREDUCE-676. Existing diagnostic rules fail for MAP ONLY jobs. |
| (Suhas Gogate via tomwhite) |
| |
| MAPREDUCE-722. Fixes a bug with tasktracker reservations for |
| high memory jobs in capacity scheduler. |
| (Vinod Kumar Vavilapalli via yhemanth) |
| |
| HADOOP-6090. Updates gridmix script to use new mapreduce api output |
| format. (Amareshwari Sriramadasu via sharad) |
| |
| MAPREDUCE-732. Removed spurious log statements in the node |
| blacklisting logic. (Sreekanth Ramakrishnan via yhemanth) |
| |
| MAPREDUCE-685. Sqoop will fail with OutOfMemory on large tables |
| using mysql. (Aaron Kimball via tomwhite) |
| |
| MAPREDUCE-734. Fix a ConcurrentModificationException in unreserving |
| unused reservations for a job when it completes. |
| (Arun Murthy and Sreekanth Ramakrishnan via yhemanth) |
| |
| MAPREDUCE-733. Fix a RuntimeException while unreserving trackers |
| that are blacklisted for a job. |
| (Arun Murthy and Sreekanth Ramakrishnan via yhemanth) |
| |
| MAPREDUCE-677. Fix timeout in TestNodeRefresh. (Amar Kamat via |
| sharad) |
| |
| MAPREDUCE-153. Fix timeout in TestJobInProgressListener. (Amar |
| Kamat via sharad) |
| |
| MAPREDUCE-742. Fix output messages and java comments in the Pi related |
| examples. (szetszwo) |
| |
| MAPREDUCE-565. Fix partitioner to work with new API. (Owen O'Malley via |
| cdouglas) |
| |
| MAPREDUCE-680. Fix so MRUnit can handle reuse of Writable objects. |
| (Aaron Kimball via johan) |
| |
| MAPREDUCE-18. Puts some checks for cross checking whether a reduce |
| task gets the correct shuffle data. (Ravi Gummadi via ddas) |
| |
| MAPREDUCE-771. Fix scheduling of setup and cleanup tasks to use |
| free slots instead of tasks for scheduling. (yhemanth) |
| |
| MAPREDUCE-717. Fixes some corner case issues in speculative |
| execution heuristics. (Devaraj Das) |
| |
| MAPREDUCE-716. Make DBInputFormat work with Oracle. (Aaron Kimball |
| via tomwhite) |
| |
| MAPREDUCE-735. Fixes a problem in the KeyFieldHelper to do with |
| the end index for some inputs (Amar Kamat via ddas) |
| |
| MAPREDUCE-682. Removes reservations on tasktrackers which are |
| blacklisted. (Sreekanth Ramakrishnan via yhemanth) |
| |
| MAPREDUCE-743. Fixes a problem to do with progress reporting |
| in the map phase. (Ravi Gummadi via ddas) |
| |
| MAPREDUCE-765. Eliminate the deprecated warnings introduced by H-5438. |
| (He Yongqiang via szetszwo) |
| |
| MAPREDUCE-383. Fix a bug in Pipes combiner due to bytes count not |
| getting reset after the spill. (Christian Kunz via sharad) |
| |
| MAPREDUCE-809. Fix job-summary logs to correctly record status of FAILED |
| and KILLED jobs. (acmurthy) |
| |
| MAPREDUCE-792. Fix unchecked warnings in DBInputFormat. (Aaron Kimball |
| via szetszwo) |
| |
| MAPREDUCE-760. Fix a timing issue in TestNodeRefresh. (Amar Kamat via |
| sharad) |
| |
| MAPREDUCE-40. Keep memory management backwards compatible for job |
| configuration parameters and limits. (Rahul Kumar Singh via yhemanth) |
| |
| MAPREDUCE-587. Fixes a OOM issue in TestStreamingExitStatus. |
| (Amar Kamat via ddas) |
| |
| MAPREDUCE-408. Fixes an assertion problem in TestKillSubProcesses |
| (Ravi Gummadi via ddas) |
| |
| MAPREDUCE-659. Fix gridmix2 compilation. (Giridharan Kesavan) |
| |
| MAPREDUCE-796. Fixes a ClassCastException in an exception log in |
| MultiThreadedMapRunner. (Amar Kamat via ddas) |
| |
| MAPREDUCE-808. Fixes a serialization problem in TypedBytes. |
| (Klaas Bosteels via ddas) |
| |
| MAPREDUCE-845. Fix a findbugs heap size problem in build.xml and add |
| a new property findbugs.heap.size. (Lee Tucker via szetszwo) |
| |
| MAPREDUCE-838. Fixes a problem in the way commit of task outputs |
| happens. The bug was that even if commit failed, the task would |
| be declared as successful. (Amareshwari Sriramadasu via ddas) |
| |
| MAPREDUCE-813. Updates Streaming and M/R tutorial documents. |
| (Corinne Chandel via ddas) |
| |
| MAPREDUCE-805. Fixes some deadlocks in the JobTracker due to the fact |
| the JobTracker lock hierarchy wasn't maintained in some JobInProgress |
| method calls. (Amar Kamat via ddas) |
| |
| MAPREDUCE-799. Fixes so all of the MRUnit self-tests run. |
| (Aaron Kimball via johan) |
| |
| MAPREDUCE-848. Fixes a problem to do with TestCapacityScheduler |
| failing (Amar Kamat via ddas) |
| |
| MAPREDUCE-840. DBInputFormat leaves open transaction. |
| (Aaron Kimball via tomwhite) |
| |
| MAPREDUCE-859. Adds Avro and its dependencies required by Hadoop |
| common. (Ravi Gummadi via sharad) |
| |
| MAPREDUCE-867. Fix ivy conf to look for avro jar from maven repo. |
| (Giridharan Kesavan) |
| |
| MAPREDUCE-877. Added avro as a dependency to contrib ivy settings. |
| (Tsz Wo (Nicholas) Sze via yhemanth) |
| |
| MAPREDUCE-852. In build.xml, remove the Main-Class, which is incorrectly |
| set in tools, and rename the target "tools-jar" to "tools". (szetszwo) |
| |
| MAPREDUCE-773. Sends progress reports for compressed gzip inputs in maps. |
| Fixes a native direct buffer leak in LineRecordReader classes. |
| (Hong Tang and ddas) |
| |
| MAPREDUCE-832. Reduce number of warning messages printed when |
| deprecated memory variables are used. (Rahul Kumar Singh via yhemanth) |
| |
| MAPREDUCE-745. Fixes a testcase problem to do with generation of JobTracker |
| IDs. (Amar Kamat via ddas) |
| |
| MAPREDUCE-834. Enables memory management on tasktrackers when old |
| memory management parameters are used in configuration. |
| (Sreekanth Ramakrishnan via yhemanth) |
| |
| MAPREDUCE-818. Fixes Counters#getGroup API. (Amareshwari Sriramadasu |
| via sharad) |
| |
| MAPREDUCE-807. Handles the AccessControlException during the deletion of |
| mapred.system.dir in the JobTracker. The JobTracker will bail out if it |
| encounters such an exception. (Amar Kamat via ddas) |
| |
| MAPREDUCE-430. Fix a bug related to task getting stuck in case of |
| OOM error. (Amar Kamat via ddas) |
| |
| MAPREDUCE-871. Fix ownership of Job/Task local files to have correct |
| group ownership according to the egid of the tasktracker. |
| (Vinod Kumar Vavilapalli via yhemanth) |
| |
| MAPREDUCE-911. Fix a bug in TestTaskFail related to speculative |
| execution. (Amareshwari Sriramadasu via sharad) |
| |
| MAPREDUCE-687. Fix an assertion in TestMiniMRMapRedDebugScript. |
| (Amareshwari Sriramadasu via sharad) |
| |
| MAPREDUCE-924. Fixes the TestPipes testcase to use Tool. |
| (Amareshwari Sriramadasu via sharad) |
| |
| MAPREDUCE-903. Add Avro jar to eclipse classpath. |
| (Philip Zeyliger via tomwhite) |
| |
| MAPREDUCE-943. Removes a testcase in TestNodeRefresh that doesn't make |
| sense in the new Job recovery model. (Amar Kamat via ddas) |
| |
| MAPREDUCE-764. TypedBytesInput's readRaw() does not preserve custom type |
| codes. (Klaas Bosteels via tomwhite) |
| |
| HADOOP-6243. Fixes a NullPointerException in handling deprecated keys. |
| (Sreekanth Ramakrishnan via yhemanth) |
| |
| MAPREDUCE-968. NPE in distcp encountered when placing _logs directory on |
| S3FileSystem. (Aaron Kimball via tomwhite) |
| |
| MAPREDUCE-826. harchive doesn't use ToolRunner / harchive returns 0 even |
| if the job fails with exception (koji Noguchi via mahadev) |
| |
| MAPREDUCE-839. unit test TestMiniMRChildTask fails on mac os-x (hong tang |
| via mahadev) |
| |
| MAPREDUCE-112. Add counters for reduce input, output records to the new API. |
| (Jothi Padmanabhan via cdouglas) |
| |
| MAPREDUCE-648. Fix two distcp bugs: (1) it should not launch a job if all |
| src paths are directories, and (2) it does not skip copying when updating |
| a single file. (Ravi Gummadi via szetszwo) |
| |
| MAPREDUCE-946. Fix a regression in LineRecordReader where the |
| maxBytesToConsume parameter is not set correctly. (cdouglas) |
| |
| MAPREDUCE-977. Missing jackson jars from Eclipse template. (tomwhite) |
| |
| MAPREDUCE-988. Fix a packaging issue in the contrib modules. (Hong Tang via |
| cdouglas) |
| |
| MAPREDUCE-971. distcp does not always remove distcp.tmp.dir. (Aaron Kimball |
| via tomwhite) |
| |
| MAPREDUCE-995. Fix a bug in JobHistory where tasks completing after the job |
| is closed cause a NPE. (Jothi Padmanabhan via cdouglas) |
| |
| MAPREDUCE-953. Fix QueueManager to dump queue configuration in JSON format. |
| (V.V. Chaitanya Krishna via yhemanth) |
| |
| MAPREDUCE-645. Prevent distcp from running a job when the destination is a |
| file, but the source is not. (Ravi Gummadi via cdouglas) |
| |
| MAPREDUCE-1002. Flushed writer in JobQueueClient so queue information is |
| printed correctly. (V.V. Chaitanya Krishna via yhemanth) |
| |
| MAPREDUCE-1003. Fix compilation problem in eclipse plugin when |
| eclipse.home is set. (Ravi Gummadi via yhemanth) |
| |
| MAPREDUCE-941. Vaidya script fails on Solaris. (Chad Metcalf |
| via tomwhite) |
| |
| MAPREDUCE-923. Sqoop classpath breaks for jar files with a |
| plus sign in their names. (Aaron Kimball via tomwhite) |
| |
| MAPREDUCE-912. Add and standardize Apache license headers. (Chad Metcalf |
| via cdouglas) |
| |
| MAPREDUCE-1022. Fix compilation of vertica testcases. (Vinod Kumar |
| Vavilapalli via acmurthy) |
| |
| MAPREDUCE-1000. Handle corrupt history files in JobHistory.initDone(). |
| (Jothi Padmanabhan via sharad) |
| |
| MAPREDUCE-1028. Fixed number of slots occupied by cleanup tasks to one |
| irrespective of slot size for the job. |
| (Ravi Gummadi via yhemanth) |
| |
| MAPREDUCE-964. Fixed start and finish times of TaskStatus to be |
| consistent, thereby fixing inconsistencies in metering tasks. |
| (Sreekanth Ramakrishnan via yhemanth) |
| |
| MAPREDUCE-1076. Deprecate ClusterStatus and add javadoc in ClusterMetrics. |
| (Amareshwari Sriramadasu via sharad) |
| |
| MAPREDUCE-979. Fixed JobConf APIs related to memory parameters to return |
| values of new configuration variables when deprecated variables are |
| disabled. (Sreekanth Ramakrishnan via yhemanth) |
| |
| MAPREDUCE-1030. Modified scheduling algorithm to return a map and reduce |
| task per heartbeat in the capacity scheduler. |
| (Rahul Kumar Singh via yhemanth) |
| |
| MAPREDUCE-1071. Use DataInputStream rather than FSDataInputStream in the |
| JobHistory EventReader. (Hong Tang via cdouglas) |
| |
| MAPREDUCE-986. Fix Rumen to work with truncated task lines. (Dick King via |
| cdouglas) |
| |
| MAPREDUCE-1029. Fix failing TestCopyFiles by restoring the unzipping of |
| HDFS webapps from the hdfs jar. (Aaron Kimball and Jothi Padmanabhan via |
| cdouglas) |
| |
| MAPREDUCE-769. Make findbugs and javac warnings to zero. |
| (Amareshwari Sriramadasu via sharad) |
| |
| MAPREDUCE-1104. Initialize RecoveryManager in JobTracker cstr called by |
| Mumak. (Hong Tang via cdouglas) |
| |
| MAPREDUCE-1061. Add unit test validating byte specifications for gridmix |
| jobs. (cdouglas) |
| |
| MAPREDUCE-1077. Fix Rumen so that truncated tasks do not mark the job as |
| successful. (Dick King via cdouglas) |
| |
| MAPREDUCE-1041. Make TaskInProgress::taskStatuses map package-private. |
| (Jothi Padmanabhan via cdouglas) |
| |
| MAPREDUCE-1070. Prevent a deadlock in the fair scheduler servlet. |
| (Todd Lipcon via cdouglas) |
| |
| MAPREDUCE-1086. Setup Hadoop logging environment for tasks to point to |
| task related parameters. (Ravi Gummadi via yhemanth) |
| |
| MAPREDUCE-1105. Remove max limit configuration in capacity scheduler in |
| favor of max capacity percentage thus allowing the limit to go over |
| queue capacity. (Rahul Kumar Singh via yhemanth) |
| |
| MAPREDUCE-1016. Make the job history log format JSON. (cutting) |
| |
| MAPREDUCE-1038. Weave Mumak aspects only if related files have changed. |
| (Aaron Kimball via cdouglas) |
| |
| MAPREDUCE-1037. Continue running contrib tests if Sqoop tests fail. (Aaron |
| Kimball via cdouglas) |
| |
| MAPREDUCE-1163. Remove unused, hard-coded paths from libhdfs. (Allen |
| Wittenauer via cdouglas) |
| |
| MAPREDUCE-962. Fix a NullPointerException while killing task process |
| trees. (Ravi Gummadi via yhemanth) |
| |
| MAPREDUCE-1177. Correct setup/cleanup inversion in |
| JobTracker::getTaskReports. (Vinod Kumar Vavilapalli via cdouglas) |
| |
| MAPREDUCE-1178. Fix ClassCastException in MultipleInputs by adding |
| a DelegatingRecordReader. (Amareshwari Sriramadasu and Jay Booth |
| via sharad) |
| |
| MAPREDUCE-1068. Fix streaming job to show proper message if file is |
| is not present. (Amareshwari Sriramadasu via sharad) |
