Apache Hadoop Changelog

Release 2.2.0 - 2013-10-15

INCOMPATIBLE CHANGES:

JIRA	Summary	Priority	Component	Reporter	Contributor
HADOOP-10020	disable symlinks temporarily	Blocker	fs	Colin Patrick McCabe	Sanjay Radia
YARN-1229	Define constraints on Auxiliary Service names. Change ShuffleHandler service name from mapreduce.shuffle to mapreduce_shuffle.	Blocker	nodemanager	Tassapol Athiapinya	Xuan Gong
YARN-1228	Clean up Fair Scheduler configuration loading	Major	scheduler	Sandy Ryza	Sandy Ryza

IMPORTANT ISSUES:

JIRA	Summary	Priority	Component	Reporter	Contributor

NEW FEATURES:

JIRA	Summary	Priority	Component	Reporter	Contributor

IMPROVEMENTS:

JIRA	Summary	Priority	Component	Reporter	Contributor
HADOOP-9758	Provide configuration option for FileSystem/FileContext symlink resolution	Major	.	Andrew Wang	Andrew Wang
HADOOP-8315	Support SASL-authenticated ZooKeeper in ActiveStandbyElector	Major	auto-failover, ha	Todd Lipcon	Todd Lipcon
HDFS-5308	Replace HttpConfig#getSchemePrefix with implicit schemes in HDFS JSP	Major	.	Haohui Mai	Haohui Mai
HDFS-5256	Use guava LoadingCache to implement DFSClientCache	Major	nfs	Haohui Mai	Haohui Mai
HDFS-5139	Remove redundant -R option from setrep	Major	tools	Arpit Agarwal	Arpit Agarwal
HDFS-4817	make HDFS advisory caching configurable on a per-file basis	Minor	hdfs-client	Colin Patrick McCabe	Colin Patrick McCabe
YARN-1246	Log application status in the rm log when app is done running	Minor	.	Arpit Gupta	Arpit Gupta
YARN-1213	Restore config to ban submitting to undeclared pools in the Fair Scheduler	Major	scheduler	Sandy Ryza	Sandy Ryza

BUG FIXES:

JIRA	Summary	Priority	Component	Reporter	Contributor
HADOOP-10012	Secure Oozie jobs fail with delegation token renewal exception in Namenode HA setup	Blocker	ha	Arpit Gupta	Suresh Srinivas
HADOOP-10003	HarFileSystem.listLocatedStatus() fails	Major	fs	Jason Dere
HADOOP-9976	Different versions of avro and avro-maven-plugin	Major	.	Karthik Kambatla	Karthik Kambatla
HADOOP-9776	HarFileSystem.listStatus() returns invalid authority if port number is empty	Major	fs	shanyu zhao	shanyu zhao
HADOOP-9761	ViewFileSystem#rename fails when using DistributedFileSystem	Blocker	viewfs	Andrew Wang	Andrew Wang
HDFS-5300	FSNameSystem#deleteSnapshot() should not check owner in case of permissions disabled	Major	namenode	Vinayakumar B	Vinayakumar B
HDFS-5299	DFS client hangs in updatePipeline RPC when failover happened	Blocker	namenode	Vinayakumar B	Vinayakumar B
HDFS-5289	Race condition in TestRetryCacheWithHA#testCreateSymlink causes spurious test failure	Major	test	Aaron T. Myers	Aaron T. Myers
HDFS-5279	Guard against NullPointerException in NameNode JSP pages before initialization of FSNamesystem.	Major	namenode	Chris Nauroth	Chris Nauroth
HDFS-5268	NFS write commit verifier is not set in a few places	Major	nfs	Brandon Li	Brandon Li
HDFS-5265	Namenode fails to start when dfs.https.port is unspecified	Major	.	Haohui Mai	Haohui Mai
HDFS-5258	Skip tests in TestHDFSCLI that are not applicable on Windows.	Minor	test	Chris Nauroth	Chuan Liu
HDFS-5255	Distcp job fails with hsftp when https is enabled in insecure cluster	Major	.	Yesha Vora	Arpit Agarwal
HDFS-5251	Race between the initialization of NameNode and the http server	Major	.	Haohui Mai	Haohui Mai
HDFS-5228	The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw NPE	Blocker	hdfs-client	Tsz Wo Nicholas Sze	Tsz Wo Nicholas Sze
HDFS-5031	BlockScanner scans the block multiple times and on restart scans everything	Blocker	datanode	Vinayakumar B	Vinayakumar B
MAPREDUCE-5554	hdfs-site.xml included in hadoop-mapreduce-client-jobclient tests jar is breaking tests for downstream components	Minor	test	Robert Kanter	Robert Kanter
MAPREDUCE-5545	org.apache.hadoop.mapred.TestTaskAttemptListenerImpl.testCommitWindow times out	Major	.	Robert Kanter	Robert Kanter
MAPREDUCE-5544	JobClient#getJob loads job conf twice	Major	.	Sandy Ryza	Sandy Ryza
MAPREDUCE-5536	mapreduce.jobhistory.webapp.https.address property is not respected	Blocker	.	Yesha Vora	Omkar Vinit Joshi
MAPREDUCE-5533	Speculative execution does not function for reduce	Major	applicationmaster	Tassapol Athiapinya	Xuan Gong
MAPREDUCE-5523	Need to add https port related property in Job history server	Major	.	Omkar Vinit Joshi	Omkar Vinit Joshi
MAPREDUCE-5515	Application Manager UI does not appear with Https enabled	Major	.	Omkar Vinit Joshi	Omkar Vinit Joshi
MAPREDUCE-5513	ConcurrentModificationException in JobControl	Major	.	Jason Lowe	Robert Parker
MAPREDUCE-5503	TestMRJobClient.testJobClient is failing	Blocker	mrv2	Jason Lowe	Jian He
MAPREDUCE-5489	MR jobs hangs as it does not use the node-blacklisting feature in RM requests	Critical	.	Yesha Vora	Zhijie Shen
MAPREDUCE-5488	Job recovery fails after killing all the running containers for the app	Major	.	Arpit Gupta	Jian He
MAPREDUCE-5459	Update the doc of running MRv1 examples jar on YARN	Major	.	Zhijie Shen	Zhijie Shen
MAPREDUCE-5442	$HADOOP_MAPRED_HOME/$HADOOP_CONF_DIR setting not working on Windows	Major	client	Yingda Chen	Yingda Chen
MAPREDUCE-5170	incorrect exception message if min node size > min rack size	Trivial	mrv2	Sangjin Lee	Sangjin Lee
YARN-1278	New AM does not start after rm restart	Blocker	.	Yesha Vora	Hitesh Shah
YARN-1274	LCE fails to run containers that don't have resources to localize	Blocker	nodemanager	Alejandro Abdelnur	Siddharth Seth
YARN-1273	Distributed shell does not account for start container failures reported asynchronously.	Major	.	Hitesh Shah	Hitesh Shah
YARN-1271	“Text file busy” errors launching containers again	Major	nodemanager	Sandy Ryza	Sandy Ryza
YARN-1262	TestApplicationCleanup relies on all containers assigned in a single heartbeat	Major	.	Sandy Ryza	Karthik Kambatla
YARN-1251	TestDistributedShell#TestDSShell failed with timeout	Major	applications/distributed-shell	Junping Du	Xuan Gong
YARN-1247	test-container-executor has gotten out of sync with the changes to container-executor	Major	nodemanager	Roman Shaposhnik	Roman Shaposhnik
YARN-1236	FairScheduler setting queue name in RMApp is not working	Major	resourcemanager	Sandy Ryza	Sandy Ryza
YARN-1221	With Fair Scheduler, reserved MB reported in RM web UI increases indefinitely	Major	resourcemanager, scheduler	Sandy Ryza	Siqi Li
YARN-1219	FSDownload changes file suffix making FileUtil.unTar() throw exception	Major	nodemanager	shanyu zhao	shanyu zhao
YARN-1215	Yarn URL should include userinfo	Major	api	Chuan Liu	Chuan Liu
YARN-1167	Submitted distributed shell application shows appMasterHost = empty	Major	applications/distributed-shell	Tassapol Athiapinya	Xuan Gong
YARN-1157	ResourceManager UI has invalid tracking URL link for distributed shell application	Major	resourcemanager	Tassapol Athiapinya	Xuan Gong
YARN-1149	NM throws InvalidStateTransitonException: Invalid event: APPLICATION_LOG_HANDLING_FINISHED at RUNNING	Major	.	Ramya Sunil	Xuan Gong
YARN-1141	Updating resource requests should be decoupled with updating blacklist	Major	.	Zhijie Shen	Zhijie Shen
YARN-1128	FifoPolicy.computeShares throws NPE on empty list of Schedulables	Major	scheduler	Sandy Ryza	Karthik Kambatla
YARN-1090	Job does not get into Pending State	Major	.	Yesha Vora	Jian He
YARN-1032	NPE in RackResolve	Critical	.	Lohit Vijayarenu	Lohit Vijayarenu
YARN-890	The roundup for memory values on resource manager UI is misleading	Major	resourcemanager	Trupti Dhavle	Xuan Gong
YARN-876	Node resource is added twice when node comes back from unhealthy to healthy	Major	resourcemanager	Peng Zhang	Peng Zhang

TESTS:

JIRA	Summary	Priority	Component	Reporter	Contributor
HADOOP-9948	Add a config value to CLITestHelper to skip tests on Windows	Minor	test	Chuan Liu	Chuan Liu
HDFS-5186	TestFileJournalManager fails on Windows due to file handle leaks	Minor	namenode, test	Chuan Liu	Chuan Liu
MAPREDUCE-5525	Increase timeout of TestDFSIO.testAppend and TestMRJobsWithHistoryService.testJobHistoryData	Minor	mrv2, test	Chuan Liu	Chuan Liu

SUB-TASKS:

JIRA	Summary	Priority	Component	Reporter	Contributor
HADOOP-10017	Fix NPE in DFSClient#getDelegationToken when doing Distcp from a secured cluster to an insecured cluster	Major	.	Jing Zhao	Haohui Mai
HDFS-5306	Datanode https port is not available at the namenode	Major	datanode, namenode	Suresh Srinivas	Suresh Srinivas
HDFS-5259	Support client which combines appended data with old data before sends it to NFS server	Major	nfs	Yesha Vora	Brandon Li
HDFS-5246	Make Hadoop nfs server port and mount daemon port configurable	Major	nfs	Jinghui Wang	Jinghui Wang
HDFS-5230	Introduce RpcInfo to decouple XDR classes from the RPC API	Major	nfs	Haohui Mai	Haohui Mai
MAPREDUCE-5562	MR AM should exit when unregister() throws exception	Major	.	Zhijie Shen	Zhijie Shen
MAPREDUCE-5551	Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes	Blocker	.	Zhijie Shen	Zhijie Shen
MAPREDUCE-5538	MRAppMaster#shutDownJob shouldn't send job end notification before checking isLastRetry	Blocker	.	Zhijie Shen	Zhijie Shen
MAPREDUCE-5531	Binary and source incompatibility in mapreduce.TaskID and mapreduce.TaskAttemptID between branch-1 and branch-2	Blocker	mrv1, mrv2	Robert Kanter	Robert Kanter
MAPREDUCE-5530	Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2	Blocker	mrv1, mrv2	Robert Kanter	Robert Kanter
MAPREDUCE-5529	Binary incompatibilities in mapred.lib.TotalOrderPartitioner between branch-1 and branch-2	Blocker	mrv1, mrv2	Robert Kanter	Robert Kanter
MAPREDUCE-5505	Clients should be notified job finished only after job successfully unregistered	Critical	.	Jian He	Zhijie Shen
YARN-1277	Add http policy support for YARN daemons	Major	.	Suresh Srinivas	Omkar Vinit Joshi
YARN-1260	RM_HOME link breaks when webapp.https.address related properties are not specified	Major	.	Yesha Vora	Omkar Vinit Joshi
YARN-1256	NM silently ignores non-existent service in StartContainerRequest	Critical	.	Bikas Saha	Xuan Gong
YARN-1254	NM is polluting container's credentials	Major	.	Vinod Kumar Vavilapalli	Omkar Vinit Joshi
YARN-1214	Register ClientToken MasterKey in SecretManager after it is saved	Critical	resourcemanager	Jian He	Jian He
YARN-1204	Need to add https port related property in Yarn	Major	.	Yesha Vora	Omkar Vinit Joshi
YARN-1203	Application Manager UI does not appear with Https enabled	Major	.	Yesha Vora	Omkar Vinit Joshi
YARN-1131	$yarn logs command should return an appropriate error message if YARN application is still running	Minor	client	Tassapol Athiapinya	Siddharth Seth
YARN-1070	ContainerImpl State Machine: Invalid event: CONTAINER_KILLED_ON_REQUEST at CONTAINER_CLEANEDUP_AFTER_KILL	Major	nodemanager	Hitesh Shah	Zhijie Shen
YARN-899	Get queue administration ACLs working	Major	scheduler	Sandy Ryza	Xuan Gong
YARN-621	RM triggers web auth failure before first job	Critical	resourcemanager	Allen Wittenauer	Omkar Vinit Joshi
YARN-49	Improve distributed shell application to work on a secure cluster	Major	applications/distributed-shell	Hitesh Shah	Vinod Kumar Vavilapalli

OTHER:

JIRA	Summary	Priority	Component	Reporter	Contributor