blob: ee67ff77a7116a185edd65b603371d33fb670fd6 [file] [log] [blame]
Hadoop HDFS Change Log
Release 2.2.0 - UNRELEASED
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
OPTIMIZATIONS
BUG FIXES
Release 2.1.2 - UNRELEASED
INCOMPATIBLE CHANGES
NEW FEATURES
HDFS-5230. Introduce RpcInfo to decouple XDR classes from the RPC API.
(Haohui Mai via brandonli)
IMPROVEMENTS
HDFS-5246. Make Hadoop nfs server port and mount daemon port
configurable. (Jinghui Wang via brandonli)
HDFS-5256. Use guava LoadingCache to implement DFSClientCache. (Haohui Mai
via brandonli)
HDFS-5308. Replace HttpConfig#getSchemePrefix with implicit schemes in HDFS
JSP. (Haohui Mai via jing9)
OPTIMIZATIONS
BUG FIXES
HDFS-5139. Remove redundant -R option from setrep.
HDFS-5251. Race between the initialization of NameNode and the http
server. (Haohui Mai via suresh)
HDFS-5258. Skip tests in TestHDFSCLI that are not applicable on Windows.
(Chuan Liu via cnauroth)
HDFS-5186. TestFileJournalManager fails on Windows due to file handle leaks.
(Chuan Liu via cnauroth)
HDFS-5031. BlockScanner scans the block multiple times. (Vinay via Arpit
Agarwal)
HDFS-5268. NFS write commit verifier is not set in a few places (brandonli)
HDFS-5265. Namenode fails to start when dfs.https.port is unspecified.
(Haohui Mai via jing9)
HDFS-5255. Distcp job fails with hsftp when https is enabled in insecure
cluster. (Arpit Agarwal)
HDFS-5279. Guard against NullPointerException in NameNode JSP pages before
initialization of FSNamesystem. (cnauroth)
HDFS-5289. Race condition in TestRetryCacheWithHA#testCreateSymlink causes
spurious test failure. (atm)
HDFS-5300. FSNameSystem#deleteSnapshot() should not check owner in case of
permissions disabled. (Vinay via jing9)
HDFS-5306. Datanode https port is not available at the namenode. (Suresh
Srinivas via brandonli)
HDFS-5299. DFS client hangs in updatePipeline RPC when failover happened.
(Vinay via jing9)
HDFS-5259. Support client which combines appended data with old data
before sends it to NFS server. (brandonli)
Release 2.1.1-beta - 2013-09-23
INCOMPATIBLE CHANGES
NEW FEATURES
HDFS-4962 Use enum for nfs constants (Nicholas SZE via jing9)
HDFS-5071 Change hdfs-nfs parent project to hadoop-project (brandonli)
HDFS-4763 Add script changes/utility for starting NFS gateway (brandonli)
HDFS-5076 Add MXBean methods to query NN's transaction information and
JournalNode's journal status. (jing9)
HDFS-5104 Support dotdot name in NFS LOOKUP operation (brandonli)
HDFS-5107 Fix array copy error in Readdir and Readdirplus responses
(brandonli)
HDFS-5110 Change FSDataOutputStream to HdfsDataOutputStream for opened
streams to fix type cast error. (brandonli)
HDFS-5069 Include hadoop-nfs and hadoop-hdfs-nfs into hadoop dist for
NFS deployment (brandonli)
HDFS-4947 Add NFS server export table to control export by hostname or
IP range (Jing Zhao via brandonli)
HDFS-5078 Support file append in NFSv3 gateway to enable data streaming
to HDFS (brandonli)
HDFS-5136 MNT EXPORT should give the full group list which can mount the
exports (brandonli)
HDFS-5118. Provide testing support for DFSClient to drop RPC responses.
(jing9)
HDFS-5085. Refactor o.a.h.nfs to support different types of
authentications. (jing9)
HDFS-5067 Support symlink operations in NFS gateway. (brandonli)
HDFS-5199 Add more debug trace for NFS READ and WRITE. (brandonli)
HDFS-5234 Move RpcFrameDecoder out of the public API.
(Haohui Mai via brandonli)
IMPROVEMENTS
HDFS-4513. Clarify in the WebHDFS REST API that all JSON respsonses may
contain additional properties. (szetszwo)
HDFS-5061. Make FSNameSystem#auditLoggers an unmodifiable list.
(Arpit Agarwal via suresh)
HDFS-4905. Add appendToFile command to "hdfs dfs". (Arpit Agarwal via
cnauroth)
HDFS-4926. Namenode webserver's page has a tooltip that is inconsistent
with the datanode HTML link. (Vivek Ganesan via jing9)
HDFS-5047. Supress logging of full stack trace of quota and lease
exceptions. (Robert Parker via kihwal)
HDFS-5111. Remove duplicated error message for snapshot commands when
processing invalid arguments. (jing9)
HDFS-5045. Add more unit tests for retry cache to cover all AtMostOnce
methods. (jing9)
HDFS-3245. Add metrics and web UI for cluster version summary. (Ravi
Prakash via kihwal)
HDFS-5128. Allow multiple net interfaces to be used with HA namenode RPC
server. (kihwal)
HDFS-5150. Allow per NN SPN for internal SPNEGO. (kihwal)
HDFS-4680. Audit logging of delegation tokens for MR tracing. (Andrew Wang)
HDFS-5212. Refactor RpcMessage and NFS3Response to support different
types of authentication information. (jing9)
HDFS-4971. Move IO operations out of locking in OpenFileCtx. (brandonli and
jing9)
OPTIMIZATIONS
BUG FIXES
HDFS-5043. For HdfsFileStatus, set default value of childrenNum to -1
instead of 0 to avoid confusing applications. (brandonli)
HDFS-5028. LeaseRenewer throws ConcurrentModificationException when timeout.
(zhaoyunjiong via szetszwo)
HDFS-4993. Fsck can fail if a file is renamed or deleted. (Robert Parker
via kihwal)
HDFS-5091. Support for spnego keytab separate from the JournalNode keytab
for secure HA. (jing9)
HDFS-5051. nn fails to download checkpointed image from snn in some
setups. (Vinay and suresh via suresh)
HDFS-4898. BlockPlacementPolicyWithNodeGroup.chooseRemoteRack() fails to
properly fallback to local rack. (szetszwo)
HDFS-4632. globStatus using backslash for escaping does not work on Windows.
(Chuan Liu via cnauroth)
HDFS-5080. BootstrapStandby not working with QJM when the existing NN is
active. (jing9)
HDFS-5099. Namenode#copyEditLogSegmentsToSharedDir should close
EditLogInputStreams upon finishing. (Chuan Liu via cnauroth)
HDFS-2994. If lease soft limit is recovered successfully
the append can fail. (Tao Luo via shv)
HDFS-5100. TestNamenodeRetryCache fails on Windows due to incorrect cleanup.
(Chuan Liu via cnauroth)
HDFS-5103. TestDirectoryScanner fails on Windows. (Chuan Liu via cnauroth)
HDFS-5102. Snapshot names should not be allowed to contain slash characters.
(jing9)
HDFS-5105. TestFsck fails on Windows. (Chuan Liu via arp)
HDFS-5106. TestDatanodeBlockScanner fails on Windows due to incorrect path
format. (Chuan Liu via cnauroth)
HDFS-4594. WebHDFS open sets Content-Length header to what is specified by
length parameter rather than how much data is actually returned. (cnauroth)
HDFS-5124. DelegationTokenSecretManager#retrievePassword can cause deadlock
in NameNode. (Daryn Sharp via jing9)
HDFS-5132. Deadlock in NameNode between SafeModeMonitor#run and
DatanodeManager#handleHeartbeat. (kihwal)
HDFS-5077. NPE in FSNamesystem.commitBlockSynchronization().
(Plamen Jeliazkov via shv)
HDFS-5140. Too many safemode monitor threads being created in the standby
namenode causing it to fail with out of memory error. (jing9)
HDFS-5159. Secondary NameNode fails to checkpoint if error occurs
downloading edits on first checkpoint. (atm)
HDFS-5192. NameNode may fail to start when
dfs.client.test.drop.namenode.response.number is set. (jing9)
HDFS-5219. Add configuration keys for retry policy in WebHDFSFileSystem.
(Haohui Mai via jing9)
HDFS-5231. Fix broken links in the document of HDFS Federation. (Haohui Mai
via jing9)
HDFS-5249. Fix dumper thread which may die silently. (brandonli)
Release 2.1.0-beta - 2013-08-22
INCOMPATIBLE CHANGES
HDFS-4053. Increase the default block size. (eli)
HDFS-4305. Add a configurable limit on number of blocks per file, and min
block size. (Andrew Wang via atm)
HDFS-4434. Provide a mapping from INodeId to INode. (suresh)
HDFS-2802. Add HDFS Snapshot feature. (See breakdown of tasks below for
subtasks and contributors)
HDFS-4866. Protocol buffer support cannot compile under C. (Arpit Agarwal via
cnauroth)
HDFS-5083. Update the HDFS compatibility version range. (kihwal)
NEW FEATURES
HDFS-1804. Add a new block-volume device choosing policy that looks at
free space. (atm)
HDFS-4296. Reserve layout version for release 1.2.0. (suresh)
HDFS-4334. Add a unique id to INode. (Brandon Li via szetszwo)
HDFS-4339. Persist inode id in fsimage and editlog. (Brandon Li via
suresh)
HDFS-4340. Update addBlock() to inculde inode id as additional argument.
(Brandon Li via suresh)
HDFS-4502. JsonUtil.toFileStatus(..) should check if the fileId property
exists. (Brandon Li via suresh)
HDFS-2576. Enhances the DistributedFileSystem's create API so that clients
can specify favored datanodes for a file's blocks. (ddas)
HDFS-347. DFS read performance suboptimal when client co-located on nodes
with data. (Colin Patrick McCabe via todd and atm)
HADOOP-8562. Enhancements to support Hadoop on Windows Server and Windows
Azure environments. (See breakdown of tasks below for subtasks and
contributors)
HDFS-3601. Add BlockPlacementPolicyWithNodeGroup to support block placement
with 4-layer network topology. (Junping Du via szetszwo)
HDFS-3495. Update Balancer to support new NetworkTopology with NodeGroup.
(Junping Du via szetszwo)
HDFS-4659 Support setting execution bit for regular files (Brandon Li via sanjay)
HDFS-4762 Provide HDFS based NFSv3 and Mountd implementation (brandonli)
HDFS-4372. Track NameNode startup progress. (cnauroth)
HDFS-4373. Add HTTP API for querying NameNode startup progress. (cnauroth)
HDFS-4374. Display NameNode startup progress in UI. (cnauroth)
HDFS-4974. Add Idempotent and AtMostOnce annotations to namenode
protocol methods. (suresh)
HDFS-4979. Implement retry cache on Namenode. (suresh)
HDFS-5025. Record ClientId and CallId in EditLog to enable rebuilding
retry cache in case of HA failover. (Jing Zhao via suresh)
IMPROVEMENTS
HDFS-4461. DirectoryScanner: volume path prefix takes up memory for every
block that is scanned (Colin Patrick McCabe)
HDFS-4222. NN is unresponsive and loses heartbeats from DNs when
configured to use LDAP and LDAP has issues. (Xiaobo Peng, suresh)
HDFS-4304. Make FSEditLogOp.MAX_OP_SIZE configurable. (Colin Patrick
McCabe via atm)
HDFS-4518. Finer grained metrics for HDFS capacity.
(Arpit Agarwal via suresh)
HDFS-4519. Support overriding jsvc binary and log file locations
when launching secure datanode. (Chris Nauroth via suresh)
HDFS-4569. Small image transfer related cleanups.
(Andrew Wang via suresh)
HDFS-4521. Invalid network toploogies should not be cached. (Colin Patrick
McCabe via atm)
HDFS-4246. The exclude node list should be more forgiving, for each output
stream. (harsh via atm)
HDFS-4635. Move BlockManager#computeCapacity to LightWeightGSet. (suresh)
HDFS-4621. Additional logging to help diagnose slow QJM syncs. (todd)
HDFS-4618. Default transaction interval for checkpoints is too low. (todd)
HDFS-4525. Provide an API for knowing that whether file is closed or not.
(SreeHari via umamahesh)
HDFS-3940. Add Gset#clear method and clear the block map when namenode is
shutdown. (suresh)
HDFS-4679. Namenode operation checks should be done in a consistent
manner. (suresh)
HDFS-4693. Some test cases in TestCheckpoint do not clean up after
themselves. (Arpit Agarwal, suresh via suresh)
HDFS-3817. Avoid printing SafeModeException stack trace.
(Brandon Li via suresh)
HDFS-4124. Refactor INodeDirectory#getExistingPathINodes() to enable
returning more than INode array. (Jing Zhao via suresh)
HDFS-4151. Change the methods in FSDirectory to pass INodesInPath instead
of INode[] as a parameter. (szetszwo)
HDFS-4129. Add utility methods to dump NameNode in memory tree for
testing. (szetszwo via suresh)
HDFS-4152. Add a new class BlocksMapUpdateInfo for the parameter in
INode.collectSubtreeBlocksAndClear(..). (Jing Zhao via szetszwo)
HDFS-4206. Change the fields in INode and its subclasses to private.
(szetszwo)
HDFS-4215. Remove locking from addToParent(..) since it is used in image
loading, and add INode.isFile(). (szetszwo)
HDFS-4243. When replacing an INodeDirectory, the parent pointers of the
children of the child have to be updated to the new child. (Jing Zhao
via szetszwo)
HDFS-4209. Clean up the addNode/addChild/addChildNoQuotaCheck methods in
FSDirectory and INodeDirectory. (szetszwo)
HDFS-4346. Add SequentialNumber as a base class for INodeId and
GenerationStamp. (szetszwo)
HDFS-4721. Speed up lease recovery by avoiding stale datanodes and choosing
the datanode with the most recent heartbeat as the primary. (Varun Sharma
via szetszwo)
HDFS-4804. WARN when users set the block balanced preference percent below
0.5 or above 1.0. (Stephen Chu via atm)
HDFS-4698. Provide client-side metrics for remote reads, local reads, and
short-circuit reads. (Colin Patrick McCabe via atm)
HDFS-3498. Support replica removal in BlockPlacementPolicy and make
BlockPlacementPolicyDefault extensible for reusing code in subclasses.
(Junping Du via szetszwo)
HDFS-4234. Use generic code for choosing datanode in Balancer. (szetszwo)
HDFS-4880. Print the image and edits file loaded by the namenode in the
logs. (Arpit Agarwal via suresh)
HDFS-2572. Remove unnecessary double-check in DN#getHostName. (harsh)
HDFS-2857. Cleanup BlockInfo class. (suresh)
HDFS-3009. Remove duplicate code in DFSClient#isLocalAddress by using
NetUtils. (Hari Mankude via suresh)
HDFS-4914. Use DFSClient.Conf instead of Configuration. (szetszwo)
HDFS-4883. complete() should verify fileId. (Tao Luo via shv)
HDFS-4772. Add number of children in HdfsFileStatus. (brandonli)
HDFS-4932. Avoid a wide line on the name node webUI if we have more Journal
nodes. (Fengdong Yu via cnauroth)
HDFS-4908. Reduce snapshot inode memory usage. (szetszwo)
HDFS-4645. Move from randomly generated block ID to sequentially generated
block ID. (Arpit Agarwal via szetszwo)
HDFS-4912. Cleanup FSNamesystem#startFileInternal. (suresh)
HDFS-4903. Print trash configuration and trash emptier state in
namenode log. (Arpit Agarwal via suresh)
HDFS-4992. Make balancer's mover thread count and dispatcher thread count
configurable. (Max Lapan via szetszwo)
HDFS-4996. ClientProtocol#metaSave can be made idempotent by overwriting the
output file instead of appending to it. (cnauroth)
HADOOP-9418. Add symlink support to DistributedFileSystem (Andrew Wang via
Colin Patrick McCabe)
HDFS-5007. Replace hard-coded property keys with DFSConfigKeys fields.
(Kousuke Saruta via jing9)
HDFS-5008. Make ClientProtocol#abandonBlock() idempotent. (jing9)
HADOOP-9760. Move GSet and related classes to common from HDFS.
(suresh)
HDFS-5020. Make DatanodeProtocol#blockReceivedAndDeleted idempotent.
(jing9)
HDFS-5024. Make DatanodeProtocol#commitBlockSynchronization idempotent.
(Arpit Agarwal via jing9)
HDFS-3880. Use Builder to build RPC server in HDFS.
(Brandon Li and Junping Du via szetszwo)
OPTIMIZATIONS
HDFS-4465. Optimize datanode ReplicasMap and ReplicaInfo. (atm)
HDFS-5027. On startup, DN should scan volumes in parallel. (atm)
BUG FIXES
HDFS-4626. ClientProtocol#getLinkTarget should throw an exception for
non-symlink and non-existent paths. (Andrew Wang via cmccabe)
HDFS-3934. duplicative dfs_hosts entries handled wrong. (Colin Patrick
McCabe)
HDFS-4470. Several HDFS tests attempt file operations on invalid HDFS
paths when running on Windows. (Chris Nauroth via suresh)
HDFS-4471. Namenode WebUI file browsing does not work with wildcard
addresses configured. (Andrew Wang via atm)
HDFS-4342. Directories configured in dfs.namenode.edits.dir.required
but not in dfs.namenode.edits.dir are silently ignored. (Arpit Agarwal
via szetszwo)
HDFS-4482. ReplicationMonitor thread can exit with NPE due to the race
between delete and replication of same file. (umamahesh)
HDFS-4269. Datanode rejects all datanode registrations from localhost
in single-node developer setup on Windows. (Chris Nauroth via suresh)
HDFS-4235. When outputting XML, OfflineEditsViewer can't handle some edits
containing non-ASCII strings. (Colin Patrick McCabe via atm)
HDFS-4541. Set hadoop.log.dir and hadoop.id.str when starting secure
datanode to write the logs to right dir by default. (Arpit Gupta via
suresh)
HDFS-4540. Namenode http server should use the web authentication
keytab for spnego principal. (Arpit Gupta via suresh)
HDFS-4544. Error in deleting blocks should not do check disk, for
all types of errors. (Arpit Agarwal via suresh)
HDFS-4565. Use DFSUtil.getSpnegoKeytabKey() to get the spnego keytab key
in secondary namenode and namenode http server. (Arpit Gupta via suresh)
HDFS-4571. WebHDFS should not set the service hostname on the server side.
(tucu)
HDFS-4013. TestHftpURLTimeouts throws NPE. (Chao Shi via suresh)
HDFS-4592. Default values for access time precision are out of sync between
hdfs-default.xml and the code. (atm)
HDFS-4522. LightWeightGSet expects incrementing a volatile to be atomic.
(Colin Patrick McCabe via atm)
HDFS-4484. libwebhdfs compilation broken with gcc 4.6.2. (Colin Patrick
McCabe via atm)
HDFS-4595. When short circuit read is fails, DFSClient does not fallback
to regular reads. (suresh)
HDFS-4583. TestNodeCount fails. (Ivan Mitic via suresh)
HDFS-4591. HA clients can fail to fail over while Standby NN is performing
long checkpoint. (atm)
HDFS-3277. fail over to loading a different FSImage if the first one we
try to load is corrupt. (Colin Patrick McCabe and Andrew Wang via atm)
HDFS-4596. Shutting down namenode during checkpointing can lead to md5sum
error. (Andrew Wang via atm)
HDFS-4614. FSNamesystem#getContentSummary should use getPermissionChecker
helper method. (atm)
HDFS-4620. Documentation for dfs.namenode.rpc-address specifies wrong
format. (Sandy Ryza via atm)
HDFS-4609. TestAuditLogs should release log handles between tests.
(Ivan Mitic via szetszwo)
HDFS-4598. Fix the default value of ConcatSourcesParam and the WebHDFS doc.
(szetszwo)
HDFS-4655. DNA_FINALIZE is logged as being an unknown command by the DN
when received from the standby NN. (atm)
HDFS-4656. DN heartbeat loop can be briefly tight. (atm)
HDFS-4658. Standby NN will log that it has received a block report "after
becoming active" (atm)
HDFS-4646. createNNProxyWithClientProtocol ignores configured timeout
value (Jagane Sundar via cos)
HDFS-3981. Fix handling of FSN lock in getBlockLocations. (Xiaobo Peng
and todd via todd)
HDFS-4676. TestHDFSFileSystemContract should set MiniDFSCluster variable
to null to free up memory. (suresh)
HDFS-4669. TestBlockPoolManager fails using IBM java. (Tian Hong Wang via
suresh)
HDFS-4643. Fix flakiness in TestQuorumJournalManager. (todd)
HDFS-4639. startFileInternal() should not increment generation stamp.
(Plamen Jeliazkov via shv)
HDFS-4695. TestEditLog leaks open file handles between tests.
(Ivan Mitic via suresh)
HDFS-4737. JVM path embedded in fuse binaries. (Sean Mackrory via atm)
HDFS-4739. NN can miscalculate the number of extra edit log segments to
retain. (atm)
HDFS-4745. TestDataTransferKeepalive#testSlowReader has race condition that
causes sporadic failure. (Chris Nauroth via suresh)
HDFS-4768. File handle leak in datanode when a block pool is removed.
(Chris Nauroth via suresh)
HDFS-4748. MiniJournalCluster#restartJournalNode leaks resources, which
causes sporadic test failures. (Chris Nauroth via suresh)
HDFS-4733. Make HttpFS username pattern configurable. (tucu via atm)
HDFS-4778. Fixes some issues that the first patch on HDFS-2576 missed.
(ddas)
HDFS-4785. Concat operation does not remove concatenated files from
InodeMap. (suresh)
HDFS-4784. NPE in FSDirectory.resolvePath(). (Brandon Li via suresh)
HDFS-4810. several HDFS HA tests have timeouts that are too short. (Chris
Nauroth via atm)
HDFS-4799. Corrupt replica can be prematurely removed from
corruptReplicas map. (todd via kihwal)
HDFS-4751. TestLeaseRenewer#testThreadName flakes. (Andrew Wang via atm)
HDFS-4533. start-dfs.sh ignores additional parameters besides -upgrade.
(Fengdong Yu via suresh)
HDFS-4765. Permission check of symlink deletion incorrectly throws
UnresolvedLinkException. (Andrew Wang via atm)
HDFS-4300. TransferFsImage.downloadEditsToStorage should use a tmp file for
destination. (Andrew Wang via atm)
HDFS-4813. Add volatile to BlocksMap.blocks so that the replication thread
can see the updated value. (Jing Zhao via szetszwo)
HDFS-3180. Add socket timeouts to WebHdfsFileSystem. (Chris Nauroth via
szetszwo)
HDFS-4787. Create a new HdfsConfiguration before each TestDFSClientRetries
testcases. (Tian Hong Wang via atm)
HDFS-4830. Typo in config settings for AvailableSpaceVolumeChoosingPolicy
in hdfs-default.xml. (atm)
HDFS-4824. FileInputStreamCache.close leaves dangling reference to
FileInputStreamCache.cacheCleaner. (Colin Patrick McCabe via todd)
HDFS-4298. StorageRetentionManager spews warnings when used with QJM. (atm)
HDFS-4725. Fix HDFS file handle leaks in FSEditLog, NameNode,
OfflineEditsBinaryLoader and some tests. (Chris Nauroth via szetszwo)
HDFS-4825. webhdfs / httpfs tests broken because of min block size change.
(Andrew Wang via suresh)
HDFS-4780. Use the correct relogin method for services. (Robert Parker via
kihwal)
HDFS-4827. Slight update to the implementation of API for handling favored
nodes in DFSClient (ddas)
HDFS-4865. Remove sub resource warning from httpfs log at startup time.
(ywskycn via tucu)
HDFS-4240. For nodegroup-aware block placement, when a node is excluded,
the nodes in the same nodegroup should also be excluded. (Junping Du
via szetszwo)
HDFS-4261. Fix bugs in Balaner causing infinite loop and
TestBalancerWithNodeGroup timeing out. (Junping Du via szetszwo)
HDFS-4382. Fix typo MAX_NOT_CHANGED_INTERATIONS. (Ted Yu via suresh)
HDFS-4840. ReplicationMonitor gets NPE during shutdown. (kihwal)
HADOOP-8957 HDFS tests for AbstractFileSystem#IsValidName should be overridden for
embedded file systems like ViewFs (Chris Nauroth via Sanjay Radia)
HDFS-4586. TestDataDirs.testGetDataDirsFromURIs fails with all directories
in dfs.datanode.data.dir are invalid. (Ivan Mitic via atm)
HDFS-3792. Fix two findbugs introduced by HDFS-3695 (todd)
HADOOP-9635 Fix potential Stack Overflow in DomainSocket.c (V. Karthik Kumar
via cmccabe)
HDFS-3163. TestHDFSCLI.testAll fails if the user name is not all lowercase.
(Brandon Li via atm)
HDFS-4845. FSNamesystem.deleteInternal should acquire write-lock before
changing the inode map. (Arpit Agarwal via szetszwo)
HDFS-4910. TestPermission failed in branch-2. (Chuan Liu via cnauroth)
HDFS-4906. HDFS Output streams should not accept writes after being
closed. (atm)
HDFS-4917. Start-dfs.sh cannot pass the parameters correctly.
(Fengdong Yu via suresh)
HDFS-4205. fsck fails with symlinks. (jlowe)
HDFS-4927. CreateEditsLog creates inodes with an invalid inode ID, which then
cannot be loaded by a namenode. (cnauroth)
HDFS-4944. WebHDFS cannot create a file path containing characters that must
be URI-encoded, such as space. (cnauroth)
HDFS-4888. Refactor and fix FSNamesystem.getTurnOffTip. (Ravi Prakash via
kihwal)
HDFS-4943. WebHdfsFileSystem does not work when original file path has
encoded chars. (Jerry He via szetszwo)
HDFS-4948. mvn site for hadoop-hdfs-nfs fails. (brandonli)
HDFS-4954. In nfs, OpenFileCtx.getFlushedOffset() should handle IOException.
(Brandon Li via szetszwo)
HDFS-4887. TestNNThroughputBenchmark exits abruptly. (kihwal)
HDFS-4980. Incorrect logging.properties file for hadoop-httpfs.
(Mark Grover via suresh)
HDFS-4999. Fix TestShortCircuitLocalRead on branch-2. (cmccabe via kihwal)
HDFS-4687. TestDelegationTokenForProxyUser#testWebHdfsDoAs is flaky with
JDK7. (Andrew Wang via atm)
HDFS-5003. TestNNThroughputBenchmark failed caused by existing directories.
(Xi Fang via cnauroth)
HDFS-5018. Misspelled DFSConfigKeys#DFS_NAMENODE_STALE_DATANODE_INTERVAL_DEFAULT
in javadoc of DatanodeInfo#isStale(). (Ted Yu via jing9)
HDFS-4602. TestBookKeeperHACheckpoints fails. (umamahesh)
HDFS-5016. Deadlock in pipeline recovery causes Datanode to be marked dead.
(suresh)
HDFS-5228. The RemoteIterator returned by DistributedFileSystem.listFiles
may throw NullPointerException. (szetszwo and cnauroth via szetszwo)
BREAKDOWN OF HDFS-347 SUBTASKS AND RELATED JIRAS
HDFS-4353. Encapsulate connections to peers in Peer and PeerServer classes.
(Colin Patrick McCabe via todd)
HDFS-4354. Create DomainSocket and DomainPeer and associated unit tests.
(Colin Patrick McCabe via todd)
HDFS-4356. BlockReaderLocal should use passed file descriptors rather than paths.
(Colin Patrick McCabe via todd)
HDFS-4388. DomainSocket should throw AsynchronousCloseException when appropriate.
(Colin Patrick McCabe via todd)
HDFS-4390. Bypass UNIX domain socket unit tests when they cannot be run.
(Colin Patrick McCabe via todd)
HDFS-4400. DFSInputStream#getBlockReader: last retries should ignore the cache
(Colin Patrick McCabe via todd)
HDFS-4401. Fix bug in DomainSocket path validation
(Colin Patrick McCabe via todd)
HDFS-4402. Some small DomainSocket fixes: avoid findbugs warning, change
log level, etc. (Colin Patrick McCabe via todd)
HDFS-4418. increase default FileInputStreamCache size (todd)
HDFS-4416. Rename dfs.datanode.domain.socket.path to dfs.domain.socket.path
(Colin Patrick McCabe via todd)
HDFS-4417. Fix case where local reads get disabled incorrectly
(Colin Patrick McCabe and todd via todd)
HDFS-4433. Make TestPeerCache not flaky (Colin Patrick McCabe via todd)
HDFS-4438. TestDomainSocket fails when system umask is set to 0002. (Colin
Patrick McCabe via atm)
HDFS-4440. Avoid annoying log message when dfs.domain.socket.path is not
set. (Colin Patrick McCabe via atm)
HDFS-4473. Don't create domain socket unless we need it. (Colin Patrick McCabe via atm)
HDFS-4485. DN should chmod socket path a+w. (Colin Patrick McCabe via atm)
HDFS-4453. Make a simple doc to describe the usage and design of the
shortcircuit read feature. (Colin Patrick McCabe via atm)
HDFS-4496. DFSClient: don't create a domain socket unless we need it (Colin
Patrick McCabe via todd)
HDFS-347: style cleanups (Colin Patrick McCabe via atm)
HDFS-4538. Allow use of legacy blockreader (Colin Patrick McCabe via todd)
HDFS-4661. A few little code cleanups of some HDFS-347-related code. (Colin
Patrick McCabe via atm)
BREAKDOWN OF HADOOP-8562 and HDFS-3602 SUBTASKS AND RELATED JIRAS
HDFS-4145. Merge hdfs cmd line scripts from branch-1-win. (David Lao,
Bikas Saha, Lauren Yang, Chuan Liu, Thejas M Nair and Ivan Mitic via suresh)
HDFS-4163. HDFS distribution build fails on Windows. (Chris Nauroth via
suresh)
HDFS-4316. branch-trunk-win contains test code accidentally added during
work on fixing tests on Windows. (Chris Nauroth via suresh)
HDFS-4297. Fix issues related to datanode concurrent reading and writing on
Windows. (Arpit Agarwal, Chuan Liu via suresh)
HDFS-4573. Fix TestINodeFile on Windows. (Arpit Agarwal via suresh)
HDFS-4572. Fix TestJournal failures on Windows. (Arpit Agarwal via suresh)
HDFS-4287. HTTPFS tests fail on Windows. (Chris Nauroth via suresh)
HDFS-4593. TestSaveNamespace fails on Windows. (Arpit Agarwal via suresh)
HDFS-4582. TestHostsFiles fails on Windows. (Ivan Mitic via suresh)
HDFS-4603. TestMiniDFSCluster fails on Windows. (Ivan Mitic via suresh)
HDFS-4604. TestJournalNode fails on Windows. (Ivan Mitic via suresh)
HDFS-4607. In TestGetConf.testGetSpecificKey(), use a platform-specific
line separator; otherwise, it fails on Windows. (Ivan Mitic via szetszwo)
HDFS-4625. Make TestNNWithQJM#testNewNamenodeTakesOverWriter work on
Windows. (Ivan Mitic via suresh)
HDFS-4674. TestBPOfferService fails on Windows due to failure parsing
datanode data directory as URI. (Chris Nauroth via suresh)
HDFS-4615. Fix TestDFSShell failures on Windows. (Arpit Agarwal
via szetszwo)
HDFS-4584. Skip TestNNWithQJM.testNewNamenodeTakesOverWriter() on Windows.
(Arpit Agarwal via szetszwo)
HDFS-4732. Fix TestDFSUpgradeFromImage which fails on Windows due to
failure to unpack old image tarball that contains hard links.
(Chris Nauroth via szetszwo)
HDFS-4741. TestStorageRestore#testStorageRestoreFailure fails on Windows.
(Arpit Agarwal via suresh)
HDFS-4743. TestNNStorageRetentionManager fails on Windows.
(Chris Nauroth via suresh)
HDFS-4740. Fixes for a few test failures on Windows.
(Arpit Agarwal via suresh)
HDFS-4722. TestGetConf#testFederation times out on Windows.
(Ivan Mitic via suresh)
HDFS-4705. Address HDFS test failures on Windows because of invalid
dfs.namenode.name.dir. (Ivan Mitic via suresh)
HDFS-4734. HDFS Tests that use ShellCommandFencer are broken on Windows.
(Arpit Agarwal via suresh)
HDFS-4610. Use common utils FileUtil#setReadable/Writable/Executable and
FileUtil#canRead/Write/Execute. (Ivan Mitic via suresh)
HDFS-4677. Editlog should support synchronous writes. (ivanmi)
HDFS-4752. TestRBWBlockInvalidation fails on Windows due to file locking.
(Chris Nauroth via suresh)
HDFS-4783. TestDelegationTokensWithHA#testHAUtilClonesDelegationTokens fails
on Windows. (cnauroth)
HDFS-4818. Several HDFS tests that attempt to make directories unusable do
not work correctly on Windows. (cnauroth)
BREAKDOWN OF HDFS-2802 HDFS SNAPSHOT SUBTASKS AND RELATED JIRAS
HDFS-4076. Support snapshot of single files. (szetszwo)
HDFS-4082. Add editlog opcodes for snapshot create and delete operations.
(suresh via szetszwo)
HDFS-4086. Add editlog opcodes to allow and disallow snapshots on a
directory. (Brandon Li via suresh)
HDFS-4083. Protocol changes for snapshots. (suresh)
HDFS-4077. Add support for Snapshottable Directory. (szetszwo via suresh)
HDFS-4087. Protocol changes for listSnapshots functionality.
(Brandon Li via suresh)
HDFS-4079. Add SnapshotManager which maintains a list for all the
snapshottable directories and supports snapshot methods such as setting a
directory to snapshottable and creating a snapshot. (szetszwo)
HDFS-4078. Handle replication in snapshots. (szetszwo)
HDFS-4084. Provide CLI support to allow and disallow snapshot
on a directory. (Brondon Li via suresh)
HDFS-4091. Add snapshot quota to limit the number of snapshots allowed.
(szetszwo)
HDFS-4097. Provide CLI support for createSnapshot. (Brandon Li via suresh)
HDFS-4092. Update file deletion logic for snapshot so that the current inode
is removed from the circular linked list; and if some blocks at the end of
the block list no longer belong to any other inode, collect them and update
the block list. (szetszwo)
HDFS-4111. Support snapshot of subtrees. (szetszwo via suresh)
HDFS-4119. Complete the allowSnapshot code and add a test for it. (szetszwo)
HDFS-4133. Add testcases for testing basic snapshot functionalities.
(Jing Zhao via suresh)
HDFS-4116. Add auditlog for some snapshot operations. (Jing Zhao via suresh)
HDFS-4095. Add some snapshot related metrics. (Jing Zhao via suresh)
HDFS-4141. Support directory diff - the difference between the current state
and a previous snapshot of an INodeDirectory. (szetszwo)
HDFS-4146. Use getter and setter in INodeFileWithLink to access blocks and
initialize root directory as snapshottable. (szetszwo)
HDFS-4149. Implement the disallowSnapshot(..) in FSNamesystem and add
resetSnapshottable(..) to SnapshotManager. (szetszwo)
HDFS-4147. When there is a snapshot in a subtree, deletion of the subtree
should fail. (Jing Zhao via szetszwo)
HDFS-4150. Update the inode in the block map when a snapshotted file or a
snapshot file is deleted. (Jing Zhao via szetszwo)
HDFS-4159. Rename should fail when the destination directory is
snapshottable and has snapshots. (Jing Zhao via szetszwo)
HDFS-4170. Add snapshot information to INodesInPath. (szetszwo)
HDFS-4177. Add a snapshot parameter to INodeDirectory.getChildrenList() for
selecting particular snapshot children list views. (szetszwo)
HDFS-4148. Disallow write/modify operations on files and directories in a
snapshot. (Brandon Li via suresh)
HDFS-4188. Add Snapshot.ID_COMPARATOR for comparing IDs and fix a bug in
ReadOnlyList.Util.binarySearch(..). (szetszwo)
HDFS-4187. Add tests for replication handling in snapshots. (Jing Zhao via
szetszwo)
HDFS-4196. Support renaming of snapshots. (Jing Zhao via szetszwo)
HDFS-4175. Additional snapshot tests for more complicated directory
structure and modifications. (Jing Zhao via suresh)
HDFS-4293. Fix TestSnapshot failure. (Jing Zhao via suresh)
HDFS-4317. Change INode and its subclasses to support HDFS-4103. (szetszwo)
HDFS-4103. Support O(1) snapshot creation. (szetszwo)
HDFS-4330. Support snapshots up to the snapshot limit. (szetszwo)
HDFS-4357. Fix a bug that if an inode is replaced, further INode operations
should apply to the new inode. (Jing Zhao via szetszwo)
HDFS-4230. Support listing of all the snapshottable directories. (Jing Zhao
via szetszwo)
HDFS-4244. Support snapshot deletion. (Jing Zhao via szetszwo)
HDFS-4245. Include snapshot related operations in TestOfflineEditsViewer.
(Jing Zhao via szetszwo)
HDFS-4395. In INodeDirectorySnapshottable's constructor, the passed-in dir
could be an INodeDirectoryWithSnapshot. (Jing Zhao via szetszwo)
HDFS-4397. Fix a bug in INodeDirectoryWithSnapshot.Diff.combinePostDiff(..)
that it may put the wrong node into the deleted list. (szetszwo)
HDFS-4407. Change INodeDirectoryWithSnapshot.Diff.combinePostDiff(..) to
merge-sort like and keep the postDiff parameter unmodified. (szetszwo)
HDFS-4098. Add FileWithSnapshot, INodeFileUnderConstructionWithSnapshot and
INodeFileUnderConstructionSnapshot for supporting append to snapshotted
files. (szetszwo)
HDFS-4126. Add reading/writing snapshot information to FSImage.
(Jing Zhao via suresh)
HDFS-4436. Change INode.recordModification(..) to return only the current
inode and remove the updateCircularList parameter from some methods in
INodeDirectoryWithSnapshot.Diff. (szetszwo)
HDFS-4429. When the latest snapshot exists, INodeFileUnderConstruction
should be replaced with INodeFileWithSnapshot but not INodeFile.
(Jing Zhao via szetszwo)
HDFS-4441. Move INodeDirectoryWithSnapshot.Diff and the related classes to a
package. (szetszwo)
HDFS-4432. Support INodeFileUnderConstructionWithSnapshot in FSImage
saving/loading. (Jing Zhao via suresh)
HDFS-4131. Add capability to namenode to get snapshot diff. (Jing Zhao via
suresh)
HDFS-4447. Refactor INodeDirectoryWithSnapshot for supporting general INode
diff lists. (szetszwo)
HDFS-4189. Renames the getMutableXxx methods to getXxx4Write and fix a bug
that some getExistingPathINodes calls should be getINodesInPath4Write.
(szetszwo)
HDFS-4361. When listing snapshottable directories, only return those
where the user has permission to take snapshots. (Jing Zhao via szetszwo)
HDFS-4464. Combine collectSubtreeBlocksAndClear with deleteDiffsForSnapshot
and rename it to destroySubtreeAndCollectBlocks. (szetszwo)
HDFS-4414. Add support for getting snapshot diff from DistributedFileSystem.
(Jing Zhao via suresh)
HDFS-4446. Support file snapshots with diff lists. (szetszwo)
HDFS-4480. Eliminate the file snapshot circular linked list. (szetszwo)
HDFS-4481. Change fsimage to support snapshot file diffs. (szetszwo)
HDFS-4500. Refactor snapshot INode methods. (szetszwo)
HDFS-4487. Fix snapshot diff report for HDFS-4446. (Jing Zhao via szetszwo)
HDFS-4431. Support snapshot in OfflineImageViewer. (Jing Zhao via szetszwo)
HDFS-4503. Update computeContentSummary(..), spaceConsumedInTree(..) and
diskspaceConsumed(..) in INode for snapshot. (szetszwo)
HDFS-4499. Fix file/directory/snapshot deletion for file diff. (Jing Zhao
via szetszwo)
HDFS-4524. Update SnapshotManager#snapshottables when loading fsimage.
(Jing Zhao via szetszwo)
HDFS-4520. Support listing snapshots under a snapshottable directory using
ls. (Jing Zhao via szetszwo)
HDFS-4514. Add CLI for supporting snapshot rename, diff report, and
snapshottable directory listing. (Jing Zhao via szetszwo)
HDFS-4523. Fix INodeFile replacement, TestQuota and javac errors from trunk
merge. (szetszwo)
HDFS-4507. Update quota verification for snapshots. (szetszwo)
HDFS-4545. With snapshots, FSDirectory.unprotectedSetReplication(..) always
changes file replication but it may or may not changes block replication.
(szetszwo)
HDFS-4557. Fix FSDirectory#delete when INode#cleanSubtree returns 0.
(Jing Zhao via szetszwo)
HDFS-4579. Annotate snapshot tests. (Arpit Agarwal via suresh)
HDFS-4574. Move Diff to the util package. (szetszwo)
HDFS-4563. Update namespace/diskspace usage after deleting snapshots.
(Jing Zhao via szetszwo)
HDFS-4144. Create test for all snapshot-related metrics.
(Jing Zhao via suresh)
HDFS-4556. Add snapshotdiff and LsSnapshottableDir tools to hdfs script.
(Arpit Agarwal via szetszwo)
HDFS-4534. Add INodeReference in order to support rename with snapshots.
(szetszwo)
HDFS-4616. Update the FilesDeleted metric while deleting file/dir in the
current tree. (Jing Zhao via szetszwo)
HDFS-4627. Fix FSImageFormat#Loader NPE and synchronization issues.
(Jing Zhao via suresh)
HDFS-4612. Not to use INode.getParent() when generating snapshot diff
report. (Jing Zhao via szetszwo)
HDFS-4636. Update quota usage when deleting files/dirs that were created
after taking the latest snapshot. (Jing Zhao via szetszwo)
HDFS-4648. For snapshot deletion, when merging the diff from to-delete
snapshot to the prior snapshot, make sure files/directories created after
the prior snapshot get deleted. (Jing Zhao via szetszwo)
HDFS-4637. INodeDirectory#replaceSelf4Quota may incorrectly convert a newly
created directory to an INodeDirectoryWithSnapshot. (Jing Zhao via szetszwo)
HDFS-4611. Update FSImage for INodeReference. (szetszwo)
HDFS-4647. Rename should call setLocalName after an inode is removed from
snapshots. (Arpit Agarwal via szetszwo)
HDFS-4684. Use INode id for image serialization when writing INodeReference.
(szetszwo)
HDFS-4675. Fix rename across snapshottable directories. (Jing Zhao via
szetszwo)
HDFS-4692. Use timestamp as default snapshot names. (szetszwo)
HDFS-4666. Define ".snapshot" as a reserved inode name so that users cannot
create a file/directory with ".snapshot" as the name. If ".snapshot" is
used in a previous version of HDFS, it must be renamed before upgrade;
otherwise, upgrade will fail. (szetszwo)
HDFS-4700. Fix the undo section of rename with snapshots. (Jing Zhao via
szetszwo)
HDFS-4529. Disallow concat when one of the src files is in some snapshot.
(szetszwo)
HDFS-4550. Refactor INodeDirectory.INodesInPath to a standalone class.
(szetszwo)
HDFS-4707. Add snapshot methods to FilterFileSystem and fix findbugs
warnings. (szetszwo)
HDFS-4706. Do not replace root inode for disallowSnapshot. (szetszwo)
HDFS-4717. Change the path parameter type of the snapshot methods in
HdfsAdmin from String to Path. (szetszwo)
HDFS-4708. Add snapshot user documentation. (szetszwo)
HDFS-4726. Fix test failures after merging the INodeId-INode mapping
from trunk. (Jing Zhao via szetszwo)
HDFS-4727. Update inodeMap after deleting files/directories/snapshots.
(Jing Zhao via szetszwo)
HDFS-4719. Remove AbstractINodeDiff.Factory and move its methods to
AbstractINodeDiffList. (Arpit Agarwal via szetszwo)
HDFS-4735. DisallowSnapshot throws IllegalStateException for nested
snapshottable directories. (Jing Zhao via szetszwo)
HDFS-4738. Changes AbstractINodeDiff to implement Comparable<Integer>, and
fix javadoc and other warnings. (szetszwo)
HDFS-4686. Update quota computation for rename and INodeReference.
(Jing Zhao via szetszwo)
HDFS-4729. Fix OfflineImageViewer and permission checking for snapshot
operations. (Jing Zhao via szetszwo)
HDFS-4749. Use INodeId to identify the corresponding directory node in
FSImage saving/loading. (Jing Zhao via szetszwo)
HDFS-4742. Fix appending to a renamed file with snapshot. (Jing Zhao via
szetszwo)
HDFS-4755. Fix AccessControlException message and moves "implements
LinkedElement" from INode to INodeWithAdditionalFields. (szetszwo)
HDFS-4650. Fix a bug in FSDirectory and add more unit tests for rename with
existence of snapshottable directories and snapshots. (Jing Zhao via
szetszwo)
HDFS-4650. When passing two non-existing snapshot names to snapshotDiff, it
returns success if the names are the same. (Jing Zhao via szetszwo)
HDFS-4767. If a directory is snapshottable, do not replace it when clearing
quota. (Jing Zhao via szetszwo)
HDFS-4578. Restrict snapshot IDs to 24-bit wide. (Arpit Agarwal via
szetszwo)
HDFS-4773. Fix bugs in quota usage computation and OfflineImageViewer.
(Jing Zhao via szetszwo)
HDFS-4760. Update inodeMap after node replacement. (Jing Zhao via szetszwo)
HDFS-4758. Disallow nested snapshottable directories and unwrap
RemoteException. (szetszwo)
HDFS-4781. Fix a NullPointerException when listing .snapshot under
a non-existing directory. (szetszwo)
HDFS-4791. Update and fix deletion of reference inode. (Jing Zhao via
szetszwo)
HDFS-4798. Update computeContentSummary() for the reference nodes in
snapshots. (szetszwo)
HDFS-4800. Fix INodeDirectoryWithSnapshot#cleanDeletedINode. (Jing Zhao via
szetszwo)
HDFS-4801. lsSnapshottableDir throws IllegalArgumentException when root is
snapshottable. (Jing Zhao via szetszwo)
HDFS-4802. Disallowing snapshot on / twice should throw SnapshotException
but not IllegalStateException. (Jing Zhao via szetszwo)
HDFS-4806. In INodeDirectoryWithSnapshot, use isInLatestSnapshot() to
determine if an added/removed child should be recorded in the snapshot diff.
(Jing Zhao via szetszwo)
HDFS-4809. When a QuotaExceededException is thrown during rename, the quota
usage should be subtracted back. (Jing Zhao via szetszwo)
HDFS-4842. Identify the correct prior snapshot when deleting a
snapshot under a renamed subtree. (jing9)
HDFS-4846. Clean up snapshot CLI commands output stacktrace for invalid
arguments. (Jing Zhao via brandonli)
HDFS-4857. Snapshot.Root and AbstractINodeDiff#snapshotINode should not be
put into INodeMap when loading FSImage. (jing9)
HDFS-4863. The root directory should be added to the snapshottable
directory list while loading fsimage. (jing9)
HDFS-4848. copyFromLocal and renaming a file to ".snapshot" should output
that ".snapshot" is a reserved name. (Jing Zhao via brandonli)
HDFS-4826. TestNestedSnapshots times out due to repeated slow edit log
flushes when running on virtualized disk. (Chris Nauroth via szetszwo)
HDFS-4876. Fix the javadoc of FileWithSnapshot and move FileDiffList to
FileWithSnapshot. (szetszwo)
HDFS-4850. Fix OfflineImageViewer to work on fsimages with empty files or
snapshots. (jing9)
HDFS-4877. Snapshot: fix the scenario where a directory is renamed under
its prior descendant. (jing9)
HDFS-4873. callGetBlockLocations returns incorrect number of blocks for
snapshotted files. (jing9)
HDFS-4819. Update Snapshot doc to clarify that nested snapshots are not
allowed. (szetszwo)
HDFS-4902. DFSClient.getSnapshotDiffReport should use string path rather
than o.a.h.fs.Path. (Binglin Chang via jing9)
HDFS-4875. Add a test for testing snapshot file length.
(Arpit Agarwal via jing9)
HDFS-4841. FsShell commands using secure webhfds fail ClientFinalizer
shutdown hook. (rkanter via tucu)
HDFS-4951. FsShell commands using secure httpfs throw exceptions due
to missing TokenRenewer. (rknater via tucu)
HDFS-4969. WebhdfsFileSystem expects non-standard WEBHDFS Json element.
(rkanter via tucu)
HDFS-4797. BlockScanInfo does not override equals(..) and hashCode()
consistently. (szetszwo)
HDFS-4978. Make disallowSnapshot idempotent. (jing9)
HDFS-5005. Move SnapshotException and SnapshotAccessControlException
to o.a.h.hdfs.protocol. (jing9)
HDFS-4982. JournalNode should relogin from keytab before fetching logs
from other JNs (todd)
Release 2.0.5-alpha - 06/06/2013
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
OPTIMIZATIONS
BUG FIXES
Release 2.0.4-alpha - 2013-04-25
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
OPTIMIZATIONS
BUG FIXES
Release 2.0.3-alpha - 2013-02-06
INCOMPATIBLE CHANGES
HDFS-4122. Cleanup HDFS logs and reduce the size of logged messages.
(suresh)
HDFS-4362. GetDelegationTokenResponseProto does not handle null token.
(suresh)
HDFS-4367. GetDataEncryptionKeyResponseProto does not handle null
response. (suresh)
HDFS-4364. GetLinkTargetResponseProto does not handle null path. (suresh)
HDFS-4369. GetBlockKeysResponseProto does not handle null response.
(suresh)
HDFS-4451. hdfs balancer command returns exit code 1 on success instead
of 0. (Joshua Blatt via suresh)
HDFS-4350. Make enabling of stale marking on read and write paths
independent. (Andrew Wang via suresh)
NEW FEATURES
HDFS-2656. Add libwebhdfs, a pure C client based on WebHDFS.
(Jaimin D Jetly and Jing Zhao via szetszwo)
HDFS-3912. Detect and avoid stale datanodes for writes.
(Jing Zhao via suresh)
HDFS-4059. Add number of stale DataNodes to metrics. (Jing Zhao via suresh)
HDFS-4155. libhdfs implementation of hsync API (Liang Xie via todd)
HDFS-4213. Add an API to hsync for updating the last block length at the
namenode. (Jing Zhao via szetszwo)
HDFS-3077. Implement QuorumJournalManager, a distributed mechanism for
reliably storing HDFS edit logs. See dedicated section below for breakdown
of subtasks.
IMPROVEMENTS
HDFS-3925. Prettify PipelineAck#toString() for printing to a log
(Andrew Wang via todd)
HDFS-3939. NN RPC address cleanup. (eli)
HDFS-3373. Change DFSClient input stream socket cache to global static and
add a thread to cleanup expired cache entries. (John George via szetszwo)
HDFS-3896. Add descriptions for dfs.namenode.rpc-address and
dfs.namenode.servicerpc-address to hdfs-default.xml. (Jeff Lord via atm)
HDFS-3996. Add debug log removed in HDFS-3873 back. (eli)
HDFS-3916. libwebhdfs (C client) code cleanups.
(Colin Patrick McCabe via eli)
HDFS-3813. Log error message if security and WebHDFS are enabled but
principal/keytab are not configured. (Stephen Chu via atm)
HDFS-3483. Better error message when hdfs fsck is run against a ViewFS
config. (Stephen Fritz via atm)
HDFS-3682. MiniDFSCluster#init should provide more info when it fails.
(todd via eli)
HDFS-4008. TestBalancerWithEncryptedTransfer needs a timeout. (eli)
HDFS-4007. Rehabilitate bit-rotted unit tests under
hadoop-hdfs-project/hadoop-hdfs/src/test/unit/
(Colin Patrick McCabe via todd)
HDFS-4041. Hadoop HDFS Maven protoc calls must not depend on external
sh script. (Chris Nauroth via suresh)
HADOOP-8911. CRLF characters in source and text files.
(Raja Aluri via suresh)
HDFS-4037. Rename the getReplication() method in BlockCollection to
getBlockReplication(). (szetszwo)
HDFS-4036. Remove "throws UnresolvedLinkException" from
FSDirectory.unprotectedAddFile(..). (Jing Zhao via szetszwo)
HDFS-2946. HA: Put a cap on the number of completed edits files retained
by the NN. (atm)
HDFS-4029. GenerationStamp should use an AtomicLong. (eli)
HDFS-4068. DatanodeID and DatanodeInfo member should be private. (eli)
HDFS-4073. Two minor improvements to FSDirectory. (Jing Zhao via szetszwo)
HDFS-4074. Remove the unused default constructor from INode. (Brandon Li
via szetszwo)
HDFS-4088. Remove "throws QuotaExceededException" from an
INodeDirectoryWithQuota constructor. (szetszwo)
HDFS-4099. Clean up replication code and add more javadoc. (szetszwo)
HDFS-4107. Add utility methods for casting INode to INodeFile and
INodeFileUnderConstruction. (szetszwo)
HDFS-4112. A few improvements on INodeDirectory include adding a utility
method for casting; avoiding creation of new empty lists; cleaning up
some code and rewriting some javadoc. (szetszwo)
HDFS-4121. Add namespace declarations in hdfs .proto files for languages
other than java. (Binglin Chang via suresh)
HDFS-3573. Supply NamespaceInfo when instantiating JournalManagers.
(todd and ivank via umamahesh)
HDFS-3695. Genericize format() to non-file JournalManagers.
(todd via umamahesh)
HDFS-3789. JournalManager#format() should be able to throw IOException.
(Ivan Kelly via umamahesh)
HDFS-3809. Make BKJM use protobufs for all serialization with ZK.
(Ivan Kelly via umamhesh)
HDFS-3916. libwebhdfs testing code cleanup. (Jing Zhao via suresh)
HDFS-4143. Change blocks to private in INodeFile and renames isLink() to
isSymlink() in INode. (szetszwo)
HDFS-4046. Rename ChecksumTypeProto enum NULL since it is illegal in
C/C++. (Binglin Chang via suresh)
HDFS-4048. Use ERROR instead of INFO for volume failure logs.
(Stephen Chu via eli)
HDFS-1322. Document umask in DistributedFileSystem#mkdirs javadocs.
(Colin Patrick McCabe via eli)
HDFS-4038. Override toString() for BookKeeperEditLogInputStream.
(Vinay via umamahesh)
HDFS-4214. OfflineEditsViewer should print out the offset at which it
encountered an error. (Colin Patrick McCabe via atm)
HDFS-4199. Provide test for HdfsVolumeId. (Ivan A. Veselovsky via atm)
HDFS-3049. During the normal NN startup process, fall back on a different
edit log if we see one that is corrupt (Colin Patrick McCabe via todd)
HDFS-3571. Allow EditLogFileInputStream to read from a remote URL (todd)
HDFS-4110. Refine a log printed in JNStorage. (Liang Xie via suresh)
HDFS-4153. Add START_MSG/SHUTDOWN_MSG for JournalNode. (liang xie via atm)
HDFS-3935. Add JournalNode to the start/stop scripts (Andy Isaacson via todd)
HDFS-4268. Remove redundant enum NNHAStatusHeartbeat.State. (shv)
HDFS-3680. Allow customized audit logging in HDFS FSNamesystem. (Marcelo
Vanzin via atm)
HDFS-4130. BKJM: The reading for editlog at NN starting using bkjm is not efficient.
(Han Xiao via umamahesh)
HDFS-4326. bump up Tomcat version for HttpFS to 6.0.36. (tucu via acmurthy)
HDFS-4270. Introduce soft and hard limits for max replication so that
replications of the highest priority are allowed to choose a source datanode
that has reached its soft limit but not the hard limit. (Derek Dagit via
szetszwo)
HADOOP-9173. Add security token protobuf definition to common and
use it in hdfs. (suresh)
HDFS-4030. BlockManager excessBlocksCount and
postponedMisreplicatedBlocksCount should be AtomicLongs. (eli)
HDFS-4031. Update findbugsExcludeFile.xml to include findbugs 2
exclusions. (eli)
HDFS-4033. Miscellaneous findbugs 2 fixes. (eli)
HDFS-4034. Remove redundant null checks. (eli)
HDFS-4035. LightWeightGSet and LightWeightHashSet increment a
volatile without synchronization. (eli)
HDFS-4032. Specify the charset explicitly rather than rely on the
default. (eli)
HDFS-4363. Combine PBHelper and HdfsProtoUtil and remove redundant
methods. (suresh)
HDFS-4377. Some trivial DN comment cleanup. (eli)
HDFS-4381. Document fsimage format details in FSImageFormat class javadoc.
(Jing Zhao via suresh)
HDFS-4375. Use token request messages defined in hadoop common.
(suresh)
HDFS-4392. Use NetUtils#getFreeSocketPort in MiniDFSCluster.
(Andrew Purtell via suresh)
HDFS-4393. Make empty request and responses in protocol translators can be
static final members. (Brandon Li via suresh)
HDFS-4403. DFSClient can infer checksum type when not provided by reading
first byte (todd)
HDFS-4259. Improve pipeline DN replacement failure message (harsh)
HDFS-3598. WebHDFS support for file concat. (Plamen Jeliazkov via shv)
HDFS-4456. Add concat to HttpFS and WebHDFS REST API docs. (plamenj2003 via tucu)
HDFS-3131. Improve TestStorageRestore. (Brandon Li via atm)
OPTIMIZATIONS
HDFS-3429. DataNode reads checksums even if client does not need them (todd)
BUG FIXES
HDFS-3919. MiniDFSCluster:waitClusterUp can hang forever.
(Andy Isaacson via eli)
HDFS-3924. Multi-byte id in HdfsVolumeId. (Andrew Wang via atm)
HDFS-3936. MiniDFSCluster shutdown races with BlocksMap usage. (eli)
HDFS-3951. datanode web ui does not work over HTTPS when datanode is started in secure mode. (tucu)
HDFS-3949. NameNodeRpcServer#join should join on both client and
server RPC servers. (eli)
HDFS-3932. NameNode Web UI broken if the rpc-address is set to the wildcard.
(Colin Patrick McCabe via eli)
HDFS-3931. TestDatanodeBlockScanner#testBlockCorruptionPolicy2 is broken.
(Andy Isaacson via eli)
HDFS-3964. Make NN log of fs.defaultFS debug rather than info. (eli)
HDFS-3992. Method org.apache.hadoop.hdfs.TestHftpFileSystem.tearDown()
sometimes throws NPEs. (Ivan A. Veselovsky via atm)
HDFS-3753. Tests don't run with native libraries.
(Colin Patrick McCabe via eli)
HDFS-4000. TestParallelLocalRead fails with "input ByteBuffers
must be direct buffers". (Colin Patrick McCabe via eli)
HDFS-3999. HttpFS OPEN operation expects len parameter, it should be length. (tucu)
HDFS-4006. TestCheckpoint#testSecondaryHasVeryOutOfDateImage
occasionally fails due to unexpected exit. (todd via eli)
HDFS-4018. testMiniDFSClusterWithMultipleNN is missing some
cluster cleanup. (eli)
HDFS-4020. TestRBWBlockInvalidation may time out. (eli)
HDFS-4021. Misleading error message when resources are low on the NameNode.
(Christopher Conner via atm)
HDFS-4044. Duplicate ChecksumType definition in HDFS .proto files.
(Binglin Chang via suresh)
HDFS-4049. Fix hflush performance regression due to nagling delays
(todd)
HDFS-3678. Edit log files are never being purged from 2NN. (atm)
HDFS-4058. DirectoryScanner may fail with IOOB if the directory
scanning threads return out of volume order. (eli)
HDFS-3985. Add timeouts to TestMulitipleNNDataBlockScanner. (todd via eli)
HDFS-4061. TestBalancer and TestUnderReplicatedBlocks need timeouts. (eli)
HDFS-3997. OfflineImageViewer incorrectly passes value of imageVersion when
visiting IS_COMPRESSED element. (Mithun Radhakrishnan via atm)
HDFS-4055. TestAuditLogs is flaky. (Binglin Chang via eli)
HDFS-4072. On file deletion remove corresponding blocks pending
replications. (Jing Zhao via suresh)
HDFS-4022. Replication not happening for appended block.
(Vinay via umamahesh)
HDFS-3948. Do not use hflush in TestWebHDFS.testNamenodeRestart() since the
out stream returned by WebHdfsFileSystem does not support it. (Jing Zhao
via szetszwo)
HDFS-3616. Fix a ConcurrentModificationException bug that BP actor threads
may not be shutdown properly in DataNode. (Jing Zhao via szetszwo)
HDFS-4127. Log message is not correct in case of short of replica.
(Junping Du via suresh)
HADOOP-8994. TestDFSShell creates file named "noFileHere", making further
tests hard to understand (Andy Isaacson via daryn)
HDFS-3804. TestHftpFileSystem fails intermittently with JDK7
(Trevor Robinson via daryn)
HDFS-4132. When libwebhdfs is not enabled, nativeMiniDfsClient frees
uninitialized memory (Colin Patrick McCabe via todd)
HDFS-1331. dfs -test should work like /bin/test (Andy Isaacson via daryn)
HDFS-3979. For hsync, datanode should wait for the local sync to complete
before sending ack. (Lars Hofhansl via szetszwo)
HDFS-3810. Implement format() for BKJM (Ivan Kelly via umamahesh)
HDFS-3625. Fix TestBackupNode by properly initializing edit log during
startup. (Junping Du via todd)
HDFS-4138. BackupNode startup fails due to uninitialized edit log.
(Kihwal Lee via shv)
HDFS-4162. Some malformed and unquoted HTML strings are returned from
datanode web ui. (Darek Dagit via suresh)
HDFS-4164. fuse_dfs: add -lrt to the compiler command line on Linux.
(Colin Patrick McCabe via eli)
HDFS-3921. NN will prematurely consider blocks missing when entering active
state while still in safe mode. (atm)
HDFS-4106. BPServiceActor#lastHeartbeat, lastBlockReport and
lastDeletedReport should be volatile. (Jing Zhao via suresh)
HDFS-4139. fuse-dfs RO mode still allows file truncation.
(Colin Patrick McCabe via eli)
HDFS-4104. dfs -test -d prints inappropriate error on nonexistent directory
(Andy Isaacson via daryn)
HDFS-3623. BKJM: zkLatchWaitTimeout hard coded to 6000. Make use of ZKSessionTimeout instead.
(umamahesh)
HDFS-4100. Fix all findbug security warings. (Liang Xie via eli)
HDFS-3507. DFS#isInSafeMode needs to execute only on Active NameNode.
(Vinay via atm)
HDFS-4105. The SPNEGO user for secondary namenode should use the web
keytab. (Arpit Gupta via jitendra)
HDFS-4156. Seeking to a negative position should throw an IOE.
(Eli Reisman via eli)
HDFS-4171. WebHDFS and HttpFs should accept only valid Unix user
names. (tucu)
HDFS-4178. Shell scripts should not close stderr (Andy Isaacson via daryn)
HDFS-4179. BackupNode: allow reads, fix checkpointing, safeMode. (shv)
HDFS-4216. Do not ignore QuotaExceededException when adding symlinks.
(szetszwo)
HDFS-4242. Map.Entry is incorrectly used in LeaseManager since the behavior
of it is undefined after the iteration or modifications of the map.
(szetszwo)
HDFS-4231. BackupNode: Introduce BackupState. (shv)
HDFS-4238. Standby namenode should not do purging of shared
storage edits. (todd)
HDFS-4282. TestEditLog.testFuzzSequences FAILED in all pre-commit test
(todd)
HDFS-4236. Remove artificial limit on username length introduced in
HDFS-4171. (tucu via suresh)
HDFS-4279. NameNode does not initialize generic conf keys when started
with -recover. (Colin Patrick McCabe via atm)
HDFS-4291. edit log unit tests leave stray test_edit_log_file around
(Colin Patrick McCabe via todd)
HDFS-4292. Sanity check not correct in RemoteBlockReader2.newBlockReader
(Binglin Chang via todd)
HDFS-4295. Using port 1023 should be valid when starting Secure DataNode
(Stephen Chu via todd)
HDFS-4294. Backwards compatibility is not maintained for TestVolumeId.
(Ivan A. Veselovsky and Robert Parker via atm)
HDFS-2264. NamenodeProtocol has the wrong value for clientPrincipal in
KerberosInfo annotation. (atm)
HDFS-4307. SocketCache should use monotonic time. (Colin Patrick McCabe
via atm)
HDFS-4315. DNs with multiple BPs can have BPOfferServices fail to start
due to unsynchronized map access. (atm)
HDFS-4140. fuse-dfs handles open(O_TRUNC) poorly. (Colin Patrick McCabe
via atm)
HDFS-4308. addBlock() should persist file blocks once.
(Plamen Jeliazkov via shv)
HDFS-4347. Avoid infinite waiting checkpoint to complete in TestBackupNode.
(Plamen Jeliazkov via shv)
HDFS-4349. Add test for reading files from BackupNode. (shv)
HDFS-4302. Fix fatal exception when starting NameNode with DEBUG logs
(Eugene Koontz via todd)
HDFS-3970. Fix bug causing rollback of HDFS upgrade to result in bad
VERSION file. (Vinay and Andrew Wang via atm)
HDFS-4306. PBHelper.convertLocatedBlock miss convert BlockToken. (Binglin
Chang via atm)
HDFS-4384. test_libhdfs_threaded gets SEGV if JNIEnv cannot be
initialized. (Colin Patrick McCabe via eli)
HDFS-4328. TestLargeBlock#testLargeBlockSize is timing out. (Chris Nauroth
via atm)
HDFS-4274. BlockPoolSliceScanner does not close verification log during
shutdown. (Chris Nauroth via suresh)
HDFS-1245. Pluggable block id generation. (shv)
HDFS-4415. HostnameFilter should handle hostname resolution failures and
continue processing. (Robert Kanter via atm)
HDFS-4359. Slow RPC responses from NN can prevent metrics collection on
DNs. (liang xie via atm)
HDFS-4444. Add space between total transaction time and number of
transactions in FSEditLog#printStatistics. (Stephen Chu via suresh)
HDFS-4428. FsDatasetImpl should disclose what the error is when a rename
fails. (Colin Patrick McCabe via atm)
HDFS-4452. getAdditionalBlock() can create multiple blocks if the client
times out and retries. (shv)
HDFS-4445. All BKJM ledgers are not checked while tailing, So failover will fail.
(Vinay via umamahesh)
HDFS-4462. 2NN will fail to checkpoint after an HDFS upgrade from a
pre-federation version of HDFS. (atm)
HDFS-4404. Create file failure when the machine of first attempted NameNode
is down. (Todd Lipcon via atm)
HDFS-4344. dfshealth.jsp throws NumberFormatException when
dfs.hosts/dfs.hosts.exclude includes port number. (Andy Isaacson via atm)
HDFS-4468. Use the new StringUtils methods added by HADOOP-9252 and fix
TestHDFSCLI and TestQuota. (szetszwo)
HDFS-4458. In DFSUtil.getNameServiceUris(..), convert default fs URI using
NetUtils.createSocketAddr(..) for being consistent with other addresses.
(Binglin Chang via szetszwo)
BREAKDOWN OF HDFS-3077 SUBTASKS
HDFS-3077. Quorum-based protocol for reading and writing edit logs.
(todd, Brandon Li, and Hari Mankude via todd)
HDFS-3694. Fix getEditLogManifest to fetch httpPort if necessary (todd)
HDFS-3692. Support purgeEditLogs() call to remotely purge logs on JNs
(todd)
HDFS-3693. JNStorage should read its storage info even before a writer
becomes active (todd)
HDFS-3725. Fix QJM startup when individual JNs have gaps (todd)
HDFS-3741. Exhaustive failure injection test for skipped RPCs (todd)
HDFS-3773. TestNNWithQJM fails after HDFS-3741. (atm)
HDFS-3793. Implement genericized format() in QJM (todd)
HDFS-3795. QJM: validate journal dir at startup (todd)
HDFS-3798. Avoid throwing NPE when finalizeSegment() is called on invalid
segment (todd)
HDFS-3799. QJM: handle empty log segments during recovery (todd)
HDFS-3797. QJM: add segment txid as a parameter to journal() RPC (todd)
HDFS-3800. improvements to QJM fault testing (todd)
HDFS-3823. QJM: TestQJMWithFaults fails occasionally because of missed
setting of HTTP port. (todd and atm)
HDFS-3826. QJM: Some trivial logging / exception text improvements. (todd
and atm)
HDFS-3839. QJM: hadoop-daemon.sh should be updated to accept "journalnode"
(eli)
HDFS-3845. Fixes for edge cases in QJM recovery protocol (todd)
HDFS-3877. QJM: Provide defaults for dfs.journalnode.*address (eli)
HDFS-3863. Track last "committed" txid in QJM (todd)
HDFS-3869. Expose non-file journal manager details in web UI (todd)
HDFS-3884. Journal format() should reset cached values (todd)
HDFS-3870. Add metrics to JournalNode (todd)
HDFS-3891. Make selectInputStreams throw IOE instead of RTE (todd)
HDFS-3726. If a logger misses an RPC, don't retry that logger until next
segment (todd)
HDFS-3893. QJM: Make QJM work with security enabled. (atm)
HDFS-3897. QJM: TestBlockToken fails after HDFS-3893. (atm)
HDFS-3898. QJM: enable TCP_NODELAY for IPC (todd)
HDFS-3885. QJM: optimize log sync when JN is lagging behind (todd)
HDFS-3900. QJM: avoid validating log segments on log rolls (todd)
HDFS-3901. QJM: send 'heartbeat' messages to JNs even when they are
out-of-sync (todd)
HDFS-3899. QJM: Add client-side metrics (todd)
HDFS-3914. QJM: acceptRecovery should abort current segment (todd)
HDFS-3915. QJM: Failover fails with auth error in secure cluster (todd)
HDFS-3906. QJM: quorum timeout on failover with large log segment (todd)
HDFS-3840. JournalNodes log JournalNotFormattedException backtrace error
before being formatted (todd)
HDFS-3894. QJM: testRecoverAfterDoubleFailures can be flaky due to IPC
client caching (todd)
HDFS-3926. QJM: Add user documentation for QJM. (atm)
HDFS-3943. QJM: remove currently-unused md5sum field (todd)
HDFS-3950. QJM: misc TODO cleanup, improved log messages, etc. (todd)
HDFS-3955. QJM: Make acceptRecovery() atomic. (todd)
HDFS-3956. QJM: purge temporary files when no longer within retention
period (todd)
HDFS-4004. TestJournalNode#testJournal fails because of test case execution
order (Chao Shi via todd)
HDFS-4017. Unclosed FileInputStream in GetJournalEditServlet
(Chao Shi via todd)
HDFS-4351. In BlockPlacementPolicyDefault.chooseTarget(..), numOfReplicas
needs to be updated when avoiding stale nodes. (Andrew Wang via szetszwo)
HDFS-2908. Add apache license header for StorageReport.java. (Brandon Li
via tgraves)
HDFS-4399. Fix RAT warnings by excluding images sub-dir in docs. (Thomas
Graves via acmurthy)
Release 2.0.2-alpha - 2012-09-07
INCOMPATIBLE CHANGES
HDFS-3446. HostsFileReader silently ignores bad includes/excludes
(Matthew Jacobs via todd)
HDFS-3755. Creating an already-open-for-write file with overwrite=true fails
(todd)
NEW FEATURES
HDFS-744. Support hsync in HDFS. (Lars Hofhansl via szetszwo)
HDFS-3042. Automatic failover support for NameNode HA (todd)
(see dedicated section below for breakdown of subtasks)
HDFS-3518. Add a utility method HdfsUtils.isHealthy(uri) for checking if
the given HDFS is healthy. (szetszwo)
HDFS-3113. httpfs does not support delegation tokens. (tucu)
HDFS-3513. HttpFS should cache filesystems. (tucu)
HDFS-3637. Add support for encrypting the DataTransferProtocol. (atm)
HDFS-3150. Add option for clients to contact DNs via hostname. (eli)
HDFS-2793. Add an admin command to trigger an edit log roll. (todd)
HDFS-3703. Datanodes are marked stale if heartbeat is not received in
configured timeout and are selected as the last location to read from.
(Jing Zhao via suresh)
IMPROVEMENTS
HDFS-3040. TestMulitipleNNDataBlockScanner is misspelled. (Madhukara Phatak
via atm)
HDFS-3390. DFSAdmin should print full stack traces of errors when DEBUG
logging is enabled. (atm)
HDFS-3341. Change minimum RPC versions to respective SNAPSHOTs instead of
final releases. (todd)
HDFS-3369. Rename {get|set|add}INode(..) methods in BlockManager and
BlocksMap to {get|set|add}BlockCollection(..). (John George via szetszwo)
HDFS-3134. harden edit log loader against malformed or malicious input.
(Colin Patrick McCabe via eli)
HDFS-3230. Cleanup DatanodeID creation in the tests. (eli)
HDFS-3401. Cleanup DatanodeDescriptor creation in the tests. (eli)
HDFS-3400. DNs should be able start with jsvc even if security is disabled.
(atm via eli)
HDFS-3404. Make putImage in GetImageServlet infer remote address to fetch
from request. (atm)
HDFS-3335. check for edit log corruption at the end of the log
(Colin Patrick McCabe via todd)
HDFS-3417. Rename BalancerDatanode#getName to getDisplayName to be
consistent with Datanode. (eli)
HDFS-3416. Cleanup DatanodeID and DatanodeRegistration
constructors used by testing. (eli)
HDFS-3419. Cleanup LocatedBlock. (eli)
HDFS-3440. More effectively limit stream memory consumption when reading
corrupt edit logs (Colin Patrick McCabe via todd)
HDFS-3438. BootstrapStandby should not require a rollEdits on active node
(todd)
HDFS-2885. Remove "federation" from the nameservice config options.
(Tsz Wo (Nicholas) Sze via eli)
HDFS-3394. Do not use generic in INodeFile.getLastBlock(): the run-time
ClassCastException check is useless since generic type information is only
available in compile-time. (szetszwo)
HDFS-3454. Balancer unconditionally logs InterruptedException at
INFO level on shutdown if security is enabled. (eli)
HDFS-1013. Miscellaneous improvements to HTML markup for web UIs
(Eugene Koontz via todd)
HDFS-3052. Change INodeFile and INodeFileUnderConstruction to package
private. (szetszwo)
HDFS-3520. Add transfer rate logging to TransferFsImage. (eli)
HDFS-3504. Support configurable retry policy in DFSClient for RPC
connections and RPC calls, and add MultipleLinearRandomRetry, a new retry
policy. (szetszwo)
HDFS-3372. offlineEditsViewer should be able to read a binary
edits file with recovery mode. (Colin Patrick McCabe via eli)
HDFS-3516. Check content-type in WebHdfsFileSystem. (szetszwo)
HDFS-3535. Audit logging should log denied accesses. (Andy Isaacson via eli)
HDFS-3481. Refactor HttpFS handling of JAX-RS query string parameters (tucu)
HDFS-3572. Cleanup code which inits SPNEGO in HttpServer (todd)
HDFS-3475. Make the replication monitor multipliers configurable.
(harsh via eli)
HDFS-3343. Improve metrics for DN read latency (Andrew Wang via todd)
HDFS-3170. Add more useful metrics for write latency (Matthew Jacobs via
todd)
HDFS-3604. Add dfs.webhdfs.enabled to hdfs-default.xml. (eli)
HDFS-2988. Improve error message when storage directory lock fails
(Miomir Boljanovic via harsh)
HDFS-2391. Newly set BalancerBandwidth value is not displayed anywhere.
(harsh)
HDFS-3067. NPE in DFSInputStream.readBuffer if read is repeated on
corrupted block. (Henry Robinson via atm)
HDFS-3555. idle client socket triggers DN ERROR log
(should be INFO or DEBUG). (Andy Isaacson via harsh)
HDFS-3568. fuse_dfs: add support for security. (Colin McCabe via atm)
HDFS-3629. Fix the typo in the error message about inconsistent
storage layout version. (Brandon Li via harsh)
HDFS-3613. GSet prints some INFO level values, which aren't
really very useful to all (Andrew Wang via harsh)
HDFS-3611. NameNode prints unnecessary WARNs about edit log normally skipping
a few bytes. (Colin Patrick McCabe via harsh)
HDFS-3582. Hook daemon process exit for testing. (eli)
HDFS-3641. Move server Util time methods to common and use now
instead of System#currentTimeMillis. (eli)
HDFS-3633. libhdfs: hdfsDelete should pass JNI_FALSE or JNI_TRUE.
(Colin Patrick McCabe via eli)
HDFS-799. libhdfs must call DetachCurrentThread when a thread is destroyed.
(Colin Patrick McCabe via eli)
HDFS-3306. fuse_dfs: don't lock release operations.
(Colin Patrick McCabe via eli)
HDFS-3612. Single namenode image directory config warning can
be improved. (Andy Isaacson via harsh)
HDFS-3606. libhdfs: create self-contained unit test.
(Colin Patrick McCabe via eli)
HDFS-3539. libhdfs code cleanups. (Colin Patrick McCabe via eli)
HDFS-3610. fuse_dfs: Provide a way to use the default (configured) NN URI.
(Colin Patrick McCabe via eli)
HDFS-3663. MiniDFSCluster should capture the code path that led to
the first ExitException. (eli)
HDFS-3659. Add missing @Override to methods across the hadoop-hdfs
project. (Brandon Li via harsh)
HDFS-3537. Move libhdfs and fuse-dfs source to native subdirectories.
(Colin Patrick McCabe via eli)
HDFS-3665. Add a test for renaming across file systems via a symlink. (eli)
HDFS-3666. Plumb more exception messages to terminate. (eli)
HDFS-3673. libhdfs: fix some compiler warnings. (Colin Patrick McCabe via eli)
HDFS-3675. libhdfs: follow documented return codes. (Colin Patrick McCabe via eli)
HDFS-1249. With fuse-dfs, chown which only has owner (or only group)
argument fails with Input/output error. (Colin Patrick McCabe via eli)
HDFS-3583. Convert remaining tests to Junit4. (Andrew Wang via atm)
HDFS-3711. Manually convert remaining tests to JUnit4. (Andrew Wang via atm)
HDFS-3650. Use MutableQuantiles to provide latency histograms for various
operations. (Andrew Wang via atm)
HDFS-3667. Add retry support to WebHdfsFileSystem. (szetszwo)
HDFS-3291. add test that covers HttpFS working w/ a non-HDFS Hadoop
filesystem (tucu)
HDFS-3634. Add self-contained, mavenized fuse_dfs test. (Colin Patrick
McCabe via atm)
HDFS-3190. Simple refactors in existing NN code to assist
QuorumJournalManager extension. (todd)
HDFS-3276. initializeSharedEdits should have a -nonInteractive flag (todd)
HDFS-3765. namenode -initializeSharedEdits should be able to initialize
all shared storages. (Vinay and todd via todd)
HDFS-3723. Add support -h, -help to all the commands. (Jing Zhao via
suresh)
HDFS-3803. Change BlockPoolSliceScanner chatty INFO log to DEBUG.
(Andrew Purtell via suresh)
HDFS-3802. StartupOption.name in HdfsServerConstants should be final.
(Jing Zhao via szetszwo)
HDFS-3796. Speed up edit log tests by avoiding fsync() (todd)
HDFS-2963. Console Output is confusing while executing metasave
(dfsadmin command). (Andrew Wang via eli)
HDFS-3672. Expose disk-location information for blocks to enable better
scheduling. (Andrew Wang via atm)
HDFS-2727. libhdfs should get the default block size from the server.
(Colin Patrick McCabe via eli)
HDFS-3832. Remove protocol methods related to DistributedUpgrade. (suresh)
HDFS-3819. Should check whether invalidate work percentage default value is
not greater than 1.0f. (Jing Zhao via jitendra)
HDFS-3177. Update DFSClient and DataXceiver to handle different checkum
types in file checksum computation. (Kihwal Lee via szetszwo)
HDFS-3844. Add @Override and remove {@inheritdoc} and unnecessary
imports. (Jing Zhao via suresh)
HDFS-3853. Port MiniDFSCluster enableManagedDfsDirsRedundancy
option to branch-2. (Colin Patrick McCabe via eli)
HDFS-3871. Change NameNodeProxies to use RetryUtils. (Arun C Murthy
via szetszwo)
HDFS-3887. Remove redundant chooseTarget methods in BlockPlacementPolicy.
(Jing Zhao via szetszwo)
HDFS-3888. Clean up BlockPlacementPolicyDefault. (Jing Zhao via szetszwo)
HDFS-3907. Allow multiple users for local block readers. (eli)
HDFS-3510. Editlog pre-allocation is performed prior to writing edits
to avoid partial edits case disk out of space. (Colin McCabe via todd)
HDFS-3910. DFSTestUtil#waitReplication should timeout. (eli)
HDFS-3920. libwebdhfs string processing and using strerror consistently
to handle all errors. (Jing Zhao via suresh)
OPTIMIZATIONS
HDFS-2982. Startup performance suffers when there are many edit log
segments. (Colin Patrick McCabe via todd)
HDFS-2834. Add a ByteBuffer-based read API to DFSInputStream.
(Henry Robinson via todd)
HDFS-3110. Use directRead API to reduce the number of buffer copies in
libhdfs (Henry Robinson via todd)
HDFS-3697. Enable fadvise readahead by default. (todd)
HDFS-2421. Improve the concurrency of SerialNumberMap in NameNode.
(Jing Zhao and Weiyan Wang via szetszwo)
HDFS-3866. HttpFS POM should have property where to download tomcat from (zero45 via tucu)
BUG FIXES
HDFS-3385. The last block of INodeFileUnderConstruction is not
necessarily a BlockInfoUnderConstruction, so do not cast it in
FSNamesystem.recoverLeaseInternal(..). (szetszwo)
HDFS-3414. Balancer does not find NameNode if rpc-address or
servicerpc-address are not set in client configs. (atm)
HDFS-3031. Fix complete() and getAdditionalBlock() RPCs to be idempotent
(todd)
HDFS-2759. Pre-allocate HDFS edit log files after writing version number.
(atm)
HDFS-3413. TestFailureToReadEdits timing out. (atm)
HDFS-3422. TestStandbyIsHot timeouts too aggressive (todd)
HDFS-3433. GetImageServlet should allow administrative requestors when
security is enabled. (atm)
HDFS-1153. dfsnodelist.jsp should handle invalid input parameters.
(Ravi Phulari via eli)
HDFS-3434. InvalidProtocolBufferException when visiting DN
browseDirectory.jsp (eli)
HDFS-2800. Fix cancellation of checkpoints in the standby node to be more
reliable. (todd)
HDFS-3391. Fix InvalidateBlocks to compare blocks including their
generation stamps. (todd)
HDFS-3444. hdfs groups command doesn't work with security enabled. (atm)
HDFS-2717. BookKeeper Journal output stream doesn't check addComplete rc.
(Ivan Kelly via umamahesh)
HDFS-3415. Make sure all layout versions are the same for all storage
directories in the Namenode. (Brandon Li via szetszwo)
HDFS-3436. In DataNode.transferReplicaForPipelineRecovery(..), it should
use the stored generation stamp to check if the block is valid. (Vinay
via szetszwo)
HDFS-3460. HttpFS proxyuser validation with Kerberos ON uses full
principal name. (tucu)
HDFS-3058. HA: Bring BookKeeperJournalManager up to date with HA changes.
(Ivan Kelly via umamahesh)
HDFS-3368. Missing blocks due to bad DataNodes coming up and down. (shv)
HDFS-3452. BKJM:Switch from standby to active fails and NN gets shut down
due to delay in clearing of lock. (umamahesh)
HDFS-3398. Client will not retry when primaryDN is down once it's just got pipeline.
(Amith D K via umamahesh)
HDFS-3474. Cleanup Exception handling in BookKeeper journal manager.
(Ivan Kelly via umamahesh)
HDFS-3468. Make BKJM-ZK session timeout configurable. (umamahesh)
HDFS-3423. BKJM: NN startup is failing, when tries to recoverUnfinalizedSegments()
a bad inProgress_ ZNodes. (Ivan Kelly and Uma via umamahesh)
HDFS-3441. Race condition between rolling logs at active NN and purging at standby.
(Rakesh R via umamahesh)
HDFS-3484. hdfs fsck doesn't work if NN HTTP address is set to
0.0.0.0 even if NN RPC address is configured. (atm via eli)
HDFS-3486. offlineimageviewer can't read fsimage files that contain
persistent delegation tokens. (Colin Patrick McCabe via eli)
HDFS-3487. offlineimageviewer should give byte offset information
when it encounters an exception. (Colin Patrick McCabe via eli)
HDFS-3442. Incorrect count for Missing Replicas in FSCK report. (Andrew
Wang via atm)
HDFS-2025. Go Back to File View link is not working in tail.jsp.
(Ashish and Sravan via umamahesh)
HDFS-3501. Checkpointing with security enabled will stop working
after ticket lifetime expires. (atm via eli)
HDFS-3266. DFSTestUtil#waitCorruptReplicas doesn't sleep between checks.
(Madhukara Phatak via atm)
HDFS-3505. DirectoryScanner does not join all threads in shutdown.
(Colin Patrick McCabe via eli)
HDFS-3485. DataTransferThrottler will over-throttle when currentTimeMillis
jumps (Andy Isaacson via todd)
HDFS-2914. HA: Standby should not enter safemode when resources are low.
(Vinay via atm)
HDFS-3235. MiniDFSClusterManager doesn't correctly support -format option.
(Henry Robinson via atm)
HDFS-3514. Add missing TestParallelLocalRead. (Henry Robinson via atm)
HDFS-3243. TestParallelRead timing out on jenkins. (Henry Robinson via todd)
HDFS-3490. DatanodeWebHdfsMethods throws NullPointerException if
NamenodeRpcAddressParam is not set. (szetszwo)
HDFS-2797. Fix misuses of InputStream#skip in the edit log code.
(Colin Patrick McCabe via eli)
HDFS-3517. TestStartup should bind ephemeral ports. (eli)
HDFS-3522. If a namenode is in safemode, it should throw SafeModeException
when getBlockLocations has zero locations. (Brandon Li via szetszwo)
HDFS-3408. BKJM : Namenode format fails, if there is no BK root. (Rakesh R via umamahesh)
HDFS-3389. Document the BKJM usage in Namenode HA. (umamahesh and Ivan Kelly via umamahesh)
HDFS-3531. EditLogFileOutputStream#preallocate should check for
incomplete writes. (Colin Patrick McCabe via eli)
HDFS-766. Error message not clear for set space quota out of boundary
values. (Jon Zuanich via atm)
HDFS-3480. Multiple SLF4J binding warning. (Vinay via eli)
HDFS-3524. Update TestFileLengthOnClusterRestart for HDFS-3522. (Brandon
Li via szetszwo)
HDFS-3559. DFSTestUtil: use Builder class to construct DFSTestUtil
instances. (Colin Patrick McCabe via atm)
HDFS-3551. WebHDFS CREATE should use client location for HTTP redirection.
(szetszwo)
HDFS-3157. Fix a bug in the case that the generation stamps of the stored
block in a namenode and the reported block from a datanode do not match.
(Ashish Singhi via szetszwo)
HDFS-3575. HttpFS does not log Exception Stacktraces (brocknoland via tucu)
HDFS-3574. Fix small race and do some cleanup in GetImageServlet (todd)
HDFS-3581. FSPermissionChecker#checkPermission sticky bit check
missing range check. (eli)
HDFS-3541. Deadlock between recovery, xceiver and packet responder.
(Vinay via umamahesh)
HDFS-3428. Move DelegationTokenRenewer to common (tucu)
HDFS-3491. HttpFs does not set permissions correctly (tucu)
HDFS-3580. incompatible types; no instance(s) of type variable(s) V exist
so that V conforms to boolean compiling HttpFSServer.java with OpenJDK
(adi2 via tucu)
HDFS-3603. Decouple TestHDFSTrash from TestTrash. (Jason Lowe via eli)
HDFS-711. hdfsUtime does not handle atime = 0 or mtime = 0 correctly.
(Colin Patrick McCabe via eli)
HDFS-3548. NamenodeFsck.copyBlock fails to create a Block Reader.
(Colin Patrick McCabe via eli)
HDFS-3615. Two BlockTokenSecretManager findbugs warnings. (atm)
HDFS-470. libhdfs should handle 0-length reads from FSInputStream
correctly. (Colin Patrick McCabe via eli)
HDFS-3492. fix some misuses of InputStream#skip.
(Colin Patrick McCabe via eli)
HDFS-3609. libhdfs: don't force the URI to look like hdfs://hostname:port.
(Colin Patrick McCabe via eli)
HDFS-2966 TestNameNodeMetrics tests can fail under load. (stevel)
HDFS-3605. Block mistakenly marked corrupt during edit log catchup
phase of failover. (todd and Brahma Reddy Battula via todd)
HDFS-3690. BlockPlacementPolicyDefault incorrectly casts LOG. (eli)
HDFS-3597. SNN fails to start after DFS upgrade. (Andy Isaacson via todd)
HDFS-3608. fuse_dfs: detect changes in UID ticket cache. (Colin Patrick
McCabe via atm)
HDFS-3709. TestStartup tests still binding to the ephemeral port. (eli)
HDFS-3720. hdfs.h must get packaged. (Colin Patrick McCabe via atm)
HDFS-3626. Creating file with invalid path can corrupt edit log (todd)
HDFS-3679. fuse_dfs notrash option sets usetrash. (Conrad Meyer via suresh)
HDFS-3732. fuse_dfs: incorrect configuration value checked for connection
expiry timer period. (Colin Patrick McCabe via atm)
HDFS-3738. TestDFSClientRetries#testFailuresArePerOperation sets incorrect
timeout config. (atm)
HDFS-3756. DelegationTokenFetcher creates 2 HTTP connections, the second
one not properly configured. (tucu)
HDFS-3579. libhdfs: fix exception handling. (Colin Patrick McCabe via atm)
HDFS-3754. BlockSender doesn't shutdown ReadaheadPool threads. (eli)
HDFS-3760. primitiveCreate is a write, not a read. (Andy Isaacson via atm)
HDFS-3710. libhdfs misuses O_RDONLY/WRONLY/RDWR. (Andy Isaacson via atm)
HDFS-3721. hsync support broke wire compatibility. (todd and atm)
HDFS-3758. TestFuseDFS test failing. (Colin Patrick McCabe via eli)
HDFS-2330. In NNStorage.java, IOExceptions of stream closures can mask
root exceptions. (umamahesh via todd)
HDFS-3790. test_fuse_dfs.c doesn't compile on centos 5. (Colin Patrick
McCabe via atm)
HDFS-3658. Fix bugs in TestDFSClientRetries and add more tests. (szetszwo)
HDFS-3794. WebHDFS OPEN returns the incorrect Content-Length in the HTTP
header when offset is specified and length is omitted.
(Ravi Prakash via szetszwo)
HDFS-3048. Small race in BlockManager#close. (Andy Isaacson via eli)
HDFS-3194. DataNode block scanner is running too frequently.
(Andy Isaacson via eli)
HDFS-3808. fuse_dfs: postpone libhdfs intialization until after fork.
(Colin Patrick McCabe via atm)
HDFS-3788. ByteRangeInputStream should not expect HTTP Content-Length header
when chunked transfer-encoding is used. (szetszwo)
HDFS-3816. Invalidate work percentage default value should be 0.32f
instead of 32. (Jing Zhao via suresh)
HDFS-3707. TestFSInputChecker: improper use of skip.
(Colin Patrick McCabe via eli)
HDFS-3830. test_libhdfs_threaded: use forceNewInstance.
(Colin Patrick McCabe via eli)
HDFS-3835. Long-lived 2NN cannot perform a checkpoint if security is
enabled and the NN restarts with outstanding delegation tokens. (atm)
HDFS-3715. Fix TestFileCreation#testFileCreationNamenodeRestart.
(Andrew Whang via eli)
HDFS-3683. Edit log replay progress indicator shows >100% complete. (Plamen
Jeliazkov via atm)
HDFS-3731. Release upgrade must handle blocks being written from 1.0.
(Colin Patrick McCabe via eli)
HDFS-3856. TestHDFSServerPorts failure is causing surefire fork failure.
(eli)
HDFS-3860. HeartbeatManager#Monitor may wrongly hold the writelock of
namesystem. (Jing Zhao via atm)
HDFS-3849. When re-loading the FSImage, we should clear the existing
genStamp and leases. (Colin Patrick McCabe via atm)
HDFS-3864. NN does not update internal file mtime for OP_CLOSE when reading
from the edit log. (atm)
HDFS-3837. Fix DataNode.recoverBlock findbugs warning. (eli)
HDFS-3733. Audit logs should include WebHDFS access. (Andy Isaacson via
eli)
HDFS-2686. Remove DistributedUpgrade related code. (suresh)
HDFS-3833. TestDFSShell fails on windows due to concurrent file
read/write. (Brandon Li via suresh)
HDFS-3466. Get HTTP kerberos principal from the web authentication keytab.
(omalley)
HDFS-3879. Fix findbugs warning in TransferFsImage on branch-2. (eli)
HDFS-3469. start-dfs.sh will start zkfc, but stop-dfs.sh will not stop zkfc similarly.
(Vinay via umamahesh)
HDFS-1490. TransferFSImage should timeout (Dmytro Molkov and Vinay via todd)
HDFS-3828. Block Scanner rescans blocks too frequently.
(Andy Isaacson via eli)
HDFS-3895. hadoop-client must include commons-cli (tucu)
HDFS-2757. Cannot read a local block that's being written to when
using the local read short circuit. (Jean-Daniel Cryans via eli)
HDFS-3664. BlockManager race when stopping active services.
(Colin Patrick McCabe via eli)
HDFS-3928. MiniDFSCluster should reset the first ExitException on shutdown. (eli)
HDFS-3938. remove current limitations from HttpFS docs. (tucu)
HDFS-3944. Httpfs resolveAuthority() is not resolving host correctly. (tucu)
HDFS-3972. Trash emptier fails in secure HA cluster. (todd via eli)
HDFS-4443. Remove a trailing '`' character from the HTML code generated by
NamenodeJspHelper.generateNodeData(..). (Christian Rohling via szetszwo)
BREAKDOWN OF HDFS-3042 SUBTASKS
HDFS-2185. HDFS portion of ZK-based FailoverController (todd)
HDFS-3200. Scope all ZKFC configurations by nameservice (todd)
HDFS-3223. add zkfc to hadoop-daemon.sh script (todd)
HDFS-3261. TestHASafeMode fails on HDFS-3042 branch (todd)
HDFS-3159. Document NN auto-failover setup and configuration (todd)
HDFS-3412. Fix findbugs warnings in auto-HA branch (todd)
HDFS-3432. TestDFSZKFailoverController tries to fail over too early (todd)
HDFS-3902. TestDatanodeBlockScanner#testBlockCorruptionPolicy is broken.
(Andy Isaacson via eli)
Release 2.0.0-alpha - 05-23-2012
INCOMPATIBLE CHANGES
HDFS-2676. Remove Avro RPC. (suresh)
HDFS-2303. Unbundle jsvc. (Roman Shaposhnik and Mingjie Lai via eli)
HDFS-3137. Bump LAST_UPGRADABLE_LAYOUT_VERSION to -16. (eli)
HDFS-3138. Move DatanodeInfo#ipcPort to DatanodeID. (eli)
HDFS-3164. Move DatanodeInfo#hostName to DatanodeID. (eli)
NEW FEATURES
HDFS-2978. The NameNode should expose name dir statuses via JMX. (atm)
HDFS-395. DFS Scalability: Incremental block reports. (Tomasz Nykiel
via hairong)
HDFS-2517. Add protobuf service for JounralProtocol. (suresh)
HDFS-2518. Add protobuf service for NamenodeProtocol. (suresh)
HDFS-2520. Add protobuf service for InterDatanodeProtocol. (suresh)
HDFS-2519. Add protobuf service for DatanodeProtocol. (suresh)
HDFS-2581. Implement protobuf service for JournalProtocol. (suresh)
HDFS-2618. Implement protobuf service for NamenodeProtocol. (suresh)
HDFS-2629. Implement protobuf service for InterDatanodeProtocol. (suresh)
HDFS-2636. Implement protobuf service for ClientDatanodeProtocol. (suresh)
HDFS-2642. Protobuf translators for DatanodeProtocol. (jitendra)
HDFS-2647. Used protobuf based RPC for InterDatanodeProtocol,
ClientDatanodeProtocol, JournalProtocol, NamenodeProtocol. (suresh)
HDFS-2661. Enable protobuf RPC for DatanodeProtocol. (jitendra)
HDFS-2697. Move RefreshAuthPolicy, RefreshUserMappings, GetUserMappings
protocol to protocol buffers. (jitendra)
HDFS-2880. Protobuf changes in DatanodeProtocol to add multiple storages.
(suresh)
HDFS-2899. Service protocol changes in DatanodeProtocol to add multiple
storages. (suresh)
HDFS-2430. The number of failed or low-resource volumes the NN can tolerate
should be configurable. (atm)
HDFS-1623. High Availability Framework for HDFS NN. Contributed by Todd
Lipcon, Aaron T. Myers, Eli Collins, Uma Maheswara Rao G, Bikas Saha,
Suresh Srinivas, Jitendra Nath Pandey, Hari Mankude, Brandon Li, Sanjay
Radia, Mingjie Lai, and Gregory Chanan
HDFS-2941. Add an administrative command to download a copy of the fsimage
from the NN. (atm)
HDFS-2413. Add an API DistributedFileSystem.isInSafeMode() and change
DistributedFileSystem to @InterfaceAudience.LimitedPrivate.
(harsh via szetszwo)
HDFS-3167. CLI-based driver for MiniDFSCluster. (Henry Robinson via atm)
HDFS-3148. The client should be able to use multiple local interfaces
for data transfer. (eli)
HDFS-3000. Add a public API for setting quotas. (atm)
HDFS-3102. Add CLI tool to initialize the shared-edits dir. (atm)
HDFS-3004. Implement Recovery Mode. (Colin Patrick McCabe via eli)
HDFS-3282. Add HdfsDataInputStream as a public API. (umamahesh)
HDFS-3298. Add HdfsDataOutputStream as a public API. (szetszwo)
HDFS-234. Integration with BookKeeper logging system. (Ivan Kelly
via umamahesh)
IMPROVEMENTS
HDFS-2018. Move all journal stream management code into one place.
(Ivan Kelly via jitendra)
HDFS-2223. Untangle depencencies between NN components (todd)
HDFS-2351. Change Namenode and Datanode to register each of their protocols
seperately (sanjay)
HDFS-2337. DFSClient shouldn't keep multiple RPC proxy references (atm)
HDFS-2181. Separate HDFS Client wire protocol data types (sanjay)
HDFS-2459. Separate datatypes for Journal Protocol. (suresh)
HDFS-2480. Separate datatypes for NamenodeProtocol. (suresh)
HDFS-2489. Move Finalize and Register to separate file out of
DatanodeCommand.java. (suresh)
HDFS-2488. Separate datatypes for InterDatanodeProtocol. (suresh)
HDFS-2496. Separate datatypes for DatanodeProtocol. (suresh)
HDFS-2479. HDFS Client Data Types in Protocol Buffers (sanjay)
HADOOP-7862. Hdfs changes to work with HADOOP-7862: Move the support for
multiple protocols to lower layer so that Writable, PB and Avro can all
use it. (sanjay)
HDFS-2597. ClientNameNodeProtocol in Protocol Buffers. (sanjay)
HDFS-2651. ClientNameNodeProtocol Translators for Protocol Buffers. (sanjay)
HDFS-2650. Replace @inheritDoc with @Override. (Hari Mankude via suresh).
HDFS-2669. Enable protobuf rpc for ClientNamenodeProtocol. (sanjay)
HDFS-2801. Provide a method in client side translators to check for a
methods supported in underlying protocol. (jitendra)
HDFS-2895. Remove Writable wire protocol types and translators to
complete transition to protocol buffers. (suresh)
HDFS-2992. Edit log failure trace should include transaction ID of
error. (Colin Patrick McCabe via eli)
HDFS-3030. Remove getProtocolVersion and getProtocolSignature from
translators. (jitendra)
HDFS-2158. Add JournalSet to manage the set of journals. (jitendra)
HDFS-2334. Add Closeable to JournalManager. (Ivan Kelly via jitendra)
HDFS-1580. Add interface for generic Write Ahead Logging mechanisms.
(Ivan Kelly via jitendra)
HDFS-3060. Bump TestDistributedUpgrade#testDistributedUpgrade timeout (eli)
HDFS-2410. Further cleanup of hardcoded configuration keys and values.
(suresh)
HDFS-2878. Fix TestBlockRecovery and move it back into main test directory.
(todd)
HDFS-3003. Remove getHostPortString() from NameNode, replace it with
NetUtils.getHostPortString(). (Brandon Li via atm)
HDFS-3014. FSEditLogOp and its subclasses should have toString() method.
(Sho Shimauchi via atm)
HDFS-3021. Use generic type to declare FSDatasetInterface. (szetszwo)
HDFS-3056. Add a new interface RollingLogs for DataBlockScanner logging.
(szetszwo)
HDFS-2731. Add command to bootstrap the Standby Node's name directories
from the Active NameNode. (todd)
HDFS-3082. Clean up FSDatasetInterface and change DataNode.data to package
private. (szetszwo)
HDFS-3057. httpfs and hdfs launcher scripts should honor CATALINA_HOME
and HADOOP_LIBEXEC_DIR (rvs via tucu)
HDFS-3088. Move FSDatasetInterface inner classes to a package. (szetszwo)
HDFS-3111. Missing license headers in trunk. (umamahesh)
HDFS-3091. Update the usage limitations of ReplaceDatanodeOnFailure policy in
the config description for the smaller clusters. (szetszwo via umamahesh)
HDFS-3105. Add DatanodeStorage information to block recovery. (szetszwo)
HDFS-3086. Change Datanode not to send storage list in registration.
(szetszwo)
HDFS-309. FSEditLog should log progress during replay. (Sho Shimauchi
via todd)
HDFS-3044. fsck move should be non-destructive by default.
(Colin Patrick McCabe via eli)
HDFS-3071. haadmin failover command does not provide enough detail when
target NN is not ready to be active. (todd)
HDFS-3089. Move FSDatasetInterface and the related classes to a package.
(szetszwo)
HDFS-3129. NetworkTopology: add test that getLeaf should check for
invalid topologies. (Colin Patrick McCabe via eli)
HDFS-3155. Clean up FSDataset implemenation related code. (szetszwo)
HDFS-3158. LiveNodes member of NameNodeMXBean should list non-DFS used
space and capacity per DN. (atm)
HDFS-3172. dfs.upgrade.permission is dead code. (eli)
HDFS-3171. The DatanodeID "name" field is overloaded. (eli)
HDFS-3144. Refactor DatanodeID#getName by use. (eli)
HDFS-3130. Move fsdataset implementation to a package. (szetszwo)
HDFS-3120. Enable hsync and hflush by default. (eli)
HDFS-3187. Upgrade guava to 11.0.2 (todd)
HDFS-3168. Remove unnecessary "throw IOException" and change fields to
final in FSNamesystem and BlockManager. (szetszwo)
HDFS-2564. Cleanup unnecessary exceptions thrown and unnecessary casts.
(Hari Mankude via eli)
HDFS-3084. FenceMethod.tryFence() and ShellCommandFencer should pass
namenodeId as well as host:port (todd)
HDFS-3050. rework OEV to share more code with the NameNode.
(Colin Patrick McCabe via eli)
HDFS-3204. Minor modification to JournalProtocol.proto to make
it generic. (suresh)
HDFS-3226. Allow GetConf tool to print arbitrary keys (todd)
HDFS-3240. Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG
(todd)
HDFS-3238. ServerCommand and friends don't need to be writables. (eli)
HDFS-3094. add -nonInteractive and -force option to namenode -format
command (Arpit Gupta via todd)
HDFS-3244. Remove dead writable code from hdfs/protocol. (eli)
HDFS-3247. Improve bootstrapStandby behavior when original NN is not active
(todd)
HDFS-3249. Use ToolRunner.confirmPrompt in NameNode (todd)
HDFS-3179. Improve the exception message thrown by DataStreamer when
it failed to add a datanode. (szetszwo)
HDFS-2983. Relax the build version check to permit rolling upgrades within a release. (atm)
HDFS-3259. NameNode#initializeSharedEdits should populate shared edits dir
with edit log segments. (atm)
HDFS-2708. Stats for the # of blocks per DN. (atm)
HDFS-3279. Move the FSEditLog constructor with @VisibleForTesting to
TestEditLog. (Arpit Gupta via szetszwo)
HDFS-3294. Fix code indentation in NamenodeWebHdfsMethods and
DatanodeWebHdfsMethods. (szetszwo)
HDFS-3263. HttpFS should read HDFS config from Hadoop site.xml files (tucu)
HDFS-3206. Miscellaneous xml cleanups for OEV.
(Colin Patrick McCabe via eli)
HDFS-3169. TestFsck should test multiple -move operations in a row.
(Colin Patrick McCabe via eli)
HDFS-3258. Test for HADOOP-8144 (pseudoSortByDistance in
NetworkTopology for first rack local node). (Junping Du via eli)
HDFS-3322. Use HdfsDataInputStream and HdfsDataOutputStream in Hdfs.
(szetszwo)
HDFS-3339. Change INode to package private. (John George via szetszwo)
HDFS-3303. Remove Writable implementation from RemoteEditLogManifest.
(Brandon Li via szetszwo)
HDFS-2617. Replaced Kerberized SSL for image transfer and fsck
with SPNEGO-based solution. (jghoman, omalley, tucu, and atm via eli)
HDFS-3365. Enable users to disable socket caching in DFS client
configuration (todd)
HDFS-3375. Put client name in DataXceiver thread name for readBlock
and keepalive (todd)
HDFS-3363. Define BlockCollection and MutableBlockCollection interfaces
so that INodeFile and INodeFileUnderConstruction do not have to be used in
block management. (John George via szetszwo)
HDFS-3211. Add fence(..) and replace NamenodeRegistration with JournalInfo
and epoch in JournalProtocol. (suresh via szetszwo)
HADOOP-8285 HDFS changes for Use ProtoBuf for RpcPayLoadHeader (sanjay radia)
HDFS-3418. Rename BlockWithLocationsProto datanodeIDs field to storageIDs.
(eli)
OPTIMIZATIONS
HDFS-2477. Optimize computing the diff between a block report and the
namenode state. (Tomasz Nykiel via hairong)
HDFS-2495. Increase granularity of write operations in ReplicationMonitor
thus reducing contention for write lock. (Tomasz Nykiel via hairong)
HDFS-2476. More CPU efficient data structure for under-replicated,
over-replicated, and invalidated blocks. (Tomasz Nykiel via todd)
HDFS-3036. Remove unused method DFSUtil#isDefaultNamenodeAddress. (atm)
HDFS-3378. Remove DFS_NAMENODE_SECONDARY_HTTPS_PORT_KEY and DEFAULT. (eli)
BUG FIXES
HDFS-2481. Unknown protocol: org.apache.hadoop.hdfs.protocol.ClientProtocol.
(sanjay)
HDFS-2497. Fix TestBackupNode failure. (suresh)
HDFS-2499. RPC client is created incorrectly introduced in HDFS-2459.
(suresh)
HDFS-2526. (Client)NamenodeProtocolTranslatorR23 do not need to keep a
reference to rpcProxyWithoutRetry (atm)
HDFS-2532. TestDfsOverAvroRpc timing out in trunk (Uma Maheswara Rao G
via todd)
HDFS-2666. Fix TestBackupNode failure. (suresh)
HDFS-2663. Optional protobuf parameters are not handled correctly. (suresh)
HDFS-2694. Removal of Avro broke non-PB NN services. (atm)
HDFS-2687. Tests failing with ClassCastException post protobuf RPC
changes. (suresh)
HDFS-2700. Fix failing TestDataNodeMultipleRegistrations in trunk
(Uma Maheswara Rao G via todd)
HDFS-2739. SecondaryNameNode doesn't start up. (jitendra)
HDFS-2768. BackupNode stop can not close proxy connections because
it is not a proxy instance. (Uma Maheswara Rao G via eli)
HDFS-2968. Protocol translator for BlockRecoveryCommand broken when
multiple blocks need recovery. (todd)
HDFS-3020. Fix editlog to automatically sync when buffer is full. (todd)
HDFS-3038. Add FSEditLog.metrics to findbugs exclude list. (todd via atm)
HDFS-2188. Make FSEditLog create its journals from a list of URIs rather
than NNStorage. (Ivan Kelly via jitendra)
HDFS-1765. Block Replication should respect under-replication
block priority. (Uma Maheswara Rao G via eli)
HDFS-2285. BackupNode should reject requests to modify namespace.
(shv and Uma Maheswara Rao)
HDFS-2764. TestBackupNode is racy. (atm)
HDFS-3093. Fix bug where namenode -format interpreted the -force flag in
reverse. (todd)
HDFS-3005. FSVolume.decDfsUsed(..) should be synchronized. (szetszwo)
HDFS-3099. SecondaryNameNode does not properly initialize metrics system.
(atm)
HDFS-3062. Fix bug which prevented MR job submission from creating
delegation tokens on an HA cluster. (Mingjie Lai via todd)
HDFS-3083. Cannot run an MR job with HA and security enabled when
second-listed NN active. (atm)
HDFS-3100. In BlockSender, throw an exception when it needs to verify
checksum but the meta data does not exist. (Brandon Li via szetszwo)
HDFS-3132. Fix findbugs warning on HDFS trunk. (todd)
HDFS-3156. TestDFSHAAdmin is failing post HADOOP-8202. (atm)
HDFS-3143. TestGetBlocks.testGetBlocks is failing. (Arpit Gupta via atm)
HDFS-3142. TestHDFSCLI.testAll is failing. (Brandon Li via atm)
HDFS-3070. HDFS balancer doesn't ensure that hdfs-site.xml is loaded. (atm)
HDFS-2995. start-dfs.sh should only start the 2NN for namenodes
with dfs.namenode.secondary.http-address configured. (eli)
HDFS-3174. Fix assert in TestPendingDataNodeMessages. (eli)
HDFS-3199. TestValidateConfigurationSettings is failing. (todd via eli)
HDFS-3202. NamespaceInfo PB translation drops build version. (atm)
HDFS-3109. Remove hsqldf exclusions from pom.xml. (Ravi Prakash
via suresh)
HDFS-3210. JsonUtil#toJsonMap for for a DatanodeInfo should use
"ipAddr" instead of "name". (eli)
HDFS-3208. Bogus entries in hosts files are incorrectly displayed
in the report. (eli)
HDFS-3136. Remove SLF4J dependency as HDFS does not need it to fix
unnecessary warnings. (Jason Lowe via suresh)
HDFS-3214. InterDatanodeProtocolServerSideTranslatorPB doesn't handle
null response from initReplicaRecovery (todd)
HDFS-3119. Overreplicated block is not deleted even after the replication
factor is reduced after sync follwed by closing that file. (Ashish Singhi
via umamahesh)
HDFS-3234. Accidentally left log message in GetConf after HDFS-3226 (todd)
HDFS-3236. NameNode does not initialize generic conf keys when started
with -initializeSharedEditsDir (atm)
HDFS-3248. bootstrapStandby repeated twice in hdfs namenode usage message
(Colin Patrick McCabe via todd)
HDFS-2696. Fix the fuse-fds build. (Bruno Mahé via eli)
HDFS-3254. Branch-2 build broken due to wrong version number in
fuse-dfs' pom.xml. (Anupam Seth via eli)
HDFS-3260. TestDatanodeRegistration should set minimum DN version in
addition to minimum NN version. (atm)
HDFS-3255. HA DFS returns wrong token service (Daryn Sharp via todd)
HDFS-3256. HDFS considers blocks under-replicated if topology script is
configured with only 1 rack. (atm)
HDFS-2799. Trim fs.checkpoint.dir values. (Amith D K via eli)
HDFS-2765. TestNameEditsConfigs is incorrectly swallowing IOE. (atm)
HDFS-3280. DFSOutputStream.sync should not be synchronized (todd)
HDFS-3268. FileContext API mishandles token service and incompatible with
HA (Daryn Sharp via todd)
HDFS-3284. bootstrapStandby fails in secure cluster (todd)
HDFS-3165. HDFS Balancer scripts are refering to wrong path of
hadoop-daemon.sh (Amith D K via eli)
HDFS-891. DataNode no longer needs to check for dfs.network.script.
(harsh via eli)
HDFS-3305. GetImageServlet should consider SBN a valid requestor in a
secure HA setup. (atm)
HDFS-3314. HttpFS operation for getHomeDirectory is incorrect. (tucu)
HDFS-3319. Change DFSOutputStream to not to start a thread in constructors.
(szetszwo)
HDFS-3222. DFSInputStream#openInfo should not silently get the length as 0
when locations length is zero for last partial block. (umamahesh)
HDFS-3181. Fix a test case in TestLeaseRecovery2. (szetszwo)
HDFS-3309. HttpFS (Hoop) chmod not supporting octal and sticky bit
permissions. (tucu)
HDFS-3326. Append enabled log message uses the wrong variable.
(Matthew Jacobs via eli)
HDFS-3275. Skip format for non-file based directories.
(Amith D K via umamahesh)
HDFS-3286. When the threshold value for balancer is zero, unexpected output is displayed.
(Ashish Singhi via umamahesh)
HDFS-3336. hdfs launcher script will be better off not special casing
namenode command with regards to hadoop.security.logger (rvs via tucu)
HDFS-3330. If GetImageServlet throws an Error or RTE, response should not
have HTTP "OK" status. (todd)
HDFS-3351. NameNode#initializeGenericKeys should always set fs.defaultFS
regardless of whether HA or Federation is enabled. (atm)
HDFS-3332. NullPointerException in DN when directoryscanner is trying to
report bad blocks. (Amith D K via umamahesh)
HDFS-3359. DFSClient.close should close cached sockets. (todd)
HDFS-3350. In INode, add final to compareTo(..), equals(..) and hashCode(),
and remove synchronized from updatePermissionStatus(..). (szetszwo)
HDFS-3357. DataXceiver reads from client socket with incorrect/no timeout
(todd)
HDFS-3376. DFSClient fails to make connection to DN if there are many
unusable cached sockets (todd)
HDFS-3328. NPE in DataNode.getIpcPort. (eli)
HDFS-3396. FUSE build fails on Ubuntu 12.04. (Colin Patrick McCabe via eli)
HDFS-3395. NN doesn't start with HA+security enabled and HTTP address set to 0.0.0.0. (atm)
HDFS-3026. HA: Handle failure during HA state transition. (atm)
HDFS-860. fuse-dfs truncate behavior causes issues with scp.
(Brian Bockelman via eli)
BREAKDOWN OF HDFS-1623 SUBTASKS
HDFS-2179. Add fencing framework and mechanisms for NameNode HA. (todd)
HDFS-1974. Introduce active and standy states to the namenode. (suresh)
HDFS-2407. getServerDefaults and getStats don't check operation category (atm)
HDFS-1973. HA: HDFS clients must handle namenode failover and switch over to
the new active namenode. (atm)
HDFS-2301. Start/stop appropriate namenode services when transition to active
and standby states. (suresh)
HDFS-2231. Configuration changes for HA namenode. (suresh)
HDFS-2418. Change ConfiguredFailoverProxyProvider to take advantage of
HDFS-2231. (atm)
HDFS-2393. Mark appropriate methods of ClientProtocol with the idempotent
annotation. (atm)
HDFS-2523. Small NN fixes to include HAServiceProtocol and prevent NPE on
shutdown. (todd)
HDFS-2577. NN fails to start since it tries to start secret manager in
safemode. (todd)
HDFS-2582. Scope dfs.ha.namenodes config by nameservice (todd)
HDFS-2591. MiniDFSCluster support to mix and match federation with HA (todd)
HDFS-1975. Support for sharing the namenode state from active to standby.
(jitendra, atm, todd)
HDFS-1971. Send block report from datanode to both active and standby
namenodes. (sanjay, todd via suresh)
HDFS-2616. Change DatanodeProtocol#sendHeartbeat() to return HeartbeatResponse.
(suresh)
HDFS-2622. Fix TestDFSUpgrade in HA branch. (todd)
HDFS-2612. Handle refreshNameNodes in federated HA clusters (todd)
HDFS-2623. Add test case for hot standby capability (todd)
HDFS-2626. BPOfferService.verifyAndSetNamespaceInfo needs to be synchronized
(todd)
HDFS-2624. ConfiguredFailoverProxyProvider doesn't correctly stop
ProtocolTranslators (todd)
HDFS-2625. TestDfsOverAvroRpc failing after introduction of HeartbeatResponse
type (todd)
HDFS-2627. Determine DN's view of which NN is active based on heartbeat
responses (todd)
HDFS-2634. Standby needs to ingest latest edit logs before transitioning to
active (todd)
HDFS-2671. NN should throw StandbyException in response to RPCs in STANDBY
state (todd)
HDFS-2680. DFSClient should construct failover proxy with exponential backoff
(todd)
HDFS-2683. Authority-based lookup of proxy provider fails if path becomes
canonicalized (todd)
HDFS-2689. HA: BookKeeperEditLogInputStream doesn't implement isInProgress()
(atm)
HDFS-2602. NN should log newly-allocated blocks without losing BlockInfo (atm)
HDFS-2667. Fix transition from active to standby (todd)
HDFS-2684. Fix up some failing unit tests on HA branch (todd)
HDFS-2679. Add interface to query current state to HAServiceProtocol (eli via
todd)
HDFS-2677. Web UI should indicate the NN state. (eli via todd)
HDFS-2678. When a FailoverProxyProvider is used, DFSClient should not retry
connection ten times before failing over (atm via todd)
HDFS-2682. When a FailoverProxyProvider is used, Client should not retry for 45
times if it is timing out to connect to server. (Uma Maheswara Rao G via todd)
HDFS-2693. Fix synchronization issues around state transition (todd)
HDFS-1972. Fencing mechanism for block invalidations and replications (todd)
HDFS-2714. Fix test cases which use standalone FSNamesystems (todd)
HDFS-2692. Fix bugs related to failover from/into safe mode. (todd)
HDFS-2716. Configuration needs to allow different dfs.http.addresses for each
HA NN (todd)
HDFS-2720. Fix MiniDFSCluster HA support to work properly on Windows. (Uma
Maheswara Rao G via todd)
HDFS-2291. Allow the StandbyNode to make checkpoints in an HA setup. (todd)
HDFS-2709. Appropriately handle error conditions in EditLogTailer (atm via
todd)
HDFS-2730. Refactor shared HA-related test code into HATestUtil class (todd)
HDFS-2762. Fix TestCheckpoint timing out on HA branch. (Uma Maheswara Rao G via
todd)
HDFS-2724. NN web UI can throw NPE after startup, before standby state is
entered. (todd)
HDFS-2753. Fix standby getting stuck in safemode when blocks are written while
SBN is down. (Hari Mankude and todd via todd)
HDFS-2773. Reading edit logs from an earlier version should not leave blocks in
under-construction state. (todd)
HDFS-2775. Fix TestStandbyCheckpoints.testBothNodesInStandbyState failing
intermittently. (todd)
HDFS-2766. Test for case where standby partially reads log and then performs
checkpoint. (atm)
HDFS-2738. FSEditLog.selectinputStreams is reading through in-progress streams
even when non-in-progress are requested. (atm)
HDFS-2789. TestHAAdmin.testFailover is failing (eli)
HDFS-2747. Entering safe mode after starting SBN can NPE. (Uma Maheswara Rao G
via todd)
HDFS-2772. On transition to active, standby should not swallow ELIE. (atm)
HDFS-2767. ConfiguredFailoverProxyProvider should support NameNodeProtocol.
(Uma Maheswara Rao G via todd)
HDFS-2795. Standby NN takes a long time to recover from a dead DN starting up.
(todd)
HDFS-2592. Balancer support for HA namenodes. (Uma Maheswara Rao G via todd)
HDFS-2367. Enable the configuration of multiple HA cluster addresses. (atm)
HDFS-2812. When becoming active, the NN should treat all leases as freshly
renewed. (todd)
HDFS-2737. Automatically trigger log rolls periodically on the active NN. (todd
and atm)
HDFS-2820. Add a simple sanity check for HA config (todd)
HDFS-2688. Add tests for quota tracking in an HA cluster. (todd)
HDFS-2804. Should not mark blocks under-replicated when exiting safemode (todd)
HDFS-2807. Service level authorizartion for HAServiceProtocol. (jitendra)
HDFS-2809. Add test to verify that delegation tokens are honored after
failover. (jitendra and atm)
HDFS-2838. NPE in FSNamesystem when in safe mode. (Gregory Chanan via eli)
HDFS-2805. Add a test for a federated cluster with HA NNs. (Brandon Li via
jitendra)
HDFS-2841. HAAdmin does not work if security is enabled. (atm)
HDFS-2691. Fixes for pipeline recovery in an HA cluster: report RBW replicas
immediately upon pipeline creation. (todd)
HDFS-2824. Fix failover when prior NN died just after creating an edit log
segment. (atm via todd)
HDFS-2853. HA: NN fails to start if the shared edits dir is marked required
(atm via eli)
HDFS-2845. SBN should not allow browsing of the file system via web UI. (Bikas
Saha via atm)
HDFS-2742. HA: observed dataloss in replication stress test. (todd via eli)
HDFS-2870. Fix log level for block debug info in processMisReplicatedBlocks
(todd)
HDFS-2859. LOCAL_ADDRESS_MATCHER.match has NPE when called from
DFSUtil.getSuffixIDs when the host is incorrect (Bikas Saha via todd)
HDFS-2861. checkpointing should verify that the dfs.http.address has been
configured to a non-loopback for peer NN (todd)
HDFS-2860. TestDFSRollback#testRollback is failing. (atm)
HDFS-2769. HA: When HA is enabled with a shared edits dir, that dir should be
marked required. (atm via eli)
HDFS-2863. Failures observed if dfs.edits.dir and shared.edits.dir have same
directories. (Bikas Saha via atm)
HDFS-2874. Edit log should log to shared dirs before local dirs. (todd)
HDFS-2890. DFSUtil#getSuffixIDs should skip unset configurations. (atm)
HDFS-2792. Make fsck work. (atm)
HDFS-2808. HA: haadmin should use namenode ids. (eli)
HDFS-2819. Document new HA-related configs in hdfs-default.xml. (eli)
HDFS-2752. HA: exit if multiple shared dirs are configured. (eli)
HDFS-2894. HA: automatically determine the nameservice Id if only one
nameservice is configured. (eli)
HDFS-2733. Document HA configuration and CLI. (atm)
HDFS-2794. Active NN may purge edit log files before standby NN has a chance to
read them (todd)
HDFS-2901. Improvements for SBN web UI - not show under-replicated/missing
blocks. (Brandon Li via jitendra)
HDFS-2905. HA: Standby NN NPE when shared edits dir is deleted. (Bikas Saha via
jitendra)
HDFS-2579. Starting delegation token manager during safemode fails. (todd)
HDFS-2510. Add HA-related metrics. (atm)
HDFS-2924. Standby checkpointing fails to authenticate in secure cluster.
(todd)
HDFS-2915. HA: TestFailureOfSharedDir.testFailureOfSharedDir() has race
condition. (Bikas Saha via jitendra)
HDFS-2912. Namenode not shutting down when shared edits dir is inaccessible.
(Bikas Saha via atm)
HDFS-2917. HA: haadmin should not work if run by regular user (eli)
HDFS-2939. TestHAStateTransitions fails on Windows. (Uma Maheswara Rao G via
atm)
HDFS-2947. On startup NN throws an NPE in the metrics system. (atm)
HDFS-2942. TestActiveStandbyElectorRealZK fails if build dir does not exist.
(atm)
HDFS-2948. NN throws NPE during shutdown if it fails to startup (todd)
HDFS-2909. HA: Inaccessible shared edits dir not getting removed from FSImage
storage dirs upon error. (Bikas Saha via jitendra)
HDFS-2934. Allow configs to be scoped to all NNs in the nameservice. (todd)
HDFS-2935. Shared edits dir property should be suffixed with nameservice and
namenodeID (todd)
HDFS-2928. ConfiguredFailoverProxyProvider should not create a NameNode proxy
with an underlying retry proxy. (Uma Maheswara Rao G via atm)
HDFS-2955. IllegalStateException during standby startup in getCurSegmentTxId.
(Hari Mankude via atm)
HDFS-2937. TestDFSHAAdmin needs tests with MiniDFSCluster. (Brandon Li via
suresh)
HDFS-2586. Add protobuf service and implementation for HAServiceProtocol.
(suresh via atm)
HDFS-2952. NN should not start with upgrade option or with a pending an
unfinalized upgrade. (atm)
HDFS-2974. MiniDFSCluster does not delete standby NN name dirs during format.
(atm)
HDFS-2929. Stress test and fixes for block synchronization (todd)
HDFS-2972. Small optimization building incremental block report (todd)
HDFS-2973. Re-enable NO_ACK optimization for block deletion. (todd)
HDFS-2922. HA: close out operation categories (eli)
HDFS-2993. HA: BackupNode#checkOperation should permit CHECKPOINT operations
(eli)
HDFS-2904. Client support for getting delegation tokens. (todd)
HDFS-3013. HA: NameNode format doesn't pick up
dfs.namenode.name.dir.NameServiceId configuration (Mingjie Lai via todd)
HDFS-3019. Fix silent failure of TestEditLogJournalFailures (todd)
HDFS-2958. Sweep for remaining proxy construction which doesn't go through
failover path. (atm)
HDFS-2920. fix remaining TODO items. (atm and todd)
HDFS-3027. Implement a simple NN health check. (atm)
HDFS-3023. Optimize entries in edits log for persistBlocks call. (todd)
HDFS-2979. Balancer should use logical uri for creating failover proxy with HA
enabled. (atm)
HDFS-3035. Fix failure of TestFileAppendRestart due to OP_UPDATE_BLOCKS (todd)
HDFS-3039. Address findbugs and javadoc warnings on branch. (todd via atm)
Release 0.23.10 - UNRELEASED
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
HDFS-5010. Reduce the frequency of getCurrentUser() calls from namenode
(kihwal)
OPTIMIZATIONS
BUG FIXES
HDFS-4998. TestUnderReplicatedBlocks fails intermittently (kihwal)
Release 0.23.9 - UNRELEASED
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
OPTIMIZATIONS
BUG FIXES
HDFS-4867. metaSave NPEs when there are invalid blocks in repl queue.
(Plamen Jeliazkov and Ravi Prakash via shv)
HDFS-4862. SafeModeInfo.isManual() returns true when resources are low even
if it wasn't entered into manually (Ravi Prakash via kihwal)
HDFS-4832. Namenode doesn't change the number of missing blocks in
safemode when DNs rejoin or leave (Ravi Prakash via kihwal)
HDFS-4878. On Remove Block, block is not removed from neededReplications
queue. (Tao Luo via shv)
HDFS-4205. fsck fails with symlinks. (jlowe)
Release 0.23.8 - 2013-06-05
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
HDFS-4714. Log short messages in Namenode RPC server for exceptions
meant for clients. (kihwal)
OPTIMIZATIONS
BUG FIXES
HDFS-4477. Secondary namenode may retain old tokens (daryn via kihwal)
HDFS-4699. TestPipelinesFailover#testPipelineRecoveryStress fails
sporadically (Chris Nauroth via kihwal)
HDFS-4805. Webhdfs client is fragile to token renewal errors
(daryn via kihwal)
HDFS-3875. Issue handling checksum errors in write pipeline. (kihwal)
HDFS-4807. createSocketForPipeline() should not include timeout extension
on connect. (Cristina L. Abad via kihwal)
Release 0.23.7 - 2013-04-18
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
OPTIMIZATIONS
HDFS-4532. RPC call queue may fill due to current user lookup (daryn)
BUG FIXES
HDFS-4288. NN accepts incremental BR as IBR in safemode (daryn via kihwal)
HDFS-4495. Allow client-side lease renewal to be retried beyond soft-limit
(kihwal)
HDFS-4128. 2NN gets stuck in inconsistent state if edit log replay fails
in the middle (kihwal via daryn)
HDFS-4542. Webhdfs doesn't support secure proxy users (Daryn Sharp via
kihwal)
HDFS-4560. Webhdfs cannot use tokens obtained by another user (daryn)
HDFS-4566. Webdhfs token cancelation should use authentication (daryn)
HDFS-4567. Webhdfs does not need a token for token operations (daryn via
kihwal)
HDFS-4577. Webhdfs operations should declare if authentication is required
(daryn via kihwal)
HDFS-3344. Unreliable corrupt blocks counting in TestProcessCorruptBlocks
(kihwal)
HDFS-3367. WebHDFS doesn't use the logged in user when opening
connections (daryn)
HDFS-4581. checkDiskError should not be called on network errors (Rohit
Kochar via kihwal)
HDFS-4649. Webhdfs cannot list large directories (daryn via kihwal)
HDFS-4548. Webhdfs doesn't renegotiate SPNEGO token (daryn via kihwal)
Release 0.23.6 - 2013-02-06
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
OPTIMIZATIONS
BUG FIXES
HDFS-4247. saveNamespace should be tolerant of dangling lease (daryn)
HDFS-4248. Renaming directories may incorrectly remove the paths in leases
under the tree. (daryn via szetszwo)
HDFS-4385. Maven RAT plugin is not checking all source files (tgraves)
HDFS-4426. Secondary namenode shuts down immediately after startup.
(Arpit Agarwal via suresh)
Release 0.23.5 - 2012-11-28
INCOMPATIBLE CHANGES
HDFS-4080. Add a separate logger for block state change logs to enable turning
off those logs. (Kihwal Lee via suresh)
NEW FEATURES
IMPROVEMENTS
OPTIMIZATIONS
HDFS-4075. Reduce recommissioning overhead (Kihwal Lee via daryn)
HDFS-3990. NN's health report has severe performance problems (daryn)
HDFS-4181. LeaseManager tries to double remove and prints extra messages
(Kihwal Lee via daryn)
BUG FIXES
HDFS-3829. TestHftpURLTimeouts fails intermittently with JDK7 (Trevor
Robinson via tgraves)
HDFS-3824. TestHftpDelegationToken fails intermittently with JDK7 (Trevor
Robinson via tgraves)
HDFS-3224. Bug in check for DN re-registration with different storage ID
(jlowe)
HDFS-4090. getFileChecksum() result incompatible when called against
zero-byte files. (Kihwal Lee via daryn)
HDFS-4172. namenode does not URI-encode parameters when building URI for
datanode request (Derek Dagit via bobby)
HDFS-4182. SecondaryNameNode leaks NameCache entries (bobby)
HDFS-4186. logSync() is called with the write lock held while releasing
lease (Kihwal Lee via daryn)
Release 0.23.4
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
OPTIMIZATIONS
BUG FIXES
HDFS-3831. Failure to renew tokens due to test-sources left in classpath
(jlowe via bobby)
Release 0.23.3
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
HDFS-2505. Add a test to verify getFileChecksum(..) with ViewFS. (Ravi
Prakash via szetszwo)
OPTIMIZATIONS
BUG FIXES
HDFS-3166. Add timeout to Hftp connections. (Daryn Sharp via szetszwo)
HDFS-3176. Use MD5MD5CRC32FileChecksum.readFields() in JsonUtil . (Kihwal
Lee via szetszwo)
HDFS-2652. Add support for host-based delegation tokens. (Daryn Sharp via
szetszwo)
HDFS-3308. Uses canonical URI to select delegation tokens in HftpFileSystem
and WebHdfsFileSystem. (Daryn Sharp via szetszwo)
HDFS-3312. In HftpFileSystem, the namenode URI is non-secure but the
delegation tokens have to use secure URI. (Daryn Sharp via szetszwo)
HDFS-3318. Use BoundedInputStream in ByteRangeInputStream, otherwise, it
hangs on transfers >2 GB. (Daryn Sharp via szetszwo)
HDFS-3321. Fix safe mode turn off tip message. (Ravi Prakash via szetszwo)
HDFS-3334. Fix ByteRangeInputStream stream leakage. (Daryn Sharp via
szetszwo)
HDFS-3331. In namenode, check superuser privilege for setBalancerBandwidth
and acquire the write lock for finalizeUpgrade. (szetszwo)
HDFS-3037. TestMulitipleNNDataBlockScanner#testBlockScannerAfterRestart is
racy. (atm)
HDFS-3577. In DatanodeWebHdfsMethods, use MessageBodyWriter instead of
StreamingOutput, otherwise, it will fail to transfer large files.
(szetszwo)
HDFS-3646. LeaseRenewer can hold reference to inactive DFSClient
instances forever. (Kihwal Lee via daryn)
HDFS-3696. Set chunked streaming mode in WebHdfsFileSystem write operations
to get around a Java library bug causing OutOfMemoryError. (szetszwo)
HDFS-3553. Hftp proxy tokens are broken (daryn)
HDFS-3718. Datanode won't shutdown because of runaway DataBlockScanner
thread (Kihwal Lee via daryn)
HDFS-3861. Deadlock in DFSClient (Kihwal Lee via daryn)
HDFS-3873. Hftp assumes security is disabled if token fetch fails (daryn)
HDFS-3852. TestHftpDelegationToken is broken after HADOOP-8225 (daryn)
HDFS-3890. filecontext mkdirs doesn't apply umask as expected
(Tom Graves via daryn)
Release 0.23.2 - UNRELEASED
INCOMPATIBLE CHANGES
HDFS-2887. FSVolume, is a part of FSDatasetInterface implementation, should
not be referred outside FSDataset. A new FSVolumeInterface is defined.
The BlockVolumeChoosingPolicy.chooseVolume(..) method signature is also
updated. (szetszwo)
NEW FEATURES
HDFS-2943. Expose last checkpoint time and transaction stats as JMX
metrics. (atm)
IMPROVEMENTS
HDFS-2931. Switch DataNode's BlockVolumeChoosingPolicy to private-audience.
(harsh via szetszwo)
HDFS-2655. BlockReaderLocal#skip performs unnecessary IO. (Brandon Li
via jitendra)
HDFS-2725. hdfs script usage information is missing the information
about "dfs" command (Prashant Sharma via stevel)
HDFS-2907. Add a conf property dfs.datanode.fsdataset.factory to make
FSDataset in Datanode pluggable. (szetszwo)
HDFS-2985. Improve logging when replicas are marked as corrupt. (todd)
HDFS-3098. Update and add tests for HADOOP-8173. (Daryn Sharp via szetszwo)
HDFS-3104. Add tests for HADOOP-8175. (Daryn Sharp via szetszwo)
HDFS-3066. Cap space usage of default log4j rolling policy.
(Patrick Hunt via eli)
OPTIMIZATIONS
HDFS-3024. Improve performance of stringification in addStoredBlock (todd)
BUG FIXES
HDFS-2923. Namenode IPC handler count uses the wrong configuration key
(todd)
HDFS-2764. TestBackupNode is racy. (atm)
HDFS-2869. Fix an error in the webhdfs docs for the mkdir op (harsh)
HDFS-776. Fix exception handling in Balancer. (Uma Maheswara Rao G
via szetszwo)
HDFS-2815. Namenode sometimes oes not come out of safemode during
NN crash + restart. (Uma Maheswara Rao via suresh)
HDFS-2950. Secondary NN HTTPS address should be listed as a
NAMESERVICE_SPECIFIC_KEY. (todd)
HDFS-2525. Race between BlockPoolSliceScanner and append. (Brandon Li
via jitendra)
HDFS-2938. Recursive delete of a large directory make namenode
unresponsive. (Hari Mankude via suresh)
HDFS-2969. ExtendedBlock.equals is incorrectly implemented (todd)
HDFS-2944. Typo in hdfs-default.xml causes
dfs.client.block.write.replace-datanode-on-failure.enable to be mistakenly
disabled. (atm)
HDFS-2981. In hdfs-default.xml, the default value of
dfs.client.block.write.replace-datanode-on-failure.enable should be true.
(szetszwo)
HDFS-3008. Negative caching of local addrs doesn't work. (eli)
HDFS-3006. In WebHDFS, when the return body is empty, set the Content-Type
to application/octet-stream instead of application/json. (szetszwo)
HDFS-2991. Fix case where OP_ADD would not be logged in append(). (todd)
HDFS-3012. Exception while renewing delegation token. (Bobby Evans via
jitendra)
HDFS-3032. Change DFSClient.renewLease() so that it only retries up to the
lease soft-limit. (Kihwal Lee via szetszwo)
HDFS-2038. Update TestHDFSCLI to handle relative paths with globs.
(Kihwal Lee via szetszwo)
HDFS-3101. Cannot read empty file using WebHDFS. (szetszwo)
HDFS-3160. httpfs should exec catalina instead of forking it.
(Roman Shaposhnik via eli)
Release 0.23.1 - 2012-02-17
INCOMPATIBLE CHANGES
NEW FEATURES
HDFS-2316. [umbrella] WebHDFS: a complete FileSystem implementation for
accessing HDFS over HTTP (szetszwo)
HDFS-2594. Support getDelegationTokens and createSymlink in WebHDFS.
(szetszwo)
HDFS-2545. Change WebHDFS to support multiple namenodes in federation.
(szetszwo)
HDFS-2178. Contributing Hoop to HDFS, replacement for HDFS proxy
with read/write capabilities. (tucu)
IMPROVEMENTS
HDFS-2560. Refactor BPOfferService to be a static inner class (todd)
HDFS-2544. Hadoop scripts unconditionally source
"$bin"/../libexec/hadoop-config.sh. (Bruno Mahé via tomwhite)
HDFS-2543. HADOOP_PREFIX cannot be overridden. (Bruno Mahé via tomwhite)
HDFS-2562. Refactor DN configuration variables out of DataNode class
(todd)
HDFS-2563. Some cleanup in BPOfferService. (todd)
HDFS-2568. Use a set to manage child sockets in XceiverServer.
(harsh via eli)
HDFS-2454. Move maxXceiverCount check to before starting the
thread in dataXceiver. (harsh via eli)
HDFS-2570. Add descriptions for dfs.*.https.address in hdfs-default.xml.
(eli)
HDFS-2536. Remove unused imports. (harsh via eli)
HDFS-2566. Move BPOfferService to be a non-inner class. (todd)
HDFS-2552. Add Forrest doc for WebHDFS REST API. (szetszwo)
HDFS-2587. Add apt doc for WebHDFS REST API. (szetszwo)
HDFS-2604. Add a log message to show if WebHDFS is enabled and a
configuration section in the forrest doc. (szetszwo)
HDFS-2511. Add dev script to generate HDFS protobufs. (tucu)
HDFS-2654. Make BlockReaderLocal not extend RemoteBlockReader2. (eli)
HDFS-2675. Reduce warning verbosity when double-closing edit logs
(todd)
HDFS-2335. DataNodeCluster and NNStorage always pull fresh entropy.
(Uma Maheswara Rao G via eli)
HDFS-2574. Remove references to some deprecated properties in conf
templates and defaults files. (Joe Crobak via harsh)