blob: b8ac77721c0a3807d8a3d2d8029684bf0037b5cb [file] [log] [blame]
Hadoop HDFS Change Log
Release 2.6.5 - 2016-10-08
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
HDFS-9629. Update the footer of Web UI to show year 2016
(Xiao Chen and Vinayakumar B via vinayakumarb)
HDFS-9688. Test the effect of nested encryption zones in HDFS downgrade.
(zhz)
HDFS-10264. Logging improvements in FSImageFormatProtobuf.Saver. (Xiaobing
Zhou via shvachko)
HDFS-2580. NameNode#main(...) can make use of GenericOptionsParser. (harsh)
HDFS-7258. CacheReplicationMonitor rescan schedule log should use DEBUG
level instead of INFO level. (Xiaoyu Yao via wheat9)
HDFS-10377. CacheReplicationMonitor shutdown log message should use INFO
level (Yiqun Lin via cmccabe)
HDFS-9669. TcpPeerServer should respect ipc.server.listen.queue.size
(Elliot Clark via cmccabe)
HDFS-10870. Wrong dfs.namenode.acls.enabled default in HdfsPermissionsGuide.apt.vm.
(John Zhuge via lei)
OPTIMIZATIONS
HDFS-10653. Optimize conversion from path string to components. (Daryn
Sharp via zhz)
HDFS-8845. DiskChecker should not traverse the entire tree (Chang Li via
Colin P. McCabe)
BUG FIXES
HDFS-9347. Invariant assumption in TestQuorumJournalManager.shutdown()
is wrong. (Wei-Chiu Chuang via zhz)
HDFS-9752. Permanent write failures may happen to slow writers during
datanode rolling upgrades. (Walter Su via kihwal)
HDFS-10182. Hedged read might overwrite user's buf. (zhouyingchao via
Walter Su)
HDFS-10271. Extra bytes are getting released from reservedSpace for append
(Brahma Reddy Battula via vinayakumarb)
HDFS-10458. getFileEncryptionInfo should return quickly for non-encrypted
cluster. (zhz)
HDFS-8581. ContentSummary on / skips further counts on yielding lock
(J.Andreina via vinayakumarb)
HDFS-10544. Balancer doesn't work with IPFailoverProxyProvider. (zhz)
HDFS-9530. ReservedSpace is not cleared for abandoned Blocks.
(Brahma Reddy Battula)
HDFS-10178. Permanent write failures can happen if pipeline recoveries
occur for the first packet (kihwal)
HDFS-9365. Balancer does not work with the HDFS-6376 HA setup. (Tsz Wo
Nicholas Sze)
HDFS-9696. Garbage snapshot records linger forever. (kihwal)
HDFS-10763. Open files can leak permanently due to inconsistent
lease update (kihwal)
HDFS-7530. Allow renaming of encryption zone roots. (Charles Lamb via wang)
Release 2.6.4 - 2016-02-11
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
HDFS-8647. Abstract BlockManager's rack policy into BlockPlacementPolicy.
(Brahma Reddy Battula via mingma)
HDFS-9313. Possible NullPointerException in BlockManager if no excess
replica can be chosen. (mingma)
HDFS-9314. Improve BlockPlacementPolicyDefault's picking of excess
replicas. (Xiao Chen via zhz)
HDFS-6945. BlockManager should remove a block from excessReplicateMap and
decrement ExcessBlocks metric when the block is removed. (aajisaka)
HDFS-9574. Reduce client failures during datanode restart (kihwal)
HDFS-9415. Document dfs.cluster.administrators and
dfs.permissions.superusergroup. (Xiaobing Zhou via Arpit Agarwal)
OPTIMIZATIONS
BUG FIXES
HDFS-9445. Datanode may deadlock while handling a bad volume.
(Wlater Su via Kihwal)
HDFS-9600. do not check replication if the block is under construction
(Phil Yang via vinayakumarb)
HDFS-8767. RawLocalFileSystem.listStatus() returns null for UNIX pipefile.
(Kanaka Kumar Avvaru via wheat9)
HDFS-9294. DFSClient deadlock when close file and failed to renew lease.
(Brahma Reddy Battula via szetszwo)
HDFS-4660. Block corruption can happen during pipeline recovery (kihwal)
HDFS-9178. Slow datanode I/O can cause a wrong node to be marked bad.
(Kihwal Lee)
HDFS-8722. Optimize datanode writes for small writes and flushes.
(Kihwal Lee)
HDFS-9220. Reading small file (< 512 bytes) that is open for append fails
due to incorrect checksum (Jing Zhao via kihwal)
Release 2.6.3 - 2015-12-17
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
OPTIMIZATIONS
BUG FIXES
HDFS-8615. Correct HTTP method in WebHDFS document.
(Brahma Reddy Battula via aajisaka)
HDFS-9431. DistributedFileSystem#concat fails if the target path is
relative. (Kazuho Fujii via aajisaka)
HDFS-9083. Replication violates block placement policy (Rushabh Shah)
HDFS-9289. Make DataStreamer#block thread safe and verify genStamp in
commitBlock. (Chang Li via zhz)
HDFS-9434. Recommission a datanode with 500k blocks may pause NN for 30
seconds for printing info log messags. (szetszwo)
HDFS-9470. Encryption zone on root not loaded from fsimage after NN
restart. (Xiao Chen via wang)
HDFS-9273. ACLs on root directory may be lost after NN restart.
(Xiao Chen via cnauroth)
Release 2.6.2 - 2015-10-28
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
OPTIMIZATIONS
BUG FIXES
Release 2.6.1 - 2015-09-23
INCOMPATIBLE CHANGES
NEW FEATURES
HDFS-7278. Add a command that allows sysadmins to manually trigger full
block reports from a DN (cmccabe)
IMPROVEMENTS
HDFS-7035. Make adding a new data directory to the DataNode an atomic
operation and improve error handling (Lei Xu via Colin P. McCabe)
HDFS-7531. Improve the concurrent access on FsVolumeList (Lei Xu via Colin
P. McCabe)
HDFS-7579. Improve log reporting during block report rpc failure.
(Charles Lamb via cnauroth)
HDFS-7182. JMX metrics aren't accessible when NN is busy. (Ming Ma via jing9)
HDFS-7596. NameNode should prune dead storages from storageMap.
(Arpit Agarwal via cnauroth)
HDFS-8046. Allow better control of getContentSummary (kihwal)
HDFS-8384. Allow NN to startup if there are files having a lease but are not
under construction. (jing9)
HDFS-7694. FSDataInputStream should support "unbuffer" (cmccabe)
OPTIMIZATIONS
HDFS-8480. Fix performance and timeout issues in HDFS-7929 by using
hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang
via Colin P. McCabe)
BUG FIXES
HDFS-7425. NameNode block deletion logging uses incorrect appender.
(cnauroth)
HDFS-4882. Prevent the Namenode's LeaseManager from looping forever in
checkLeases (Ravi Prakash via Colin P. McCabe)
HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang datanodes
(Noah Lorang via Colin P. McCabe)
HDFS-7503. Namenode restart after large deletions can cause slow
processReport. (Arpit Agarwal)
HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate
block files are present in the same volume (cmccabe)
HDFS-3443. Fix NPE when namenode transition to active during startup by
adding checkNNStartup() in NameNodeRpcServer. (Vinayakumar B via szetszwo)
HDFS-7733. NFS: readdir/readdirplus return null directory
attribute on failure. (Arpit Agarwal)
HDFS-8486. DN startup may cause severe data loss. (daryn via cmccabe)
HDFS-7213. processIncrementalBlockReport performance degradation.
(Eric Payne via kihwal)
HDFS-7235. DataNode#transferBlock should report blocks that don't exist
using reportBadBlock (yzhang via cmccabe)
HDFS-7263. Snapshot read can reveal future bytes for appended files.
(Tao Luo via shv)
HDFS-7225. Remove stale block invalidation work when DN re-registers with
different UUID. (Zhe Zhang and Andrew Wang)
HDFS-7533. Datanode sometimes does not shutdown on receiving upgrade
shutdown command (Eric Payne via kihwal)
HDFS-7575. Upgrade should generate a unique storage ID for each
volume. (Arpit Agarwal)
HDFS-7707. Edit log corruption due to delayed block removal again.
(Yongjun Zhang via kihwal)
HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause
DataNode to register successfully with only one NameNode.(vinayakumarb)
HDFS-7788. Post-2.6 namenode may not start up with an image containing
inodes created with an old release. (Rushabh Shah via kihwal)
HDFS-7009. Active NN and standby NN have different live nodes.
(Ming Ma via cnauroth)
HDFS-7763. fix zkfc hung issue due to not catching exception in a corner
case. (Liang Xie via wang)
HDFS-7871. NameNodeEditLogRoller can keep printing "Swallowing exception"
message. (jing9)
HDFS-7885. Datanode should not trust the generation stamp provided by
client. (Tsz Wo Nicholas Sze via jing9)
HDFS-7610. Fix removal of dynamically added DN volumes (Lei (Eddy) Xu via
Colin P. McCabe)
HDFS-7830. DataNode does not release the volume lock when adding a volume
fails. (Lei Xu via Colin P. Mccabe)
HDFS-7587. Edit log corruption can happen if append fails with a quota
violation. (jing9)
HDFS-7929. inotify unable fetch pre-upgrade edit log segments once upgrade
starts (Zhe Zhang via Colin P. McCabe)
HDFS-7930. commitBlockSynchronization() does not remove locations. (yliu)
HDFS-7884. Fix NullPointerException in BlockSender when the generation stamp
provided by the client is larger than the one stored in the datanode.
(Brahma Reddy Battula via szetszwo)
HDFS-7960. The full block report should prune zombie storages even if
they're not empty. (cmccabe and Eddy Xu via wang)
HDFS-7742. Favoring decommissioning node for replication can cause a block
to stay underreplicated for long periods (Nathan Roberts via kihwal)
HDFS-7999. FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl
lock for a very long time (sinago via cmccabe)
HDFS-8072. Reserved RBW space is not released if client terminates while
writing block. (Arpit Agarwal)
HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to
finalize upgrade. (jing9)
HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and
fail to tell the DFSClient about it because of a network error (cmccabe)
HDFS-8070. Pre-HDFS-7915 DFSClient cannot use short circuit on
post-HDFS-7915 DataNode (cmccabe)
HDFS-8219. setStoragePolicy with folder behavior is different after cluster restart.
(surendra singh lilhore via Xiaoyu Yao)
HDFS-7894. Rolling upgrade readiness is not updated in jmx until query
command is issued. (Brahma Reddy Battula via kihwal)
HDFS-8254. Standby namenode doesn't process DELETED_BLOCK if the add block
request is in edit log. (Rushabh S Shah via kihwal)
HDFS-8404. Pending block replication can get stuck using older genstamp
(Nathan Roberts via kihwal)
HDFS-8431. hdfs crypto class not found in Windows.
(Anu Engineer via cnauroth)
HDFS-7609. Avoid retry cache collision when Standby NameNode loading edits.
(Ming Ma via jing9)
HDFS-8270. create() always retried with hardcoded timeout when file already
exists with open lease (J.Andreina via vinayakumarb)
HDFS-7980. Incremental BlockReport will dramatically slow down namenode
startup. (Walter Su via szetszwo)
HDFS-7446. HDFS inotify should have the ability to determine what txid it
has read up to (cmccabe)
HDFS-8846. Add a unit test for INotify functionality across a layout
version upgrade (Zhe Zhang via Colin P. McCabe)
HDFS-7470. SecondaryNameNode need twice memory when calling
reloadFromImageFile. (zhaoyunjiong via cnauroth)
HDFS-8863. The remaining space check in BlockPlacementPolicyDefault is
flawed. (Kihwal Lee via yliu)
HDFS-7552. Change FsVolumeList toString() to fix
TestDataNodeVolumeFailureToleration (Liang Xie via Colin P. McCabe)
Release 2.6.0 - 2014-11-18
INCOMPATIBLE CHANGES
NEW FEATURES
HDFS-6584. Support archival storage. (See breakdown of tasks below for
features and contributors)
HDFS-6581. Support writing to a single replica in RAM. (See breakdown of
tasks below for features and contributors).
IMPROVEMENTS
HDFS-6613. Improve logging in caching classes. (wang)
HDFS-6511. BlockManager#computeInvalidateWork() could do nothing. (Juan Yu via wang)
HDFS-6638. Shorten test run time with a smaller retry timeout setting.
(Liang Xie via cnauroth)
HDFS-6627. Rename DataNode#checkWriteAccess to checkReadAccess.
(Liang Xie via cnauroth)
HDFS-6645. Add test for successive Snapshots between XAttr modifications.
(Stephen Chu via jing9)
HDFS-6643. Refactor INodeWithAdditionalFields.PermissionStatusFormat and
INodeFile.HeaderFormat. (szetszwo)
HDFS-6640. Syntax for MKDIRS, CREATESYMLINK, and SETXATTR are given wrongly
in WebHdfs document (missed webhdfs/v1). (Stephen Chu via jing9)
HDFS-5202. Support Centralized Cache Management on Windows. (cnauroth)
HDFS-2976. Remove unnecessary method (tokenRefetchNeeded) in DFSClient.
(Uma Maheswara Rao G)
HDFS-3851. DFSOutputStream class code cleanup. (Jing Zhao via suresh)
HDFS-2856. Fix block protocol so that Datanodes don't require root or jsvc.
(cnauroth)
HDFS-5624. Add HDFS tests for ACLs in combination with viewfs.
(Stephen Chu via cnauroth)
HDFS-6655. Add 'header banner' to 'explorer.html' also in Namenode UI
(vinayakumarb)
HDFS-4120. Add a new "-skipSharedEditsCheck" option for BootstrapStandby
(Liang Xie and Rakesh R via vinayakumarb)
HDFS-6597. Add a new option to NN upgrade to terminate the process after
upgrade on NN is completed. (Danilo Vunjak via cnauroth)
HDFS-6700. BlockPlacementPolicy shoud choose storage but not datanode for
deletion. (szetszwo)
HDFS-6616. Add exclude-datanodes feature to WebHDFS redirection so that it
will not redirect retries to the same datanode. (zhaoyunjiong via szetszwo)
HDFS-6702. Change DFSClient to pass the StorageType from the namenode to
datanodes and change datanode to write block replicas using the specified
storage type. (szetszwo)
HDFS-6701. Make seed optional in NetworkTopology#sortByDistance.
(Ashwin Shankar via wang)
HDFS-6755. There is an unnecessary sleep in the code path where
DFSOutputStream#close gives up its attempt to contact the namenode
(mitdesai21 via cmccabe)
HDFS-6750. The DataNode should use its shared memory segment to mark
short-circuit replicas that have been unlinked as stale (cmccabe)
HDFS-6739. Add getDatanodeStorageReport to ClientProtocol. (szetszwo)
HDFS-6665. Add tests for XAttrs in combination with viewfs.
(Stephen Chu via wang)
HDFS-6778. The extended attributes javadoc should simply refer to the
user docs. (clamb via wang)
HDFS-6570. add api that enables checking if a user has certain permissions on
a file. (Jitendra Pandey via cnauroth)
HDFS-6441. Add ability to exclude/include specific datanodes while
balancing. (Benoy Antony and Yu Li via Arpit Agarwal)
HDFS-6685. Balancer should preserve storage type of replicas. (szetszwo)
HDFS-6798. Add test case for incorrect data node condition during
balancing. (Benoy Antony via Arpit Agarwal)
HDFS-6796. Improve the argument check during balancer command line parsing.
(Benoy Antony via szetszwo)
HDFS-6794. Update BlockManager methods to use DatanodeStorageInfo
where possible (Arpit Agarwal)
HDFS-6802. Some tests in TestDFSClientFailover are missing @Test
annotation. (Akira Ajisaka via wang)
HDFS-6788. Improve synchronization in BPOfferService with read write lock.
(Yongjun Zhang via wang)
HDFS-6787. Remove duplicate code in FSDirectory#unprotectedConcat. (Yi Liu via umamahesh)
HDFS-6809. Move Balancer's inner classes MovedBlocks and Matcher as to
standalone classes and separates KeyManager from NameNodeConnector.
(szetszwo)
HDFS-6812. Remove addBlock and replaceBlock from DatanodeDescriptor.
(szetszwo)
HDFS-6781. Separate HDFS commands from CommandsManual.apt.vm. (Akira
Ajisaka via Arpit Agarwal)
HDFS-6728. Dynamically add new volumes to DataStorage, formatted if
necessary. (Lei Xu vi atm)
HDFS-6740. Make FSDataset support adding data volumes dynamically. (Lei
Xu via atm)
HDFS-6722. Display readable last contact time for dead nodes on NN webUI.
(Ming Ma via wheat9)
HDFS-6772. Get DN storages out of blockContentsStale state faster after
NN restarts. (Ming Ma via Arpit Agarwal)
HDFS-573. Porting libhdfs to Windows. (cnauroth)
HDFS-6828. Separate block replica dispatching from Balancer. (szetszwo via
jing9)
HDFS-6837. Code cleanup for Balancer and Dispatcher. (szetszwo via
jing9)
HDFS-6838. Code cleanup for unnecessary INode replacement.
(Jing Zhao via wheat9)
HDFS-6836. HDFS INFO logging is verbose & uses file appenders. (Xiaoyu
Yao via Arpit Agarwal)
HDFS-6567. Normalize the order of public final in HdfsFileStatus.
(Tassapol Athiapinya via wheat9)
HDFS-6849. Replace HttpFS custom proxyuser handling with common
implementation. (tucu)
HDFS-6850. Move NFS out of order write unit tests into TestWrites class.
(Zhe Zhang via atm)
HDFS-6188. An ip whitelist based implementation of TrustedChannelResolver.
(Benoy Antony via Arpit Agarwal)
HDFS-6858. Allow dfs.data.transfer.saslproperties.resolver.class default to
hadoop.security.saslproperties.resolver.class. (Benoy Antony via cnauroth)
HDFS-6878. Change MiniDFSCluster to support StorageType configuration
for individual directories. (Arpit Agarwal)
HDFS-6758. block writer should pass the expected block size to
DataXceiverServer. (Arpit Agarwal)
HDFS-6899. Allow changing MiniDFSCluster volumes per DN and capacity
per volume. (Arpit Agarwal)
HDFS-4486. Add log category for long-running DFSClient notices (Zhe Zhang
via Colin Patrick McCabe)
HDFS-6879. Adding tracing to Hadoop RPC (Masatake Iwasaki via Colin Patrick
McCabe)
HDFS-6774. Make FsDataset and DataStore support removing volumes. (Lei Xu
via atm)
HDFS-6634. inotify in HDFS. (James Thomas via wang)
HDFS-4257. The ReplaceDatanodeOnFailure policies could have a forgiving
option (szetszwo via cmccabe)
HDFS-6959. Make the HDFS home directory location customizable. (yzhang via
cmccabe)
HDFS-6609. Use DirectorySnapshottableFeature to represent a snapshottable
directory. (Jing Zhao via wheat9)
HDFS-6886. Use single editlog record for creating file + overwrite. (Yi Liu
via jing9)
HDFS-6376. Distcp data between two HA clusters requires another configuration.
(Dave Marion and Haohui Mai via jing9)
HDFS-6943. Improve NN allocateBlock log to include replicas' datanode IPs.
(Ming Ma via wheat9)
HDFS-6036. Forcibly timeout misbehaving DFSClients that try to do
no-checksum reads that extend too long (cmccabe)
HDFS-6482. Use block ID-based block layout on datanodes (James Thomas via
Colin Patrick McCabe)
HDFS-7061. Add test to verify encryption zone creation after NameNode
restart without saving namespace. (Stephen Chu via wang)
HDFS-7059. HAadmin transtionToActive with forceActive option can show
confusing message.
HDFS-6880. Adding tracing to DataNode data transfer protocol. (iwasakims
via cmccabe)
HDFS-7006. Test encryption zones with KMS. (Anthony Young-Garner and tucu)
HDFS-6851. Refactor EncryptionZoneWithId and EncryptionZone. (clamb via wang)
HDFS-6705. Create an XAttr that disallows the HDFS admin from accessing a
file. (clamb via wang)
HDFS-6843. Create FileStatus isEncrypted() method (clamb via cmccabe)
HDFS-7004. Update KeyProvider instantiation to create by URI. (wang)
HDFS-7047. Expose FileStatus#isEncrypted in libhdfs (cmccabe)
HDFS-7003. Add NFS Gateway support for reading and writing to
encryption zones. (clamb via wang)
HDFS-6727. Refresh data volumes on DataNode based on configuration changes
(Lei Xu via cmccabe)
HDFS-6970. Move startFile EDEK retries to the DFSClient. (wang)
HDFS-6948. DN rejects blocks if it has older UC block
(Eric Payne via kihwal)
HDFS-6987. Move CipherSuite xattr information up to the encryption zone
root. (Zhe Zhang via wang)
HDFS-7139. Unit test for creating encryption zone on root path. (Zhe Zhang via wang)
HDFS-7138. Fix hftp to work with encryption. (clamb via wang)
HDFS-7118. Improve diagnostics on storage directory rename operations by
using NativeIO#renameTo in Storage#rename. (cnauroth)
HDFS-6808. Add command line option to ask DataNode reload configuration.
(Lei Xu via Colin Patrick McCabe)
HDFS-7119. Split error checks in AtomicFileOutputStream#close into separate
conditions to improve diagnostics. (cnauroth)
HDFS-7077. Separate CipherSuite from crypto protocol version. (wang)
HDFS-6956. Allow dynamically changing the tracing level in Hadoop servers
(cmccabe)
HDFS-7156. Update fsck documentation. (Masahiro Yamaguch via shv)
HDFS-7093. Add config key to restrict setStoragePolicy. (Arpit Agarwal)
HDFS-6519. Document oiv_legacy command (Akira AJISAKA via aw)
HDFS-4165. Faulty sanity check in FsDirectory.unprotectedSetQuota.
(Binglin Chang via suresh)
HDFS-7104. Fix and clarify INodeInPath getter functions. (Zhe Zhang via wang)
HDFS-7124. Remove EncryptionZoneManager.NULL_EZ. (clamb via wang)
HDFS-6779. Add missing version subcommand for hdfs.
(Sasaki Toru via wheat9)
HDFS-7153. Add storagePolicy to NN edit log during file creation.
(Arpit Agarwal)
HDFS-7158. Reduce the memory usage of WebImageViewer. (wheat9)
HDFS-6894. Add XDR parser method for each NFS response.
(Brandon Li via wheat9)
HDFS-7217. Better batching of IBRs. (kihwal)
HDFS-7195. Update user doc of secure mode about Datanodes don't require root
or jsvc. (cnauroth)
HDFS-7228. Add an SSD policy into the default BlockStoragePolicySuite.
(jing9)
HDFS-6904. YARN unable to renew delegation token fetched via webhdfs
due to incorrect service port. (jitendra)
HDFS-6988. Improve HDFS-6581 eviction configuration (Xiaoyu Yao via Colin
P. McCabe)
HDFS-7230. Add rolling downgrade documentation. (szetszwo via jing9)
HDFS-6385. Show when block deletion will start after NameNode startup in
WebUI. (cnauroth)
HDFS-7313. Support optional configuration of AES cipher suite on
DataTransferProtocol. (cnauroth)
HDFS-7276. Limit the number of byte arrays used by DFSOutputStream and
provide a mechanism for recycling arrays. (szetszwo)
HDFS-7233. NN logs unnecessary org.apache.hadoop.hdfs.protocol.UnresolvedPathException.
(Rushabh S Shah via jing9)
HDFS-7221. TestDNFencingWithReplication fails consistently. (Charles Lamb via wang)
OPTIMIZATIONS
HDFS-6690. Deduplicate xattr names in memory. (wang)
HDFS-6773. MiniDFSCluster should skip edit log fsync by default (Stephen
Chu via Colin Patrick McCabe)
HDFS-6865. Byte array native checksumming on client side
(James Thomas via todd)
HDFS-7122. Use of ThreadLocal<Random> results in poor block placement.
(wang)
HDFS-6606. Optimize HDFS Encrypted Transport performance. (yliu)
BUG FIXES
HDFS-7309. XMLUtils.mangleXmlString doesn't seem to handle less than sign
(Colin Patrick McCabe via raviprak)
HDFS-6823. dfs.web.authentication.kerberos.principal shows up in logs for
insecure HDFS (Allen Wittenauer via raviprak)
HDFS-6517. Remove hadoop-metrics2.properties from hdfs project (Akira
AJISAKA via aw)
HDFS-6617. Flake TestDFSZKFailoverController.testManualFailoverWithDFSHAAdmin
due to a long edit log sync op. (Liang Xie via cnauroth)
HDFS-6646. [ HDFS Rolling Upgrade - Shell ] shutdownDatanode and getDatanodeInfo
usage is missed ( Brahma Reddy Battula via vinayakumarb)
HDFS-6630. Unable to fetch the block information by Browsing the file system on
Namenode UI through IE9 ( Haohui Mai via vinayakumarb)
HADOOP-8158. Interrupting hadoop fs -put from the command line
causes a LeaseExpiredException. (daryn via harsh)
HDFS-6678. MiniDFSCluster may still be partially running after initialization
fails. (cnauroth)
HDFS-5809. BlockPoolSliceScanner and high speed hdfs appending make
datanode to drop into infinite loop (cmccabe)
HDFS-6456. NFS should throw error for invalid entry in
dfs.nfs.exports.allowed.hosts (Abhiraj Butala via brandonli)
HDFS-6689. NFS doesn't return correct lookup access for direcories (brandonli)
HDFS-6478. RemoteException can't be retried properly for non-HA scenario.
(Ming Ma via jing9)
HDFS-6693. TestDFSAdminWithHA fails on windows ( vinayakumarb )
HDFS-6667. In HDFS HA mode, Distcp/SLive with webhdfs on secure cluster fails
with Client cannot authenticate via:[TOKEN, KERBEROS] error. (jing9)
HDFS-6704. Fix the command to launch JournalNode in HDFS-HA document.
(Akira AJISAKA via jing9)
HDFS-6731. Run "hdfs zkfc-formatZK" on a server in a non-namenode will cause
a null pointer exception. (Masatake Iwasaki via brandonli)
HDFS-6114. Block Scan log rolling will never happen if blocks written
continuously leading to huge size of dncp_block_verification.log.curr
(vinayakumarb via cmccabe)
HDFS-6455. NFS: Exception should be added in NFS log for invalid separator in
nfs.exports.allowed.hosts. (Abhiraj Butala via brandonli)
HDFS-6715. Webhdfs wont fail over when it gets java.io.IOException: Namenode
is in startup mode. (jing9)
HDFS-5919. FileJournalManager doesn't purge empty and corrupt inprogress edits
files (vinayakumarb)
HDFS-6752. Avoid Address bind errors in TestDatanodeConfig#testMemlockLimit
(vinayakumarb)
HDFS-6749. FSNamesystem methods should call resolvePath.
(Charles Lamb via cnauroth)
HDFS-4629. Using com.sun.org.apache.xml.internal.serialize.* in
XmlEditsVisitor.java is JVM vendor specific. Breaks IBM JAVA.
(Amir Sanjar via stevel)
HDFS-3482. hdfs balancer throws ArrayIndexOutOfBoundsException
if option is specified without values. ( Madhukara Phatak via umamahesh)
HDFS-6797. DataNode logs wrong layoutversion during upgrade. (Benoy Antony
via Arpit Agarwal)
HDFS-6810. StorageReport array is initialized with wrong size in
DatanodeDescriptor#getStorageReports. (szetszwo via Arpit Agarwal)
HDFS-5723. Append failed FINALIZED replica should not be accepted as valid
when that block is underconstruction (vinayakumarb)
HDFS-5185. DN fails to startup if one of the data dir is full. (vinayakumarb)
HDFS-6451. NFS should not return NFS3ERR_IO for AccessControlException
(Abhiraj Butala via brandonli)
HDFS-6717. JIRA HDFS-5804 breaks default nfs-gateway behavior for unsecured config
(brandonli)
HDFS-6790. DFSUtil Should Use configuration.getPassword for SSL passwords
(Larry McCay via brandonli)
HDFS-6791. A block could remain under replicated if all of its replicas are on
decommissioned nodes. (Ming Ma via jing9)
HDFS-6582. Missing null check in RpcProgramNfs3#read(XDR, SecurityHandler)
(Abhiraj Butala via brandonli)
HDFS-6830. BlockInfo.addStorage fails when DN changes the storage for a
block replica (Arpit Agarwal)
HDFS-6247. Avoid timeouts for replaceBlock() call by sending intermediate
responses to Balancer (vinayakumarb)
HDFS-6783. Fix HDFS CacheReplicationMonitor rescan logic. (Yi Liu and Colin Patrick McCabe via umamahesh)
HDFS-6825. Edit log corruption due to delayed block removal.
(Yongjun Zhang via wang)
HDFS-6569. OOB message can't be sent to the client when DataNode shuts down for upgrade
(brandonli)
HDFS-6868. portmap and nfs3 are documented as hadoop commands instead of hdfs
(brandonli)
HDFS-6870. Blocks and INodes could leak for Rename with overwrite flag. (Yi
Liu via jing9)
HDFS-6890. NFS readdirplus doesn't return dotdot attributes (brandonli)
HDFS-6829. DFSAdmin refreshSuperUserGroupsConfiguration failed in
security cluster (zhaoyunjiong via Arpit Agarwal)
HDFS-4852. libhdfs documentation is out of date. (cnauroth)
HDFS-6908. Incorrect snapshot directory diff generated by snapshot deletion.
(Juan Yu and jing9 via jing9)
HDFS-6892. Add XDR packaging method for each NFS request (brandonli)
HDFS-6902. FileWriter should be closed in finally block in
BlockReceiver#receiveBlock() (Tsuyoshi OZAWA via Colin Patrick McCabe)
HDFS-6972. TestRefreshUserMappings.testRefreshSuperUserGroupsConfiguration
doesn't decode url correctly. (Yongjun Zhang via wang)
HDFS-6942. Fix typos in log messages. (Ray Chiang via wheat9)
HDFS-6848. Lack of synchronization on access to datanodeUuid in
DataStorage#format(). (Xiaoyu Yao via Arpit Agarwal)
HDFS-6996. SnapshotDiff report can hit IndexOutOfBoundsException when there
are nested renamed directory/file. (jing9)
HDFS-6831. Inconsistency between 'hdfs dfsadmin' and 'hdfs dfsadmin -help'.
(Xiaoyu Yao via Arpit Agarwal)
HDFS-6979. hdfs.dll does not produce .pdb files. (cnauroth)
HDFS-6862. Add missing timeout annotations to tests. (Xiaoyu Yao via
Arpit Agarwal)
HDFS-6898. DN must reserve space for a full block when an RBW block is
created. (Arpit Agarwal)
HDFS-7025. HDFS Credential Provider related Unit Test Failure.
(Xiaoyu Yao via cnauroth)
HDFS-7005. DFS input streams do not timeout.
HDFS-6951. Correctly persist raw namespace xattrs to edit log and fsimage.
(clamb via wang)
HDFS-6800. Support Datanode layout changes with rolling upgrade.
(James Thomas via Arpit Agarwal)
HDFS-6981. Fix DN upgrade with layout version change. (Arpit Agarwal)
HDFS-6506. Newly moved block replica been invalidated and deleted in
TestBalancer. (Binglin Chang via cnauroth)
HDFS-6966. Add additional unit tests for encryption zones.
(Stephen Chu via wang)
HDFS-6621. Hadoop Balancer prematurely exits iterations.
(Rafal Wojdyla and Benjamin Bowman via wang)
HDFS-7045. Fix NameNode deadlock when opening file under /.reserved path.
(Yi Liu via wang)
HDFS-7032. Add WebHDFS support for reading and writing to encryption zones.
(clamb via wang)
HDFS-6965. NN continues to issue block locations for DNs with full disks.
(Rushabh Shah via kihwal)
HDFS-6789. TestDFSClientFailover.testFileContextDoesntDnsResolveLogicalURI
and TestDFSClientFailover.testDoesntDnsResolveLogicalURI failing on jdk7.
(Akira Ajisaka via wang)
HDFS-6912. SharedFileDescriptorFactory should not allocate sparse files
(cmccabe)
HDFS-7075. hadoop-fuse-dfs fails because it cannot find
JavaKeyStoreProvider$Factory (cmccabe)
HDFS-7078. Fix listEZs to work correctly with snapshots. (wang)
HDFS-6840. Clients are always sent to the same datanode when read
is off rack. (wang)
HDFS-7065. Pipeline close recovery race can cause block corruption (kihwal)
HDFS-7096. Fix TestRpcProgramNfs3 to use DFS_ENCRYPTION_KEY_PROVIDER_URI
(clamb via cmccabe)
HDFS-7046. HA NN can NPE upon transition to active. (kihwal)
HDFS-7106. Reconfiguring DataNode volumes does not release the lock files
in removed volumes. (cnauroth via cmccabe)
HDFS-7001. Tests in TestTracing depends on the order of execution
(iwasakims via cmccabe)
HDFS-7132. hdfs namenode -metadataVersion command does not honor
configured name dirs. (Charles Lamb via wang)
HDFS-7049. TestByteRangeInputStream.testPropagatedClose fails and throw
NPE on branch-2. (Juan Yu via wheat9)
HDFS-7148. TestEncryptionZones#testIsEncryptedMethod fails on branch-2
after archival storage merge. (wang)
HDFS-7157. Using Time.now() for recording start/end time of reconfiguration
tasks (Lei Xu via cmccabe)
HDFS-6664. HDFS permissions guide documentation states incorrect default
group mapping class. (Ray Chiang via aw)
HDFS-4227. Document dfs.namenode.resource.* (Daisuke Kobayashi via aw)
HDFS-6754. TestNamenodeCapacityReport may sometimes fail due to lack of
retry. (Mit Desai via kihwal)
HDFS-7178. Additional unit test for replica write with full disk.
(Arpit Agarwal)
HDFS-7179. DFSClient should instantiate a KeyProvider, not a
KeyProviderCryptoExtension. (wang)
HDFS-7181. Remove incorrect precondition check on key length in
FileEncryptionInfo. (wang)
HDFS-7162. Wrong path when deleting through fuse-dfs a file which already
exists in trash (Chenging Liu via cmccabe)
HDFS-7176. The namenode usage message doesn't include "-rollingupgrade
started" (cmccabe)
HDFS-7172. Test data files may be checked out of git with incorrect line
endings, causing test failures in TestHDFSCLI. (Chris Nauroth via wheat9)
HDFS-7203. Concurrent appending to the same file can cause data corruption
(kihwal)
HDFS-7236. Fix TestOpenFilesWithSnapshot#testOpenFilesWithMultipleSnapshots.
(Yongjun Zhang via jing9)
HDFS-6544. Broken Link for GFS in package.html. (Suraj Nayak M via wheat9)
HDFS-7237. The command "hdfs namenode -rollingUpgrade" throws
ArrayIndexOutOfBoundsException. (szetszwo)
HDFS-7185. The active NameNode will not accept an fsimage sent from the
standby during rolling upgrade. (jing9)
HDFS-7208. NN doesn't schedule replication when a DN storage fails.
(Ming Ma via szetszwo)
HDFS-5089. When a LayoutVersion support SNAPSHOT, it must support
FSIMAGE_NAME_OPTIMIZATION. (szetszwo)
HDFS-7260. Change DFSOutputStream.MAX_PACKETS to be configurable. (szetszwo)
HDFS-7259. Unresponseive NFS mount point due to deferred COMMIT response.
(brandonli)
HDFS-7215.Add JvmPauseMonitor to NFS gateway (brandonli)
HDFS-7180. NFSv3 gateway frequently gets stuck due to GC (brandonli)
HDFS-7243. HDFS concat operation should not be allowed in Encryption Zone.
(clamb via yliu)
HADOOP-11233. hadoop.security.kms.client.encrypted.key.cache.expiry
property spelled wrong in core-default. (Stephen Chu via yliu)
HDFS-7218. FSNamesystem ACL operations should write to audit log on
failure. (clamb via yliu)
HDFS-7199. DFSOutputStream should not silently drop data if DataStreamer
crashes with an unchecked exception (rushabhs via cmccabe)
HDFS-7383. DataNode.requestShortCircuitFdsForRead may throw
NullPointerException. (szetszwo via suresh)
HDFS-7147. Update archival storage user documentation.
(Tsz Wo Nicholas Sze via wheat9)
HDFS-7340. Make rollingUpgrade start/finalize idempotent. (jing9)
HDFS-7334. Fix periodic failures of TestCheckpoint
#testTooManyEditReplayFailures. (Charles Lamb via wheat9)
HDFS-7355. TestDataNodeVolumeFailure#testUnderReplicationAfterVolFailure
fails on Windows, because we cannot deny access to the file owner.
(Chris Nauroth via wheat9)
HDFS-7367. HDFS short-circuit read cannot negotiate shared memory slot and
file descriptors when SASL is enabled on DataTransferProtocol.
(Chris Nauroth via wheat9)
HDFS-7364. Balancer always shows zero Bytes Already Moved.
(Tsz Wo Nicholas Sze via jing9)
HDFS-7379. TestBalancer#testBalancerWithRamDisk creates test files
incorrectly. (Xiaoyu Yao via wheat9)
BREAKDOWN OF HDFS-6134 AND HADOOP-10150 SUBTASKS AND RELATED JIRAS
HDFS-6387. HDFS CLI admin tool for creating & deleting an
encryption zone. (clamb)
HDFS-6386. HDFS Encryption Zones (clamb)
HDFS-6388. HDFS integration with KeyProvider. (clamb)
HDFS-6473. Protocol and API for Encryption Zones (clamb)
HDFS-6392. Wire crypto streams for encrypted files in
DFSClient. (clamb and yliu)
HDFS-6476. Print out the KeyProvider after finding KP successfully on
startup. (Juan Yu via wang)
HDFS-6391. Get the Key/IV from the NameNode for encrypted files in
DFSClient. (Charles Lamb and wang)
HDFS-6389. Rename restrictions for encryption zones. (clamb)
HDFS-6605. Client server negotiation of cipher suite. (wang)
HDFS-6625. Remove the Delete Encryption Zone function (clamb)
HDFS-6516. List of Encryption Zones should be based on inodes (clamb)
HDFS-6629. Not able to create symlinks after HDFS-6516 (umamaheswararao)
HDFS-6635. Refactor encryption zone functionality into new
EncryptionZoneManager class. (wang)
HDFS-6474. Namenode needs to get the actual keys and iv from the
KeyProvider. (wang)
HDFS-6619. Clean up encryption-related tests. (wang)
HDFS-6405. Test Crypto streams in HDFS. (yliu via wang)
HDFS-6490. Fix the keyid format for generated keys in
FSNamesystem.createEncryptionZone (clamb)
HDFS-6716. Update usage of KeyProviderCryptoExtension APIs on NameNode.
(wang)
HDFS-6718. Remove EncryptionZoneManager lock. (wang)
HDFS-6720. Remove KeyProvider in EncryptionZoneManager. (wang)
HDFS-6738. Remove unnecessary getEncryptionZoneForPath call in
EZManager#createEncryptionZone. (clamb)
HDFS-6724. Decrypt EDEK before creating
CryptoInputStream/CryptoOutputStream. (wang)
HDFS-6509. Create a special /.reserved/raw directory for raw access to
encrypted data. (clamb via wang)
HDFS-6771. Require specification of an encryption key when creating
an encryption zone. (wang)
HDFS-6730. Create a .RAW extended attribute namespace. (clamb)
HDFS-6692. Add more HDFS encryption tests. (wang)
HDFS-6780. Batch the encryption zones listing API. (wang)
HDFS-6394. HDFS encryption documentation. (wang)
HDFS-6834. Improve the configuration guidance in DFSClient when there
are no Codec classes found in configs. (umamahesh)
HDFS-6546. Add non-superuser capability to get the encryption zone
for a specific path. (clamb)
HDFS-6733. Creating encryption zone results in NPE when
KeyProvider is null. (clamb)
HDFS-6785. Should not be able to create encryption zone using path
to a non-directory file. (clamb)
HDFS-6807. Fix TestReservedRawPaths. (clamb)
HDFS-6814. Mistakenly dfs.namenode.list.encryption.zones.num.responses configured
as boolean. (umamahesh)
HDFS-6817. Fix findbugs and other warnings. (yliu)
HDFS-6839. Fix TestCLI to expect new output. (clamb)
HDFS-6954. With crypto, no native lib systems are too verbose. (clamb via wang)
HDFS-2975. Rename with overwrite flag true can make NameNode to stuck in safemode
on NN (crash + restart). (Yi Liu via umamahesh)
HDFS-6905. fs-encryption merge triggered release audit failures. (clamb via tucu)
HDFS-6714. TestBlocksScheduledCounter#testBlocksScheduledCounter should
shutdown cluster (vinayakumarb)
HDFS-6986. DistributedFileSystem must get delegation tokens from configured
KeyProvider. (zhz via tucu)
HDFS-6776. Using distcp to copy data between insecure and secure cluster via webdhfs
doesn't work. (yzhangal via tucu)
HDFS-7042. Upgrade fails for Windows HA cluster due to file locks held during
rename in JournalNode. (cnauroth)
HDFS-7051. TestDataNodeRollingUpgrade#isBlockFileInPrevious assumes Unix file
path separator. (cnauroth)
HDFS-7105. Fix TestJournalNode#testFailToStartWithBadConfig to match log
output change. (Ray Chiang via cnauroth)
HDFS-7105. Allow falling back to a non-SASL connection on
DataTransferProtocol in several edge cases. (cnauroth)
HDFS-7107. Avoid Findbugs warning for synchronization on
AbstractNNFailoverProxyProvider#fallbackToSimpleAuth. (cnauroth)
HDFS-7109. TestDataStorage does not release file locks between tests.
(cnauroth)
HDFS-7110. Skip tests related to short-circuit read on platforms that do not
currently implement short-circuit read. (cnauroth)
HDFS-7115. TestEncryptionZones assumes Unix path separator for KMS key store
path. (Xiaoyu Yao via cnauroth)
HDFS-7115. TestEncryptionZonesWithHA assumes Unix path separator for KMS key
store path. (Xiaoyu Yao via cnauroth)
HDFS-7130. TestDataTransferKeepalive fails intermittently on Windows.
(cnauroth)
HDFS-6534. Fix build on macosx: HDFS parts (Binglin Chang via aw)
HDFS-7111. TestSafeMode assumes Unix line endings in safe mode tip.
(cnauroth)
HDFS-7127. TestLeaseRecovery leaks MiniDFSCluster instances. (cnauroth)
HDFS-7131. During HA upgrade, JournalNode should create a new committedTxnId
file in the current directory. (jing9)
HDFS-6995. Block should be placed in the client's 'rack-local' node
if 'client-local' node is not available (vinayakumarb)
HDFS-7128. Decommission slows way down when it gets towards the end.
(Ming Ma via cnauroth)
HDFS-7287. The OfflineImageViewer (OIV) can output invalid XML depending on
the filename (Ravi Prakash via Colin P. McCabe)
HDFS-7300. The getMaxNodesPerRack() method in BlockPlacementPolicyDefault
is flawed (kihwal)
HDFS-7305. NPE seen in wbhdfs FS while running SLive. (jing9)
HDFS-7199. DFSOutputStream should not silently drop data if DataStreamer
crashes with an unchecked exception (rushabhs via cmccabe)
BREAKDOWN OF HDFS-6584 ARCHIVAL STORAGE
HDFS-6677. Change INodeFile and FSImage to support storage policy ID.
(szetszwo)
HDFS-6670. Add block storage policy support with default HOT, WARM and COLD
policies. (szetszwo)
HDFS-6671. Change BlockPlacementPolicy to consider block storage policy
in replicaiton. (szetszwo)
HDFS-6710. Change BlockPlacementPolicy to consider block storage policy
in replica deletion. (szetszwo)
HDFS-6679. Bump NameNodeLayoutVersion and update editsStored test files.
(vinayakumarb via szetszwo)
HDFS-6686. Change BlockPlacementPolicy to use fallback when some storage
types are unavailable. (szetszwo)
HDFS-6835. Add a new API to set storage policy. (jing9)
HDFS-6847. Support storage policy on directories and include storage policy
in HdfsFileStatus. (Jing Zhao via szetszwo)
HDFS-6801. Add a new data migration tool, Mover, for archiving data.
(szetszwo via jing9)
HDFS-6863. Support migration for snapshot paths. (jing9)
HDFS-6906. Add more tests for BlockStoragePolicy. (szetszwo via jing9)
HDFS-6911. check if a block is already scheduled in Mover.
(szetszwo via jing9)
HDFS-6920. Check the storage type of delNodeHintStorage when deleting
a replica. (szetszwo via jing9)
HDFS-6944. Add retry and termination logic for Mover. (jing9)
HDFS-6969. INode#getStoragePolicyID should always return the latest
storage policy. (jing9)
HDFS-6961. BlockPlacementPolicy#chooseTarget should check each valid
storage type in each choosing round. (jing9)
HDFS-6876. support set/get storage policy in DFSAdmin. (jing9)
HDFS-6997. Add more tests for data migration and replicaion. (szetszwo)
HDFS-6875. Support migration for a list of specified paths. (jing9)
HDFS-7027. Mover does not terminate when some storage type is out of space.
(szetszwo via jing9)
HDFS-7029. Fix TestDFSInotifyEventInputStream and TestDistributedFileSystem.
(szetszwo via jing9)
HDFS-7028. FSDirectory should not get storage policy id from symlinks.
(szetszwo)
HDFS-7034. Fix TestBlockPlacement and TestStorageMover. (jing9)
HDFS-7039. Fix Balancer tests. (szetszwo via jing9)
HDFS-7062. Skip under construction block for migration. (jing9)
HDFS-7052. Add Mover into hdfs script. (jing9)
HDFS-7072. Fix TestBlockManager and TestStorageMover. (jing9 via szetszwo)
HDFS-6864. Archival Storage: add user documentation. (szetszwo via jing9)
HDFS-7088. Archival Storage: fix TestBalancer and
TestBalancerWithMultipleNameNodes. (szetszwo via jing9)
HDFS-7095. TestStorageMover often fails in Jenkins. (jing9)
HDFS-7081. Add new DistributedFileSystem API for getting all the existing
storage policies. (jing9)
HDFS-7140. Add a tool to list all the existing block storage policies.
(jing9)
HDFS-7167. NPE while running Mover if the given path is for a file. (jing9)
HDFS-7154. Fix returning value of starting reconfiguration task (Lei Xu via
Colin P. McCabe)
BREAKDOWN OF HDFS-6581 SUBTASKS AND RELATED JIRAS
HDFS-6921. Add LazyPersist flag to FileStatus. (Arpit Agarwal)
HDFS-6924. Add new RAM_DISK storage type. (Arpit Agarwal)
HDFS-6922. Add LazyPersist flag to INodeFile, save it in FsImage and
edit logs. (Arpit Agarwal)
HDFS-6923. Propagate LazyPersist flag to DNs via DataTransferProtocol.
(Arpit Agarwal)
HDFS-6925. DataNode should attempt to place replicas on transient storage
first if lazyPersist flag is received. (Arpit Agarwal)
HDFS-6926. DN support for saving replicas to persistent storage and
evicting in-memory replicas. (Arpit Agarwal)
HDFS-6927. Initial unit tests for lazy persist files. (Arpit Agarwal)
HDFS-6929. NN periodically unlinks lazy persist files with missing
replicas from namespace. (Arpit Agarwal)
HDFS-6928. 'hdfs put' command should accept lazyPersist flag for testing.
(Arpit Agarwal)
HDFS-6960. Bugfix in LazyWriter, fix test case and some refactoring.
(Arpit Agarwal)
HDFS-6931. Move lazily persisted replicas to finalized directory on DN
startup. (Arpit Agarwal)
HDFS-6950. Add Additional unit tests for HDFS-6581. (Xiaoyu Yao via
Arpit Agarwal)
HDFS-6930. Improve replica eviction from RAM disk. (Arpit Agarwal)
HDFS-6977. Delete all copies when a block is deleted from the block space.
(Arpit Agarwal)
HDFS-6991. Notify NN of evicted block before deleting it from RAM disk.
(Arpit Agarwal)
HDFS-6978. Directory scanner should correctly reconcile blocks on RAM
disk. (Arpit Agarwal)
HDFS-7066. LazyWriter#evictBlocks misses a null check for replicaState.
(Xiaoyu Yao via Arpit Agarwal)
HDFS-7064. Fix unit test failures in HDFS-6581 branch. (Xiaoyu Yao via
Arpit Agarwal)
HDFS-6581. Few more unit test fixes for HDFS-6581. (Arpit Agarwal)
HDFS-7080. Fix finalize and upgrade unit test failures. (Arpit Agarwal)
HDFS-7084. FsDatasetImpl#copyBlockFiles debug log can be improved.
(Xiaoyu Yao via Arpit Agarwal)
HDFS-7091. Add forwarding constructor for INodeFile for existing callers.
(Arpit Agarwal)
HDFS-7100. Make eviction scheme pluggable. (Arpit Agarwal)
HDFS-7108. Fix unit test failures in SimulatedFsDataset. (Arpit Agarwal)
HDFS-7071. Updated editsStored and editsStored.xml to bump layout
version and add LazyPersist flag. (Xiaoyu Yao and Arpit Agarwal via
Arpit Agarwal)
HDFS-6990. Add unit test for evict/delete RAM_DISK block with open
handle. (Xiaoyu Yao via Arpit Agarwal)
HDFS-7143. Fix findbugs warnings in HDFS-6581 branch. (szetszwo via
Arpit Agarwal)
HDFS-6932. Balancer and Mover tools should ignore replicas on RAM_DISK.
(Xiaoyu Yao via Arpit Agarwal)
HDFS-7144. Fix findbugs warnings in RamDiskReplicaTracker. (szetszwo via
Arpit Agarwal)
HDFS-7155. Bugfix in createLocatedFileStatus caused by bad merge.
(Arpit Agarwal)
HDFS-7153. Add storagePolicy to NN edit log during file creation.
(Arpit Agarwal)
HDFS-7159. Use block storage policy to set lazy persist preference.
(Arpit Agarwal)
HDFS-7129. Metrics to track usage of memory for writes. (Xiaoyu Yao
via Arpit Agarwal)
HDFS-7171. Fix Jenkins failures in HDFS-6581 branch. (Arpit Agarwal)
HDFS-7112. LazyWriter should use either async IO or one thread per physical
disk. (Xiaoyu Yao via cnauroth)
HDFS-7090. Use unbuffered writes when persisting in-memory replicas.
(Xiaoyu Yao via cnauroth)
HDFS-6934. Move checksum computation off the hot path when writing to RAM
disk. (cnauroth)
HDFS-7291. Persist in-memory replicas with appropriate unbuffered copy API
on POSIX and Windows. (Xiaoyu Yao via cnauroth)
HDFS-7328. TestTraceAdmin assumes Unix line endings. (cnauroth)
HDFS-7359. NameNode in secured HA cluster fails to start if
dfs.namenode.secondary.http-address cannot be interpreted as a network
address. (cnauroth)
HDFS-7226. Fix TestDNFencing.testQueueingWithAppend. (Yongjun Zhang via jing9)
HDFS-7382. DataNode in secure mode may throw NullPointerException if client
connects before DataNode registers itself with NameNode. (cnauroth)
HDFS-7387. NFS may only do partial commit due to a race between COMMIT and write
(brandonli)
HDFS-7391. Renable SSLv2Hello in HttpFS. (rkanter via acmurthy)
HDFS-7385. ThreadLocal used in FSEditLog class causes FSImage permission mess
up. (jiangyu via cnauroth)
Release 2.5.2 - 2014-11-19
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
OPTIMIZATIONS
BUG FIXES
HDFS-7274. Disable SSLv3 in HttpFS. (Robert Kanter via kasha)
Release 2.5.1 - 2014-09-05
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
OPTIMIZATIONS
BUG FIXES
HADOOP-10957. The globber will sometimes erroneously return a permission
denied exception when there is a non-terminal wildcard (cmccabe)
Release 2.5.0 - 2014-08-11
INCOMPATIBLE CHANGES
HDFS-6168. Remove a deprecated constructor and the deprecated methods reportChecksumFailure,
getDelegationToken(Text), renewDelegationToken and cancelDelegationToken from
DistributedFileSystem. (szetszwo)
NEW FEATURES
HDFS-6281. Provide option to use the NFS Gateway without having to use the
Hadoop portmapper. (atm)
HDFS-5168. Add cross node dependency support to BlockPlacementPolicy.
(Nikola Vujic via szetszwo)
HDFS-6334. Client failover proxy provider for IP failover based NN HA.
(kihwal)
HDFS-6406. Add capability for NFS gateway to reject connections from
unprivileged ports. (atm)
HDFS-2006. Ability to support storing extended attributes per file.
HDFS-5978. Create a tool to take fsimage and expose read-only WebHDFS API.
(Akira Ajisaka via wheat9)
HDFS-6278. Create HTML5-based UI for SNN. (wheat9)
HDFS-6279. Create new index page for JN / DN. (wheat9)
IMPROVEMENTS
HDFS-6007. Update documentation about short-circuit local reads (iwasakims
via cmccabe)
HDFS-6125. Cleanup unnecessary cast in HDFS code base. (suresh)
HDFS-5196. Provide more snapshot information in WebUI.
(Shinichi Yamashita via wheat9)
HDFS-6155. Fix Boxing/unboxing to parse a primitive findbugs warnings.
(suresh)
HDFS-6119. FSNamesystem code cleanup. (suresh)
HDFS-6158. Clean up dead code for OfflineImageViewer. (wheat9)
HDFS-6164. Remove lsr in OfflineImageViewer. (wheat9)
HDFS-6167. Relocate the non-public API classes in the hdfs.client package.
(szetszwo)
HDFS-6191. Disable quota checks when replaying edit log. (kihwal)
HDFS-6170. Support GETFILESTATUS operation in WebImageViewer.
(Akira Ajisaka via wheat9)
HDFS-6225. Remove the o.a.h.hdfs.server.common.UpgradeStatusReport.
(wheat9)
HDFS-6224. Add a unit test to TestAuditLogger for file permissions
passed to logAuditEvent. (Charles Lamb via wang)
HDFS-6194. Create new tests for ByteRangeInputStream.
(Akira Ajisaka via wheat9)
HDFS-6219. Proxy superuser configuration should use true client IP for
address checks. (daryn via kihwal)
HDFS-6256. Clean up ImageVisitor and SpotCheckImageVisitor.
(Akira Ajisaka via wheat9)
HDFS-6265. Prepare HDFS codebase for JUnit 4.11. (cnauroth)
HDFS-5693. Few NN metrics data points were collected via JMX when NN
is under heavy load. (Ming Ma via jing9)
HDFS-6273. Config options to allow wildcard endpoints for namenode HTTP
and HTTPS servers. (Arpit Agarwal)
HDFS-6282. Re-add testIncludeByRegistrationName. (cmccabe)
HDFS-6266. Identify full path for a given INode. (jing9)
HDFS-6210. Support GETACLSTATUS operation in WebImageViewer.
(Akira Ajisaka via wheat9)
HDFS-6269. NameNode Audit Log should differentiate between webHDFS open and
HDFS open. (Eric Payne via jeagles)
HDFS-6304. Consolidate the logic of path resolution in FSDirectory.
(wheat9)
HDFS-6295. Add "decommissioning" state and node state filtering to
dfsadmin. (wang)
HDFS-6294. Use INode IDs to avoid conflicts when a file open for write is
renamed. (cmccabe)
HDFS-6328. Clean up dead code in FSDirectory. (wheat9)
HDFS-6230. Expose upgrade status through NameNode web UI.
(Mit Desai via wheat9)
HDFS-6186. Pause deletion of blocks when the namenode starts up. (jing9)
HDFS-6293. Issues with OIV processing PB-based fsimages. (kihwal)
HDFS-2949. Add check to active state transition to prevent operator-induced
split brain. (Rushabh S Shah via kihwal)
HDFS-6287. Add vecsum test of libhdfs read access times (cmccabe)
HDFS-5683. Better audit log messages for caching operations.
(Abhiraj Butala via wang)
HDFS-6345. DFS.listCacheDirectives() should allow filtering based on
cache directive ID. (wang)
HDFS-6432. Add snapshot related APIs to webhdfs. (jing9)
HDFS-6396. Remove support for ACL feature from INodeSymlink.
(Charles Lamb via wang)
HDFS-6435. Add support for specifying a static uid/gid mapping for the NFS
gateway. (atm via wang)
HDFS-6110 adding more slow action log in critical write path
(Liang Xie via stack)
HDFS-6416. Use Time#monotonicNow in OpenFileCtx and OpenFileCtxCatch to
avoid system clock bugs (Abhiraj Butala via brandonli)
HDFS-6356. Fix typo in DatanodeLayoutVersion. (Tulasi G via wang)
HDFS-6447. balancer should timestamp the completion message.
(Juan Yu via wang)
HDFS-6463. Clarify behavior of AclStorage#createFsPermissionForExtendedAcl
in comments. (cnauroth)
HDFS-6472. fix typo in webapps/hdfs/explorer.js. (Juan Yu via wang)
HDFS-6056. Clean up NFS config settings (brandonli)
HDFS-6109 let sync_file_range() system call run in background
(Liang Xie via stack)
HDFS-6268. Better sorting in NetworkTopology#pseudoSortByDistance when
no local node is found. (wang)
HDFS-6369. Document that BlockReader#available() can return more bytes than
are remaining in the block (Ted Yu via Colin Patrick McCabe)
HDFS-6487. TestStandbyCheckpoint#testSBNCheckpoints is racy.
(Mit Desai via wang)
HDFS-6297. Add CLI testcases to reflect new features of dfs and dfsadmin
(Dasha Boudnik via cos)
HDFS-6399. Add note about setfacl in HDFS permissions guide.
(cnauroth via wang)
HDFS-6315. Decouple recording edit logs from FSDirectory. (wheat9)
HDFS-6379. HTTPFS - Implement ACLs support. (yoderme via tucu)
HDFS-6471. Make moveFromLocal CLI testcases to be non-disruptive
(Dasha Boudnik via cos)
HDFS-6395. Skip checking xattr limits for non-user-visible namespaces.
(Yi Liu via wang).
HDFS-3493. Invalidate excess corrupted blocks as long as minimum
replication is satisfied. (Juan Yu and Vinayakumar B via wang)
HDFS-6330. Move mkdirs() to FSNamesystem. (wheat9)
HDFS-6470. TestBPOfferService.testBPInitErrorHandling is flaky.
(Ming Ma via wang)
HDFS-6529. Trace logging for RemoteBlockReader2 to identify remote datanode
and file being read. (Anubhav Dhoot via atm)
HDFS-6499. Use NativeIO#renameTo instead of File#renameTo in
FileJournalManager. (Yongjun Zhang via atm)
HDFS-6518. TestCacheDirectives#testExceedsCapacity should
take FSN read lock when accessing pendingCached list.
(wang)
HDFS-6528. Add XAttrs to TestOfflineImageViewer. (Stephen Chu via wang)
HDFS-6545. Finalizing rolling upgrade can make NN unavailable for a long
duration. (kihwal)
HDFS-6530. Fix Balancer documentation. (szetszwo)
HDFS-6480. Move waitForReady() from FSDirectory to FSNamesystem. (wheat9)
HDFS-6403. Add metrics for log warnings reported by JVM pauses. (Yongjun
Zhang via atm)
HDFS-6557. Move the reference of fsimage to FSNamesystem. (wheat9)
HDFS-4667. Capture renamed files/directories in snapshot diff report. (jing9
and Binglin Chang via jing9)
HDFS-6507. Improve DFSAdmin to support HA cluster better.
(Zesheng Wu via vinayakumarb)
HDFS-6578. add toString method to DatanodeStorage for easier debugging.
(Yongjun Zhang via Arpit Agarwal)
HDFS-6562. Refactor rename() in FSDirectory. (wheat9)
HDFS-6486. Add user doc for XAttrs via WebHDFS. (Yi Liu via umamahesh)
HDFS-6430. HTTPFS - Implement XAttr support. (Yi Liu via tucu)
HDFS-6593. Move SnapshotDiffInfo out of INodeDirectorySnapshottable.
(Jing Zhao via wheat9)
HDFS-6595. Allow the maximum threads for balancing on datanodes to be
configurable. (Benoy Antony via szetszwo)
HDFS-6572. Add an option to the NameNode that prints the software and
on-disk image versions. (Charles Lamb via cnauroth)
HDFS-6603. Add XAttr with ACL test. (Stephen Chu via cnauroth)
HDFS-6612. MiniDFSNNTopology#simpleFederatedTopology(int)
always hardcode nameservice ID. (Juan Yu via wang)
HDFS-6614. shorten TestPread run time with a smaller retry timeout setting.
(Liang Xie via cnauroth)
HDFS-6610. TestShortCircuitLocalRead tests sometimes timeout on slow
machines. (Charles Lamb via wang)
HDFS-6620. Snapshot docs should specify about preserve options with cp command
(Stephen Chu via umamahesh)
HDFS-6493. Change dfs.namenode.startup.delay.block.deletion to second
instead of millisecond. (Juan Yu via wang)
HDFS-6680. BlockPlacementPolicyDefault does not choose favored nodes
correctly. (szetszwo)
HDFS-6712. Document HDFS Multihoming Settings. (Arpit Agarwal)
OPTIMIZATIONS
HDFS-6214. Webhdfs has poor throughput for files >2GB (daryn)
HDFS-6460. Ignore stale and decommissioned nodes in
NetworkTopology#sortByDistance. (Yongjun Zhang via wang)
HDFS-6583. Remove clientNode in FileUnderConstructionFeature. (wheat9)
HDFS-6599. 2.4 addBlock is 10 to 20 times slower compared to 0.23 (daryn)
BUG FIXES
HDFS-6112. NFS Gateway docs are incorrect for allowed hosts configuration.
(atm)
HDFS-3087. Decomissioning on NN restart can complete without blocks being
replicated. (Rushabh S Shah via kihwal)
HDFS-6162. Format strings should use platform independent line separator.
(suresh)
HDFS-6156. Simplify the JMX API that provides snapshot information.
(wheat9)
HDFS-6173. Move the default processor from Ls to Web in OfflineImageViewer.
(Akira Ajisaka via wheat9)
HDFS-5591. Checkpointing should use monotonic time when calculating period.
(Charles Lamb via wang)
HDFS-6190. Minor textual fixes in DFSClient. (Charles Lamb via wheat9)
HDFS-6159. TestBalancerWithNodeGroup.testBalancerWithNodeGroup fails if
there is block missing after balancer success (Chen He via kihwal)
HDFS-6181. Fix the wrong property names in NFS user guide (brandonli)
HDFS-6180. dead node count / listing is very broken in JMX and old GUI.
(wheat9)
HDFS-6143. WebHdfsFileSystem open should throw FileNotFoundException for
non-existing paths. (Gera Shegalov via wheat9)
HDFS-6169. Move the address in WebImageViewer. (Akira Ajisaka via wheat9)
HDFS-6160. TestSafeMode occasionally fails. (Arpit Agarwal)
HDFS-5669. Storage#tryLock() should check for null before logging successfull message
(Vinayakumar B via umamahesh)
HDFS-6237. TestDFSShell#testGet fails on Windows due to invalid file system
path. (cnauroth)
HDFS-6238. TestDirectoryScanner leaks file descriptors. (cnauroth)
HDFS-6243. HA NameNode transition to active or shutdown may leave lingering
image transfer thread. (cnauroth)
HDFS-5409. TestOfflineEditsViewer#testStored fails on Windows due to CRLF
line endings in editsStored.xml from git checkout. (cnauroth)
HDFS-4909. Avoid protocol buffer RPC namespace clashes. (cmccabe)
HDFS-6178. Decommission on standby NN couldn't finish. (Ming Ma via jing9)
HDFS-6213. TestDataNodeConfig failing on Jenkins runs due to DN web port
in use. (wang)
HDFS-6274. Cleanup javadoc warnings in HDFS code. (suresh)
HDFS-6275. Fix warnings - type arguments can be inferred and redudant
local variable. (suresh)
HDFS-6217. Webhdfs PUT operations may not work via a http proxy.
(Daryn Sharp via kihwal)
HDFS-6276. Remove unnecessary conditions and null check. (suresh)
HDFS-5865. Update OfflineImageViewer document. (Akira Ajisaka via wheat9)
HDFS-6270. Secondary namenode status page shows transaction count in bytes.
(Benoy Antony via wheat9)
HDFS-6218. Audit log should use true client IP for proxied webhdfs
operations. (daryn via kihwal)
HDFS-6288. DFSInputStream Pread doesn't update ReadStatistics.
(Juan Yu via wang)
HDFS-5892. TestDeleteBlockPool fails in branch-2. (Ted Yu via wheat9)
HDFS-6289. HA failover can fail if there are pending DN messages for DNs
which no longer exist. (atm)
HDFS-6337. Setfacl testcase is failing due to dash character in username
in TestAclCLI (umamahesh)
HDFS-5381. ExtendedBlock#hashCode should use both blockId and block pool ID
(Benoy Antony via Colin Patrick McCabe)
HDFS-6240. WebImageViewer returns 404 if LISTSTATUS to an empty directory.
(Akira Ajisaka via wheat9)
HDFS-6351. Command hdfs dfs -rm -r can't remove empty directory.
(Yongjun Zhang via wang)
HDFS-5522. Datanode disk error check may be incorrectly skipped.
(Rushabh S Shah via kihwal)
HDFS-6367. EnumSetParam$Domain#parse fails for parameter containing more than one enum.
(Yi Liu via umamahesh)
HDFS-6305. WebHdfs response decoding may throw RuntimeExceptions (Daryn
Sharp via jeagles)
HDFS-6355. Fix divide-by-zero, improper use of wall-clock time in
BlockPoolSliceScanner (cmccabe)
HDFS-6370. Web UI fails to display in intranet under IE.
(Haohui Mai via cnauroth)
HDFS-6381. Fix a typo in INodeReference.java. (Binglin Chang via jing9)
HDFS-6400. Cannot execute hdfs oiv_legacy. (Akira AJISAKA via kihwal)
HDFS-6250. Fix test failed in TestBalancerWithNodeGroup.testBalancerWithRackLocality
(Binglin Chang and Chen He via junping_du)
HDFS-4913. Deleting file through fuse-dfs when using trash fails requiring
root permissions (cmccabe)
HDFS-6421. Fix vecsum.c compile on BSD and some other systems. (Mit Desai
via Colin Patrick McCabe)
HDFS-6419. TestBookKeeperHACheckpoints#TestSBNCheckpoints fails on trunk.
(Akira AJISAKA via kihwal)
HDFS-6409. Fix typo in log message about NameNode layout version upgrade.
(Chen He via cnauroth)
HDFS-6433. Replace BytesMoved class with AtomicLong.
(Benoy Antony via cnauroth)
HDFS-6438. DeleteSnapshot should be a DELETE request in WebHdfs. (jing9)
HDFS-6423. Diskspace quota usage should be updated when appending data to
partial block. (jing9)
HDFS-6443. Fix MiniQJMHACluster related test failures. (Zesheng Wu via
Arpit Agarwal)
HDFS-6227. ShortCircuitCache#unref should purge ShortCircuitReplicas whose
streams have been closed by java interrupts. (Colin Patrick McCabe via jing9)
HDFS-6442. Fix TestEditLogAutoroll and TestStandbyCheckpoints failure
caused by port conficts. (Zesheng Wu via Arpit Agarwal)
HDFS-6448. BlockReaderLocalLegacy should set socket timeout based on
conf.socketTimeout (liangxie via cmccabe)
HDFS-6453. Use Time#monotonicNow to avoid system clock reset.
(Liang Xie via wang)
HDFS-6461. Use Time#monotonicNow to compute duration in DataNode#shutDown.
(James Thomas via wang)
HDFS-6462. NFS: fsstat request fails with the secure hdfs (brandonli)
HDFS-6404. HttpFS should use a 000 umask for mkdir and create
operations. (yoderme via tucu)
HDFS-6424. blockReport doesn't need to invalidate blocks on SBN. (Ming Ma
via jing9)
HDFS-6497. Make TestAvailableSpaceVolumeChoosingPolicy deterministic
(cmccabe)
HDFS-6500. Snapshot shouldn't be removed silently after renaming to an
existing snapshot. (Nicholas SZE via junping_du)
HDFS-6257. TestCacheDirectives#testExceedsCapacity fails occasionally
(cmccabe)
HDFS-6364. Incorrect check for unknown datanode in Balancer. (Benoy
Antony via Arpit Agarwal)
HDFS-6503. Fix typo of DFSAdmin restoreFailedStorage.
(Zesheng Wu via wheat9)
HDFS-6464. Support multiple xattr.name parameters for WebHDFS getXAttrs.
(Yi Liu via umamahesh)
HDFS-6539. test_native_mini_dfs is skipped in hadoop-hdfs/pom.xml
(decstery via cmccabe)
HDFS-6527. Edit log corruption due to defered INode removal. (kihwal and
jing9 via jing9)
HDFS-6552. add DN storage to a BlockInfo will not replace the different
storage from same DN. (Amir Langer via Arpit Agarwal)
HDFS-6551. Rename with OVERWRITE option may throw NPE when the target
file/directory is a reference INode. (jing9)
HDFS-6439. NFS should not reject NFS requests to the NULL procedure whether
port monitoring is enabled or not. (brandonli)
HDFS-6559. Fix wrong option "dfsadmin -rollingUpgrade start" in the
document. (Akira Ajisaka via Arpit Agarwal)
HDFS-6553. Add missing DeprecationDeltas for NFS Kerberos configurations
(Stephen Chu via brandonli)
HDFS-6563. NameNode cannot save fsimage in certain circumstances when
snapshots are in use. (atm)
HDFS-3848. A Bug in recoverLeaseInternal method of FSNameSystem class
(Hooman Peiro Sajjad and Chen He via kihwal)
HDFS-6549. Add support for accessing the NFS gateway from the AIX NFS
client. (atm)
HDFS-6535. HDFS quota update is wrong when file is appended. (George Wong
via jing9)
HDFS-6222. Remove background token renewer from webhdfs.
(Rushabh Shah and Daryn Sharp via cnauroth)
HDFS-6580. FSNamesystem.mkdirsInt should call the getAuditFileInfo()
wrapper. (Zhilei Xu via wheat9)
HDFS-6587. Bug in TestBPOfferService can cause test failure. (Zhilei Xu
via Arpit Agarwal)
HDFS-6598. Fix a typo in message issued from explorer.js. (Yongjun Zhang
via wheat9)
HDFS-6475. WebHdfs clients fail without retry because incorrect handling
of StandbyException. (Yongjun Zhang via atm)
HADOOP-10701. NFS should not validate the access premission only based on
the user's primary group (Harsh J via atm)
HDFS-6556. Refine XAttr permissions (umamahesh)
HDFS-6601. Issues in finalizing rolling upgrade when there is a layout
version change (kihwal)
HDFS-6418. Regression: DFS_NAMENODE_USER_NAME_KEY missing
(szetszwo via stevel)
HDFS-6558. Missing newline in the description of dfsadmin -rollingUpgrade.
(Chen He via kihwal)
HDFS-6591. while loop is executed tens of thousands of times in Hedged Read
(Liang Xie via cnauroth)
HDFS-6604. The short-circuit cache doesn't correctly time out replicas that
haven't been used in a while (cmccabe)
HDFS-4286. Changes from BOOKKEEPER-203 broken capability of including
bookkeeper-server jar in hidden package of BKJM (Rakesh R via umamahesh)
HDFS-4221. Remove the format limitation point from BKJM documentation as HDFS-3810
closed. (Rakesh R via umamahesh)
HDFS-5411. Update Bookkeeper dependency to 4.2.3. (Rakesh R via umamahesh)
HDFS-6631. TestPread#testHedgedReadLoopTooManyTimes fails intermittently.
(Liang Xie via cnauroth)
HDFS-6647. Edit log corruption when pipeline recovery occurs for deleted
file present in snapshot (kihwal)
HDFS-6378. NFS registration should timeout instead of hanging when
portmap/rpcbind is not available (Abhiraj Butala via brandonli)
HDFS-6632. Reintroduce dfs.http.port / dfs.https.port in branch-2.
(Yongjun Zhang via wheat9)
HDFS-6703. NFS: Files can be deleted from a read-only mount
(Srikanth Upputuri via brandonli)
HDFS-6422. getfattr in CLI doesn't throw exception or return non-0 return code
when xattr doesn't exist. (Charles Lamb via umamahesh)
HDFS-6696. Name node cannot start if the path of a file under
construction contains ".snapshot". (wang)
HDFS-6312. WebHdfs HA failover is broken on secure clusters.
(daryn via tucu)
HDFS-6618. FSNamesystem#delete drops the FSN lock between removing INodes
from the tree and deleting them from the inode map (kihwal via cmccabe)
HDFS-6622. Rename and AddBlock may race and produce invalid edits (kihwal
via cmccabe)
HDFS-6723. New NN webUI no longer displays decommissioned state for dead node.
(Ming Ma via wheat9)
HDFS-6768. Fix a few unit tests that use hard-coded port numbers. (Arpit
Agarwal)
HDFS-6793. Missing changes in HftpFileSystem when Reintroduce
dfs.http.port / dfs.https.port in branch-2. (Juan Yu via wang).
BREAKDOWN OF HDFS-2006 SUBTASKS AND RELATED JIRAS
HDFS-6299. Protobuf for XAttr and client-side implementation. (Yi Liu via umamahesh)
HDFS-6302. Implement XAttr as a INode feature. (Yi Liu via umamahesh)
HDFS-6309. Javadocs for Xattrs apis in DFSClient and other minor fixups. (Charles Lamb via umamahesh)
HDFS-6258. Namenode server-side storage for XAttrs. (Yi Liu via umamahesh)
HDFS-6303. HDFS implementation of FileContext API for XAttrs. (Yi Liu and Charles Lamb via umamahesh)
HDFS-6324. Shift XAttr helper code out for reuse. (Yi Liu via umamahesh)
HDFS-6301. NameNode: persist XAttrs in fsimage and record XAttrs modifications to edit log.
(Yi Liu via umamahesh)
HDFS-6298. XML based End-to-End test for getfattr and setfattr commands. (Yi Liu via umamahesh)
HDFS-6314. Test cases for XAttrs. (Yi Liu via umamahesh)
HDFS-6344. Maximum limit on the size of an xattr. (Yi Liu via umamahesh)
HDFS-6377. Unify xattr name and value limits into a single limit. (wang)
HDFS-6373. Remove support for extended attributes on symlinks. (Charles Lamb via wang)
HDFS-6283. Write end user documentation for xattrs. (wang)
HDFS-6412. Interface audience and stability annotations missing from
several new classes related to xattrs. (wang)
HDFS-6259. Support extended attributes via WebHDFS. (yliu)
HDFS-6346. Optimize OP_SET_XATTRS by persisting single Xattr entry per setXattr/removeXattr api call
(Yi Liu via umamahesh)
HDFS-6331. ClientProtocol#setXattr should not be annotated idempotent.
(umamahesh via wang)
HDFS-6335. TestOfflineEditsViewer for XAttr. (Yi Liu via umamahesh)
HDFS-6343. fix TestNamenodeRetryCache and TestRetryCacheWithHA failures. (umamahesh)
HDFS-6366. FsImage loading failed with RemoveXattr op (umamahesh)
HDFS-6357. SetXattr should persist rpcIDs for handling retrycache with Namenode restart and HA
(umamahesh)
HDFS-6372. Handle setXattr rpcIDs for OfflineEditsViewer. (umamahesh)
HDFS-6410. DFSClient unwraps AclException in xattr methods, but those
methods cannot throw AclException. (wang)
HDFS-6413. xattr names erroneously handled as case-insensitive.
(Charles Lamb via cnauroth)
HDFS-6414. xattr modification operations are based on state of latest
snapshot instead of current version of inode. (Andrew Wang via cnauroth)
HDFS-6374. setXAttr should require the user to be the owner of the file
or directory (Charles Lamb via wang)
HDFS-6492. Support create-time xattrs and atomically setting multiple
xattrs. (wang)
Release 2.4.1 - 2014-06-23
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
HDFS-6215. Wrong error message for upgrade. (Kihwal Lee via jeagles)
OPTIMIZATIONS
BUG FIXES
HDFS-6189. Multiple HDFS tests fail on Windows attempting to use a test
root path containing a colon. (cnauroth via szetszwo)
HDFS-6197. Rolling upgrade rollback on Windows can fail attempting to rename
edit log segment files to a destination that already exists. (cnauroth)
HDFS-6198. DataNode rolling upgrade does not correctly identify current
block pool directory and replace with trash on Windows. (cnauroth)
HDFS-6206. Fix NullPointerException in DFSUtil.substituteForWildcardAddress.
(szetszwo)
HDFS-6204. Fix TestRBWBlockInvalidation: change the last sleep to a loop.
(szetszwo)
HDFS-6209. TestValidateConfigurationSettings should use random ports.
(Arpit Agarwal via szetszwo)
HDFS-6208. DataNode caching can leak file descriptors. (cnauroth)
HDFS-6231. DFSClient hangs infinitely if using hedged reads and all eligible
datanodes die. (cnauroth)
HDFS-6234. TestDatanodeConfig#testMemlockLimit fails on Windows due to
invalid file path. (cnauroth)
HDFS-6235. TestFileJournalManager can fail on Windows due to file locking if
tests run out of order. (cnauroth)
HDFS-6229. Race condition in failover can cause RetryCache fail to work.
(jing9)
HDFS-6232. OfflineEditsViewer throws a NPE on edits containing ACL
modifications (ajisakaa via cmccabe)
HDFS-6236. ImageServlet should use Time#monotonicNow to measure latency.
(cnauroth)
HDFS-6245. datanode fails to start with a bad disk even when failed
volumes is set. (Arpit Agarwal)
HDFS-2882. DN continues to start up, even if block pool fails to initialize
(vinayakumarb)
HDFS-6340. DN can't finalize upgrade. (Rahul Singhal via Arpit Agarwal)
HDFS-6329. WebHdfs does not work if HA is enabled on NN but logical URI is
not configured. (kihwal)
HDFS-6313. WebHdfs may use the wrong NN when configured for multiple HA NNs
(kihwal)
HDFS-6326. WebHdfs ACL compatibility is broken. (cnauroth)
HDFS-6361. TestIdUserGroup.testUserUpdateSetting failed due to out of range
nfsnobody Id. (Yongjun Zhang via brandonli)
HDFS-4052. BlockManager#invalidateWork should print log outside the lock.
(Jing Zhao via suresh)
HDFS-6362. InvalidateBlocks is inconsistent in usage of DatanodeUuid and
StorageID. (Arpit Agarwal)
HDFS-6402. Suppress findbugs warning for failure to override equals and
hashCode in FsAclPermission. (cnauroth)
HDFS-6325. Append should fail if the last block has insufficient number of
replicas (Keith Pak via cos)
HDFS-6397. NN shows inconsistent value in deadnode count.
(Mohammad Kamrul Islam via kihwal)
HDFS-6411. nfs-hdfs-gateway mount raises I/O error and hangs when a
unauthorized user attempts to access it (brandonli)
Release 2.4.0 - 2014-04-07
INCOMPATIBLE CHANGES
NEW FEATURES
HDFS-5698. Use protobuf to serialize / deserialize FSImage. (See breakdown
of tasks below for features and contributors)
HDFS-5776 Support 'hedged' reads in DFSClient (Liang Xie via stack)
HDFS-4685. Implementation of ACLs in HDFS. (See breakdown of tasks below for
features and contributors)
IMPROVEMENTS
HDFS-5781. Use an array to record the mapping between FSEditLogOpCode and
the corresponding byte value. (jing9)
HDFS-5153. Datanode should send block reports for each storage in a
separate message. (Arpit Agarwal)
HDFS-5804. HDFS NFS Gateway fails to mount and proxy when using Kerberos.
(Abin Shahab via jing9)
HDFS-5859. DataNode#checkBlockToken should check block tokens even if
security is not enabled. (cmccabe)
HDFS-5746. Add ShortCircuitSharedMemorySegment (cmccabe)
HDFS-4911. Reduce PeerCache timeout to be commensurate with
dfs.datanode.socket.reuse.keepalive (cmccabe)
HDFS-4370. Fix typo Blanacer in DataNode. (Chu Tong via shv)
HDFS-5929. Add blockpool % usage to HDFS federated nn page.
(Siqi Li via suresh)
HDFS-5810. Unify mmap cache and short-circuit file descriptor cache
(cmccabe)
HDFS-5940. Minor cleanups to ShortCircuitReplica, FsDatasetCache, and
DomainSocketWatcher (cmccabe)
HDFS-5531. Combine the getNsQuota() and getDsQuota() methods in INode.
(szetszwo)
HDFS-5285. Flatten INodeFile hierarchy: Replace INodeFileUnderConstruction
and INodeFileUnderConstructionWithSnapshot with FileUnderContructionFeature.
(jing9 via szetszwo)
HDFS-5286. Flatten INodeDirectory hierarchy: Replace INodeDirectoryWithQuota
with DirectoryWithQuotaFeature. (szetszwo)
HDFS-5537. Remove FileWithSnapshot interface. (jing9 via szetszwo)
HDFS-5554. Flatten INodeFile hierarchy: Replace INodeFileWithSnapshot with
FileWithSnapshotFeature. (jing9 via szetszwo)
HDFS-5647. Merge INodeDirectory.Feature and INodeFile.Feature. (Haohui Mai
via jing9)
HDFS-5632. Flatten INodeDirectory hierarchy: Replace
INodeDirectoryWithSnapshot with DirectoryWithSnapshotFeature.
(jing9 via szetszwo)
HDFS-5715. Use Snapshot ID to indicate the corresponding Snapshot for a
FileDiff/DirectoryDiff. (jing9)
HDFS-5726. Fix compilation error in AbstractINodeDiff for JDK7. (jing9)
HDFS-5768. Consolidate the serialization code in DelegationTokenSecretManager
(Haohui Mai via brandonli)
HDFS-5775. Consolidate the code for serialization in CacheManager
(Haohui Mai via brandonli)
HDFS-5768. Consolidate the serialization code in DelegationTokenSecretManager
(Haohui Mai via brandonli)
HDFS-5973. add DomainSocket#shutdown method (cmccabe)
HDFS-5318. Support read-only and read-write paths to shared replicas.
(Eric Sirianni via Arpit Agarwal)
HDFS-5868. Make hsync implementation pluggable on the DataNode.
(Buddy Taylor via Arpit Agarwal)
HDFS-5935. New Namenode UI FS browser should throw smarter error messages.
(Travis Thompson via jing9)
HDFS-5939. WebHdfs returns misleading error code and logs nothing if trying
to create a file with no DNs in cluster. (Yongjun Zhang via jing9)
HDFS-6006. Remove duplicate code in FSNameSystem#getFileInfo.
(Akira Ajisaka via cnauroth)
HDFS-6018. Exception recorded in LOG when IPCLoggerChannel#close is called.
(jing9)
HDFS-3969. Small bug fixes and improvements for disk locations API.
(Todd Lipcon and Andrew Wang)
HDFS-6025. Update findbugsExcludeFile.xml. (szetszwo)
HDFS-6030. Remove an unused constructor in INode.java. (yzhang via
cmccabe)
HDFS-4200. Reduce the size of synchronized sections in PacketResponder.
(Suresh Srinivas, backported by Andrew Wang, committed by jing9)
HDFS-5950. The DFSClient and DataNode should use shared memory segments to
communicate short-circuit information. (cmccabe)
HDFS-6046. add dfs.client.mmap.enabled (cmccabe)
HDFS-5321. Clean up the HTTP-related configuration in HDFS (wheat9)
HDFS-5167. Add metrics about the NameNode retry cache. (Tsuyoshi OZAWA via
jing9)
HDFS-6043. Give HDFS daemons NFS3 and Portmap their own OPTS (brandonli)
HDFS-6044. Add property for setting the NFS look up time for users
(brandonli)
HDFS-6061. Allow dfs.datanode.shared.file.descriptor.path to contain
multiple entries and fall back when needed (cmccabe)
HDFS-5986. Capture the number of blocks pending deletion on namenode webUI.
(cnauroth)
HDFS-6070. Cleanup use of ReadStatistics in DFSInputStream. (wang)
HDFS-6055. Change default configuration to limit file name length in HDFS.
(cnauroth)
HDFS-3405. Checkpointing should use HTTP POST or PUT instead of GET-GET
to send merged fsimages. (Vinayakumar B via wang)
HDFS-6085. Improve CacheReplicationMonitor log messages a bit (cmccabe)
HDFS-6072. Clean up dead code of FSImage. (wheat9)
HDFS-6080. Improve NFS gateway performance by making rtmax and wtmax
configurable. (Abin Shahab via brandonli)
HDFS-6084. Namenode UI - "Hadoop" logo link shouldn't go to hadoop
homepage. (Travis Thompson via wheat9)
HDFS-6106. Reduce default for
dfs.namenode.path.based.cache.refresh.interval.ms (cmccabe)
HDFS-6090. Use MiniDFSCluster.Builder instead of deprecated constructors.
(Akira AJISAKA via jing9)
HDFS-6068. Disallow snapshot names that are also invalid directory names.
(sathish via szetszwo)
HDFS-6123. Do not log stack trace for ReplicaAlreadyExistsException and
SocketTimeoutException. (szetszwo)
HDFS-6129. When a replica is not found for deletion, do not throw an
exception. (szetszwo)
HDFS-6138. Add a user guide for how to use viewfs with federation.
(sanjay and szetszwo via szetszwo)
HDFS-6120. Fix and improve safe mode log messages. (Arpit Agarwal)
HDFS-6050. NFS does not handle exceptions correctly in a few places
(brandonli)
HDFS-5138. Support HDFS upgrade in HA. (atm via todd)
HDFS-6124. Add final modifier to class members. (Suresh Srinivas via
Arpit Agarwal)
HDFS-5910. Enhance DataTransferProtocol to allow per-connection choice
of encryption/plain-text. (Benoy Antony via Arpit Agarwal)
HDFS-6150. Add inode id information in the logs to make debugging easier.
(suresh)
OPTIMIZATIONS
HDFS-5790. LeaseManager.findPath is very slow when many leases need recovery
(todd)
BUG FIXES
HDFS-5492. Port HDFS-2069 (Incorrect default trash interval in the
docs) to trunk. (Akira Ajisaka via Arpit Agarwal)
HDFS-5843. DFSClient.getFileChecksum() throws IOException if checksum is
disabled. (Laurent Goujon via jing9)
HDFS-5856. DataNode.checkDiskError might throw NPE.
(Josh Elser via suresh)
HDFS-5828. BlockPlacementPolicyWithNodeGroup can place multiple replicas on
the same node group when dfs.namenode.avoid.write.stale.datanode is true.
(Buddy via junping_du)
HDFS-5767. NFS implementation assumes userName userId mapping to be unique,
which is not true sometimes (Yongjun Zhang via brandonli)
HDFS-5791. TestHttpsFileSystem should use a random port to avoid binding
error during testing (Haohui Mai via brandonli)
HDFS-5709. Improve NameNode upgrade with existing reserved paths and path
components. (Andrew Wang via atm)
HDFS-5881. Fix skip() of the short-circuit local reader(legacy). (kihwal)
HDFS-5895. HDFS cacheadmin -listPools has exit_code of 1 when the command
returns 0 result. (Tassapol Athiapinya via cnauroth)
HDFS-5807. TestBalancerWithNodeGroup.testBalancerWithNodeGroup fails
intermittently. (Chen He via kihwal)
HDFS-5882. TestAuditLogs is flaky (jxiang via cmccabe)
HDFS-5900. Cannot set cache pool limit of "unlimited" via CacheAdmin.
(wang)
HDFS-5886. Potential null pointer deference in RpcProgramNfs3#readlink()
(brandonli)
HDFS-4858. HDFS DataNode to NameNode RPC should timeout.
(Henry Wang via shv)
HDFS-5879. Some TestHftpFileSystem tests do not close streams.
(Gera Shegalov via suresh)
HDFS-5938. Make BlockReaderFactory#BlockReaderPeer a static class to avoid
a findbugs warning. (cmccabe)
HDFS-5891. webhdfs should not try connecting the DN during redirection
(Haohui Mai via brandonli)
HDFS-5904. TestFileStatus fails intermittently. (Mit Desai via kihwal)
HDFS-5941. add dfs.namenode.secondary.https-address and
dfs.namenode.secondary.https-address in hdfs-default.xml.
(Haohui Mai via cnauroth)
HDFS-5913. Nfs3Utils#getWccAttr() should check attr parameter against null
(brandonli)
HDFS-5934. New Namenode UI back button doesn't work as expected
(Travis Thompson via brandonli)
HDFS-5901. NameNode new UI doesn't support IE8 and IE9 on windows 7
(Vinayakumar B via brandonli)
HDFS-5943. 'dfs.namenode.https-address' property is not loaded from
configuration in federation setup. (suresh)
HDFS-3128. Unit tests should not use a test root in /tmp. (wang)
HDFS-5948. TestBackupNode flakes with port in use error. (Haohui Mai
via Arpit Agarwal)
HDFS-5949. New Namenode UI when trying to download a file, the browser
doesn't know the file name. (Haohui Mai via brandonli)
HDFS-5716. Allow WebHDFS to use pluggable authentication filter
(Haohui Mai via brandonli)
HDFS-5953. TestBlockReaderFactory fails if libhadoop.so has not been built.
(Akira Ajisaka via wang)
HDFS-5759. Web UI does not show up during the period of loading FSImage.
(Haohui Mai via Arpit Agarwal)
HDFS-5942. Fix javadoc in OfflineImageViewer. (Akira Ajisaka via cnauroth)
HDFS-5780. TestRBWBlockInvalidation times out intemittently. (Mit Desai
via kihwal)
HDFS-5803. TestBalancer.testBalancer0 fails. (Chen He via kihwal)
HDFS-5893. HftpFileSystem.RangeHeaderUrlOpener uses the default
URLConnectionFactory which does not import SSL certificates. (Haohui Mai via
jing9)
HDFS-5961. OIV cannot load fsimages containing a symbolic link. (kihwal)
HDFS-5483. NN should gracefully handle multiple block replicas on same DN.
(Arpit Agarwal)
HDFS-5742. DatanodeCluster (mini cluster of DNs) fails to start.
(Arpit Agarwal)
HDFS-5979. Typo and logger fix for fsimage PB code. (wang)
HDFS-5962. Mtime and atime are not persisted for symbolic links. (Akira
Ajisaka via kihwal)
HDFS-5944. LeaseManager:findLeaseWithPrefixPath can't handle path like /a/b/
and cause SecondaryNameNode failed do checkpoint (Yunjiong Zhao via brandonli)
HDFS-5982. Need to update snapshot manager when applying editlog for deleting
a snapshottable directory. (jing9)
HDFS-5988. Bad fsimage always generated after upgrade. (wang)
HDFS-5922. DN heartbeat thread can get stuck in tight loop. (Arpit Agarwal)
HDFS-6008. Namenode dead node link is giving HTTP error 500.
(Benoy Antony via cnauroth)
HDFS-5936. MiniDFSCluster does not clean data left behind by
SecondaryNameNode. (Binglin Chang via cnauroth)
HDFS-5339. WebHDFS URI does not accept logical nameservices when security is
enabled. (Haohui Mai via jing9)
HDFS-6033. PBImageXmlWriter incorrectly handles processing cache
directives. (atm)
HDFS-5821. TestHDFSCLI fails for user names with the dash character.
(Gera Shegalov via Arpit Agarwal)
HDFS-5956. A file size is multiplied by the replication factor in 'hdfs oiv
-p FileDistribution' option. (Akira Ajisaka via wheat9)
HDFS-5866. '-maxSize' and '-step' option fail in OfflineImageViewer.
(Akira Ajisaka via wheat9)
HDFS-6040. fix DFSClient issue without libhadoop.so and some other
ShortCircuitShm cleanups (cmccabe)
HDFS-6053. Fix TestDecommissioningStatus and TestDecommission in branch-2.
(jing9)
HDFS-6047 TestPread NPE inside in DFSInputStream hedgedFetchBlockByteRange
(stack)
HDFS-6051. HDFS cannot run on Windows since short-circuit shared memory
segment changes. (cmccabe)
HDFS-5857. TestWebHDFS#testNamenodeRestart fails intermittently with NPE.
(Mit Desai via wheat9)
HDFS-5898. Allow NFS gateway to login/relogin from its kerberos keytab.
(Abin Shahab via atm)
HDFS-6057. DomainSocketWatcher.watcherThread should be marked as daemon
thread (cmccabe)
HDFS-6058. Fix TestHDFSCLI failures after HADOOP-8691 change.
(Akira Ajisaka via wheat9)
HDFS-6062. TestRetryCacheWithHA#testConcat is flaky. (Jing Zhao via wheat9)
HDFS-6059. TestBlockReaderLocal fails if native library is not available.
(Akira AJISAKA via Colin Patrick McCabe)
HDFS-6084. DFSConfigKeys.DFS_BLOCKREPORT_INTERVAL_MSEC_DEFAULT is
not updated with latest block report interval of 6 hrs.
(Vinayakumar B via wheat9)
HDFS-6067. TestPread.testMaxOutHedgedReadPool is flaky (cmccabe)
HDFS-6065. HDFS zero-copy reads should return null on EOF when doing ZCR
(cmccabe)
HDFS-5064. Standby checkpoints should not block concurrent readers.
(atm via wang)
HDFS-6078. TestIncrementalBlockReports is flaky. (Arpit Agarwal)
HDFS-6071. BlockReaderLocal doesn't return -1 on EOF when doing a
zero-length read on a short file (cmccabe)
HDFS-6077. Running slive with webhdfs on secure HA cluster fails with unkown
host exception. (jing9)
HDFS-6086. Fix a case where zero-copy or no-checksum reads were not allowed
even when the block was cached (cmccabe)
HDFS-6079. Timeout for getFileBlockStorageLocations does not work. (wang)
HDFS-5705. TestSecondaryNameNodeUpgrade#testChangeNsIDFails may fail due
to ConcurrentModificationException. (Ted Yu via brandonli)
HDFS-6096. TestWebHdfsTokens may timeout. (szetszwo via Arpit Agarwal)
HDFS-5244. TestNNStorageRetentionManager#testPurgeMultipleDirs fails.
(Jinghui Wang via suresh)
HDFS-6097. zero-copy reads are incorrectly disabled on file offsets above
2GB (cmccabe)
HDFS-6102. Lower the default maximum items per directory to fix PB fsimage
loading. (wang)
HDFS-6094. The same block can be counted twice towards safe mode
threshold. (Arpit Agarwal)
HDFS-5516. WebHDFS does not require user name when anonymous http requests
are disallowed. (Miodrag Radulovic via cnauroth)
HDFS-6117. Print file path information in FileNotFoundException on INode
ID mismatch. (suresh)
HDFS-6099. HDFS file system limits not enforced on renames. (cnauroth)
HDFS-6100. DataNodeWebHdfsMethods does not failover in HA mode. (Haohui Mai
via jing9)
HDFS-6105. NN web UI for DN list loads the same jmx page multiple times.
(wheat9)
HDFS-6127. WebHDFS tokens cannot be renewed in HA setup. (wheat9)
HDFS-6131. Move HDFSHighAvailabilityWithNFS.apt.vm and
HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS. (jing9)
HDFS-6140. WebHDFS cannot create a file with spaces in the name after HA
failover changes. (cnauroth)
HDFS-6135. In HDFS upgrade with HA setup, JournalNode cannot handle layout
version bump when rolling back. (jing9)
HDFS-5846. Assigning DEFAULT_RACK in resolveNetworkLocation method can break
data resiliency. (Nikola Vujic via cnauroth)
HDFS-5840. Follow-up to HDFS-5138 to improve error handling during partial
upgrade failures. (atm, jing9 and suresh via jing9)
HDFS-6130. NPE when upgrading namenode from fsimages older than -32.
(wheat9)
HDFS-6115. Call flush() for every append on block scan verification log.
(Vinayakumar B via szetszwo)
HDFS-5672. TestHASafeMode#testSafeBlockTracking fails in trunk. (jing9)
HDFS-6157. Fix the entry point of OfflineImageViewer for hdfs.cmd. (wheat9)
HDFS-6163. Fix a minor bug in the HA upgrade document. (Fengdong Yu via
jing9)
HDFS-6166. Change Balancer socket read timeout to 20 minutes and add
10 seconds delay after error. (Nathan Roberts via szetszwo)
BREAKDOWN OF HDFS-5698 SUBTASKS AND RELATED JIRAS
HDFS-5717. Save FSImage header in protobuf. (Haohui Mai via jing9)
HDFS-5738. Serialize INode information in protobuf. (Haohui Mai via jing9)
HDFS-5772. Serialize under-construction file information in FSImage. (jing9)
HDFS-5783. Compute the digest before loading FSImage. (Haohui Mai via jing9)
HDFS-5785. Serialize symlink in protobuf. (Haohui Mai via jing9)
HDFS-5793. Optimize the serialization of PermissionStatus. (Haohui Mai via
jing9)
HDFS-5743. Use protobuf to serialize snapshot information. (jing9)
HDFS-5774. Serialize CachePool directives in protobuf. (Haohui Mai via jing9)
HDFS-5744. Serialize information for token managers in protobuf. (Haohui Mai
via jing9)
HDFS-5824. Add a Type field in Snapshot DiffEntry's protobuf definition.
(jing9)
HDFS-5808. Implement cancellation when saving FSImage. (Haohui Mai via jing9)
HDFS-5826. Update the stored edit logs to be consistent with the changes in
HDFS-5698 branch. (Haohui Mai via jing9)
HDFS-5797. Implement offline image viewer. (Haohui Mai via jing9)
HDFS-5771. Track progress when loading fsimage. (Haohui Mai via cnauroth)
HDFS-5871. Use PBHelper to serialize CacheDirectiveInfoExpirationProto.
(Haohui Mai via jing9)
HDFS-5884. LoadDelegator should use IOUtils.readFully() to read the magic
header. (Haohui Mai via jing9)
HDFS-5885. Add annotation for repeated fields in the protobuf definition.
(Haohui Mai via jing9)
HDFS-5906. Fixing findbugs and javadoc warnings in the HDFS-5698 branch.
(Haohui Mai via jing9)
HDFS-5911. The id of a CacheDirective instance does not get serialized in
the protobuf-fsimage. (Haohui Mai via jing9)
HDFS-5915. Refactor FSImageFormatProtobuf to simplify cross section reads.
(Haohui Mai via cnauroth)
HDFS-5847. Consolidate INodeReference into a separate section. (jing9)
HDFS-5959. Fix typo at section name in FSImageFormatProtobuf.java.
(Akira Ajisaka via suresh)
HDFS-5981. PBImageXmlWriter generates malformed XML.
(Haohui Mai via cnauroth)
HDFS-6107. When a block can't be cached due to limited space on the
DataNode, that block becomes uncacheable (cmccabe)
HDFS-6089. Standby NN while transitioning to active throws a connection
refused error when the prior active NN process is suspended.
(Jing Zhao via Andrew Wang)
BREAKDOWN OF HDFS-4685 SUBTASKS AND RELATED JIRAS
HDFS-5596. Implement RPC stubs. (Haohui Mai via cnauroth)
HDFS-5685. Implement ACL as a INode feature. (Haohui Mai via cnauroth)
HDFS-5618. NameNode: persist ACLs in fsimage. (Haohui Mai via cnauroth)
HDFS-5619. NameNode: record ACL modifications to edit log.
(Haohui Mai via cnauroth)
HDFS-5673. Implement logic for modification of ACLs. (cnauroth)
HDFS-5758. NameNode: complete implementation of inode modifications for
ACLs. (Chris Nauroth via wheat9)
HDFS-5612. NameNode: change all permission checks to enforce ACLs in
addition to permissions. (Chris Nauroth via wheat9)
HDFS-5613. NameNode: implement handling of ACLs in combination with
symlinks. (Chris Nauroth via wheat9)
HDFS-5615. NameNode: implement handling of ACLs in combination with sticky
bit. (Chris Nauroth via wheat9)
HDFS-5702. FsShell Cli: Add XML based End-to-End test for getfacl and
setfacl commands. (Vinay via cnauroth)
HDFS-5608. WebHDFS: implement ACL APIs.
(Sachin Jose and Renil Joseph via cnauroth)
HDFS-5614. NameNode: implement handling of ACLs in combination with
snapshots. (cnauroth)
HDFS-5858. Refactor common ACL test cases to be run through multiple
FileSystem implementations. (cnauroth)
HDFS-5860. Refactor INodeDirectory getDirectoryXFeature methods to use
common getFeature helper method. (Jing Zhao via cnauroth)
HDFS-5861. Add CLI test for Ls output for extended ACL marker.
(Vinay via cnauroth)
HDFS-5616. NameNode: implement default ACL handling. (cnauroth)
HDFS-5899. Add configuration flag to disable/enable support for ACLs.
(cnauroth)
HDFS-5914. Incorporate ACLs with the changes from HDFS-5698.
(Haohui Mai via cnauroth)
HDFS-5625. Write end user documentation for HDFS ACLs. (cnauroth)
HDFS-5925. ACL configuration flag must only reject ACL API calls, not ACLs
present in fsimage or edits. (cnauroth)
HDFS-5923. Do not persist the ACL bit in the FsPermission.
(Haohui Mai via cnauroth)
HDFS-5933. Optimize the FSImage layout for ACLs (Haohui Mai via cnauroth)
HDFS-5932. Ls should display the ACL bit (Chris Nauroth via wheat9)
HDFS-5937. Fix TestOfflineEditsViewer on HDFS-4685 branch. (cnauroth)
HDFS-5737. Replacing only the default ACL can fail to copy unspecified base
entries from the access ACL. (cnauroth)
HDFS-5739. ACL RPC must allow null name or unspecified permissions in ACL
entries. (cnauroth)
HDFS-5799. Make audit logging consistent across ACL APIs. (cnauroth)
HDFS-5849. Removing ACL from an inode fails if it has only a default ACL.
(cnauroth)
HDFS-5623. NameNode: add tests for skipping ACL enforcement when permission
checks are disabled, user is superuser or user is member of supergroup.
(cnauroth)
HDFS-5908. Change AclFeature to capture list of ACL entries in an
ImmutableList. (cnauroth)
HDFS-6028. Print clearer error message when user attempts to delete required
mask entry from ACL. (cnauroth)
HDFS-6039. Uploading a File under a Dir with default acls throws "Duplicated
ACLFeature". (cnauroth)
HDFS-6063. TestAclCLI fails intermittently when running test 24:
copyFromLocal. (cnauroth)
HDFS-6069. Quash stack traces when ACLs are disabled. (cnauroth)
HDFS-5638. HDFS implementation of FileContext API for ACLs.
(Vinayakumar B via cnauroth)
HDFS-5535 subtasks:
HDFS-5496. Make replication queue initialization asynchronous. (Vinay via
jing9)
HDFS-5645. Support upgrade marker in editlog streams. (szetszwo)
HDFS-5752. Add a new DFSAdmin command to query, start and finalize rolling
upgrade. (szetszwo)
HDFS-5786. Support QUERY and FINALIZE actions of rolling upgrade. (szetszwo)
HDFS-5753. Add new Namenode startup options for downgrade and rollback using
upgrade marker. (szetszwo)
HDFS-5835. Add a new option for starting Namenode when rolling upgrade is
in progress. (szetszwo)
HDFS-5754. Split LayoutVerion into NameNodeLayoutVersion and
DataNodeLayoutVersion. (Brandon Li via szetszwo)
HDFS-5848. Add rolling upgrade status to heartbeat response. (szetszwo)
HDFS-5890. Avoid NPE in Datanode heartbeat. (Vinay via brandonli)
HDFS-5869. When starting rolling upgrade or NN restarts, NN should create
a checkpoint right before the upgrade marker. (szetszwo)
HDFS-5874. Should not compare DataNode current layout version with that of
NameNode in DataStrorage. (brandonli)
HDFS-5889. When starting rolling upgrade, create a fs image for rollback
so that the standby namenode can create checkpoints during upgrade.
(szetszwo & jing9)
HDFS-5907. Add BlockPoolSliceStorage 'trash' to handle block deletions
during rolling upgrades. (Arpit Agarwal)
HDFS-5494. Merge Protobuf-based-FSImage code from trunk - fix build
break after the merge. (Jing Zhao via Arpit Agarwal)
HDFS-5585. Provide admin commands for data node upgrade (kihwal)
HDFS-5920. Support rollback of rolling upgrade in NameNode and JournalNodes.
(jing9)
HDFS-5945. Add rolling upgrade information to fsimage; and disallow upgrade
and rolling upgrade to be started simultaneously. (szetszwo & jing9)
HDFS-5966. Fix rollback of rolling upgrade in NameNode HA setup. (jing9
via szetszwo)
HDFS-5974. Fix compilation error, NameNodeLayoutVersion and
DataNodeLayoutVersion after merge from trunk. (szetszwo)
HDFS-5963. TestRollingUpgrade#testSecondaryNameNode causes subsequent
tests to fail. (szetszwo via Arpit Agarwal)
HDFS-5976. Create unit tests for downgrade and finalize rolling upgrade.
(Haohui Mai via Arpit Agarwal)
HDFS-5980. Rollback does not need to load edits. (jing9 via szetszwo)
HDFS-5984. Fix TestEditLog and TestStandbyCheckpoints. (jing9 via szetszwo)
HDFS-5985. SimulatedFSDataset#disableAndPurgeTrashStorage should not throw
UnsupportedOperationException. (jing9 via kihwal)
HDFS-5987. Fix findbugs warnings in Rolling Upgrade branch. (seztszwo via
Arpit Agarwal)
HDFS-5992. Fix NPE in MD5FileUtils and update editsStored for
TestOfflineEditsViewer. (szetszwo)
HDFS-5994. Fix TestDataNodeRollingUpgrade. (Arpit Agarwal via szetszwo)
HDFS-5999. Do not create rollback fsimage when it already exists. (jing9)
HDFS-6005. Simplify Datanode rollback and downgrade. (Suresh Srinivas via
Arpit Agarwal)
HDFS-6004. Change DFSAdmin for rolling upgrade commands. (szetszwo via
Arpit Agarwal)
HDFS-5583. Make DN send an OOB Ack on shutdown before restarting. (kihwal)
HDFS-5778. Add rolling upgrade user document. (szetszwo)
HDFS-6003. Add the new -rollingUpgrade startup option to the namenode
usage message. (Vinayakumar B via szetszwo)
HDFS-6014. Fix findbug warnings introduced by HDFS-5583. (kihwal)
HDFS-6015. Fix TestBlockRecovery
#testRaceBetweenReplicaRecoveryAndFinalizeBlock. (kihwal)
HDFS-5924. Utilize OOB upgrade message processing for writes. (kihwal)
HDFS-5498. Improve datanode startup time. (kihwal)
HDFS-6000. Avoid saving namespace when starting rolling upgrade. (jing9)
HDFS-6017. Query the status of rolling upgrade in the preparation stage in
TestRollingUpgrade and TestRollingUpgradeRollback. (Haohui Mai via
Arpit Agarwal)
HDFS-6020. Fix the five findbugs warnings. (kihwal)
HDFS-6019. Standby NN might not checkpoint when processing the rolling
upgrade marker. (Haohui Mai via jing9)
HDFS-6023. Test whether the standby NN continues to checkpoint after the
prepare stage. (Haohui Mai via jing9)
HDFS-6024. Test whether the NN will reject the downgrade if it has a
fsimage from a newer release. (Haohui Mai via jing9)
HDFS-6026. Fix TestDFSUpgrade and TestDataNodeRollingUpgrade.
(jing9 via szetszwo)
HDFS-6029. Secondary NN fails to checkpoint after -rollingUpgrade prepare.
(jing9)
HDFS-6032. -rollingUpgrade query hits NPE after the NN restarts. (Haohui Mai
via jing9)
HDFS-6031. Add back the "-rollingUpgrade started" namenode startup option;
otherwise, namenode cannot start when the layout version is changed.
(szetszwo)
HDFS-6034. Use DataNodeLayoutVersion for DN registration check and do not
verify layout version if there is a rolling upgrade in progress. (szetszwo)
HDFS-6013. add rollingUpgrade information to latest UI.
(Vinayakumar B via wheat9)
HDFS-6042. Fix rolling upgrade documentation and error messages. (szetszwo
via Arpit Agarwal)
HDFS-6041. Downgrade/Finalize should rename the rollback image instead of
purging it. (jing9)
HDFS-6060. NameNode should not check DataNode layout version (brandonli)
HDFS-6076. DataNode with SimulatedDataSet should not create
DatanodeRegistration with namenode layout version and namenode node type.
(szetszwo)
HDFS-6038. Allow JournalNode to handle editlog produced by new release with
future layoutversion. (jing9)
HDFS-4564. Ensure webhdfs returns correct HTTP response codes for denied
operations. (daryn via acmurthy)
Release 2.3.1 - UNRELEASED
INCOMPATIBLE CHANGES
NEW FEATURES
IMPROVEMENTS
OPTIMIZATIONS
BUG FIXES
Release 2.3.0 - 2014-02-18
INCOMPATIBLE CHANGES
NEW FEATURES
HDFS-5122. Support failover and retry in WebHdfsFileSystem for NN HA.
(Haohui Mai via jing9)
HDFS-4953. Enable HDFS local reads via mmap.
(Colin Patrick McCabe via wang).
HDFS-5342. Provide more information in the FSNamesystem JMX interfaces.
(Haohui Mai via jing9)
HDFS-5334. Implement dfshealth.jsp in HTML pages. (Haohui Mai via jing9)
HDFS-5379. Update links to datanode information in dfshealth.html. (Haohui
Mai via jing9)
HDFS-5382. Implement the UI of browsing filesystems in HTML 5 page. (Haohui
Mai via jing9)
HDFS-3987. Support webhdfs over HTTPS. (Haohui Mai via jing9)
HDFS-5444. Choose default web UI based on browser capabilities. (Haohui Mai
via jing9)
HDFS-5514. FSNamesystem's fsLock should allow custom implementation (daryn)
HDFS-2832. Heterogeneous Storages support in HDFS phase 1 - treat DataNode
as a collection of storages (see breakdown of tasks below for features and
contributors).
HDFS-5703. Add support for HTTPS and swebhdfs to HttpFS. (tucu)
HDFS-4949. Centralized cache management in HDFS (wang and cmccabe)
IMPROVEMENTS
HDFS-5360. Improvement of usage message of renameSnapshot and
deleteSnapshot. (Shinichi Yamashita via wang)
HDFS-5331. make SnapshotDiff.java to a o.a.h.util.Tool interface implementation.
(Vinayakumar B via umamahesh)
HDFS-4657. Limit the number of blocks logged by the NN after a block
report to a configurable value. (Aaron T. Myers via Colin Patrick
McCabe)
HDFS-5344. Make LsSnapshottableDir as Tool interface implementation. (Sathish via umamahesh)
HDFS-5544. Adding Test case For Checking dfs.checksum type as NULL value. (Sathish via umamahesh)
HDFS-5568. Support includeSnapshots option with Fsck command. (Vinayakumar B via umamahesh)
HDFS-4983. Numeric usernames do not work with WebHDFS FS. (Yongjun Zhang via
jing9)
HDFS-5592. statechangeLog of completeFile should be logged only in case of success.
(Vinayakumar via umamahesh)
HDFS-5662. Can't decommission a DataNode due to file's replication factor
larger than the rest of the cluster size. (brandonli)
HDFS-5068. Convert NNThroughputBenchmark to a Tool to allow generic options.
(shv)
HDFS-5675. Add Mkdirs operation to NNThroughputBenchmark.
(Plamen Jeliazkov via shv)
HDFS-5677. Need error checking for HA cluster configuration.
(Vincent Sheffer via cos)
HDFS-5825. Use FileUtils.copyFile() to implement DFSTestUtils.copyFile().
(Haohui Mai via Arpit Agarwal)
HDFS-5267. Remove volatile from LightWeightHashSet. (Junping Du via llu)
HDFS-4278. Log an ERROR when DFS_BLOCK_ACCESS_TOKEN_ENABLE config is
disabled but security is turned on. (Kousuke Saruta via harsh)
HDFS-5004. Add additional JMX bean for NameNode status data. Contributed
by Trevor Lorimer.
HDFS-4994. Audit log getContentSummary() calls. (Robert Parker via kihwal)
HDFS-5144. Document time unit to NameNodeMetrics. (Akira Ajisaka via
suresh)
HDFS-4491. Parallel testing HDFS. (Andrey Klochkov via cnauroth)
HDFS-4879. Add "blocked ArrayList" collection to avoid CMS full GCs
(Todd Lipcon via Colin Patrick McCabe)
HDFS-4096. Add snapshot information to namenode WebUI. (Haohui Mai via
jing9)
HDFS-5188. In BlockPlacementPolicy, reduce the number of chooseTarget(..)
methods; replace HashMap with Map in parameter declarations and cleanup
some related code. (szetszwo)
HDFS-5207. In BlockPlacementPolicy.chooseTarget(..), change the writer
and the excludedNodes parameter types respectively to Node and Set.
(Junping Du via szetszwo)
HDFS-5240. Separate formatting from logging in the audit logger API (daryn)
HDFS-5191. Revisit zero-copy API in FSDataInputStream to make it more
intuitive. (Contributed by Colin Patrick McCabe)
HDFS-5260. Merge zero-copy memory-mapped HDFS client reads to trunk and
branch-2. (cnauroth)
HDFS-4517. Cover class RemoteBlockReader with unit tests. (Vadim Bondarev
and Dennis Y via kihwal)
HDFS-4512. Cover package org.apache.hadoop.hdfs.server.common with tests.
(Vadim Bondarev via kihwal)
HDFS-4510. Cover classes ClusterJspHelper/NamenodeJspHelper with unit
tests. (Andrey Klochkov via kihwal)
HDFS-5323. Remove some deadcode in BlockManager. (Colin Patrick McCabe)
HDFS-5338. Add a conf to disable hostname check in datanode registration.
(szetszwo)
HDFS-5130. Add test for snapshot related FsShell and DFSAdmin commands.
(Binglin Chang via jing9)
HDFS-5374. Remove deadcode in DFSOutputStream. (suresh)
HDFS-4511. Cover package org.apache.hadoop.hdfs.tools with unit test
(Andrey Klochkov via jeagles)
HDFS-4885. Improve the verifyBlockPlacement() API in BlockPlacementPolicy.
(Junping Du via szetszwo)
HDFS-5363. Refactor WebHdfsFileSystem: move SPENGO-authenticated connection
creation to URLConnectionFactory. (Haohui Mai via jing9)
HDFS-5436. Move HsFtpFileSystem and HFtpFileSystem into org.apache.hdfs.web
(Haohui Mai via Arpit Agarwal)
HDFS-5371. Let client retry the same NN when
"dfs.client.test.drop.namenode.response.number" is enabled. (jing9)
HDFS-5467. Remove tab characters in hdfs-default.xml.
(Shinichi Yamashita via Andrew Wang)
HDFS-5495. Remove further JUnit3 usages from HDFS.
(Jarek Jarcec Cecho via Andrew Wang)
HDFS-5325. Remove WebHdfsFileSystem#ConnRunner. (Haohui Mai via jing9)
HDFS-5488. Clean up TestHftpURLTimeout. (Haohui Mai via jing9)
HDFS-5440. Extract the logic of handling delegation tokens in HftpFileSystem
to the TokenAspect class. (Haohui Mai via jing9)
HDFS-5487. Introduce unit test for TokenAspect. (Haohui Mai via jing9)
HDFS-4995. Make getContentSummary less expensive. (kihwal)
HDFS-5506. Use URLConnectionFactory in DelegationTokenFetcher. (Haohui Mai
via jing9)
HDFS-5489. Use TokenAspect in WebHDFSFileSystem. (Haohui Mai via jing9)
HDFS-5393. Serve bootstrap and jQuery locally. (Haohui Mai via jing9)
HDFS-5073. TestListCorruptFileBlocks fails intermittently. (Arpit Agarwal)
HDFS-1386. TestJMXGet fails in jdk7 (jeagles)
HDFS-5532. Enable the webhdfs by default to support new HDFS web UI. (Vinay
via jing9)
HDFS-5525. Inline dust templates for new Web UI. (Haohui Mai via jing9)
HDFS-5561. FSNameSystem#getNameJournalStatus() in JMX should return plain
text instead of HTML. (Haohui Mai via jing9)
HDFS-5581. NameNodeFsck should use only one instance of
BlockPlacementPolicy. (vinay via cmccabe)
HDFS-5633. Improve OfflineImageViewer to use less memory. (jing9)
HDFS-5023. TestSnapshotPathINodes.testAllowSnapshot is failing with jdk7
(Mit Desai via jeagles)
HDFS-5637. Try to refeatchToken while local read InvalidToken occurred.
(Liang Xie via junping_du)
HDFS-5652. Refactor invalid block token exception handling in DFSInputStream.
(Liang Xie via junping_du)
HDFS-5350. Name Node should report fsimage transfer time as a metric.
(Jimmy Xiang via wang)
HDFS-5538. URLConnectionFactory should pick up the SSL related configuration
by default. (Haohui Mai via jing9)
HDFS-5545. Allow specifying endpoints for listeners in HttpServer. (Haohui
Mai via jing9)
HDFS-5536. Implement HTTP policy for Namenode and DataNode. (Haohui Mai via
jing9)
HDFS-5312. Generate HTTP / HTTPS URL in DFSUtil#getInfoServer() based on the
configured http policy. (Haohui Mai via jing9)
HDFS-5629. Support HTTPS in JournalNode and SecondaryNameNode.
(Haohui Mai via jing9)
HDFS-5674. Editlog code cleanup: remove @SuppressWarnings("deprecation") in
FSEditLogOp; change FSEditLogOpCodes.fromByte(..) to be more efficient; and
change Some fields in FSEditLog to final. (szetszwo)
HDFS-5634. Allow BlockReaderLocal to switch between checksumming and not
(cmccabe)
HDFS-5663 make the retry time and interval value configurable in openInfo()
(Liang Xie via stack)
HDFS-5540. Fix intermittent failure in TestBlocksWithNotEnoughRacks.
(Binglin Chang via junping_du)
HDFS-5695. Clean up TestOfflineEditsViewer and OfflineEditsViewerHelper.
(Haohui Mai via jing9)
HDFS-5220. Expose group resolution time as metric (jxiang via cmccabe)
HDFS-5762. BlockReaderLocal doesn't return -1 on EOF when doing zero-length
reads (Colin Patrick McCabe)
HDFS-5766. In DFSInputStream, do not add datanode to deadNodes after
InvalidEncryptionKeyException in fetchBlockByteRange (Liang Xie via Colin
Patrick McCabe)
HDFS-5704. Change OP_UPDATE_BLOCKS with a new OP_ADD_BLOCK. (jing9)
HDFS-5784. Reserve space in edit log header and fsimage header for feature
flag section. (Colin Patrick McCabe)
HDFS-5434. Change block placement policy constructors from package private
to protected. (Buddy Taylor via Arpit Agarwal)
HDFS-5788. listLocatedStatus response can be very large. (Nathan Roberts
via kihwal)
HDFS-5841. Update HDFS caching documentation with new changes. (wang)
OPTIMIZATIONS
HDFS-5239. Allow FSNamesystem lock fairness to be configurable (daryn)
HDFS-5341. Reduce fsdataset lock duration during directory scanning.
(Qus-Jiawei via kihwal)
HDFS-5681. renewLease should not hold fsn write lock. (daryn via Kihwal)
HDFS-5241. Provide alternate queuing audit logger to reduce logging
contention (daryn)
BUG FIXES
HDFS-5307. Support both HTTP and HTTPS in jsp pages (Haohui Mai via
brandonli)
HDFS-5291. Standby namenode after transition to active goes into safemode.
(jing9)
HDFS-5317. Go back to DFS Home link does not work on datanode webUI
(Haohui Mai via brandonli)
HDFS-5316. Namenode ignores the default https port (Haohui Mai via
brandonli)
HDFS-5281. COMMIT request should not block. (brandonli)
HDFS-5337. should do hsync for a commit request even there is no pending
writes (brandonli)
HDFS-5335. Hive query failed with possible race in dfs output stream.
(Haohui Mai via suresh)
HDFS-5322. HDFS delegation token not found in cache errors seen on secure HA
clusters. (jing9)
HDFS-5329. Update FSNamesystem#getListing() to handle inode path in startAfter
token. (brandonli)
HDFS-5330. fix readdir and readdirplus for large directories (brandonli)
HDFS-5370. Typo in Error Message: different between range in condition
and range in error message. (Kousuke Saruta via suresh)
HDFS-5365. Fix libhdfs compile error on FreeBSD9. (Radim Kolar via cnauroth)
HDFS-5347. Add HDFS NFS user guide. (brandonli)
HDFS-5403. WebHdfs client cannot communicate with older WebHdfs servers
post HDFS-5306. (atm)
HDFS-5171. NFS should create input stream for a file and try to share it
with multiple read requests. (Haohui Mai via brandonli)
HDFS-5413. hdfs.cmd does not support passthrough to any arbitrary class.
(cnauroth)
HDFS-5433. When reloading fsimage during checkpointing, we should clear
existing snapshottable directories. (Aaron T. Myers via wang)
HDFS-5432. TestDatanodeJsp fails on Windows due to assumption that loopback
address resolves to host name localhost. (cnauroth)
HDFS-5065. TestSymlinkHdfsDisable fails on Windows. (ivanmi)
HDFS-4633 TestDFSClientExcludedNodes fails sporadically if excluded nodes
cache expires too quickly (Chris Nauroth via Sanjay)
HDFS-5037. Active NN should trigger its own edit log rolls. (wang)
HDFS-5035. getFileLinkStatus and rename do not correctly check permissions
of symlinks. (Andrew Wang via Colin Patrick McCabe)
HDFS-5456. NameNode startup progress creates new steps if caller attempts to
create a counter for a step that doesn't already exist. (cnauroth)
HDFS-5458. Datanode failed volume threshold ignored if exception is thrown
in getDataDirsFromURIs. (Mike Mellenthin via wang)
HDFS-5252. Stable write is not handled correctly in someplace. (brandonli)
HDFS-5364. Add OpenFileCtx cache. (brandonli)
HDFS-5469. Add configuration property for the sub-directroy export path
(brandonli)
HDFS-5519. COMMIT handler should update the commit status after sync
(brandonli)
HDFS-5372. In FSNamesystem, hasReadLock() returns false if the current
thread holds the write lock (Vinaykumar B via umamahesh)
HDFS-4516. Client crash after block allocation and NN switch before lease recovery for
the same file can cause readers to fail forever (VinaayKumar B via umamahesh)
HDFS-5014. Process register commands with out holding BPOfferService lock.
(Vinaykumar B via umamahesh)
HDFS-5288. Close idle connections in portmap (Haohui Mai via brandonli)
HDFS-5407. Fix typos in DFSClientCache (Haohui Mai via brandonli)
HDFS-5548. Use ConcurrentHashMap in portmap (Haohui Mai via brandonli)
HDFS-5577. NFS user guide update (brandonli)
HDFS-5563. NFS gateway should commit the buffered data when read request comes
after write to the same file (brandonli)
HDFS-4997. libhdfs doesn't return correct error codes in most cases (cmccabe)
HDFS-5587. add debug information when NFS fails to start with duplicate user
or group names (brandonli)
HDFS-5590. Block ID and generation stamp may be reused when persistBlocks is
set to false. (jing9)
HDFS-5353. Short circuit reads fail when dfs.encrypt.data.transfer is
enabled. (Colin Patrick McCabe via jing9)
HDFS-5283. Under construction blocks only inside snapshots should not be
counted in safemode threshhold. (Vinay via szetszwo)
HDFS-5257. addBlock() retry should return LocatedBlock with locations else client
will get AIOBE. (Vinay via jing9)
HDFS-5427. Not able to read deleted files from snapshot directly under
snapshottable dir after checkpoint and NN restart. (Vinay via jing9)
HDFS-5443. Delete 0-sized block when deleting an under-construction file that
is included in snapshot. (jing9)
HDFS-5476. Snapshot: clean the blocks/files/directories under a renamed
file/directory while deletion. (jing9)
HDFS-5425. Renaming underconstruction file with snapshots can make NN failure on
restart. (jing9 and Vinay)
HDFS-5474. Deletesnapshot can make Namenode in safemode on NN restarts.
(Sathish via jing9)
HDFS-5504. In HA mode, OP_DELETE_SNAPSHOT is not decrementing the safemode threshold,
leads to NN safemode. (Vinay via jing9)
HDFS-5428. Under construction files deletion after snapshot+checkpoint+nn restart
leads nn safemode. (jing9)
HDFS-5074. Allow starting up from an fsimage checkpoint in the middle of a
segment. (Todd Lipcon via atm)
HDFS-4201. NPE in BPServiceActor#sendHeartBeat. (jxiang via cmccabe)
HDFS-5666. Fix inconsistent synchronization in BPOfferService (jxiang via cmccabe)
HDFS-5657. race condition causes writeback state error in NFS gateway (brandonli)
HDFS-5661. Browsing FileSystem via web ui, should use datanode's fqdn instead of ip
address. (Benoy Antony via jing9)
HDFS-5582. hdfs getconf -excludeFile or -includeFile always failed (sathish
via cmccabe)
HDFS-5671. Fix socket leak in DFSInputStream#getBlockReader. (JamesLi via umamahesh)
HDFS-5649. Unregister NFS and Mount service when NFS gateway is shutting down.
(brandonli)
HDFS-5789. Some of snapshot APIs missing checkOperation double check in fsn. (umamahesh)
HDFS-5343. When cat command is issued on snapshot files getting unexpected result.
(Sathish via umamahesh)
HDFS-5297. Fix dead links in HDFS site documents. (Akira Ajisaka via
Arpit Agarwal)
HDFS-5830. WebHdfsFileSystem.getFileBlockLocations throws
IllegalArgumentException when accessing another cluster. (Yongjun Zhang via
Colin Patrick McCabe)
HDFS-5833. Fix SecondaryNameNode javadoc. (Bangtao Zhou via Arpit Agarwal)
HDFS-5844. Fix broken link in WebHDFS.apt.vm. (Akira Ajisaka via
Arpit Agarwal)
HDFS-5034. Remove debug prints from GetFileLinkInfo (Andrew Wang via Colin
Patrick McCabe)
HDFS-4816. transitionToActive blocks if the SBN is doing checkpoint image
transfer. (Andrew Wang)
HDFS-5164. deleteSnapshot should check if OperationCategory.WRITE is
possible before taking write lock. (Colin Patrick McCabe)
HDFS-5170. BlockPlacementPolicyDefault uses the wrong classname when
alerting to enable debug logging. (Andrew Wang)
HDFS-5266. ElasticByteBufferPool#Key does not implement equals. (cnauroth)
HDFS-5352. Server#initLog() doesn't close InputStream in httpfs. (Ted Yu via
jing9)
HDFS-4376. Fix race conditions in Balancer. (Junping Du via szetszwo)
HDFS-5375. hdfs.cmd does not expose several snapshot commands. (cnauroth)
HDFS-5336. DataNode should not output 'StartupProgress' metrics.
(Akira Ajisaka via cnauroth)
HDFS-5400. DFS_CLIENT_MMAP_CACHE_THREAD_RUNS_PER_TIMEOUT constant is set
to the wrong value. (Colin Patrick McCabe)
HDFS-5075. httpfs-config.sh calls out incorrect env script name
(Timothy St. Clair via stevel)
HDFS-5438. Flaws in block report processing can cause data loss. (kihwal)
HDFS-5502. Fix HTTPS support in HsftpFileSystem. (Haohui Mai via jing9)
HDFS-5552. Fix wrong information of "Cluster summay" in dfshealth.html.
(Haohui Mai via jing9)
HDFS-5533. Symlink delete/create should be treated as DELETE/CREATE in snapshot diff
report. (Binglin Chang via jing9)
HDFS-5580. Fix infinite loop in Balancer.waitForMoveCompletion.
(Binglin Chang via junping_du)
HDFS-5676. fix inconsistent synchronization of CachingStrategy (cmccabe)
HDFS-5691. Fix typo in ShortCircuitLocalRead document.
(Akira Ajisaka via suresh)
HDFS-5690. DataNode fails to start in secure mode when dfs.http.policy equals to
HTTP_ONLY. (Haohui Mai via jing9)
HDFS-5449. WebHdfs compatibility broken between 2.2 and 1.x / 23.x (kihwal)
HDFS-5756. hadoopRzOptionsSetByteBufferPool does not accept NULL argument,
contrary to docs. (cmccabe via wang)
HDFS-5747. Fix NPEs in BlockManager. (Arpit Agarwal)
HDFS-5710. FSDirectory#getFullPathName should check inodes against null.
(Uma Maheswara Rao G via jing9)
HDFS-5579. Under construction files make DataNode decommission take very long
hours. (zhaoyunjiong via jing9)
HDFS-5777. Update LayoutVersion for the new editlog op OP_ADD_BLOCK. (jing9)
HDFS-5800. Fix a typo in DFSClient.renewLease(). (Kousuke Saruta
via szetszwo)
HDFS-5748. Too much information shown in the dfs health page.
(Haohui Mai via brandonli)
HDFS-5806. Balancer should set SoTimeout to avoid indefinite hangs.
(Nathan Roberts via Andrew Wang)
HDFS-5728. Block recovery will fail if the metafile does not have crc
for all chunks of the block (Vinay via kihwal)
HDFS-5719. FSImage#doRollback() should close prevState before return
(Ted Yu via todd)
HDFS-5721. sharedEditsImage in Namenode#initializeSharedEdits() should be
closed before method returns (Ted Yu via todd)
HDFS-5845. SecondaryNameNode dies when checkpointing with cache pools.
(wang)
HDFS-5842. Cannot create hftp filesystem when using a proxy user ugi and a doAs
on a secure cluster. (jing9)
HDFS-5399. Revisit SafeModeException and corresponding retry policies.
(Jing Zhao via todd)
HDFS-5876. SecureDataNodeStarter does not pick up configuration in
hdfs-site.xml. (Haohui Mai via jing9)
HDFS-5873. dfs.http.policy should have higher precedence over dfs.https.enable.
(Haohui Mai via jing9)
HDFS-5837. dfs.namenode.replication.considerLoad should consider
decommissioned nodes. (Tao Luo via shv)
HDFS-5921. Cannot browse file system via NN web UI if any directory has
the sticky bit set. (atm)
BREAKDOWN OF HDFS-2832 SUBTASKS AND RELATED JIRAS
HDFS-4985. Add storage type to the protocol and expose it in block report
and block locations. (Arpit Agarwal)
HDFS-5115. Make StorageID a UUID. (Arpit Agarwal)
HDFS-5000. DataNode configuration should allow specifying storage type.
(Arpit Agarwal)
HDFS-4987. Namenode changes to track multiple storages per datanode.
(szetszwo)
HDFS-5154. Fix TestBlockManager and TestDatanodeDescriptor after HDFS-4987.
(Junping Du via szetszwo)
HDFS-5009. Include storage information in the LocatedBlock. (szetszwo)
HDFS-5134. Move blockContentsStale, heartbeatedSinceFailover and
firstBlockReport from DatanodeDescriptor to DatanodeStorageInfo; and
fix a synchronization problem in DatanodeStorageInfo. (szetszwo)
HDFS-5157. Add StorageType to FsVolume. (Junping Du via szetszwo)
HDFS-4990. Change BlockPlacementPolicy to choose storages instead of
datanodes. (szetszwo)
HDFS-5232. Protocol changes to transmit StorageUuid. (Arpit Agarwal)
HDFS-5233. Use Datanode UUID to identify Datanodes. (Arpit Agarwal)
HDFS-5222. Move block schedule information from DatanodeDescriptor to
DatanodeStorageInfo. (szetszwo)
HDFS-4988. Datanode must support all the volumes as individual storages.
(Arpit Agarwal)
HDFS-5377. Heartbeats from Datandode should include one storage report
per storage directory. (Arpit Agarwal)
HDFS-5398. NameNode changes to process storage reports per storage
directory. (Arpit Agarwal)
HDFS-5390. Send one incremental block report per storage directory.
(Arpit Agarwal)
HDFS-5401. Fix NPE in Directory Scanner. (Arpit Agarwal)
HDFS-5417. Fix storage IDs in PBHelper and UpgradeUtilities. (szetszwo)
HDFS-5214. Fix NPEs in BlockManager and DirectoryScanner. (Arpit Agarwal)
HDFS-5435. File append fails to initialize storageIDs. (Junping Du via
Arpit Agarwal)
HDFS-5437. Fix TestBlockReport and TestBPOfferService failures. (Arpit
Agarwal)
HDFS-5447. Fix TestJspHelper. (Arpit Agarwal)
HDFS-5452. Fix TestReplicationPolicy and TestBlocksScheduledCounter.
HDFS-5448. Datanode should generate its ID on first registration. (Arpit
Agarwal)
HDFS-5448. Fix break caused by previous checkin for HDFS-5448. (Arpit
Agarwal)
HDFS-5455. NN should update storageMap on first heartbeat. (Arpit Agarwal)
HDFS-5457. Fix TestDatanodeRegistration, TestFsck and TestAddBlockRetry.
(Contributed by szetszwo)
HDFS-5466. Update storage IDs when the pipeline is updated. (Contributed
by szetszwo)
HDFS-5439. Fix TestPendingReplication. (Contributed by Junping Du, Arpit
Agarwal)
HDFS-5470. Add back trunk's reportDiff algorithm to the branch.
(Contributed by szetszwo)
HDFS-5472. Fix TestDatanodeManager, TestSafeMode and
TestNNThroughputBenchmark (Contributed by szetszwo)
HDFS-5475. NN incorrectly tracks more than one replica per DN. (Arpit
Agarwal)
HDFS-5481. Fix TestDataNodeVolumeFailure in branch HDFS-2832. (Contributed
by Junping Du)
HDFS-5480. Update Balancer for HDFS-2832. (Contributed by szetszwo)
HDFS-5486. Fix TestNameNodeMetrics for HDFS-2832. (Arpit Agarwal)
HDFS-5491. Update editsStored for HDFS-2832. (Arpit Agarwal)
HDFS-5494. Fix findbugs warnings for HDFS-2832. (Arpit Agarwal)
HDFS-5508. Fix compilation error after merge. (Contributed by szetszwo)
HDFS-5501. Fix pendingReceivedRequests tracking in BPServiceActor. (Arpit
Agarwal)
HDFS-5510. Fix a findbug warning in DataStorage.java on HDFS-2832 branch.
(Junping Du via Arpit Agarwal)
HDFS-5515. Fix TestDFSStartupVersions for HDFS-2832. (Arpit Agarwal)
HDFS-5527. Fix TestUnderReplicatedBlocks on branch HDFS-2832. (Arpit
Agarwal)
HDFS-5547. Fix build break after merge from trunk to HDFS-2832. (Arpit
Agarwal)
HDFS-5542. Fix TODO and clean up the code in HDFS-2832. (Contributed by
szetszwo)
HDFS-5559. Fix TestDatanodeConfig in HDFS-2832. (Contributed by szetszwo)
HDFS-5484. StorageType and State in DatanodeStorageInfo in NameNode is
not accurate. (Eric Sirianni via Arpit Agarwal)
HDFS-5648. Get rid of FsDatasetImpl#perVolumeReplicaMap. (Arpit Agarwal)
HDFS-5454. DataNode UUID should be assigned prior to FsDataset
initialization. (Arpit Agarwal)
HDFS-5406. Send incremental block reports for all storages in a
single call. (Arpit Agarwal)
HDFS-5667. Include DatanodeStorage in StorageReport. (Arpit Agarwal)
BREAKDOWN OF HDFS-4949 SUBTASKS AND RELATED JIRAS
HDFS-5049. Add JNI mlock support. (Andrew Wang via Colin Patrick McCabe)
HDFS-5051. Propagate cache status information from the DataNode to the
NameNode (Andrew Wang via Colin Patrick McCabe)
HDFS-5052. Add cacheRequest/uncacheRequest support to NameNode.
(Contributed by Colin Patrick McCabe.)
HDFS-5050. Add DataNode support for mlock and munlock (contributed by
Andrew Wang)
HDFS-5141. Add cache status information to datanode heartbeat. (Contributed
by Andrew Wang)
HDFS-5121. Add RPCs for creating and manipulating cache pools.
(Contributed by Colin Patrick McCabe)
HDFS-5163. Miscellaneous cache pool RPC fixes (Contributed by Colin Patrick
McCabe)
HDFS-5169. hdfs.c: translateZCRException: null pointer deref when
translating some exceptions (Contributed by Colin Patrick McCabe)
HDFS-5120. Add command-line support for manipulating cache pools. (cmccabe)
HDFS-5158. Add command-line support for manipulating cache directives.
(cmccabe)
HDFS-5198. NameNodeRpcServer must not send back DNA_FINALIZE in reply to a
cache report. (cmccabe)
HDFS-5195. Prevent passing null pointer to mlock and munlock. Contributed
by Chris Nauroth.
HDFS-5053. NameNode should invoke DataNode APIs to coordinate caching.
(Andrew Wang)
HDFS-5201. NativeIO: consolidate getrlimit into NativeIO#getMemlockLimit.
(Contributed by Colin Patrick McCabe)
HDFS-5197. Document dfs.cachereport.intervalMsec in hdfs-default.xml.
Contributed by Chris Nauroth.
HDFS-5210. Fix some failing unit tests on HDFS-4949 branch. (Contributed by
Andrew Wang)
HDFS-5213. Separate PathBasedCacheEntry and PathBasedCacheDirectiveWithId.
Contributed by Colin Patrick McCabe.
HDFS-5236. Change PathBasedCacheDirective APIs to be a single value rather
than batch. (Contributed by Andrew Wang)
HDFS-5119. Persist CacheManager state in the edit log. (Contributed by
Andrew Wang)
HDFS-5190. Move cache pool related CLI commands to CacheAdmin. (Contributed
by Andrew Wang)
HDFS-5309. Fix failing caching unit tests. (Andrew Wang)
HDFS-5314. Do not expose CachePool type in AddCachePoolOp (Colin Patrick
McCabe)
HDFS-5304. Expose if a block replica is cached in getFileBlockLocations.
(Contributed by Andrew Wang)
HDFS-5224. Refactor PathBasedCache* methods to use a Path rather than a
String. Contributed by Chris Nauroth.
HDFS-5348. Fix error message when dfs.datanode.max.locked.memory is
improperly configured. (Contributed by Colin Patrick McCabe)
HDFS-5349. DNA_CACHE and DNA_UNCACHE should be by blockId only (cmccabe)
HDFS-5358. Add replication field to PathBasedCacheDirective. (Contributed
by Colin Patrick McCabe)
HDFS-5359. Allow LightWeightGSet#Iterator to remove elements. (Contributed
by Colin Patrick McCabe)
HDFS-5373. hdfs cacheadmin -addDirective short usage does not mention
-replication parameter. Contributed by Chris Nauroth.
HDFS-5096. Automatically cache new data added to a cached path (contributed
by Colin Patrick McCabe)
HDFS-5383. fix broken caching unit tests (Andrew Wang)
HDFS-5388. Loading fsimage fails to find cache pools during namenode
startup (Chris Nauroth via Colin Patrick McCabe)
HDFS-5203. Concurrent clients that add a cache directive on the same path
may prematurely uncache each other. (Chris Nauroth via Colin Patrick McCabe)
HDFS-5378. In CacheReport, don't send genstamp and length on the wire
(Contributed by Colin Patrick McCabe)
HDFS-5385. Caching RPCs are AtMostOnce, but do not persist client ID and
call ID to edit log. (Chris Nauroth via Colin Patrick McCabe)
HDFS-5404 Resolve regressions in Windows compatibility on HDFS-4949 branch.
Contributed by Chris Nauroth.
HDFS-5405. Fix possible RetryCache hang for caching RPC handlers in
FSNamesystem. (Contributed by Andrew Wang)
HDFS-5419. Fixup test-patch.sh warnings on HDFS-4949 branch. (wang)
HDFS-5386. Add feature documentation for datanode caching. Contributed by
Colin Patrick McCabe.
HDFS-5468. CacheAdmin help command does not recognize commands (Stephen
Chu via Colin Patrick McCabe)
HDFS-5326. add modifyDirective to cacheAdmin (cmccabe)
HDFS-5394: Fix race conditions in DN caching and uncaching (cmccabe)
HDFS-5320. Add datanode caching metrics. Contributed by Andrew Wang.
HDFS-5482. DistributedFileSystem#listPathBasedCacheDirectives must support
relative paths. Contributed by Colin Patrick McCabe.
HDFS-5471. CacheAdmin -listPools fails when user lacks permissions to view
all pools (Andrew Wang via Colin Patrick McCabe)
HDFS-5450. better API for getting the cached blocks locations. Contributed
by Andrew Wang.
HDFS-5485. add command-line support for modifyDirective (cmccabe)
HDFS-5366. recaching improvements (cmccabe)
HDFS-5520. loading cache path directives from edit log doesnt update
nextEntryId (cmccabe)
HDFS-5512. CacheAdmin -listPools fails with NPE when user lacks permissions
to view all pools (awang via cmccabe)
HDFS-5513. CacheAdmin commands fail when using . as the path. Contributed
by Andrew Wang.
HDFS-5511. improve CacheManipulator interface to allow better unit testing
(cmccabe)
HDFS-5451. Add byte and file statistics to PathBasedCacheEntry. Contributed
by Colin Patrick McCabe.
HDFS-5473. Consistent naming of user-visible caching classes and methods
(cmccabe)
HDFS-5543. Fix narrow race condition in TestPathBasedCacheRequests
(cmccabe)
HDFS-5565. CacheAdmin help should match against non-dashed commands (wang
via cmccabe)
HDFS-5556. Add some more NameNode cache statistics, cache pool stats
(cmccabe)
HDFS-5562. TestCacheDirectives and TestFsDatasetCache should stub out
native mlock. Contributed by Colin Patrick McCabe and Akira Ajisaka.
HDFS-5430. Support TTL on CacheDirectives. Contributed by Andrew Wang.
HDFS-5555. CacheAdmin commands fail when first listed NameNode is in
Standby (jxiang via cmccabe)
HDFS-5626. dfsadmin report shows incorrect values (cmccabe)
HDFS-5630. Hook up cache directive and pool usage statistics. (wang)
HDFS-5665. Remove the unnecessary writeLock while initializing CacheManager
in FsNameSystem Ctor. (Uma Maheswara Rao G via Andrew Wang)
HDFS-5431. Support cachepool-based limit management in path-based caching.
(awang via cmccabe)