| Hadoop HDFS Change Log |
| |
| Release 2.6.4 - UNRELEASED |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| HDFS-8647. Abstract BlockManager's rack policy into BlockPlacementPolicy. |
| (Brahma Reddy Battula via mingma) |
| |
| HDFS-9313. Possible NullPointerException in BlockManager if no excess |
| replica can be chosen. (mingma) |
| |
| HDFS-9314. Improve BlockPlacementPolicyDefault's picking of excess |
| replicas. (Xiao Chen via zhz) |
| |
| HDFS-6945. BlockManager should remove a block from excessReplicateMap and |
| decrement ExcessBlocks metric when the block is removed. (aajisaka) |
| |
| HDFS-9574. Reduce client failures during datanode restart (kihwal) |
| |
| HDFS-9415. Document dfs.cluster.administrators and |
| dfs.permissions.superusergroup. (Xiaobing Zhou via Arpit Agarwal) |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| HDFS-9445. Datanode may deadlock while handling a bad volume. |
| (Wlater Su via Kihwal) |
| |
| HDFS-9600. do not check replication if the block is under construction |
| (Phil Yang via vinayakumarb) |
| |
| HDFS-8767. RawLocalFileSystem.listStatus() returns null for UNIX pipefile. |
| (Kanaka Kumar Avvaru via wheat9) |
| |
| HDFS-9294. DFSClient deadlock when close file and failed to renew lease. |
| (Brahma Reddy Battula via szetszwo) |
| |
| HDFS-4660. Block corruption can happen during pipeline recovery (kihwal) |
| |
| HDFS-9178. Slow datanode I/O can cause a wrong node to be marked bad. |
| (Kihwal Lee) |
| |
| HDFS-8722. Optimize datanode writes for small writes and flushes. |
| (Kihwal Lee) |
| |
| HDFS-9220. Reading small file (< 512 bytes) that is open for append fails |
| due to incorrect checksum (Jing Zhao via kihwal) |
| |
| Release 2.6.3 - 2015-12-17 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| HDFS-8615. Correct HTTP method in WebHDFS document. |
| (Brahma Reddy Battula via aajisaka) |
| |
| HDFS-9431. DistributedFileSystem#concat fails if the target path is |
| relative. (Kazuho Fujii via aajisaka) |
| |
| HDFS-9083. Replication violates block placement policy (Rushabh Shah) |
| |
| HDFS-9289. Make DataStreamer#block thread safe and verify genStamp in |
| commitBlock. (Chang Li via zhz) |
| |
| HDFS-9434. Recommission a datanode with 500k blocks may pause NN for 30 |
| seconds for printing info log messags. (szetszwo) |
| |
| HDFS-9470. Encryption zone on root not loaded from fsimage after NN |
| restart. (Xiao Chen via wang) |
| |
| HDFS-9273. ACLs on root directory may be lost after NN restart. |
| (Xiao Chen via cnauroth) |
| |
| Release 2.6.2 - 2015-10-28 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| Release 2.6.1 - 2015-09-23 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| HDFS-7278. Add a command that allows sysadmins to manually trigger full |
| block reports from a DN (cmccabe) |
| |
| IMPROVEMENTS |
| |
| HDFS-7035. Make adding a new data directory to the DataNode an atomic |
| operation and improve error handling (Lei Xu via Colin P. McCabe) |
| |
| HDFS-7531. Improve the concurrent access on FsVolumeList (Lei Xu via Colin |
| P. McCabe) |
| |
| HDFS-7579. Improve log reporting during block report rpc failure. |
| (Charles Lamb via cnauroth) |
| |
| HDFS-7182. JMX metrics aren't accessible when NN is busy. (Ming Ma via jing9) |
| |
| HDFS-7596. NameNode should prune dead storages from storageMap. |
| (Arpit Agarwal via cnauroth) |
| |
| HDFS-8046. Allow better control of getContentSummary (kihwal) |
| |
| HDFS-8384. Allow NN to startup if there are files having a lease but are not |
| under construction. (jing9) |
| |
| HDFS-7694. FSDataInputStream should support "unbuffer" (cmccabe) |
| |
| OPTIMIZATIONS |
| |
| HDFS-8480. Fix performance and timeout issues in HDFS-7929 by using |
| hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang |
| via Colin P. McCabe) |
| |
| BUG FIXES |
| |
| HDFS-7425. NameNode block deletion logging uses incorrect appender. |
| (cnauroth) |
| |
| HDFS-4882. Prevent the Namenode's LeaseManager from looping forever in |
| checkLeases (Ravi Prakash via Colin P. McCabe) |
| |
| HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang datanodes |
| (Noah Lorang via Colin P. McCabe) |
| |
| HDFS-7503. Namenode restart after large deletions can cause slow |
| processReport. (Arpit Agarwal) |
| |
| HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate |
| block files are present in the same volume (cmccabe) |
| |
| HDFS-3443. Fix NPE when namenode transition to active during startup by |
| adding checkNNStartup() in NameNodeRpcServer. (Vinayakumar B via szetszwo) |
| |
| HDFS-7733. NFS: readdir/readdirplus return null directory |
| attribute on failure. (Arpit Agarwal) |
| |
| HDFS-8486. DN startup may cause severe data loss. (daryn via cmccabe) |
| |
| HDFS-7213. processIncrementalBlockReport performance degradation. |
| (Eric Payne via kihwal) |
| |
| HDFS-7235. DataNode#transferBlock should report blocks that don't exist |
| using reportBadBlock (yzhang via cmccabe) |
| |
| HDFS-7263. Snapshot read can reveal future bytes for appended files. |
| (Tao Luo via shv) |
| |
| HDFS-7225. Remove stale block invalidation work when DN re-registers with |
| different UUID. (Zhe Zhang and Andrew Wang) |
| |
| HDFS-7533. Datanode sometimes does not shutdown on receiving upgrade |
| shutdown command (Eric Payne via kihwal) |
| |
| HDFS-7575. Upgrade should generate a unique storage ID for each |
| volume. (Arpit Agarwal) |
| |
| HDFS-7707. Edit log corruption due to delayed block removal again. |
| (Yongjun Zhang via kihwal) |
| |
| HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause |
| DataNode to register successfully with only one NameNode.(vinayakumarb) |
| |
| HDFS-7788. Post-2.6 namenode may not start up with an image containing |
| inodes created with an old release. (Rushabh Shah via kihwal) |
| |
| HDFS-7009. Active NN and standby NN have different live nodes. |
| (Ming Ma via cnauroth) |
| |
| HDFS-7763. fix zkfc hung issue due to not catching exception in a corner |
| case. (Liang Xie via wang) |
| |
| HDFS-7871. NameNodeEditLogRoller can keep printing "Swallowing exception" |
| message. (jing9) |
| |
| HDFS-7885. Datanode should not trust the generation stamp provided by |
| client. (Tsz Wo Nicholas Sze via jing9) |
| |
| HDFS-7610. Fix removal of dynamically added DN volumes (Lei (Eddy) Xu via |
| Colin P. McCabe) |
| |
| HDFS-7830. DataNode does not release the volume lock when adding a volume |
| fails. (Lei Xu via Colin P. Mccabe) |
| |
| HDFS-7587. Edit log corruption can happen if append fails with a quota |
| violation. (jing9) |
| |
| HDFS-7929. inotify unable fetch pre-upgrade edit log segments once upgrade |
| starts (Zhe Zhang via Colin P. McCabe) |
| |
| HDFS-7930. commitBlockSynchronization() does not remove locations. (yliu) |
| |
| HDFS-7884. Fix NullPointerException in BlockSender when the generation stamp |
| provided by the client is larger than the one stored in the datanode. |
| (Brahma Reddy Battula via szetszwo) |
| |
| HDFS-7960. The full block report should prune zombie storages even if |
| they're not empty. (cmccabe and Eddy Xu via wang) |
| |
| HDFS-7742. Favoring decommissioning node for replication can cause a block |
| to stay underreplicated for long periods (Nathan Roberts via kihwal) |
| |
| HDFS-7999. FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl |
| lock for a very long time (sinago via cmccabe) |
| |
| HDFS-8072. Reserved RBW space is not released if client terminates while |
| writing block. (Arpit Agarwal) |
| |
| HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to |
| finalize upgrade. (jing9) |
| |
| HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and |
| fail to tell the DFSClient about it because of a network error (cmccabe) |
| |
| HDFS-8070. Pre-HDFS-7915 DFSClient cannot use short circuit on |
| post-HDFS-7915 DataNode (cmccabe) |
| |
| HDFS-8219. setStoragePolicy with folder behavior is different after cluster restart. |
| (surendra singh lilhore via Xiaoyu Yao) |
| |
| HDFS-7894. Rolling upgrade readiness is not updated in jmx until query |
| command is issued. (Brahma Reddy Battula via kihwal) |
| |
| HDFS-8254. Standby namenode doesn't process DELETED_BLOCK if the add block |
| request is in edit log. (Rushabh S Shah via kihwal) |
| |
| HDFS-8404. Pending block replication can get stuck using older genstamp |
| (Nathan Roberts via kihwal) |
| |
| HDFS-8431. hdfs crypto class not found in Windows. |
| (Anu Engineer via cnauroth) |
| |
| HDFS-7609. Avoid retry cache collision when Standby NameNode loading edits. |
| (Ming Ma via jing9) |
| |
| HDFS-8270. create() always retried with hardcoded timeout when file already |
| exists with open lease (J.Andreina via vinayakumarb) |
| |
| HDFS-7980. Incremental BlockReport will dramatically slow down namenode |
| startup. (Walter Su via szetszwo) |
| |
| HDFS-7446. HDFS inotify should have the ability to determine what txid it |
| has read up to (cmccabe) |
| |
| HDFS-8846. Add a unit test for INotify functionality across a layout |
| version upgrade (Zhe Zhang via Colin P. McCabe) |
| |
| HDFS-7470. SecondaryNameNode need twice memory when calling |
| reloadFromImageFile. (zhaoyunjiong via cnauroth) |
| |
| HDFS-8863. The remaining space check in BlockPlacementPolicyDefault is |
| flawed. (Kihwal Lee via yliu) |
| |
| HDFS-7552. Change FsVolumeList toString() to fix |
| TestDataNodeVolumeFailureToleration (Liang Xie via Colin P. McCabe) |
| |
| Release 2.6.0 - 2014-11-18 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| HDFS-6584. Support archival storage. (See breakdown of tasks below for |
| features and contributors) |
| |
| HDFS-6581. Support writing to a single replica in RAM. (See breakdown of |
| tasks below for features and contributors). |
| |
| IMPROVEMENTS |
| |
| HDFS-6613. Improve logging in caching classes. (wang) |
| |
| HDFS-6511. BlockManager#computeInvalidateWork() could do nothing. (Juan Yu via wang) |
| |
| HDFS-6638. Shorten test run time with a smaller retry timeout setting. |
| (Liang Xie via cnauroth) |
| |
| HDFS-6627. Rename DataNode#checkWriteAccess to checkReadAccess. |
| (Liang Xie via cnauroth) |
| |
| HDFS-6645. Add test for successive Snapshots between XAttr modifications. |
| (Stephen Chu via jing9) |
| |
| HDFS-6643. Refactor INodeWithAdditionalFields.PermissionStatusFormat and |
| INodeFile.HeaderFormat. (szetszwo) |
| |
| HDFS-6640. Syntax for MKDIRS, CREATESYMLINK, and SETXATTR are given wrongly |
| in WebHdfs document (missed webhdfs/v1). (Stephen Chu via jing9) |
| |
| HDFS-5202. Support Centralized Cache Management on Windows. (cnauroth) |
| |
| HDFS-2976. Remove unnecessary method (tokenRefetchNeeded) in DFSClient. |
| (Uma Maheswara Rao G) |
| |
| HDFS-3851. DFSOutputStream class code cleanup. (Jing Zhao via suresh) |
| |
| HDFS-2856. Fix block protocol so that Datanodes don't require root or jsvc. |
| (cnauroth) |
| |
| HDFS-5624. Add HDFS tests for ACLs in combination with viewfs. |
| (Stephen Chu via cnauroth) |
| |
| HDFS-6655. Add 'header banner' to 'explorer.html' also in Namenode UI |
| (vinayakumarb) |
| |
| HDFS-4120. Add a new "-skipSharedEditsCheck" option for BootstrapStandby |
| (Liang Xie and Rakesh R via vinayakumarb) |
| |
| HDFS-6597. Add a new option to NN upgrade to terminate the process after |
| upgrade on NN is completed. (Danilo Vunjak via cnauroth) |
| |
| HDFS-6700. BlockPlacementPolicy shoud choose storage but not datanode for |
| deletion. (szetszwo) |
| |
| HDFS-6616. Add exclude-datanodes feature to WebHDFS redirection so that it |
| will not redirect retries to the same datanode. (zhaoyunjiong via szetszwo) |
| |
| HDFS-6702. Change DFSClient to pass the StorageType from the namenode to |
| datanodes and change datanode to write block replicas using the specified |
| storage type. (szetszwo) |
| |
| HDFS-6701. Make seed optional in NetworkTopology#sortByDistance. |
| (Ashwin Shankar via wang) |
| |
| HDFS-6755. There is an unnecessary sleep in the code path where |
| DFSOutputStream#close gives up its attempt to contact the namenode |
| (mitdesai21 via cmccabe) |
| |
| HDFS-6750. The DataNode should use its shared memory segment to mark |
| short-circuit replicas that have been unlinked as stale (cmccabe) |
| |
| HDFS-6739. Add getDatanodeStorageReport to ClientProtocol. (szetszwo) |
| |
| HDFS-6665. Add tests for XAttrs in combination with viewfs. |
| (Stephen Chu via wang) |
| |
| HDFS-6778. The extended attributes javadoc should simply refer to the |
| user docs. (clamb via wang) |
| |
| HDFS-6570. add api that enables checking if a user has certain permissions on |
| a file. (Jitendra Pandey via cnauroth) |
| |
| HDFS-6441. Add ability to exclude/include specific datanodes while |
| balancing. (Benoy Antony and Yu Li via Arpit Agarwal) |
| |
| HDFS-6685. Balancer should preserve storage type of replicas. (szetszwo) |
| |
| HDFS-6798. Add test case for incorrect data node condition during |
| balancing. (Benoy Antony via Arpit Agarwal) |
| |
| HDFS-6796. Improve the argument check during balancer command line parsing. |
| (Benoy Antony via szetszwo) |
| |
| HDFS-6794. Update BlockManager methods to use DatanodeStorageInfo |
| where possible (Arpit Agarwal) |
| |
| HDFS-6802. Some tests in TestDFSClientFailover are missing @Test |
| annotation. (Akira Ajisaka via wang) |
| |
| HDFS-6788. Improve synchronization in BPOfferService with read write lock. |
| (Yongjun Zhang via wang) |
| |
| HDFS-6787. Remove duplicate code in FSDirectory#unprotectedConcat. (Yi Liu via umamahesh) |
| |
| HDFS-6809. Move Balancer's inner classes MovedBlocks and Matcher as to |
| standalone classes and separates KeyManager from NameNodeConnector. |
| (szetszwo) |
| |
| HDFS-6812. Remove addBlock and replaceBlock from DatanodeDescriptor. |
| (szetszwo) |
| |
| HDFS-6781. Separate HDFS commands from CommandsManual.apt.vm. (Akira |
| Ajisaka via Arpit Agarwal) |
| |
| HDFS-6728. Dynamically add new volumes to DataStorage, formatted if |
| necessary. (Lei Xu vi atm) |
| |
| HDFS-6740. Make FSDataset support adding data volumes dynamically. (Lei |
| Xu via atm) |
| |
| HDFS-6722. Display readable last contact time for dead nodes on NN webUI. |
| (Ming Ma via wheat9) |
| |
| HDFS-6772. Get DN storages out of blockContentsStale state faster after |
| NN restarts. (Ming Ma via Arpit Agarwal) |
| |
| HDFS-573. Porting libhdfs to Windows. (cnauroth) |
| |
| HDFS-6828. Separate block replica dispatching from Balancer. (szetszwo via |
| jing9) |
| |
| HDFS-6837. Code cleanup for Balancer and Dispatcher. (szetszwo via |
| jing9) |
| |
| HDFS-6838. Code cleanup for unnecessary INode replacement. |
| (Jing Zhao via wheat9) |
| |
| HDFS-6836. HDFS INFO logging is verbose & uses file appenders. (Xiaoyu |
| Yao via Arpit Agarwal) |
| |
| HDFS-6567. Normalize the order of public final in HdfsFileStatus. |
| (Tassapol Athiapinya via wheat9) |
| |
| HDFS-6849. Replace HttpFS custom proxyuser handling with common |
| implementation. (tucu) |
| |
| HDFS-6850. Move NFS out of order write unit tests into TestWrites class. |
| (Zhe Zhang via atm) |
| |
| HDFS-6188. An ip whitelist based implementation of TrustedChannelResolver. |
| (Benoy Antony via Arpit Agarwal) |
| |
| HDFS-6858. Allow dfs.data.transfer.saslproperties.resolver.class default to |
| hadoop.security.saslproperties.resolver.class. (Benoy Antony via cnauroth) |
| |
| HDFS-6878. Change MiniDFSCluster to support StorageType configuration |
| for individual directories. (Arpit Agarwal) |
| |
| HDFS-6758. block writer should pass the expected block size to |
| DataXceiverServer. (Arpit Agarwal) |
| |
| HDFS-6899. Allow changing MiniDFSCluster volumes per DN and capacity |
| per volume. (Arpit Agarwal) |
| |
| HDFS-4486. Add log category for long-running DFSClient notices (Zhe Zhang |
| via Colin Patrick McCabe) |
| |
| HDFS-6879. Adding tracing to Hadoop RPC (Masatake Iwasaki via Colin Patrick |
| McCabe) |
| |
| HDFS-6774. Make FsDataset and DataStore support removing volumes. (Lei Xu |
| via atm) |
| |
| HDFS-6634. inotify in HDFS. (James Thomas via wang) |
| |
| HDFS-4257. The ReplaceDatanodeOnFailure policies could have a forgiving |
| option (szetszwo via cmccabe) |
| |
| HDFS-6959. Make the HDFS home directory location customizable. (yzhang via |
| cmccabe) |
| |
| HDFS-6609. Use DirectorySnapshottableFeature to represent a snapshottable |
| directory. (Jing Zhao via wheat9) |
| |
| HDFS-6886. Use single editlog record for creating file + overwrite. (Yi Liu |
| via jing9) |
| |
| HDFS-6376. Distcp data between two HA clusters requires another configuration. |
| (Dave Marion and Haohui Mai via jing9) |
| |
| HDFS-6943. Improve NN allocateBlock log to include replicas' datanode IPs. |
| (Ming Ma via wheat9) |
| |
| HDFS-6036. Forcibly timeout misbehaving DFSClients that try to do |
| no-checksum reads that extend too long (cmccabe) |
| |
| HDFS-6482. Use block ID-based block layout on datanodes (James Thomas via |
| Colin Patrick McCabe) |
| |
| HDFS-7061. Add test to verify encryption zone creation after NameNode |
| restart without saving namespace. (Stephen Chu via wang) |
| |
| HDFS-7059. HAadmin transtionToActive with forceActive option can show |
| confusing message. |
| |
| HDFS-6880. Adding tracing to DataNode data transfer protocol. (iwasakims |
| via cmccabe) |
| |
| HDFS-7006. Test encryption zones with KMS. (Anthony Young-Garner and tucu) |
| |
| HDFS-6851. Refactor EncryptionZoneWithId and EncryptionZone. (clamb via wang) |
| |
| HDFS-6705. Create an XAttr that disallows the HDFS admin from accessing a |
| file. (clamb via wang) |
| |
| HDFS-6843. Create FileStatus isEncrypted() method (clamb via cmccabe) |
| |
| HDFS-7004. Update KeyProvider instantiation to create by URI. (wang) |
| |
| HDFS-7047. Expose FileStatus#isEncrypted in libhdfs (cmccabe) |
| |
| HDFS-7003. Add NFS Gateway support for reading and writing to |
| encryption zones. (clamb via wang) |
| |
| HDFS-6727. Refresh data volumes on DataNode based on configuration changes |
| (Lei Xu via cmccabe) |
| |
| HDFS-6970. Move startFile EDEK retries to the DFSClient. (wang) |
| |
| HDFS-6948. DN rejects blocks if it has older UC block |
| (Eric Payne via kihwal) |
| |
| HDFS-6987. Move CipherSuite xattr information up to the encryption zone |
| root. (Zhe Zhang via wang) |
| |
| HDFS-7139. Unit test for creating encryption zone on root path. (Zhe Zhang via wang) |
| |
| HDFS-7138. Fix hftp to work with encryption. (clamb via wang) |
| |
| HDFS-7118. Improve diagnostics on storage directory rename operations by |
| using NativeIO#renameTo in Storage#rename. (cnauroth) |
| |
| HDFS-6808. Add command line option to ask DataNode reload configuration. |
| (Lei Xu via Colin Patrick McCabe) |
| |
| HDFS-7119. Split error checks in AtomicFileOutputStream#close into separate |
| conditions to improve diagnostics. (cnauroth) |
| |
| HDFS-7077. Separate CipherSuite from crypto protocol version. (wang) |
| |
| HDFS-6956. Allow dynamically changing the tracing level in Hadoop servers |
| (cmccabe) |
| |
| HDFS-7156. Update fsck documentation. (Masahiro Yamaguch via shv) |
| |
| HDFS-7093. Add config key to restrict setStoragePolicy. (Arpit Agarwal) |
| |
| HDFS-6519. Document oiv_legacy command (Akira AJISAKA via aw) |
| |
| HDFS-4165. Faulty sanity check in FsDirectory.unprotectedSetQuota. |
| (Binglin Chang via suresh) |
| |
| HDFS-7104. Fix and clarify INodeInPath getter functions. (Zhe Zhang via wang) |
| |
| HDFS-7124. Remove EncryptionZoneManager.NULL_EZ. (clamb via wang) |
| |
| HDFS-6779. Add missing version subcommand for hdfs. |
| (Sasaki Toru via wheat9) |
| |
| HDFS-7153. Add storagePolicy to NN edit log during file creation. |
| (Arpit Agarwal) |
| |
| HDFS-7158. Reduce the memory usage of WebImageViewer. (wheat9) |
| |
| HDFS-6894. Add XDR parser method for each NFS response. |
| (Brandon Li via wheat9) |
| |
| HDFS-7217. Better batching of IBRs. (kihwal) |
| |
| HDFS-7195. Update user doc of secure mode about Datanodes don't require root |
| or jsvc. (cnauroth) |
| |
| HDFS-7228. Add an SSD policy into the default BlockStoragePolicySuite. |
| (jing9) |
| |
| HDFS-6904. YARN unable to renew delegation token fetched via webhdfs |
| due to incorrect service port. (jitendra) |
| |
| HDFS-6988. Improve HDFS-6581 eviction configuration (Xiaoyu Yao via Colin |
| P. McCabe) |
| |
| HDFS-7230. Add rolling downgrade documentation. (szetszwo via jing9) |
| |
| HDFS-6385. Show when block deletion will start after NameNode startup in |
| WebUI. (cnauroth) |
| |
| HDFS-7313. Support optional configuration of AES cipher suite on |
| DataTransferProtocol. (cnauroth) |
| |
| HDFS-7276. Limit the number of byte arrays used by DFSOutputStream and |
| provide a mechanism for recycling arrays. (szetszwo) |
| |
| HDFS-7233. NN logs unnecessary org.apache.hadoop.hdfs.protocol.UnresolvedPathException. |
| (Rushabh S Shah via jing9) |
| |
| HDFS-7221. TestDNFencingWithReplication fails consistently. (Charles Lamb via wang) |
| |
| OPTIMIZATIONS |
| |
| HDFS-6690. Deduplicate xattr names in memory. (wang) |
| |
| HDFS-6773. MiniDFSCluster should skip edit log fsync by default (Stephen |
| Chu via Colin Patrick McCabe) |
| |
| HDFS-6865. Byte array native checksumming on client side |
| (James Thomas via todd) |
| |
| HDFS-7122. Use of ThreadLocal<Random> results in poor block placement. |
| (wang) |
| |
| HDFS-6606. Optimize HDFS Encrypted Transport performance. (yliu) |
| |
| BUG FIXES |
| |
| HDFS-7309. XMLUtils.mangleXmlString doesn't seem to handle less than sign |
| (Colin Patrick McCabe via raviprak) |
| |
| HDFS-6823. dfs.web.authentication.kerberos.principal shows up in logs for |
| insecure HDFS (Allen Wittenauer via raviprak) |
| |
| HDFS-6517. Remove hadoop-metrics2.properties from hdfs project (Akira |
| AJISAKA via aw) |
| |
| HDFS-6617. Flake TestDFSZKFailoverController.testManualFailoverWithDFSHAAdmin |
| due to a long edit log sync op. (Liang Xie via cnauroth) |
| |
| HDFS-6646. [ HDFS Rolling Upgrade - Shell ] shutdownDatanode and getDatanodeInfo |
| usage is missed ( Brahma Reddy Battula via vinayakumarb) |
| |
| HDFS-6630. Unable to fetch the block information by Browsing the file system on |
| Namenode UI through IE9 ( Haohui Mai via vinayakumarb) |
| |
| HADOOP-8158. Interrupting hadoop fs -put from the command line |
| causes a LeaseExpiredException. (daryn via harsh) |
| |
| HDFS-6678. MiniDFSCluster may still be partially running after initialization |
| fails. (cnauroth) |
| |
| HDFS-5809. BlockPoolSliceScanner and high speed hdfs appending make |
| datanode to drop into infinite loop (cmccabe) |
| |
| HDFS-6456. NFS should throw error for invalid entry in |
| dfs.nfs.exports.allowed.hosts (Abhiraj Butala via brandonli) |
| |
| HDFS-6689. NFS doesn't return correct lookup access for direcories (brandonli) |
| |
| HDFS-6478. RemoteException can't be retried properly for non-HA scenario. |
| (Ming Ma via jing9) |
| |
| HDFS-6693. TestDFSAdminWithHA fails on windows ( vinayakumarb ) |
| |
| HDFS-6667. In HDFS HA mode, Distcp/SLive with webhdfs on secure cluster fails |
| with Client cannot authenticate via:[TOKEN, KERBEROS] error. (jing9) |
| |
| HDFS-6704. Fix the command to launch JournalNode in HDFS-HA document. |
| (Akira AJISAKA via jing9) |
| |
| HDFS-6731. Run "hdfs zkfc-formatZK" on a server in a non-namenode will cause |
| a null pointer exception. (Masatake Iwasaki via brandonli) |
| |
| HDFS-6114. Block Scan log rolling will never happen if blocks written |
| continuously leading to huge size of dncp_block_verification.log.curr |
| (vinayakumarb via cmccabe) |
| |
| HDFS-6455. NFS: Exception should be added in NFS log for invalid separator in |
| nfs.exports.allowed.hosts. (Abhiraj Butala via brandonli) |
| |
| HDFS-6715. Webhdfs wont fail over when it gets java.io.IOException: Namenode |
| is in startup mode. (jing9) |
| |
| HDFS-5919. FileJournalManager doesn't purge empty and corrupt inprogress edits |
| files (vinayakumarb) |
| |
| HDFS-6752. Avoid Address bind errors in TestDatanodeConfig#testMemlockLimit |
| (vinayakumarb) |
| |
| HDFS-6749. FSNamesystem methods should call resolvePath. |
| (Charles Lamb via cnauroth) |
| |
| HDFS-4629. Using com.sun.org.apache.xml.internal.serialize.* in |
| XmlEditsVisitor.java is JVM vendor specific. Breaks IBM JAVA. |
| (Amir Sanjar via stevel) |
| |
| HDFS-3482. hdfs balancer throws ArrayIndexOutOfBoundsException |
| if option is specified without values. ( Madhukara Phatak via umamahesh) |
| |
| HDFS-6797. DataNode logs wrong layoutversion during upgrade. (Benoy Antony |
| via Arpit Agarwal) |
| |
| HDFS-6810. StorageReport array is initialized with wrong size in |
| DatanodeDescriptor#getStorageReports. (szetszwo via Arpit Agarwal) |
| |
| HDFS-5723. Append failed FINALIZED replica should not be accepted as valid |
| when that block is underconstruction (vinayakumarb) |
| |
| HDFS-5185. DN fails to startup if one of the data dir is full. (vinayakumarb) |
| |
| HDFS-6451. NFS should not return NFS3ERR_IO for AccessControlException |
| (Abhiraj Butala via brandonli) |
| |
| HDFS-6717. JIRA HDFS-5804 breaks default nfs-gateway behavior for unsecured config |
| (brandonli) |
| |
| HDFS-6790. DFSUtil Should Use configuration.getPassword for SSL passwords |
| (Larry McCay via brandonli) |
| |
| HDFS-6791. A block could remain under replicated if all of its replicas are on |
| decommissioned nodes. (Ming Ma via jing9) |
| |
| HDFS-6582. Missing null check in RpcProgramNfs3#read(XDR, SecurityHandler) |
| (Abhiraj Butala via brandonli) |
| |
| HDFS-6830. BlockInfo.addStorage fails when DN changes the storage for a |
| block replica (Arpit Agarwal) |
| |
| HDFS-6247. Avoid timeouts for replaceBlock() call by sending intermediate |
| responses to Balancer (vinayakumarb) |
| |
| HDFS-6783. Fix HDFS CacheReplicationMonitor rescan logic. (Yi Liu and Colin Patrick McCabe via umamahesh) |
| |
| HDFS-6825. Edit log corruption due to delayed block removal. |
| (Yongjun Zhang via wang) |
| |
| HDFS-6569. OOB message can't be sent to the client when DataNode shuts down for upgrade |
| (brandonli) |
| |
| HDFS-6868. portmap and nfs3 are documented as hadoop commands instead of hdfs |
| (brandonli) |
| |
| HDFS-6870. Blocks and INodes could leak for Rename with overwrite flag. (Yi |
| Liu via jing9) |
| |
| HDFS-6890. NFS readdirplus doesn't return dotdot attributes (brandonli) |
| |
| HDFS-6829. DFSAdmin refreshSuperUserGroupsConfiguration failed in |
| security cluster (zhaoyunjiong via Arpit Agarwal) |
| |
| HDFS-4852. libhdfs documentation is out of date. (cnauroth) |
| |
| HDFS-6908. Incorrect snapshot directory diff generated by snapshot deletion. |
| (Juan Yu and jing9 via jing9) |
| |
| HDFS-6892. Add XDR packaging method for each NFS request (brandonli) |
| |
| HDFS-6902. FileWriter should be closed in finally block in |
| BlockReceiver#receiveBlock() (Tsuyoshi OZAWA via Colin Patrick McCabe) |
| |
| HDFS-6972. TestRefreshUserMappings.testRefreshSuperUserGroupsConfiguration |
| doesn't decode url correctly. (Yongjun Zhang via wang) |
| |
| HDFS-6942. Fix typos in log messages. (Ray Chiang via wheat9) |
| |
| HDFS-6848. Lack of synchronization on access to datanodeUuid in |
| DataStorage#format(). (Xiaoyu Yao via Arpit Agarwal) |
| |
| HDFS-6996. SnapshotDiff report can hit IndexOutOfBoundsException when there |
| are nested renamed directory/file. (jing9) |
| |
| HDFS-6831. Inconsistency between 'hdfs dfsadmin' and 'hdfs dfsadmin -help'. |
| (Xiaoyu Yao via Arpit Agarwal) |
| |
| HDFS-6979. hdfs.dll does not produce .pdb files. (cnauroth) |
| |
| HDFS-6862. Add missing timeout annotations to tests. (Xiaoyu Yao via |
| Arpit Agarwal) |
| |
| HDFS-6898. DN must reserve space for a full block when an RBW block is |
| created. (Arpit Agarwal) |
| |
| HDFS-7025. HDFS Credential Provider related Unit Test Failure. |
| (Xiaoyu Yao via cnauroth) |
| |
| HDFS-7005. DFS input streams do not timeout. |
| |
| HDFS-6951. Correctly persist raw namespace xattrs to edit log and fsimage. |
| (clamb via wang) |
| |
| HDFS-6800. Support Datanode layout changes with rolling upgrade. |
| (James Thomas via Arpit Agarwal) |
| |
| HDFS-6981. Fix DN upgrade with layout version change. (Arpit Agarwal) |
| |
| HDFS-6506. Newly moved block replica been invalidated and deleted in |
| TestBalancer. (Binglin Chang via cnauroth) |
| |
| HDFS-6966. Add additional unit tests for encryption zones. |
| (Stephen Chu via wang) |
| |
| HDFS-6621. Hadoop Balancer prematurely exits iterations. |
| (Rafal Wojdyla and Benjamin Bowman via wang) |
| |
| HDFS-7045. Fix NameNode deadlock when opening file under /.reserved path. |
| (Yi Liu via wang) |
| |
| HDFS-7032. Add WebHDFS support for reading and writing to encryption zones. |
| (clamb via wang) |
| |
| HDFS-6965. NN continues to issue block locations for DNs with full disks. |
| (Rushabh Shah via kihwal) |
| |
| HDFS-6789. TestDFSClientFailover.testFileContextDoesntDnsResolveLogicalURI |
| and TestDFSClientFailover.testDoesntDnsResolveLogicalURI failing on jdk7. |
| (Akira Ajisaka via wang) |
| |
| HDFS-6912. SharedFileDescriptorFactory should not allocate sparse files |
| (cmccabe) |
| |
| HDFS-7075. hadoop-fuse-dfs fails because it cannot find |
| JavaKeyStoreProvider$Factory (cmccabe) |
| |
| HDFS-7078. Fix listEZs to work correctly with snapshots. (wang) |
| |
| HDFS-6840. Clients are always sent to the same datanode when read |
| is off rack. (wang) |
| |
| HDFS-7065. Pipeline close recovery race can cause block corruption (kihwal) |
| |
| HDFS-7096. Fix TestRpcProgramNfs3 to use DFS_ENCRYPTION_KEY_PROVIDER_URI |
| (clamb via cmccabe) |
| |
| HDFS-7046. HA NN can NPE upon transition to active. (kihwal) |
| |
| HDFS-7106. Reconfiguring DataNode volumes does not release the lock files |
| in removed volumes. (cnauroth via cmccabe) |
| |
| HDFS-7001. Tests in TestTracing depends on the order of execution |
| (iwasakims via cmccabe) |
| |
| HDFS-7132. hdfs namenode -metadataVersion command does not honor |
| configured name dirs. (Charles Lamb via wang) |
| |
| HDFS-7049. TestByteRangeInputStream.testPropagatedClose fails and throw |
| NPE on branch-2. (Juan Yu via wheat9) |
| |
| HDFS-7148. TestEncryptionZones#testIsEncryptedMethod fails on branch-2 |
| after archival storage merge. (wang) |
| |
| HDFS-7157. Using Time.now() for recording start/end time of reconfiguration |
| tasks (Lei Xu via cmccabe) |
| |
| HDFS-6664. HDFS permissions guide documentation states incorrect default |
| group mapping class. (Ray Chiang via aw) |
| |
| HDFS-4227. Document dfs.namenode.resource.* (Daisuke Kobayashi via aw) |
| |
| HDFS-6754. TestNamenodeCapacityReport may sometimes fail due to lack of |
| retry. (Mit Desai via kihwal) |
| |
| HDFS-7178. Additional unit test for replica write with full disk. |
| (Arpit Agarwal) |
| |
| HDFS-7179. DFSClient should instantiate a KeyProvider, not a |
| KeyProviderCryptoExtension. (wang) |
| |
| HDFS-7181. Remove incorrect precondition check on key length in |
| FileEncryptionInfo. (wang) |
| |
| HDFS-7162. Wrong path when deleting through fuse-dfs a file which already |
| exists in trash (Chenging Liu via cmccabe) |
| |
| HDFS-7176. The namenode usage message doesn't include "-rollingupgrade |
| started" (cmccabe) |
| |
| HDFS-7172. Test data files may be checked out of git with incorrect line |
| endings, causing test failures in TestHDFSCLI. (Chris Nauroth via wheat9) |
| |
| HDFS-7203. Concurrent appending to the same file can cause data corruption |
| (kihwal) |
| |
| HDFS-7236. Fix TestOpenFilesWithSnapshot#testOpenFilesWithMultipleSnapshots. |
| (Yongjun Zhang via jing9) |
| |
| HDFS-6544. Broken Link for GFS in package.html. (Suraj Nayak M via wheat9) |
| |
| HDFS-7237. The command "hdfs namenode -rollingUpgrade" throws |
| ArrayIndexOutOfBoundsException. (szetszwo) |
| |
| HDFS-7185. The active NameNode will not accept an fsimage sent from the |
| standby during rolling upgrade. (jing9) |
| |
| HDFS-7208. NN doesn't schedule replication when a DN storage fails. |
| (Ming Ma via szetszwo) |
| |
| HDFS-5089. When a LayoutVersion support SNAPSHOT, it must support |
| FSIMAGE_NAME_OPTIMIZATION. (szetszwo) |
| |
| HDFS-7260. Change DFSOutputStream.MAX_PACKETS to be configurable. (szetszwo) |
| |
| HDFS-7259. Unresponseive NFS mount point due to deferred COMMIT response. |
| (brandonli) |
| |
| HDFS-7215.Add JvmPauseMonitor to NFS gateway (brandonli) |
| |
| HDFS-7180. NFSv3 gateway frequently gets stuck due to GC (brandonli) |
| |
| HDFS-7243. HDFS concat operation should not be allowed in Encryption Zone. |
| (clamb via yliu) |
| |
| HADOOP-11233. hadoop.security.kms.client.encrypted.key.cache.expiry |
| property spelled wrong in core-default. (Stephen Chu via yliu) |
| |
| HDFS-7218. FSNamesystem ACL operations should write to audit log on |
| failure. (clamb via yliu) |
| |
| HDFS-7199. DFSOutputStream should not silently drop data if DataStreamer |
| crashes with an unchecked exception (rushabhs via cmccabe) |
| |
| HDFS-7383. DataNode.requestShortCircuitFdsForRead may throw |
| NullPointerException. (szetszwo via suresh) |
| |
| HDFS-7147. Update archival storage user documentation. |
| (Tsz Wo Nicholas Sze via wheat9) |
| |
| HDFS-7340. Make rollingUpgrade start/finalize idempotent. (jing9) |
| |
| HDFS-7334. Fix periodic failures of TestCheckpoint |
| #testTooManyEditReplayFailures. (Charles Lamb via wheat9) |
| |
| HDFS-7355. TestDataNodeVolumeFailure#testUnderReplicationAfterVolFailure |
| fails on Windows, because we cannot deny access to the file owner. |
| (Chris Nauroth via wheat9) |
| |
| HDFS-7367. HDFS short-circuit read cannot negotiate shared memory slot and |
| file descriptors when SASL is enabled on DataTransferProtocol. |
| (Chris Nauroth via wheat9) |
| |
| HDFS-7364. Balancer always shows zero Bytes Already Moved. |
| (Tsz Wo Nicholas Sze via jing9) |
| |
| HDFS-7379. TestBalancer#testBalancerWithRamDisk creates test files |
| incorrectly. (Xiaoyu Yao via wheat9) |
| |
| BREAKDOWN OF HDFS-6134 AND HADOOP-10150 SUBTASKS AND RELATED JIRAS |
| |
| HDFS-6387. HDFS CLI admin tool for creating & deleting an |
| encryption zone. (clamb) |
| |
| HDFS-6386. HDFS Encryption Zones (clamb) |
| |
| HDFS-6388. HDFS integration with KeyProvider. (clamb) |
| |
| HDFS-6473. Protocol and API for Encryption Zones (clamb) |
| |
| HDFS-6392. Wire crypto streams for encrypted files in |
| DFSClient. (clamb and yliu) |
| |
| HDFS-6476. Print out the KeyProvider after finding KP successfully on |
| startup. (Juan Yu via wang) |
| |
| HDFS-6391. Get the Key/IV from the NameNode for encrypted files in |
| DFSClient. (Charles Lamb and wang) |
| |
| HDFS-6389. Rename restrictions for encryption zones. (clamb) |
| |
| HDFS-6605. Client server negotiation of cipher suite. (wang) |
| |
| HDFS-6625. Remove the Delete Encryption Zone function (clamb) |
| |
| HDFS-6516. List of Encryption Zones should be based on inodes (clamb) |
| |
| HDFS-6629. Not able to create symlinks after HDFS-6516 (umamaheswararao) |
| |
| HDFS-6635. Refactor encryption zone functionality into new |
| EncryptionZoneManager class. (wang) |
| |
| HDFS-6474. Namenode needs to get the actual keys and iv from the |
| KeyProvider. (wang) |
| |
| HDFS-6619. Clean up encryption-related tests. (wang) |
| |
| HDFS-6405. Test Crypto streams in HDFS. (yliu via wang) |
| |
| HDFS-6490. Fix the keyid format for generated keys in |
| FSNamesystem.createEncryptionZone (clamb) |
| |
| HDFS-6716. Update usage of KeyProviderCryptoExtension APIs on NameNode. |
| (wang) |
| |
| HDFS-6718. Remove EncryptionZoneManager lock. (wang) |
| |
| HDFS-6720. Remove KeyProvider in EncryptionZoneManager. (wang) |
| |
| HDFS-6738. Remove unnecessary getEncryptionZoneForPath call in |
| EZManager#createEncryptionZone. (clamb) |
| |
| HDFS-6724. Decrypt EDEK before creating |
| CryptoInputStream/CryptoOutputStream. (wang) |
| |
| HDFS-6509. Create a special /.reserved/raw directory for raw access to |
| encrypted data. (clamb via wang) |
| |
| HDFS-6771. Require specification of an encryption key when creating |
| an encryption zone. (wang) |
| |
| HDFS-6730. Create a .RAW extended attribute namespace. (clamb) |
| |
| HDFS-6692. Add more HDFS encryption tests. (wang) |
| |
| HDFS-6780. Batch the encryption zones listing API. (wang) |
| |
| HDFS-6394. HDFS encryption documentation. (wang) |
| |
| HDFS-6834. Improve the configuration guidance in DFSClient when there |
| are no Codec classes found in configs. (umamahesh) |
| |
| HDFS-6546. Add non-superuser capability to get the encryption zone |
| for a specific path. (clamb) |
| |
| HDFS-6733. Creating encryption zone results in NPE when |
| KeyProvider is null. (clamb) |
| |
| HDFS-6785. Should not be able to create encryption zone using path |
| to a non-directory file. (clamb) |
| |
| HDFS-6807. Fix TestReservedRawPaths. (clamb) |
| |
| HDFS-6814. Mistakenly dfs.namenode.list.encryption.zones.num.responses configured |
| as boolean. (umamahesh) |
| |
| HDFS-6817. Fix findbugs and other warnings. (yliu) |
| |
| HDFS-6839. Fix TestCLI to expect new output. (clamb) |
| |
| HDFS-6954. With crypto, no native lib systems are too verbose. (clamb via wang) |
| |
| HDFS-2975. Rename with overwrite flag true can make NameNode to stuck in safemode |
| on NN (crash + restart). (Yi Liu via umamahesh) |
| |
| HDFS-6905. fs-encryption merge triggered release audit failures. (clamb via tucu) |
| |
| HDFS-6714. TestBlocksScheduledCounter#testBlocksScheduledCounter should |
| shutdown cluster (vinayakumarb) |
| |
| HDFS-6986. DistributedFileSystem must get delegation tokens from configured |
| KeyProvider. (zhz via tucu) |
| |
| HDFS-6776. Using distcp to copy data between insecure and secure cluster via webdhfs |
| doesn't work. (yzhangal via tucu) |
| |
| HDFS-7042. Upgrade fails for Windows HA cluster due to file locks held during |
| rename in JournalNode. (cnauroth) |
| |
| HDFS-7051. TestDataNodeRollingUpgrade#isBlockFileInPrevious assumes Unix file |
| path separator. (cnauroth) |
| |
| HDFS-7105. Fix TestJournalNode#testFailToStartWithBadConfig to match log |
| output change. (Ray Chiang via cnauroth) |
| |
| HDFS-7105. Allow falling back to a non-SASL connection on |
| DataTransferProtocol in several edge cases. (cnauroth) |
| |
| HDFS-7107. Avoid Findbugs warning for synchronization on |
| AbstractNNFailoverProxyProvider#fallbackToSimpleAuth. (cnauroth) |
| |
| HDFS-7109. TestDataStorage does not release file locks between tests. |
| (cnauroth) |
| |
| HDFS-7110. Skip tests related to short-circuit read on platforms that do not |
| currently implement short-circuit read. (cnauroth) |
| |
| HDFS-7115. TestEncryptionZones assumes Unix path separator for KMS key store |
| path. (Xiaoyu Yao via cnauroth) |
| |
| HDFS-7115. TestEncryptionZonesWithHA assumes Unix path separator for KMS key |
| store path. (Xiaoyu Yao via cnauroth) |
| |
| HDFS-7130. TestDataTransferKeepalive fails intermittently on Windows. |
| (cnauroth) |
| |
| HDFS-6534. Fix build on macosx: HDFS parts (Binglin Chang via aw) |
| |
| HDFS-7111. TestSafeMode assumes Unix line endings in safe mode tip. |
| (cnauroth) |
| |
| HDFS-7127. TestLeaseRecovery leaks MiniDFSCluster instances. (cnauroth) |
| |
| HDFS-7131. During HA upgrade, JournalNode should create a new committedTxnId |
| file in the current directory. (jing9) |
| |
| HDFS-6995. Block should be placed in the client's 'rack-local' node |
| if 'client-local' node is not available (vinayakumarb) |
| |
| HDFS-7128. Decommission slows way down when it gets towards the end. |
| (Ming Ma via cnauroth) |
| |
| HDFS-7287. The OfflineImageViewer (OIV) can output invalid XML depending on |
| the filename (Ravi Prakash via Colin P. McCabe) |
| |
| HDFS-7300. The getMaxNodesPerRack() method in BlockPlacementPolicyDefault |
| is flawed (kihwal) |
| |
| HDFS-7305. NPE seen in wbhdfs FS while running SLive. (jing9) |
| |
| HDFS-7199. DFSOutputStream should not silently drop data if DataStreamer |
| crashes with an unchecked exception (rushabhs via cmccabe) |
| |
| BREAKDOWN OF HDFS-6584 ARCHIVAL STORAGE |
| |
| HDFS-6677. Change INodeFile and FSImage to support storage policy ID. |
| (szetszwo) |
| |
| HDFS-6670. Add block storage policy support with default HOT, WARM and COLD |
| policies. (szetszwo) |
| |
| HDFS-6671. Change BlockPlacementPolicy to consider block storage policy |
| in replicaiton. (szetszwo) |
| |
| HDFS-6710. Change BlockPlacementPolicy to consider block storage policy |
| in replica deletion. (szetszwo) |
| |
| HDFS-6679. Bump NameNodeLayoutVersion and update editsStored test files. |
| (vinayakumarb via szetszwo) |
| |
| HDFS-6686. Change BlockPlacementPolicy to use fallback when some storage |
| types are unavailable. (szetszwo) |
| |
| HDFS-6835. Add a new API to set storage policy. (jing9) |
| |
| HDFS-6847. Support storage policy on directories and include storage policy |
| in HdfsFileStatus. (Jing Zhao via szetszwo) |
| |
| HDFS-6801. Add a new data migration tool, Mover, for archiving data. |
| (szetszwo via jing9) |
| |
| HDFS-6863. Support migration for snapshot paths. (jing9) |
| |
| HDFS-6906. Add more tests for BlockStoragePolicy. (szetszwo via jing9) |
| |
| HDFS-6911. check if a block is already scheduled in Mover. |
| (szetszwo via jing9) |
| |
| HDFS-6920. Check the storage type of delNodeHintStorage when deleting |
| a replica. (szetszwo via jing9) |
| |
| HDFS-6944. Add retry and termination logic for Mover. (jing9) |
| |
| HDFS-6969. INode#getStoragePolicyID should always return the latest |
| storage policy. (jing9) |
| |
| HDFS-6961. BlockPlacementPolicy#chooseTarget should check each valid |
| storage type in each choosing round. (jing9) |
| |
| HDFS-6876. support set/get storage policy in DFSAdmin. (jing9) |
| |
| HDFS-6997. Add more tests for data migration and replicaion. (szetszwo) |
| |
| HDFS-6875. Support migration for a list of specified paths. (jing9) |
| |
| HDFS-7027. Mover does not terminate when some storage type is out of space. |
| (szetszwo via jing9) |
| |
| HDFS-7029. Fix TestDFSInotifyEventInputStream and TestDistributedFileSystem. |
| (szetszwo via jing9) |
| |
| HDFS-7028. FSDirectory should not get storage policy id from symlinks. |
| (szetszwo) |
| |
| HDFS-7034. Fix TestBlockPlacement and TestStorageMover. (jing9) |
| |
| HDFS-7039. Fix Balancer tests. (szetszwo via jing9) |
| |
| HDFS-7062. Skip under construction block for migration. (jing9) |
| |
| HDFS-7052. Add Mover into hdfs script. (jing9) |
| |
| HDFS-7072. Fix TestBlockManager and TestStorageMover. (jing9 via szetszwo) |
| |
| HDFS-6864. Archival Storage: add user documentation. (szetszwo via jing9) |
| |
| HDFS-7088. Archival Storage: fix TestBalancer and |
| TestBalancerWithMultipleNameNodes. (szetszwo via jing9) |
| |
| HDFS-7095. TestStorageMover often fails in Jenkins. (jing9) |
| |
| HDFS-7081. Add new DistributedFileSystem API for getting all the existing |
| storage policies. (jing9) |
| |
| HDFS-7140. Add a tool to list all the existing block storage policies. |
| (jing9) |
| |
| HDFS-7167. NPE while running Mover if the given path is for a file. (jing9) |
| |
| HDFS-7154. Fix returning value of starting reconfiguration task (Lei Xu via |
| Colin P. McCabe) |
| |
| BREAKDOWN OF HDFS-6581 SUBTASKS AND RELATED JIRAS |
| |
| HDFS-6921. Add LazyPersist flag to FileStatus. (Arpit Agarwal) |
| |
| HDFS-6924. Add new RAM_DISK storage type. (Arpit Agarwal) |
| |
| HDFS-6922. Add LazyPersist flag to INodeFile, save it in FsImage and |
| edit logs. (Arpit Agarwal) |
| |
| HDFS-6923. Propagate LazyPersist flag to DNs via DataTransferProtocol. |
| (Arpit Agarwal) |
| |
| HDFS-6925. DataNode should attempt to place replicas on transient storage |
| first if lazyPersist flag is received. (Arpit Agarwal) |
| |
| HDFS-6926. DN support for saving replicas to persistent storage and |
| evicting in-memory replicas. (Arpit Agarwal) |
| |
| HDFS-6927. Initial unit tests for lazy persist files. (Arpit Agarwal) |
| |
| HDFS-6929. NN periodically unlinks lazy persist files with missing |
| replicas from namespace. (Arpit Agarwal) |
| |
| HDFS-6928. 'hdfs put' command should accept lazyPersist flag for testing. |
| (Arpit Agarwal) |
| |
| HDFS-6960. Bugfix in LazyWriter, fix test case and some refactoring. |
| (Arpit Agarwal) |
| |
| HDFS-6931. Move lazily persisted replicas to finalized directory on DN |
| startup. (Arpit Agarwal) |
| |
| HDFS-6950. Add Additional unit tests for HDFS-6581. (Xiaoyu Yao via |
| Arpit Agarwal) |
| |
| HDFS-6930. Improve replica eviction from RAM disk. (Arpit Agarwal) |
| |
| HDFS-6977. Delete all copies when a block is deleted from the block space. |
| (Arpit Agarwal) |
| |
| HDFS-6991. Notify NN of evicted block before deleting it from RAM disk. |
| (Arpit Agarwal) |
| |
| HDFS-6978. Directory scanner should correctly reconcile blocks on RAM |
| disk. (Arpit Agarwal) |
| |
| HDFS-7066. LazyWriter#evictBlocks misses a null check for replicaState. |
| (Xiaoyu Yao via Arpit Agarwal) |
| |
| HDFS-7064. Fix unit test failures in HDFS-6581 branch. (Xiaoyu Yao via |
| Arpit Agarwal) |
| |
| HDFS-6581. Few more unit test fixes for HDFS-6581. (Arpit Agarwal) |
| |
| HDFS-7080. Fix finalize and upgrade unit test failures. (Arpit Agarwal) |
| |
| HDFS-7084. FsDatasetImpl#copyBlockFiles debug log can be improved. |
| (Xiaoyu Yao via Arpit Agarwal) |
| |
| HDFS-7091. Add forwarding constructor for INodeFile for existing callers. |
| (Arpit Agarwal) |
| |
| HDFS-7100. Make eviction scheme pluggable. (Arpit Agarwal) |
| |
| HDFS-7108. Fix unit test failures in SimulatedFsDataset. (Arpit Agarwal) |
| |
| HDFS-7071. Updated editsStored and editsStored.xml to bump layout |
| version and add LazyPersist flag. (Xiaoyu Yao and Arpit Agarwal via |
| Arpit Agarwal) |
| |
| HDFS-6990. Add unit test for evict/delete RAM_DISK block with open |
| handle. (Xiaoyu Yao via Arpit Agarwal) |
| |
| HDFS-7143. Fix findbugs warnings in HDFS-6581 branch. (szetszwo via |
| Arpit Agarwal) |
| |
| HDFS-6932. Balancer and Mover tools should ignore replicas on RAM_DISK. |
| (Xiaoyu Yao via Arpit Agarwal) |
| |
| HDFS-7144. Fix findbugs warnings in RamDiskReplicaTracker. (szetszwo via |
| Arpit Agarwal) |
| |
| HDFS-7155. Bugfix in createLocatedFileStatus caused by bad merge. |
| (Arpit Agarwal) |
| |
| HDFS-7153. Add storagePolicy to NN edit log during file creation. |
| (Arpit Agarwal) |
| |
| HDFS-7159. Use block storage policy to set lazy persist preference. |
| (Arpit Agarwal) |
| |
| HDFS-7129. Metrics to track usage of memory for writes. (Xiaoyu Yao |
| via Arpit Agarwal) |
| |
| HDFS-7171. Fix Jenkins failures in HDFS-6581 branch. (Arpit Agarwal) |
| |
| HDFS-7112. LazyWriter should use either async IO or one thread per physical |
| disk. (Xiaoyu Yao via cnauroth) |
| |
| HDFS-7090. Use unbuffered writes when persisting in-memory replicas. |
| (Xiaoyu Yao via cnauroth) |
| |
| HDFS-6934. Move checksum computation off the hot path when writing to RAM |
| disk. (cnauroth) |
| |
| HDFS-7291. Persist in-memory replicas with appropriate unbuffered copy API |
| on POSIX and Windows. (Xiaoyu Yao via cnauroth) |
| |
| HDFS-7328. TestTraceAdmin assumes Unix line endings. (cnauroth) |
| |
| HDFS-7359. NameNode in secured HA cluster fails to start if |
| dfs.namenode.secondary.http-address cannot be interpreted as a network |
| address. (cnauroth) |
| |
| HDFS-7226. Fix TestDNFencing.testQueueingWithAppend. (Yongjun Zhang via jing9) |
| |
| HDFS-7382. DataNode in secure mode may throw NullPointerException if client |
| connects before DataNode registers itself with NameNode. (cnauroth) |
| |
| HDFS-7387. NFS may only do partial commit due to a race between COMMIT and write |
| (brandonli) |
| |
| HDFS-7391. Renable SSLv2Hello in HttpFS. (rkanter via acmurthy) |
| |
| HDFS-7385. ThreadLocal used in FSEditLog class causes FSImage permission mess |
| up. (jiangyu via cnauroth) |
| |
| Release 2.5.2 - 2014-11-19 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| HDFS-7274. Disable SSLv3 in HttpFS. (Robert Kanter via kasha) |
| |
| |
| Release 2.5.1 - 2014-09-05 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| HADOOP-10957. The globber will sometimes erroneously return a permission |
| denied exception when there is a non-terminal wildcard (cmccabe) |
| |
| Release 2.5.0 - 2014-08-11 |
| |
| INCOMPATIBLE CHANGES |
| |
| HDFS-6168. Remove a deprecated constructor and the deprecated methods reportChecksumFailure, |
| getDelegationToken(Text), renewDelegationToken and cancelDelegationToken from |
| DistributedFileSystem. (szetszwo) |
| |
| NEW FEATURES |
| |
| HDFS-6281. Provide option to use the NFS Gateway without having to use the |
| Hadoop portmapper. (atm) |
| |
| HDFS-5168. Add cross node dependency support to BlockPlacementPolicy. |
| (Nikola Vujic via szetszwo) |
| |
| HDFS-6334. Client failover proxy provider for IP failover based NN HA. |
| (kihwal) |
| |
| HDFS-6406. Add capability for NFS gateway to reject connections from |
| unprivileged ports. (atm) |
| |
| HDFS-2006. Ability to support storing extended attributes per file. |
| |
| HDFS-5978. Create a tool to take fsimage and expose read-only WebHDFS API. |
| (Akira Ajisaka via wheat9) |
| |
| HDFS-6278. Create HTML5-based UI for SNN. (wheat9) |
| |
| HDFS-6279. Create new index page for JN / DN. (wheat9) |
| |
| IMPROVEMENTS |
| |
| HDFS-6007. Update documentation about short-circuit local reads (iwasakims |
| via cmccabe) |
| |
| HDFS-6125. Cleanup unnecessary cast in HDFS code base. (suresh) |
| |
| HDFS-5196. Provide more snapshot information in WebUI. |
| (Shinichi Yamashita via wheat9) |
| |
| HDFS-6155. Fix Boxing/unboxing to parse a primitive findbugs warnings. |
| (suresh) |
| |
| HDFS-6119. FSNamesystem code cleanup. (suresh) |
| |
| HDFS-6158. Clean up dead code for OfflineImageViewer. (wheat9) |
| |
| HDFS-6164. Remove lsr in OfflineImageViewer. (wheat9) |
| |
| HDFS-6167. Relocate the non-public API classes in the hdfs.client package. |
| (szetszwo) |
| |
| HDFS-6191. Disable quota checks when replaying edit log. (kihwal) |
| |
| HDFS-6170. Support GETFILESTATUS operation in WebImageViewer. |
| (Akira Ajisaka via wheat9) |
| |
| HDFS-6225. Remove the o.a.h.hdfs.server.common.UpgradeStatusReport. |
| (wheat9) |
| |
| HDFS-6224. Add a unit test to TestAuditLogger for file permissions |
| passed to logAuditEvent. (Charles Lamb via wang) |
| |
| HDFS-6194. Create new tests for ByteRangeInputStream. |
| (Akira Ajisaka via wheat9) |
| |
| HDFS-6219. Proxy superuser configuration should use true client IP for |
| address checks. (daryn via kihwal) |
| |
| HDFS-6256. Clean up ImageVisitor and SpotCheckImageVisitor. |
| (Akira Ajisaka via wheat9) |
| |
| HDFS-6265. Prepare HDFS codebase for JUnit 4.11. (cnauroth) |
| |
| HDFS-5693. Few NN metrics data points were collected via JMX when NN |
| is under heavy load. (Ming Ma via jing9) |
| |
| HDFS-6273. Config options to allow wildcard endpoints for namenode HTTP |
| and HTTPS servers. (Arpit Agarwal) |
| |
| HDFS-6282. Re-add testIncludeByRegistrationName. (cmccabe) |
| |
| HDFS-6266. Identify full path for a given INode. (jing9) |
| |
| HDFS-6210. Support GETACLSTATUS operation in WebImageViewer. |
| (Akira Ajisaka via wheat9) |
| |
| HDFS-6269. NameNode Audit Log should differentiate between webHDFS open and |
| HDFS open. (Eric Payne via jeagles) |
| |
| HDFS-6304. Consolidate the logic of path resolution in FSDirectory. |
| (wheat9) |
| |
| HDFS-6295. Add "decommissioning" state and node state filtering to |
| dfsadmin. (wang) |
| |
| HDFS-6294. Use INode IDs to avoid conflicts when a file open for write is |
| renamed. (cmccabe) |
| |
| HDFS-6328. Clean up dead code in FSDirectory. (wheat9) |
| |
| HDFS-6230. Expose upgrade status through NameNode web UI. |
| (Mit Desai via wheat9) |
| |
| HDFS-6186. Pause deletion of blocks when the namenode starts up. (jing9) |
| |
| HDFS-6293. Issues with OIV processing PB-based fsimages. (kihwal) |
| |
| HDFS-2949. Add check to active state transition to prevent operator-induced |
| split brain. (Rushabh S Shah via kihwal) |
| |
| HDFS-6287. Add vecsum test of libhdfs read access times (cmccabe) |
| |
| HDFS-5683. Better audit log messages for caching operations. |
| (Abhiraj Butala via wang) |
| |
| HDFS-6345. DFS.listCacheDirectives() should allow filtering based on |
| cache directive ID. (wang) |
| |
| HDFS-6432. Add snapshot related APIs to webhdfs. (jing9) |
| |
| HDFS-6396. Remove support for ACL feature from INodeSymlink. |
| (Charles Lamb via wang) |
| |
| HDFS-6435. Add support for specifying a static uid/gid mapping for the NFS |
| gateway. (atm via wang) |
| |
| HDFS-6110 adding more slow action log in critical write path |
| (Liang Xie via stack) |
| |
| HDFS-6416. Use Time#monotonicNow in OpenFileCtx and OpenFileCtxCatch to |
| avoid system clock bugs (Abhiraj Butala via brandonli) |
| |
| HDFS-6356. Fix typo in DatanodeLayoutVersion. (Tulasi G via wang) |
| |
| HDFS-6447. balancer should timestamp the completion message. |
| (Juan Yu via wang) |
| |
| HDFS-6463. Clarify behavior of AclStorage#createFsPermissionForExtendedAcl |
| in comments. (cnauroth) |
| |
| HDFS-6472. fix typo in webapps/hdfs/explorer.js. (Juan Yu via wang) |
| |
| HDFS-6056. Clean up NFS config settings (brandonli) |
| |
| HDFS-6109 let sync_file_range() system call run in background |
| (Liang Xie via stack) |
| |
| HDFS-6268. Better sorting in NetworkTopology#pseudoSortByDistance when |
| no local node is found. (wang) |
| |
| HDFS-6369. Document that BlockReader#available() can return more bytes than |
| are remaining in the block (Ted Yu via Colin Patrick McCabe) |
| |
| HDFS-6487. TestStandbyCheckpoint#testSBNCheckpoints is racy. |
| (Mit Desai via wang) |
| |
| HDFS-6297. Add CLI testcases to reflect new features of dfs and dfsadmin |
| (Dasha Boudnik via cos) |
| |
| HDFS-6399. Add note about setfacl in HDFS permissions guide. |
| (cnauroth via wang) |
| |
| HDFS-6315. Decouple recording edit logs from FSDirectory. (wheat9) |
| |
| HDFS-6379. HTTPFS - Implement ACLs support. (yoderme via tucu) |
| |
| HDFS-6471. Make moveFromLocal CLI testcases to be non-disruptive |
| (Dasha Boudnik via cos) |
| |
| HDFS-6395. Skip checking xattr limits for non-user-visible namespaces. |
| (Yi Liu via wang). |
| |
| HDFS-3493. Invalidate excess corrupted blocks as long as minimum |
| replication is satisfied. (Juan Yu and Vinayakumar B via wang) |
| |
| HDFS-6330. Move mkdirs() to FSNamesystem. (wheat9) |
| |
| HDFS-6470. TestBPOfferService.testBPInitErrorHandling is flaky. |
| (Ming Ma via wang) |
| |
| HDFS-6529. Trace logging for RemoteBlockReader2 to identify remote datanode |
| and file being read. (Anubhav Dhoot via atm) |
| |
| HDFS-6499. Use NativeIO#renameTo instead of File#renameTo in |
| FileJournalManager. (Yongjun Zhang via atm) |
| |
| HDFS-6518. TestCacheDirectives#testExceedsCapacity should |
| take FSN read lock when accessing pendingCached list. |
| (wang) |
| |
| HDFS-6528. Add XAttrs to TestOfflineImageViewer. (Stephen Chu via wang) |
| |
| HDFS-6545. Finalizing rolling upgrade can make NN unavailable for a long |
| duration. (kihwal) |
| |
| HDFS-6530. Fix Balancer documentation. (szetszwo) |
| |
| HDFS-6480. Move waitForReady() from FSDirectory to FSNamesystem. (wheat9) |
| |
| HDFS-6403. Add metrics for log warnings reported by JVM pauses. (Yongjun |
| Zhang via atm) |
| |
| HDFS-6557. Move the reference of fsimage to FSNamesystem. (wheat9) |
| |
| HDFS-4667. Capture renamed files/directories in snapshot diff report. (jing9 |
| and Binglin Chang via jing9) |
| |
| HDFS-6507. Improve DFSAdmin to support HA cluster better. |
| (Zesheng Wu via vinayakumarb) |
| |
| HDFS-6578. add toString method to DatanodeStorage for easier debugging. |
| (Yongjun Zhang via Arpit Agarwal) |
| |
| HDFS-6562. Refactor rename() in FSDirectory. (wheat9) |
| |
| HDFS-6486. Add user doc for XAttrs via WebHDFS. (Yi Liu via umamahesh) |
| |
| HDFS-6430. HTTPFS - Implement XAttr support. (Yi Liu via tucu) |
| |
| HDFS-6593. Move SnapshotDiffInfo out of INodeDirectorySnapshottable. |
| (Jing Zhao via wheat9) |
| |
| HDFS-6595. Allow the maximum threads for balancing on datanodes to be |
| configurable. (Benoy Antony via szetszwo) |
| |
| HDFS-6572. Add an option to the NameNode that prints the software and |
| on-disk image versions. (Charles Lamb via cnauroth) |
| |
| HDFS-6603. Add XAttr with ACL test. (Stephen Chu via cnauroth) |
| |
| HDFS-6612. MiniDFSNNTopology#simpleFederatedTopology(int) |
| always hardcode nameservice ID. (Juan Yu via wang) |
| |
| HDFS-6614. shorten TestPread run time with a smaller retry timeout setting. |
| (Liang Xie via cnauroth) |
| |
| HDFS-6610. TestShortCircuitLocalRead tests sometimes timeout on slow |
| machines. (Charles Lamb via wang) |
| |
| HDFS-6620. Snapshot docs should specify about preserve options with cp command |
| (Stephen Chu via umamahesh) |
| |
| HDFS-6493. Change dfs.namenode.startup.delay.block.deletion to second |
| instead of millisecond. (Juan Yu via wang) |
| |
| HDFS-6680. BlockPlacementPolicyDefault does not choose favored nodes |
| correctly. (szetszwo) |
| |
| HDFS-6712. Document HDFS Multihoming Settings. (Arpit Agarwal) |
| |
| OPTIMIZATIONS |
| |
| HDFS-6214. Webhdfs has poor throughput for files >2GB (daryn) |
| |
| HDFS-6460. Ignore stale and decommissioned nodes in |
| NetworkTopology#sortByDistance. (Yongjun Zhang via wang) |
| |
| HDFS-6583. Remove clientNode in FileUnderConstructionFeature. (wheat9) |
| |
| HDFS-6599. 2.4 addBlock is 10 to 20 times slower compared to 0.23 (daryn) |
| |
| BUG FIXES |
| |
| HDFS-6112. NFS Gateway docs are incorrect for allowed hosts configuration. |
| (atm) |
| |
| HDFS-3087. Decomissioning on NN restart can complete without blocks being |
| replicated. (Rushabh S Shah via kihwal) |
| |
| HDFS-6162. Format strings should use platform independent line separator. |
| (suresh) |
| |
| HDFS-6156. Simplify the JMX API that provides snapshot information. |
| (wheat9) |
| |
| HDFS-6173. Move the default processor from Ls to Web in OfflineImageViewer. |
| (Akira Ajisaka via wheat9) |
| |
| HDFS-5591. Checkpointing should use monotonic time when calculating period. |
| (Charles Lamb via wang) |
| |
| HDFS-6190. Minor textual fixes in DFSClient. (Charles Lamb via wheat9) |
| |
| HDFS-6159. TestBalancerWithNodeGroup.testBalancerWithNodeGroup fails if |
| there is block missing after balancer success (Chen He via kihwal) |
| |
| HDFS-6181. Fix the wrong property names in NFS user guide (brandonli) |
| |
| HDFS-6180. dead node count / listing is very broken in JMX and old GUI. |
| (wheat9) |
| |
| HDFS-6143. WebHdfsFileSystem open should throw FileNotFoundException for |
| non-existing paths. (Gera Shegalov via wheat9) |
| |
| HDFS-6169. Move the address in WebImageViewer. (Akira Ajisaka via wheat9) |
| |
| HDFS-6160. TestSafeMode occasionally fails. (Arpit Agarwal) |
| |
| HDFS-5669. Storage#tryLock() should check for null before logging successfull message |
| (Vinayakumar B via umamahesh) |
| |
| HDFS-6237. TestDFSShell#testGet fails on Windows due to invalid file system |
| path. (cnauroth) |
| |
| HDFS-6238. TestDirectoryScanner leaks file descriptors. (cnauroth) |
| |
| HDFS-6243. HA NameNode transition to active or shutdown may leave lingering |
| image transfer thread. (cnauroth) |
| |
| HDFS-5409. TestOfflineEditsViewer#testStored fails on Windows due to CRLF |
| line endings in editsStored.xml from git checkout. (cnauroth) |
| |
| HDFS-4909. Avoid protocol buffer RPC namespace clashes. (cmccabe) |
| |
| HDFS-6178. Decommission on standby NN couldn't finish. (Ming Ma via jing9) |
| |
| HDFS-6213. TestDataNodeConfig failing on Jenkins runs due to DN web port |
| in use. (wang) |
| |
| HDFS-6274. Cleanup javadoc warnings in HDFS code. (suresh) |
| |
| HDFS-6275. Fix warnings - type arguments can be inferred and redudant |
| local variable. (suresh) |
| |
| HDFS-6217. Webhdfs PUT operations may not work via a http proxy. |
| (Daryn Sharp via kihwal) |
| |
| HDFS-6276. Remove unnecessary conditions and null check. (suresh) |
| |
| HDFS-5865. Update OfflineImageViewer document. (Akira Ajisaka via wheat9) |
| |
| HDFS-6270. Secondary namenode status page shows transaction count in bytes. |
| (Benoy Antony via wheat9) |
| |
| HDFS-6218. Audit log should use true client IP for proxied webhdfs |
| operations. (daryn via kihwal) |
| |
| HDFS-6288. DFSInputStream Pread doesn't update ReadStatistics. |
| (Juan Yu via wang) |
| |
| HDFS-5892. TestDeleteBlockPool fails in branch-2. (Ted Yu via wheat9) |
| |
| HDFS-6289. HA failover can fail if there are pending DN messages for DNs |
| which no longer exist. (atm) |
| |
| HDFS-6337. Setfacl testcase is failing due to dash character in username |
| in TestAclCLI (umamahesh) |
| |
| HDFS-5381. ExtendedBlock#hashCode should use both blockId and block pool ID |
| (Benoy Antony via Colin Patrick McCabe) |
| |
| HDFS-6240. WebImageViewer returns 404 if LISTSTATUS to an empty directory. |
| (Akira Ajisaka via wheat9) |
| |
| HDFS-6351. Command hdfs dfs -rm -r can't remove empty directory. |
| (Yongjun Zhang via wang) |
| |
| HDFS-5522. Datanode disk error check may be incorrectly skipped. |
| (Rushabh S Shah via kihwal) |
| |
| HDFS-6367. EnumSetParam$Domain#parse fails for parameter containing more than one enum. |
| (Yi Liu via umamahesh) |
| |
| HDFS-6305. WebHdfs response decoding may throw RuntimeExceptions (Daryn |
| Sharp via jeagles) |
| |
| HDFS-6355. Fix divide-by-zero, improper use of wall-clock time in |
| BlockPoolSliceScanner (cmccabe) |
| |
| HDFS-6370. Web UI fails to display in intranet under IE. |
| (Haohui Mai via cnauroth) |
| |
| HDFS-6381. Fix a typo in INodeReference.java. (Binglin Chang via jing9) |
| |
| HDFS-6400. Cannot execute hdfs oiv_legacy. (Akira AJISAKA via kihwal) |
| |
| HDFS-6250. Fix test failed in TestBalancerWithNodeGroup.testBalancerWithRackLocality |
| (Binglin Chang and Chen He via junping_du) |
| |
| HDFS-4913. Deleting file through fuse-dfs when using trash fails requiring |
| root permissions (cmccabe) |
| |
| HDFS-6421. Fix vecsum.c compile on BSD and some other systems. (Mit Desai |
| via Colin Patrick McCabe) |
| |
| HDFS-6419. TestBookKeeperHACheckpoints#TestSBNCheckpoints fails on trunk. |
| (Akira AJISAKA via kihwal) |
| |
| HDFS-6409. Fix typo in log message about NameNode layout version upgrade. |
| (Chen He via cnauroth) |
| |
| HDFS-6433. Replace BytesMoved class with AtomicLong. |
| (Benoy Antony via cnauroth) |
| |
| HDFS-6438. DeleteSnapshot should be a DELETE request in WebHdfs. (jing9) |
| |
| HDFS-6423. Diskspace quota usage should be updated when appending data to |
| partial block. (jing9) |
| |
| HDFS-6443. Fix MiniQJMHACluster related test failures. (Zesheng Wu via |
| Arpit Agarwal) |
| |
| HDFS-6227. ShortCircuitCache#unref should purge ShortCircuitReplicas whose |
| streams have been closed by java interrupts. (Colin Patrick McCabe via jing9) |
| |
| HDFS-6442. Fix TestEditLogAutoroll and TestStandbyCheckpoints failure |
| caused by port conficts. (Zesheng Wu via Arpit Agarwal) |
| |
| HDFS-6448. BlockReaderLocalLegacy should set socket timeout based on |
| conf.socketTimeout (liangxie via cmccabe) |
| |
| HDFS-6453. Use Time#monotonicNow to avoid system clock reset. |
| (Liang Xie via wang) |
| |
| HDFS-6461. Use Time#monotonicNow to compute duration in DataNode#shutDown. |
| (James Thomas via wang) |
| |
| HDFS-6462. NFS: fsstat request fails with the secure hdfs (brandonli) |
| |
| HDFS-6404. HttpFS should use a 000 umask for mkdir and create |
| operations. (yoderme via tucu) |
| |
| HDFS-6424. blockReport doesn't need to invalidate blocks on SBN. (Ming Ma |
| via jing9) |
| |
| HDFS-6497. Make TestAvailableSpaceVolumeChoosingPolicy deterministic |
| (cmccabe) |
| |
| HDFS-6500. Snapshot shouldn't be removed silently after renaming to an |
| existing snapshot. (Nicholas SZE via junping_du) |
| |
| HDFS-6257. TestCacheDirectives#testExceedsCapacity fails occasionally |
| (cmccabe) |
| |
| HDFS-6364. Incorrect check for unknown datanode in Balancer. (Benoy |
| Antony via Arpit Agarwal) |
| |
| HDFS-6503. Fix typo of DFSAdmin restoreFailedStorage. |
| (Zesheng Wu via wheat9) |
| |
| HDFS-6464. Support multiple xattr.name parameters for WebHDFS getXAttrs. |
| (Yi Liu via umamahesh) |
| |
| HDFS-6539. test_native_mini_dfs is skipped in hadoop-hdfs/pom.xml |
| (decstery via cmccabe) |
| |
| HDFS-6527. Edit log corruption due to defered INode removal. (kihwal and |
| jing9 via jing9) |
| |
| HDFS-6552. add DN storage to a BlockInfo will not replace the different |
| storage from same DN. (Amir Langer via Arpit Agarwal) |
| |
| HDFS-6551. Rename with OVERWRITE option may throw NPE when the target |
| file/directory is a reference INode. (jing9) |
| |
| HDFS-6439. NFS should not reject NFS requests to the NULL procedure whether |
| port monitoring is enabled or not. (brandonli) |
| |
| HDFS-6559. Fix wrong option "dfsadmin -rollingUpgrade start" in the |
| document. (Akira Ajisaka via Arpit Agarwal) |
| |
| HDFS-6553. Add missing DeprecationDeltas for NFS Kerberos configurations |
| (Stephen Chu via brandonli) |
| |
| HDFS-6563. NameNode cannot save fsimage in certain circumstances when |
| snapshots are in use. (atm) |
| |
| HDFS-3848. A Bug in recoverLeaseInternal method of FSNameSystem class |
| (Hooman Peiro Sajjad and Chen He via kihwal) |
| |
| HDFS-6549. Add support for accessing the NFS gateway from the AIX NFS |
| client. (atm) |
| |
| HDFS-6535. HDFS quota update is wrong when file is appended. (George Wong |
| via jing9) |
| |
| HDFS-6222. Remove background token renewer from webhdfs. |
| (Rushabh Shah and Daryn Sharp via cnauroth) |
| |
| HDFS-6580. FSNamesystem.mkdirsInt should call the getAuditFileInfo() |
| wrapper. (Zhilei Xu via wheat9) |
| |
| HDFS-6587. Bug in TestBPOfferService can cause test failure. (Zhilei Xu |
| via Arpit Agarwal) |
| |
| HDFS-6598. Fix a typo in message issued from explorer.js. (Yongjun Zhang |
| via wheat9) |
| |
| HDFS-6475. WebHdfs clients fail without retry because incorrect handling |
| of StandbyException. (Yongjun Zhang via atm) |
| |
| HADOOP-10701. NFS should not validate the access premission only based on |
| the user's primary group (Harsh J via atm) |
| |
| HDFS-6556. Refine XAttr permissions (umamahesh) |
| |
| HDFS-6601. Issues in finalizing rolling upgrade when there is a layout |
| version change (kihwal) |
| |
| HDFS-6418. Regression: DFS_NAMENODE_USER_NAME_KEY missing |
| (szetszwo via stevel) |
| |
| HDFS-6558. Missing newline in the description of dfsadmin -rollingUpgrade. |
| (Chen He via kihwal) |
| |
| HDFS-6591. while loop is executed tens of thousands of times in Hedged Read |
| (Liang Xie via cnauroth) |
| |
| HDFS-6604. The short-circuit cache doesn't correctly time out replicas that |
| haven't been used in a while (cmccabe) |
| |
| HDFS-4286. Changes from BOOKKEEPER-203 broken capability of including |
| bookkeeper-server jar in hidden package of BKJM (Rakesh R via umamahesh) |
| |
| HDFS-4221. Remove the format limitation point from BKJM documentation as HDFS-3810 |
| closed. (Rakesh R via umamahesh) |
| |
| HDFS-5411. Update Bookkeeper dependency to 4.2.3. (Rakesh R via umamahesh) |
| |
| HDFS-6631. TestPread#testHedgedReadLoopTooManyTimes fails intermittently. |
| (Liang Xie via cnauroth) |
| |
| HDFS-6647. Edit log corruption when pipeline recovery occurs for deleted |
| file present in snapshot (kihwal) |
| |
| HDFS-6378. NFS registration should timeout instead of hanging when |
| portmap/rpcbind is not available (Abhiraj Butala via brandonli) |
| |
| HDFS-6632. Reintroduce dfs.http.port / dfs.https.port in branch-2. |
| (Yongjun Zhang via wheat9) |
| |
| HDFS-6703. NFS: Files can be deleted from a read-only mount |
| (Srikanth Upputuri via brandonli) |
| |
| HDFS-6422. getfattr in CLI doesn't throw exception or return non-0 return code |
| when xattr doesn't exist. (Charles Lamb via umamahesh) |
| |
| HDFS-6696. Name node cannot start if the path of a file under |
| construction contains ".snapshot". (wang) |
| |
| HDFS-6312. WebHdfs HA failover is broken on secure clusters. |
| (daryn via tucu) |
| |
| HDFS-6618. FSNamesystem#delete drops the FSN lock between removing INodes |
| from the tree and deleting them from the inode map (kihwal via cmccabe) |
| |
| HDFS-6622. Rename and AddBlock may race and produce invalid edits (kihwal |
| via cmccabe) |
| |
| HDFS-6723. New NN webUI no longer displays decommissioned state for dead node. |
| (Ming Ma via wheat9) |
| |
| HDFS-6768. Fix a few unit tests that use hard-coded port numbers. (Arpit |
| Agarwal) |
| |
| HDFS-6793. Missing changes in HftpFileSystem when Reintroduce |
| dfs.http.port / dfs.https.port in branch-2. (Juan Yu via wang). |
| |
| BREAKDOWN OF HDFS-2006 SUBTASKS AND RELATED JIRAS |
| |
| HDFS-6299. Protobuf for XAttr and client-side implementation. (Yi Liu via umamahesh) |
| |
| HDFS-6302. Implement XAttr as a INode feature. (Yi Liu via umamahesh) |
| |
| HDFS-6309. Javadocs for Xattrs apis in DFSClient and other minor fixups. (Charles Lamb via umamahesh) |
| |
| HDFS-6258. Namenode server-side storage for XAttrs. (Yi Liu via umamahesh) |
| |
| HDFS-6303. HDFS implementation of FileContext API for XAttrs. (Yi Liu and Charles Lamb via umamahesh) |
| |
| HDFS-6324. Shift XAttr helper code out for reuse. (Yi Liu via umamahesh) |
| |
| HDFS-6301. NameNode: persist XAttrs in fsimage and record XAttrs modifications to edit log. |
| (Yi Liu via umamahesh) |
| |
| HDFS-6298. XML based End-to-End test for getfattr and setfattr commands. (Yi Liu via umamahesh) |
| |
| HDFS-6314. Test cases for XAttrs. (Yi Liu via umamahesh) |
| |
| HDFS-6344. Maximum limit on the size of an xattr. (Yi Liu via umamahesh) |
| |
| HDFS-6377. Unify xattr name and value limits into a single limit. (wang) |
| |
| HDFS-6373. Remove support for extended attributes on symlinks. (Charles Lamb via wang) |
| |
| HDFS-6283. Write end user documentation for xattrs. (wang) |
| |
| HDFS-6412. Interface audience and stability annotations missing from |
| several new classes related to xattrs. (wang) |
| |
| HDFS-6259. Support extended attributes via WebHDFS. (yliu) |
| |
| HDFS-6346. Optimize OP_SET_XATTRS by persisting single Xattr entry per setXattr/removeXattr api call |
| (Yi Liu via umamahesh) |
| |
| HDFS-6331. ClientProtocol#setXattr should not be annotated idempotent. |
| (umamahesh via wang) |
| |
| HDFS-6335. TestOfflineEditsViewer for XAttr. (Yi Liu via umamahesh) |
| |
| HDFS-6343. fix TestNamenodeRetryCache and TestRetryCacheWithHA failures. (umamahesh) |
| |
| HDFS-6366. FsImage loading failed with RemoveXattr op (umamahesh) |
| |
| HDFS-6357. SetXattr should persist rpcIDs for handling retrycache with Namenode restart and HA |
| (umamahesh) |
| |
| HDFS-6372. Handle setXattr rpcIDs for OfflineEditsViewer. (umamahesh) |
| |
| HDFS-6410. DFSClient unwraps AclException in xattr methods, but those |
| methods cannot throw AclException. (wang) |
| |
| HDFS-6413. xattr names erroneously handled as case-insensitive. |
| (Charles Lamb via cnauroth) |
| |
| HDFS-6414. xattr modification operations are based on state of latest |
| snapshot instead of current version of inode. (Andrew Wang via cnauroth) |
| |
| HDFS-6374. setXAttr should require the user to be the owner of the file |
| or directory (Charles Lamb via wang) |
| |
| HDFS-6492. Support create-time xattrs and atomically setting multiple |
| xattrs. (wang) |
| |
| Release 2.4.1 - 2014-06-23 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| HDFS-6215. Wrong error message for upgrade. (Kihwal Lee via jeagles) |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| HDFS-6189. Multiple HDFS tests fail on Windows attempting to use a test |
| root path containing a colon. (cnauroth via szetszwo) |
| |
| HDFS-6197. Rolling upgrade rollback on Windows can fail attempting to rename |
| edit log segment files to a destination that already exists. (cnauroth) |
| |
| HDFS-6198. DataNode rolling upgrade does not correctly identify current |
| block pool directory and replace with trash on Windows. (cnauroth) |
| |
| HDFS-6206. Fix NullPointerException in DFSUtil.substituteForWildcardAddress. |
| (szetszwo) |
| |
| HDFS-6204. Fix TestRBWBlockInvalidation: change the last sleep to a loop. |
| (szetszwo) |
| |
| HDFS-6209. TestValidateConfigurationSettings should use random ports. |
| (Arpit Agarwal via szetszwo) |
| |
| HDFS-6208. DataNode caching can leak file descriptors. (cnauroth) |
| |
| HDFS-6231. DFSClient hangs infinitely if using hedged reads and all eligible |
| datanodes die. (cnauroth) |
| |
| HDFS-6234. TestDatanodeConfig#testMemlockLimit fails on Windows due to |
| invalid file path. (cnauroth) |
| |
| HDFS-6235. TestFileJournalManager can fail on Windows due to file locking if |
| tests run out of order. (cnauroth) |
| |
| HDFS-6229. Race condition in failover can cause RetryCache fail to work. |
| (jing9) |
| |
| HDFS-6232. OfflineEditsViewer throws a NPE on edits containing ACL |
| modifications (ajisakaa via cmccabe) |
| |
| HDFS-6236. ImageServlet should use Time#monotonicNow to measure latency. |
| (cnauroth) |
| |
| HDFS-6245. datanode fails to start with a bad disk even when failed |
| volumes is set. (Arpit Agarwal) |
| |
| HDFS-2882. DN continues to start up, even if block pool fails to initialize |
| (vinayakumarb) |
| |
| HDFS-6340. DN can't finalize upgrade. (Rahul Singhal via Arpit Agarwal) |
| |
| HDFS-6329. WebHdfs does not work if HA is enabled on NN but logical URI is |
| not configured. (kihwal) |
| |
| HDFS-6313. WebHdfs may use the wrong NN when configured for multiple HA NNs |
| (kihwal) |
| |
| HDFS-6326. WebHdfs ACL compatibility is broken. (cnauroth) |
| |
| HDFS-6361. TestIdUserGroup.testUserUpdateSetting failed due to out of range |
| nfsnobody Id. (Yongjun Zhang via brandonli) |
| |
| HDFS-4052. BlockManager#invalidateWork should print log outside the lock. |
| (Jing Zhao via suresh) |
| |
| HDFS-6362. InvalidateBlocks is inconsistent in usage of DatanodeUuid and |
| StorageID. (Arpit Agarwal) |
| |
| HDFS-6402. Suppress findbugs warning for failure to override equals and |
| hashCode in FsAclPermission. (cnauroth) |
| |
| HDFS-6325. Append should fail if the last block has insufficient number of |
| replicas (Keith Pak via cos) |
| |
| HDFS-6397. NN shows inconsistent value in deadnode count. |
| (Mohammad Kamrul Islam via kihwal) |
| |
| HDFS-6411. nfs-hdfs-gateway mount raises I/O error and hangs when a |
| unauthorized user attempts to access it (brandonli) |
| |
| Release 2.4.0 - 2014-04-07 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| HDFS-5698. Use protobuf to serialize / deserialize FSImage. (See breakdown |
| of tasks below for features and contributors) |
| |
| HDFS-5776 Support 'hedged' reads in DFSClient (Liang Xie via stack) |
| |
| HDFS-4685. Implementation of ACLs in HDFS. (See breakdown of tasks below for |
| features and contributors) |
| |
| IMPROVEMENTS |
| |
| HDFS-5781. Use an array to record the mapping between FSEditLogOpCode and |
| the corresponding byte value. (jing9) |
| |
| HDFS-5153. Datanode should send block reports for each storage in a |
| separate message. (Arpit Agarwal) |
| |
| HDFS-5804. HDFS NFS Gateway fails to mount and proxy when using Kerberos. |
| (Abin Shahab via jing9) |
| |
| HDFS-5859. DataNode#checkBlockToken should check block tokens even if |
| security is not enabled. (cmccabe) |
| |
| HDFS-5746. Add ShortCircuitSharedMemorySegment (cmccabe) |
| |
| HDFS-4911. Reduce PeerCache timeout to be commensurate with |
| dfs.datanode.socket.reuse.keepalive (cmccabe) |
| |
| HDFS-4370. Fix typo Blanacer in DataNode. (Chu Tong via shv) |
| |
| HDFS-5929. Add blockpool % usage to HDFS federated nn page. |
| (Siqi Li via suresh) |
| |
| HDFS-5810. Unify mmap cache and short-circuit file descriptor cache |
| (cmccabe) |
| |
| HDFS-5940. Minor cleanups to ShortCircuitReplica, FsDatasetCache, and |
| DomainSocketWatcher (cmccabe) |
| |
| HDFS-5531. Combine the getNsQuota() and getDsQuota() methods in INode. |
| (szetszwo) |
| |
| HDFS-5285. Flatten INodeFile hierarchy: Replace INodeFileUnderConstruction |
| and INodeFileUnderConstructionWithSnapshot with FileUnderContructionFeature. |
| (jing9 via szetszwo) |
| |
| HDFS-5286. Flatten INodeDirectory hierarchy: Replace INodeDirectoryWithQuota |
| with DirectoryWithQuotaFeature. (szetszwo) |
| |
| HDFS-5537. Remove FileWithSnapshot interface. (jing9 via szetszwo) |
| |
| HDFS-5554. Flatten INodeFile hierarchy: Replace INodeFileWithSnapshot with |
| FileWithSnapshotFeature. (jing9 via szetszwo) |
| |
| HDFS-5647. Merge INodeDirectory.Feature and INodeFile.Feature. (Haohui Mai |
| via jing9) |
| |
| HDFS-5632. Flatten INodeDirectory hierarchy: Replace |
| INodeDirectoryWithSnapshot with DirectoryWithSnapshotFeature. |
| (jing9 via szetszwo) |
| |
| HDFS-5715. Use Snapshot ID to indicate the corresponding Snapshot for a |
| FileDiff/DirectoryDiff. (jing9) |
| |
| HDFS-5726. Fix compilation error in AbstractINodeDiff for JDK7. (jing9) |
| |
| HDFS-5768. Consolidate the serialization code in DelegationTokenSecretManager |
| (Haohui Mai via brandonli) |
| |
| HDFS-5775. Consolidate the code for serialization in CacheManager |
| (Haohui Mai via brandonli) |
| |
| HDFS-5768. Consolidate the serialization code in DelegationTokenSecretManager |
| (Haohui Mai via brandonli) |
| |
| HDFS-5973. add DomainSocket#shutdown method (cmccabe) |
| |
| HDFS-5318. Support read-only and read-write paths to shared replicas. |
| (Eric Sirianni via Arpit Agarwal) |
| |
| HDFS-5868. Make hsync implementation pluggable on the DataNode. |
| (Buddy Taylor via Arpit Agarwal) |
| |
| HDFS-5935. New Namenode UI FS browser should throw smarter error messages. |
| (Travis Thompson via jing9) |
| |
| HDFS-5939. WebHdfs returns misleading error code and logs nothing if trying |
| to create a file with no DNs in cluster. (Yongjun Zhang via jing9) |
| |
| HDFS-6006. Remove duplicate code in FSNameSystem#getFileInfo. |
| (Akira Ajisaka via cnauroth) |
| |
| HDFS-6018. Exception recorded in LOG when IPCLoggerChannel#close is called. |
| (jing9) |
| |
| HDFS-3969. Small bug fixes and improvements for disk locations API. |
| (Todd Lipcon and Andrew Wang) |
| |
| HDFS-6025. Update findbugsExcludeFile.xml. (szetszwo) |
| |
| HDFS-6030. Remove an unused constructor in INode.java. (yzhang via |
| cmccabe) |
| |
| HDFS-4200. Reduce the size of synchronized sections in PacketResponder. |
| (Suresh Srinivas, backported by Andrew Wang, committed by jing9) |
| |
| HDFS-5950. The DFSClient and DataNode should use shared memory segments to |
| communicate short-circuit information. (cmccabe) |
| |
| HDFS-6046. add dfs.client.mmap.enabled (cmccabe) |
| |
| HDFS-5321. Clean up the HTTP-related configuration in HDFS (wheat9) |
| |
| HDFS-5167. Add metrics about the NameNode retry cache. (Tsuyoshi OZAWA via |
| jing9) |
| |
| HDFS-6043. Give HDFS daemons NFS3 and Portmap their own OPTS (brandonli) |
| |
| HDFS-6044. Add property for setting the NFS look up time for users |
| (brandonli) |
| |
| HDFS-6061. Allow dfs.datanode.shared.file.descriptor.path to contain |
| multiple entries and fall back when needed (cmccabe) |
| |
| HDFS-5986. Capture the number of blocks pending deletion on namenode webUI. |
| (cnauroth) |
| |
| HDFS-6070. Cleanup use of ReadStatistics in DFSInputStream. (wang) |
| |
| HDFS-6055. Change default configuration to limit file name length in HDFS. |
| (cnauroth) |
| |
| HDFS-3405. Checkpointing should use HTTP POST or PUT instead of GET-GET |
| to send merged fsimages. (Vinayakumar B via wang) |
| |
| HDFS-6085. Improve CacheReplicationMonitor log messages a bit (cmccabe) |
| |
| HDFS-6072. Clean up dead code of FSImage. (wheat9) |
| |
| HDFS-6080. Improve NFS gateway performance by making rtmax and wtmax |
| configurable. (Abin Shahab via brandonli) |
| |
| HDFS-6084. Namenode UI - "Hadoop" logo link shouldn't go to hadoop |
| homepage. (Travis Thompson via wheat9) |
| |
| HDFS-6106. Reduce default for |
| dfs.namenode.path.based.cache.refresh.interval.ms (cmccabe) |
| |
| HDFS-6090. Use MiniDFSCluster.Builder instead of deprecated constructors. |
| (Akira AJISAKA via jing9) |
| |
| HDFS-6068. Disallow snapshot names that are also invalid directory names. |
| (sathish via szetszwo) |
| |
| HDFS-6123. Do not log stack trace for ReplicaAlreadyExistsException and |
| SocketTimeoutException. (szetszwo) |
| |
| HDFS-6129. When a replica is not found for deletion, do not throw an |
| exception. (szetszwo) |
| |
| HDFS-6138. Add a user guide for how to use viewfs with federation. |
| (sanjay and szetszwo via szetszwo) |
| |
| HDFS-6120. Fix and improve safe mode log messages. (Arpit Agarwal) |
| |
| HDFS-6050. NFS does not handle exceptions correctly in a few places |
| (brandonli) |
| |
| HDFS-5138. Support HDFS upgrade in HA. (atm via todd) |
| |
| HDFS-6124. Add final modifier to class members. (Suresh Srinivas via |
| Arpit Agarwal) |
| |
| HDFS-5910. Enhance DataTransferProtocol to allow per-connection choice |
| of encryption/plain-text. (Benoy Antony via Arpit Agarwal) |
| |
| HDFS-6150. Add inode id information in the logs to make debugging easier. |
| (suresh) |
| |
| OPTIMIZATIONS |
| |
| HDFS-5790. LeaseManager.findPath is very slow when many leases need recovery |
| (todd) |
| |
| BUG FIXES |
| |
| HDFS-5492. Port HDFS-2069 (Incorrect default trash interval in the |
| docs) to trunk. (Akira Ajisaka via Arpit Agarwal) |
| |
| HDFS-5843. DFSClient.getFileChecksum() throws IOException if checksum is |
| disabled. (Laurent Goujon via jing9) |
| |
| HDFS-5856. DataNode.checkDiskError might throw NPE. |
| (Josh Elser via suresh) |
| |
| HDFS-5828. BlockPlacementPolicyWithNodeGroup can place multiple replicas on |
| the same node group when dfs.namenode.avoid.write.stale.datanode is true. |
| (Buddy via junping_du) |
| |
| HDFS-5767. NFS implementation assumes userName userId mapping to be unique, |
| which is not true sometimes (Yongjun Zhang via brandonli) |
| |
| HDFS-5791. TestHttpsFileSystem should use a random port to avoid binding |
| error during testing (Haohui Mai via brandonli) |
| |
| HDFS-5709. Improve NameNode upgrade with existing reserved paths and path |
| components. (Andrew Wang via atm) |
| |
| HDFS-5881. Fix skip() of the short-circuit local reader(legacy). (kihwal) |
| |
| HDFS-5895. HDFS cacheadmin -listPools has exit_code of 1 when the command |
| returns 0 result. (Tassapol Athiapinya via cnauroth) |
| |
| HDFS-5807. TestBalancerWithNodeGroup.testBalancerWithNodeGroup fails |
| intermittently. (Chen He via kihwal) |
| |
| HDFS-5882. TestAuditLogs is flaky (jxiang via cmccabe) |
| |
| HDFS-5900. Cannot set cache pool limit of "unlimited" via CacheAdmin. |
| (wang) |
| |
| HDFS-5886. Potential null pointer deference in RpcProgramNfs3#readlink() |
| (brandonli) |
| |
| HDFS-4858. HDFS DataNode to NameNode RPC should timeout. |
| (Henry Wang via shv) |
| |
| HDFS-5879. Some TestHftpFileSystem tests do not close streams. |
| (Gera Shegalov via suresh) |
| |
| HDFS-5938. Make BlockReaderFactory#BlockReaderPeer a static class to avoid |
| a findbugs warning. (cmccabe) |
| |
| HDFS-5891. webhdfs should not try connecting the DN during redirection |
| (Haohui Mai via brandonli) |
| |
| HDFS-5904. TestFileStatus fails intermittently. (Mit Desai via kihwal) |
| |
| HDFS-5941. add dfs.namenode.secondary.https-address and |
| dfs.namenode.secondary.https-address in hdfs-default.xml. |
| (Haohui Mai via cnauroth) |
| |
| HDFS-5913. Nfs3Utils#getWccAttr() should check attr parameter against null |
| (brandonli) |
| |
| HDFS-5934. New Namenode UI back button doesn't work as expected |
| (Travis Thompson via brandonli) |
| |
| HDFS-5901. NameNode new UI doesn't support IE8 and IE9 on windows 7 |
| (Vinayakumar B via brandonli) |
| |
| HDFS-5943. 'dfs.namenode.https-address' property is not loaded from |
| configuration in federation setup. (suresh) |
| |
| HDFS-3128. Unit tests should not use a test root in /tmp. (wang) |
| |
| HDFS-5948. TestBackupNode flakes with port in use error. (Haohui Mai |
| via Arpit Agarwal) |
| |
| HDFS-5949. New Namenode UI when trying to download a file, the browser |
| doesn't know the file name. (Haohui Mai via brandonli) |
| |
| HDFS-5716. Allow WebHDFS to use pluggable authentication filter |
| (Haohui Mai via brandonli) |
| |
| HDFS-5953. TestBlockReaderFactory fails if libhadoop.so has not been built. |
| (Akira Ajisaka via wang) |
| |
| HDFS-5759. Web UI does not show up during the period of loading FSImage. |
| (Haohui Mai via Arpit Agarwal) |
| |
| HDFS-5942. Fix javadoc in OfflineImageViewer. (Akira Ajisaka via cnauroth) |
| |
| HDFS-5780. TestRBWBlockInvalidation times out intemittently. (Mit Desai |
| via kihwal) |
| |
| HDFS-5803. TestBalancer.testBalancer0 fails. (Chen He via kihwal) |
| |
| HDFS-5893. HftpFileSystem.RangeHeaderUrlOpener uses the default |
| URLConnectionFactory which does not import SSL certificates. (Haohui Mai via |
| jing9) |
| |
| HDFS-5961. OIV cannot load fsimages containing a symbolic link. (kihwal) |
| |
| HDFS-5483. NN should gracefully handle multiple block replicas on same DN. |
| (Arpit Agarwal) |
| |
| HDFS-5742. DatanodeCluster (mini cluster of DNs) fails to start. |
| (Arpit Agarwal) |
| |
| HDFS-5979. Typo and logger fix for fsimage PB code. (wang) |
| |
| HDFS-5962. Mtime and atime are not persisted for symbolic links. (Akira |
| Ajisaka via kihwal) |
| |
| HDFS-5944. LeaseManager:findLeaseWithPrefixPath can't handle path like /a/b/ |
| and cause SecondaryNameNode failed do checkpoint (Yunjiong Zhao via brandonli) |
| |
| HDFS-5982. Need to update snapshot manager when applying editlog for deleting |
| a snapshottable directory. (jing9) |
| |
| HDFS-5988. Bad fsimage always generated after upgrade. (wang) |
| |
| HDFS-5922. DN heartbeat thread can get stuck in tight loop. (Arpit Agarwal) |
| |
| HDFS-6008. Namenode dead node link is giving HTTP error 500. |
| (Benoy Antony via cnauroth) |
| |
| HDFS-5936. MiniDFSCluster does not clean data left behind by |
| SecondaryNameNode. (Binglin Chang via cnauroth) |
| |
| HDFS-5339. WebHDFS URI does not accept logical nameservices when security is |
| enabled. (Haohui Mai via jing9) |
| |
| HDFS-6033. PBImageXmlWriter incorrectly handles processing cache |
| directives. (atm) |
| |
| HDFS-5821. TestHDFSCLI fails for user names with the dash character. |
| (Gera Shegalov via Arpit Agarwal) |
| |
| HDFS-5956. A file size is multiplied by the replication factor in 'hdfs oiv |
| -p FileDistribution' option. (Akira Ajisaka via wheat9) |
| |
| HDFS-5866. '-maxSize' and '-step' option fail in OfflineImageViewer. |
| (Akira Ajisaka via wheat9) |
| |
| HDFS-6040. fix DFSClient issue without libhadoop.so and some other |
| ShortCircuitShm cleanups (cmccabe) |
| |
| HDFS-6053. Fix TestDecommissioningStatus and TestDecommission in branch-2. |
| (jing9) |
| |
| HDFS-6047 TestPread NPE inside in DFSInputStream hedgedFetchBlockByteRange |
| (stack) |
| |
| HDFS-6051. HDFS cannot run on Windows since short-circuit shared memory |
| segment changes. (cmccabe) |
| |
| HDFS-5857. TestWebHDFS#testNamenodeRestart fails intermittently with NPE. |
| (Mit Desai via wheat9) |
| |
| HDFS-5898. Allow NFS gateway to login/relogin from its kerberos keytab. |
| (Abin Shahab via atm) |
| |
| HDFS-6057. DomainSocketWatcher.watcherThread should be marked as daemon |
| thread (cmccabe) |
| |
| HDFS-6058. Fix TestHDFSCLI failures after HADOOP-8691 change. |
| (Akira Ajisaka via wheat9) |
| |
| HDFS-6062. TestRetryCacheWithHA#testConcat is flaky. (Jing Zhao via wheat9) |
| |
| HDFS-6059. TestBlockReaderLocal fails if native library is not available. |
| (Akira AJISAKA via Colin Patrick McCabe) |
| |
| HDFS-6084. DFSConfigKeys.DFS_BLOCKREPORT_INTERVAL_MSEC_DEFAULT is |
| not updated with latest block report interval of 6 hrs. |
| (Vinayakumar B via wheat9) |
| |
| HDFS-6067. TestPread.testMaxOutHedgedReadPool is flaky (cmccabe) |
| |
| HDFS-6065. HDFS zero-copy reads should return null on EOF when doing ZCR |
| (cmccabe) |
| |
| HDFS-5064. Standby checkpoints should not block concurrent readers. |
| (atm via wang) |
| |
| HDFS-6078. TestIncrementalBlockReports is flaky. (Arpit Agarwal) |
| |
| HDFS-6071. BlockReaderLocal doesn't return -1 on EOF when doing a |
| zero-length read on a short file (cmccabe) |
| |
| HDFS-6077. Running slive with webhdfs on secure HA cluster fails with unkown |
| host exception. (jing9) |
| |
| HDFS-6086. Fix a case where zero-copy or no-checksum reads were not allowed |
| even when the block was cached (cmccabe) |
| |
| HDFS-6079. Timeout for getFileBlockStorageLocations does not work. (wang) |
| |
| HDFS-5705. TestSecondaryNameNodeUpgrade#testChangeNsIDFails may fail due |
| to ConcurrentModificationException. (Ted Yu via brandonli) |
| |
| HDFS-6096. TestWebHdfsTokens may timeout. (szetszwo via Arpit Agarwal) |
| |
| HDFS-5244. TestNNStorageRetentionManager#testPurgeMultipleDirs fails. |
| (Jinghui Wang via suresh) |
| |
| HDFS-6097. zero-copy reads are incorrectly disabled on file offsets above |
| 2GB (cmccabe) |
| |
| HDFS-6102. Lower the default maximum items per directory to fix PB fsimage |
| loading. (wang) |
| |
| HDFS-6094. The same block can be counted twice towards safe mode |
| threshold. (Arpit Agarwal) |
| |
| HDFS-5516. WebHDFS does not require user name when anonymous http requests |
| are disallowed. (Miodrag Radulovic via cnauroth) |
| |
| HDFS-6117. Print file path information in FileNotFoundException on INode |
| ID mismatch. (suresh) |
| |
| HDFS-6099. HDFS file system limits not enforced on renames. (cnauroth) |
| |
| HDFS-6100. DataNodeWebHdfsMethods does not failover in HA mode. (Haohui Mai |
| via jing9) |
| |
| HDFS-6105. NN web UI for DN list loads the same jmx page multiple times. |
| (wheat9) |
| |
| HDFS-6127. WebHDFS tokens cannot be renewed in HA setup. (wheat9) |
| |
| HDFS-6131. Move HDFSHighAvailabilityWithNFS.apt.vm and |
| HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS. (jing9) |
| |
| HDFS-6140. WebHDFS cannot create a file with spaces in the name after HA |
| failover changes. (cnauroth) |
| |
| HDFS-6135. In HDFS upgrade with HA setup, JournalNode cannot handle layout |
| version bump when rolling back. (jing9) |
| |
| HDFS-5846. Assigning DEFAULT_RACK in resolveNetworkLocation method can break |
| data resiliency. (Nikola Vujic via cnauroth) |
| |
| HDFS-5840. Follow-up to HDFS-5138 to improve error handling during partial |
| upgrade failures. (atm, jing9 and suresh via jing9) |
| |
| HDFS-6130. NPE when upgrading namenode from fsimages older than -32. |
| (wheat9) |
| |
| HDFS-6115. Call flush() for every append on block scan verification log. |
| (Vinayakumar B via szetszwo) |
| |
| HDFS-5672. TestHASafeMode#testSafeBlockTracking fails in trunk. (jing9) |
| |
| HDFS-6157. Fix the entry point of OfflineImageViewer for hdfs.cmd. (wheat9) |
| |
| HDFS-6163. Fix a minor bug in the HA upgrade document. (Fengdong Yu via |
| jing9) |
| |
| HDFS-6166. Change Balancer socket read timeout to 20 minutes and add |
| 10 seconds delay after error. (Nathan Roberts via szetszwo) |
| |
| BREAKDOWN OF HDFS-5698 SUBTASKS AND RELATED JIRAS |
| |
| HDFS-5717. Save FSImage header in protobuf. (Haohui Mai via jing9) |
| |
| HDFS-5738. Serialize INode information in protobuf. (Haohui Mai via jing9) |
| |
| HDFS-5772. Serialize under-construction file information in FSImage. (jing9) |
| |
| HDFS-5783. Compute the digest before loading FSImage. (Haohui Mai via jing9) |
| |
| HDFS-5785. Serialize symlink in protobuf. (Haohui Mai via jing9) |
| |
| HDFS-5793. Optimize the serialization of PermissionStatus. (Haohui Mai via |
| jing9) |
| |
| HDFS-5743. Use protobuf to serialize snapshot information. (jing9) |
| |
| HDFS-5774. Serialize CachePool directives in protobuf. (Haohui Mai via jing9) |
| |
| HDFS-5744. Serialize information for token managers in protobuf. (Haohui Mai |
| via jing9) |
| |
| HDFS-5824. Add a Type field in Snapshot DiffEntry's protobuf definition. |
| (jing9) |
| |
| HDFS-5808. Implement cancellation when saving FSImage. (Haohui Mai via jing9) |
| |
| HDFS-5826. Update the stored edit logs to be consistent with the changes in |
| HDFS-5698 branch. (Haohui Mai via jing9) |
| |
| HDFS-5797. Implement offline image viewer. (Haohui Mai via jing9) |
| |
| HDFS-5771. Track progress when loading fsimage. (Haohui Mai via cnauroth) |
| |
| HDFS-5871. Use PBHelper to serialize CacheDirectiveInfoExpirationProto. |
| (Haohui Mai via jing9) |
| |
| HDFS-5884. LoadDelegator should use IOUtils.readFully() to read the magic |
| header. (Haohui Mai via jing9) |
| |
| HDFS-5885. Add annotation for repeated fields in the protobuf definition. |
| (Haohui Mai via jing9) |
| |
| HDFS-5906. Fixing findbugs and javadoc warnings in the HDFS-5698 branch. |
| (Haohui Mai via jing9) |
| |
| HDFS-5911. The id of a CacheDirective instance does not get serialized in |
| the protobuf-fsimage. (Haohui Mai via jing9) |
| |
| HDFS-5915. Refactor FSImageFormatProtobuf to simplify cross section reads. |
| (Haohui Mai via cnauroth) |
| |
| HDFS-5847. Consolidate INodeReference into a separate section. (jing9) |
| |
| HDFS-5959. Fix typo at section name in FSImageFormatProtobuf.java. |
| (Akira Ajisaka via suresh) |
| |
| HDFS-5981. PBImageXmlWriter generates malformed XML. |
| (Haohui Mai via cnauroth) |
| |
| HDFS-6107. When a block can't be cached due to limited space on the |
| DataNode, that block becomes uncacheable (cmccabe) |
| |
| HDFS-6089. Standby NN while transitioning to active throws a connection |
| refused error when the prior active NN process is suspended. |
| (Jing Zhao via Andrew Wang) |
| |
| BREAKDOWN OF HDFS-4685 SUBTASKS AND RELATED JIRAS |
| |
| HDFS-5596. Implement RPC stubs. (Haohui Mai via cnauroth) |
| |
| HDFS-5685. Implement ACL as a INode feature. (Haohui Mai via cnauroth) |
| |
| HDFS-5618. NameNode: persist ACLs in fsimage. (Haohui Mai via cnauroth) |
| |
| HDFS-5619. NameNode: record ACL modifications to edit log. |
| (Haohui Mai via cnauroth) |
| |
| HDFS-5673. Implement logic for modification of ACLs. (cnauroth) |
| |
| HDFS-5758. NameNode: complete implementation of inode modifications for |
| ACLs. (Chris Nauroth via wheat9) |
| |
| HDFS-5612. NameNode: change all permission checks to enforce ACLs in |
| addition to permissions. (Chris Nauroth via wheat9) |
| |
| HDFS-5613. NameNode: implement handling of ACLs in combination with |
| symlinks. (Chris Nauroth via wheat9) |
| |
| HDFS-5615. NameNode: implement handling of ACLs in combination with sticky |
| bit. (Chris Nauroth via wheat9) |
| |
| HDFS-5702. FsShell Cli: Add XML based End-to-End test for getfacl and |
| setfacl commands. (Vinay via cnauroth) |
| |
| HDFS-5608. WebHDFS: implement ACL APIs. |
| (Sachin Jose and Renil Joseph via cnauroth) |
| |
| HDFS-5614. NameNode: implement handling of ACLs in combination with |
| snapshots. (cnauroth) |
| |
| HDFS-5858. Refactor common ACL test cases to be run through multiple |
| FileSystem implementations. (cnauroth) |
| |
| HDFS-5860. Refactor INodeDirectory getDirectoryXFeature methods to use |
| common getFeature helper method. (Jing Zhao via cnauroth) |
| |
| HDFS-5861. Add CLI test for Ls output for extended ACL marker. |
| (Vinay via cnauroth) |
| |
| HDFS-5616. NameNode: implement default ACL handling. (cnauroth) |
| |
| HDFS-5899. Add configuration flag to disable/enable support for ACLs. |
| (cnauroth) |
| |
| HDFS-5914. Incorporate ACLs with the changes from HDFS-5698. |
| (Haohui Mai via cnauroth) |
| |
| HDFS-5625. Write end user documentation for HDFS ACLs. (cnauroth) |
| |
| HDFS-5925. ACL configuration flag must only reject ACL API calls, not ACLs |
| present in fsimage or edits. (cnauroth) |
| |
| HDFS-5923. Do not persist the ACL bit in the FsPermission. |
| (Haohui Mai via cnauroth) |
| |
| HDFS-5933. Optimize the FSImage layout for ACLs (Haohui Mai via cnauroth) |
| |
| HDFS-5932. Ls should display the ACL bit (Chris Nauroth via wheat9) |
| |
| HDFS-5937. Fix TestOfflineEditsViewer on HDFS-4685 branch. (cnauroth) |
| |
| HDFS-5737. Replacing only the default ACL can fail to copy unspecified base |
| entries from the access ACL. (cnauroth) |
| |
| HDFS-5739. ACL RPC must allow null name or unspecified permissions in ACL |
| entries. (cnauroth) |
| |
| HDFS-5799. Make audit logging consistent across ACL APIs. (cnauroth) |
| |
| HDFS-5849. Removing ACL from an inode fails if it has only a default ACL. |
| (cnauroth) |
| |
| HDFS-5623. NameNode: add tests for skipping ACL enforcement when permission |
| checks are disabled, user is superuser or user is member of supergroup. |
| (cnauroth) |
| |
| HDFS-5908. Change AclFeature to capture list of ACL entries in an |
| ImmutableList. (cnauroth) |
| |
| HDFS-6028. Print clearer error message when user attempts to delete required |
| mask entry from ACL. (cnauroth) |
| |
| HDFS-6039. Uploading a File under a Dir with default acls throws "Duplicated |
| ACLFeature". (cnauroth) |
| |
| HDFS-6063. TestAclCLI fails intermittently when running test 24: |
| copyFromLocal. (cnauroth) |
| |
| HDFS-6069. Quash stack traces when ACLs are disabled. (cnauroth) |
| |
| HDFS-5638. HDFS implementation of FileContext API for ACLs. |
| (Vinayakumar B via cnauroth) |
| |
| HDFS-5535 subtasks: |
| |
| HDFS-5496. Make replication queue initialization asynchronous. (Vinay via |
| jing9) |
| |
| HDFS-5645. Support upgrade marker in editlog streams. (szetszwo) |
| |
| HDFS-5752. Add a new DFSAdmin command to query, start and finalize rolling |
| upgrade. (szetszwo) |
| |
| HDFS-5786. Support QUERY and FINALIZE actions of rolling upgrade. (szetszwo) |
| |
| HDFS-5753. Add new Namenode startup options for downgrade and rollback using |
| upgrade marker. (szetszwo) |
| |
| HDFS-5835. Add a new option for starting Namenode when rolling upgrade is |
| in progress. (szetszwo) |
| |
| HDFS-5754. Split LayoutVerion into NameNodeLayoutVersion and |
| DataNodeLayoutVersion. (Brandon Li via szetszwo) |
| |
| HDFS-5848. Add rolling upgrade status to heartbeat response. (szetszwo) |
| |
| HDFS-5890. Avoid NPE in Datanode heartbeat. (Vinay via brandonli) |
| |
| HDFS-5869. When starting rolling upgrade or NN restarts, NN should create |
| a checkpoint right before the upgrade marker. (szetszwo) |
| |
| HDFS-5874. Should not compare DataNode current layout version with that of |
| NameNode in DataStrorage. (brandonli) |
| |
| HDFS-5889. When starting rolling upgrade, create a fs image for rollback |
| so that the standby namenode can create checkpoints during upgrade. |
| (szetszwo & jing9) |
| |
| HDFS-5907. Add BlockPoolSliceStorage 'trash' to handle block deletions |
| during rolling upgrades. (Arpit Agarwal) |
| |
| HDFS-5494. Merge Protobuf-based-FSImage code from trunk - fix build |
| break after the merge. (Jing Zhao via Arpit Agarwal) |
| |
| HDFS-5585. Provide admin commands for data node upgrade (kihwal) |
| |
| HDFS-5920. Support rollback of rolling upgrade in NameNode and JournalNodes. |
| (jing9) |
| |
| HDFS-5945. Add rolling upgrade information to fsimage; and disallow upgrade |
| and rolling upgrade to be started simultaneously. (szetszwo & jing9) |
| |
| HDFS-5966. Fix rollback of rolling upgrade in NameNode HA setup. (jing9 |
| via szetszwo) |
| |
| HDFS-5974. Fix compilation error, NameNodeLayoutVersion and |
| DataNodeLayoutVersion after merge from trunk. (szetszwo) |
| |
| HDFS-5963. TestRollingUpgrade#testSecondaryNameNode causes subsequent |
| tests to fail. (szetszwo via Arpit Agarwal) |
| |
| HDFS-5976. Create unit tests for downgrade and finalize rolling upgrade. |
| (Haohui Mai via Arpit Agarwal) |
| |
| HDFS-5980. Rollback does not need to load edits. (jing9 via szetszwo) |
| |
| HDFS-5984. Fix TestEditLog and TestStandbyCheckpoints. (jing9 via szetszwo) |
| |
| HDFS-5985. SimulatedFSDataset#disableAndPurgeTrashStorage should not throw |
| UnsupportedOperationException. (jing9 via kihwal) |
| |
| HDFS-5987. Fix findbugs warnings in Rolling Upgrade branch. (seztszwo via |
| Arpit Agarwal) |
| |
| HDFS-5992. Fix NPE in MD5FileUtils and update editsStored for |
| TestOfflineEditsViewer. (szetszwo) |
| |
| HDFS-5994. Fix TestDataNodeRollingUpgrade. (Arpit Agarwal via szetszwo) |
| |
| HDFS-5999. Do not create rollback fsimage when it already exists. (jing9) |
| |
| HDFS-6005. Simplify Datanode rollback and downgrade. (Suresh Srinivas via |
| Arpit Agarwal) |
| |
| HDFS-6004. Change DFSAdmin for rolling upgrade commands. (szetszwo via |
| Arpit Agarwal) |
| |
| HDFS-5583. Make DN send an OOB Ack on shutdown before restarting. (kihwal) |
| |
| HDFS-5778. Add rolling upgrade user document. (szetszwo) |
| |
| HDFS-6003. Add the new -rollingUpgrade startup option to the namenode |
| usage message. (Vinayakumar B via szetszwo) |
| |
| HDFS-6014. Fix findbug warnings introduced by HDFS-5583. (kihwal) |
| |
| HDFS-6015. Fix TestBlockRecovery |
| #testRaceBetweenReplicaRecoveryAndFinalizeBlock. (kihwal) |
| |
| HDFS-5924. Utilize OOB upgrade message processing for writes. (kihwal) |
| |
| HDFS-5498. Improve datanode startup time. (kihwal) |
| |
| HDFS-6000. Avoid saving namespace when starting rolling upgrade. (jing9) |
| |
| HDFS-6017. Query the status of rolling upgrade in the preparation stage in |
| TestRollingUpgrade and TestRollingUpgradeRollback. (Haohui Mai via |
| Arpit Agarwal) |
| |
| HDFS-6020. Fix the five findbugs warnings. (kihwal) |
| |
| HDFS-6019. Standby NN might not checkpoint when processing the rolling |
| upgrade marker. (Haohui Mai via jing9) |
| |
| HDFS-6023. Test whether the standby NN continues to checkpoint after the |
| prepare stage. (Haohui Mai via jing9) |
| |
| HDFS-6024. Test whether the NN will reject the downgrade if it has a |
| fsimage from a newer release. (Haohui Mai via jing9) |
| |
| HDFS-6026. Fix TestDFSUpgrade and TestDataNodeRollingUpgrade. |
| (jing9 via szetszwo) |
| |
| HDFS-6029. Secondary NN fails to checkpoint after -rollingUpgrade prepare. |
| (jing9) |
| |
| HDFS-6032. -rollingUpgrade query hits NPE after the NN restarts. (Haohui Mai |
| via jing9) |
| |
| HDFS-6031. Add back the "-rollingUpgrade started" namenode startup option; |
| otherwise, namenode cannot start when the layout version is changed. |
| (szetszwo) |
| |
| HDFS-6034. Use DataNodeLayoutVersion for DN registration check and do not |
| verify layout version if there is a rolling upgrade in progress. (szetszwo) |
| |
| HDFS-6013. add rollingUpgrade information to latest UI. |
| (Vinayakumar B via wheat9) |
| |
| HDFS-6042. Fix rolling upgrade documentation and error messages. (szetszwo |
| via Arpit Agarwal) |
| |
| HDFS-6041. Downgrade/Finalize should rename the rollback image instead of |
| purging it. (jing9) |
| |
| HDFS-6060. NameNode should not check DataNode layout version (brandonli) |
| |
| HDFS-6076. DataNode with SimulatedDataSet should not create |
| DatanodeRegistration with namenode layout version and namenode node type. |
| (szetszwo) |
| |
| HDFS-6038. Allow JournalNode to handle editlog produced by new release with |
| future layoutversion. (jing9) |
| |
| HDFS-4564. Ensure webhdfs returns correct HTTP response codes for denied |
| operations. (daryn via acmurthy) |
| |
| Release 2.3.1 - UNRELEASED |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| Release 2.3.0 - 2014-02-18 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| HDFS-5122. Support failover and retry in WebHdfsFileSystem for NN HA. |
| (Haohui Mai via jing9) |
| |
| HDFS-4953. Enable HDFS local reads via mmap. |
| (Colin Patrick McCabe via wang). |
| |
| HDFS-5342. Provide more information in the FSNamesystem JMX interfaces. |
| (Haohui Mai via jing9) |
| |
| HDFS-5334. Implement dfshealth.jsp in HTML pages. (Haohui Mai via jing9) |
| |
| HDFS-5379. Update links to datanode information in dfshealth.html. (Haohui |
| Mai via jing9) |
| |
| HDFS-5382. Implement the UI of browsing filesystems in HTML 5 page. (Haohui |
| Mai via jing9) |
| |
| HDFS-3987. Support webhdfs over HTTPS. (Haohui Mai via jing9) |
| |
| HDFS-5444. Choose default web UI based on browser capabilities. (Haohui Mai |
| via jing9) |
| |
| HDFS-5514. FSNamesystem's fsLock should allow custom implementation (daryn) |
| |
| HDFS-2832. Heterogeneous Storages support in HDFS phase 1 - treat DataNode |
| as a collection of storages (see breakdown of tasks below for features and |
| contributors). |
| |
| HDFS-5703. Add support for HTTPS and swebhdfs to HttpFS. (tucu) |
| |
| HDFS-4949. Centralized cache management in HDFS (wang and cmccabe) |
| |
| IMPROVEMENTS |
| |
| HDFS-5360. Improvement of usage message of renameSnapshot and |
| deleteSnapshot. (Shinichi Yamashita via wang) |
| |
| HDFS-5331. make SnapshotDiff.java to a o.a.h.util.Tool interface implementation. |
| (Vinayakumar B via umamahesh) |
| |
| HDFS-4657. Limit the number of blocks logged by the NN after a block |
| report to a configurable value. (Aaron T. Myers via Colin Patrick |
| McCabe) |
| |
| HDFS-5344. Make LsSnapshottableDir as Tool interface implementation. (Sathish via umamahesh) |
| |
| HDFS-5544. Adding Test case For Checking dfs.checksum type as NULL value. (Sathish via umamahesh) |
| |
| HDFS-5568. Support includeSnapshots option with Fsck command. (Vinayakumar B via umamahesh) |
| |
| HDFS-4983. Numeric usernames do not work with WebHDFS FS. (Yongjun Zhang via |
| jing9) |
| |
| HDFS-5592. statechangeLog of completeFile should be logged only in case of success. |
| (Vinayakumar via umamahesh) |
| |
| HDFS-5662. Can't decommission a DataNode due to file's replication factor |
| larger than the rest of the cluster size. (brandonli) |
| |
| HDFS-5068. Convert NNThroughputBenchmark to a Tool to allow generic options. |
| (shv) |
| |
| HDFS-5675. Add Mkdirs operation to NNThroughputBenchmark. |
| (Plamen Jeliazkov via shv) |
| |
| HDFS-5677. Need error checking for HA cluster configuration. |
| (Vincent Sheffer via cos) |
| |
| HDFS-5825. Use FileUtils.copyFile() to implement DFSTestUtils.copyFile(). |
| (Haohui Mai via Arpit Agarwal) |
| |
| HDFS-5267. Remove volatile from LightWeightHashSet. (Junping Du via llu) |
| |
| HDFS-4278. Log an ERROR when DFS_BLOCK_ACCESS_TOKEN_ENABLE config is |
| disabled but security is turned on. (Kousuke Saruta via harsh) |
| |
| HDFS-5004. Add additional JMX bean for NameNode status data. Contributed |
| by Trevor Lorimer. |
| |
| HDFS-4994. Audit log getContentSummary() calls. (Robert Parker via kihwal) |
| |
| HDFS-5144. Document time unit to NameNodeMetrics. (Akira Ajisaka via |
| suresh) |
| |
| HDFS-4491. Parallel testing HDFS. (Andrey Klochkov via cnauroth) |
| |
| HDFS-4879. Add "blocked ArrayList" collection to avoid CMS full GCs |
| (Todd Lipcon via Colin Patrick McCabe) |
| |
| HDFS-4096. Add snapshot information to namenode WebUI. (Haohui Mai via |
| jing9) |
| |
| HDFS-5188. In BlockPlacementPolicy, reduce the number of chooseTarget(..) |
| methods; replace HashMap with Map in parameter declarations and cleanup |
| some related code. (szetszwo) |
| |
| HDFS-5207. In BlockPlacementPolicy.chooseTarget(..), change the writer |
| and the excludedNodes parameter types respectively to Node and Set. |
| (Junping Du via szetszwo) |
| |
| HDFS-5240. Separate formatting from logging in the audit logger API (daryn) |
| |
| HDFS-5191. Revisit zero-copy API in FSDataInputStream to make it more |
| intuitive. (Contributed by Colin Patrick McCabe) |
| |
| HDFS-5260. Merge zero-copy memory-mapped HDFS client reads to trunk and |
| branch-2. (cnauroth) |
| |
| HDFS-4517. Cover class RemoteBlockReader with unit tests. (Vadim Bondarev |
| and Dennis Y via kihwal) |
| |
| HDFS-4512. Cover package org.apache.hadoop.hdfs.server.common with tests. |
| (Vadim Bondarev via kihwal) |
| |
| HDFS-4510. Cover classes ClusterJspHelper/NamenodeJspHelper with unit |
| tests. (Andrey Klochkov via kihwal) |
| |
| HDFS-5323. Remove some deadcode in BlockManager. (Colin Patrick McCabe) |
| |
| HDFS-5338. Add a conf to disable hostname check in datanode registration. |
| (szetszwo) |
| |
| HDFS-5130. Add test for snapshot related FsShell and DFSAdmin commands. |
| (Binglin Chang via jing9) |
| |
| HDFS-5374. Remove deadcode in DFSOutputStream. (suresh) |
| |
| HDFS-4511. Cover package org.apache.hadoop.hdfs.tools with unit test |
| (Andrey Klochkov via jeagles) |
| |
| HDFS-4885. Improve the verifyBlockPlacement() API in BlockPlacementPolicy. |
| (Junping Du via szetszwo) |
| |
| HDFS-5363. Refactor WebHdfsFileSystem: move SPENGO-authenticated connection |
| creation to URLConnectionFactory. (Haohui Mai via jing9) |
| |
| HDFS-5436. Move HsFtpFileSystem and HFtpFileSystem into org.apache.hdfs.web |
| (Haohui Mai via Arpit Agarwal) |
| |
| HDFS-5371. Let client retry the same NN when |
| "dfs.client.test.drop.namenode.response.number" is enabled. (jing9) |
| |
| HDFS-5467. Remove tab characters in hdfs-default.xml. |
| (Shinichi Yamashita via Andrew Wang) |
| |
| HDFS-5495. Remove further JUnit3 usages from HDFS. |
| (Jarek Jarcec Cecho via Andrew Wang) |
| |
| HDFS-5325. Remove WebHdfsFileSystem#ConnRunner. (Haohui Mai via jing9) |
| |
| HDFS-5488. Clean up TestHftpURLTimeout. (Haohui Mai via jing9) |
| |
| HDFS-5440. Extract the logic of handling delegation tokens in HftpFileSystem |
| to the TokenAspect class. (Haohui Mai via jing9) |
| |
| HDFS-5487. Introduce unit test for TokenAspect. (Haohui Mai via jing9) |
| |
| HDFS-4995. Make getContentSummary less expensive. (kihwal) |
| |
| HDFS-5506. Use URLConnectionFactory in DelegationTokenFetcher. (Haohui Mai |
| via jing9) |
| |
| HDFS-5489. Use TokenAspect in WebHDFSFileSystem. (Haohui Mai via jing9) |
| |
| HDFS-5393. Serve bootstrap and jQuery locally. (Haohui Mai via jing9) |
| |
| HDFS-5073. TestListCorruptFileBlocks fails intermittently. (Arpit Agarwal) |
| |
| HDFS-1386. TestJMXGet fails in jdk7 (jeagles) |
| |
| HDFS-5532. Enable the webhdfs by default to support new HDFS web UI. (Vinay |
| via jing9) |
| |
| HDFS-5525. Inline dust templates for new Web UI. (Haohui Mai via jing9) |
| |
| HDFS-5561. FSNameSystem#getNameJournalStatus() in JMX should return plain |
| text instead of HTML. (Haohui Mai via jing9) |
| |
| HDFS-5581. NameNodeFsck should use only one instance of |
| BlockPlacementPolicy. (vinay via cmccabe) |
| |
| HDFS-5633. Improve OfflineImageViewer to use less memory. (jing9) |
| |
| HDFS-5023. TestSnapshotPathINodes.testAllowSnapshot is failing with jdk7 |
| (Mit Desai via jeagles) |
| |
| HDFS-5637. Try to refeatchToken while local read InvalidToken occurred. |
| (Liang Xie via junping_du) |
| |
| HDFS-5652. Refactor invalid block token exception handling in DFSInputStream. |
| (Liang Xie via junping_du) |
| |
| HDFS-5350. Name Node should report fsimage transfer time as a metric. |
| (Jimmy Xiang via wang) |
| |
| HDFS-5538. URLConnectionFactory should pick up the SSL related configuration |
| by default. (Haohui Mai via jing9) |
| |
| HDFS-5545. Allow specifying endpoints for listeners in HttpServer. (Haohui |
| Mai via jing9) |
| |
| HDFS-5536. Implement HTTP policy for Namenode and DataNode. (Haohui Mai via |
| jing9) |
| |
| HDFS-5312. Generate HTTP / HTTPS URL in DFSUtil#getInfoServer() based on the |
| configured http policy. (Haohui Mai via jing9) |
| |
| HDFS-5629. Support HTTPS in JournalNode and SecondaryNameNode. |
| (Haohui Mai via jing9) |
| |
| HDFS-5674. Editlog code cleanup: remove @SuppressWarnings("deprecation") in |
| FSEditLogOp; change FSEditLogOpCodes.fromByte(..) to be more efficient; and |
| change Some fields in FSEditLog to final. (szetszwo) |
| |
| HDFS-5634. Allow BlockReaderLocal to switch between checksumming and not |
| (cmccabe) |
| |
| HDFS-5663 make the retry time and interval value configurable in openInfo() |
| (Liang Xie via stack) |
| |
| HDFS-5540. Fix intermittent failure in TestBlocksWithNotEnoughRacks. |
| (Binglin Chang via junping_du) |
| |
| HDFS-5695. Clean up TestOfflineEditsViewer and OfflineEditsViewerHelper. |
| (Haohui Mai via jing9) |
| |
| HDFS-5220. Expose group resolution time as metric (jxiang via cmccabe) |
| |
| HDFS-5762. BlockReaderLocal doesn't return -1 on EOF when doing zero-length |
| reads (Colin Patrick McCabe) |
| |
| HDFS-5766. In DFSInputStream, do not add datanode to deadNodes after |
| InvalidEncryptionKeyException in fetchBlockByteRange (Liang Xie via Colin |
| Patrick McCabe) |
| |
| HDFS-5704. Change OP_UPDATE_BLOCKS with a new OP_ADD_BLOCK. (jing9) |
| |
| HDFS-5784. Reserve space in edit log header and fsimage header for feature |
| flag section. (Colin Patrick McCabe) |
| |
| HDFS-5434. Change block placement policy constructors from package private |
| to protected. (Buddy Taylor via Arpit Agarwal) |
| |
| HDFS-5788. listLocatedStatus response can be very large. (Nathan Roberts |
| via kihwal) |
| |
| HDFS-5841. Update HDFS caching documentation with new changes. (wang) |
| |
| OPTIMIZATIONS |
| |
| HDFS-5239. Allow FSNamesystem lock fairness to be configurable (daryn) |
| |
| HDFS-5341. Reduce fsdataset lock duration during directory scanning. |
| (Qus-Jiawei via kihwal) |
| |
| HDFS-5681. renewLease should not hold fsn write lock. (daryn via Kihwal) |
| |
| HDFS-5241. Provide alternate queuing audit logger to reduce logging |
| contention (daryn) |
| |
| BUG FIXES |
| |
| HDFS-5307. Support both HTTP and HTTPS in jsp pages (Haohui Mai via |
| brandonli) |
| |
| HDFS-5291. Standby namenode after transition to active goes into safemode. |
| (jing9) |
| |
| HDFS-5317. Go back to DFS Home link does not work on datanode webUI |
| (Haohui Mai via brandonli) |
| |
| HDFS-5316. Namenode ignores the default https port (Haohui Mai via |
| brandonli) |
| |
| HDFS-5281. COMMIT request should not block. (brandonli) |
| |
| HDFS-5337. should do hsync for a commit request even there is no pending |
| writes (brandonli) |
| |
| HDFS-5335. Hive query failed with possible race in dfs output stream. |
| (Haohui Mai via suresh) |
| |
| HDFS-5322. HDFS delegation token not found in cache errors seen on secure HA |
| clusters. (jing9) |
| |
| HDFS-5329. Update FSNamesystem#getListing() to handle inode path in startAfter |
| token. (brandonli) |
| |
| HDFS-5330. fix readdir and readdirplus for large directories (brandonli) |
| |
| HDFS-5370. Typo in Error Message: different between range in condition |
| and range in error message. (Kousuke Saruta via suresh) |
| |
| HDFS-5365. Fix libhdfs compile error on FreeBSD9. (Radim Kolar via cnauroth) |
| |
| HDFS-5347. Add HDFS NFS user guide. (brandonli) |
| |
| HDFS-5403. WebHdfs client cannot communicate with older WebHdfs servers |
| post HDFS-5306. (atm) |
| |
| HDFS-5171. NFS should create input stream for a file and try to share it |
| with multiple read requests. (Haohui Mai via brandonli) |
| |
| HDFS-5413. hdfs.cmd does not support passthrough to any arbitrary class. |
| (cnauroth) |
| |
| HDFS-5433. When reloading fsimage during checkpointing, we should clear |
| existing snapshottable directories. (Aaron T. Myers via wang) |
| |
| HDFS-5432. TestDatanodeJsp fails on Windows due to assumption that loopback |
| address resolves to host name localhost. (cnauroth) |
| |
| HDFS-5065. TestSymlinkHdfsDisable fails on Windows. (ivanmi) |
| |
| HDFS-4633 TestDFSClientExcludedNodes fails sporadically if excluded nodes |
| cache expires too quickly (Chris Nauroth via Sanjay) |
| |
| HDFS-5037. Active NN should trigger its own edit log rolls. (wang) |
| |
| HDFS-5035. getFileLinkStatus and rename do not correctly check permissions |
| of symlinks. (Andrew Wang via Colin Patrick McCabe) |
| |
| HDFS-5456. NameNode startup progress creates new steps if caller attempts to |
| create a counter for a step that doesn't already exist. (cnauroth) |
| |
| HDFS-5458. Datanode failed volume threshold ignored if exception is thrown |
| in getDataDirsFromURIs. (Mike Mellenthin via wang) |
| |
| HDFS-5252. Stable write is not handled correctly in someplace. (brandonli) |
| |
| HDFS-5364. Add OpenFileCtx cache. (brandonli) |
| |
| HDFS-5469. Add configuration property for the sub-directroy export path |
| (brandonli) |
| |
| HDFS-5519. COMMIT handler should update the commit status after sync |
| (brandonli) |
| |
| HDFS-5372. In FSNamesystem, hasReadLock() returns false if the current |
| thread holds the write lock (Vinaykumar B via umamahesh) |
| |
| HDFS-4516. Client crash after block allocation and NN switch before lease recovery for |
| the same file can cause readers to fail forever (VinaayKumar B via umamahesh) |
| |
| HDFS-5014. Process register commands with out holding BPOfferService lock. |
| (Vinaykumar B via umamahesh) |
| |
| HDFS-5288. Close idle connections in portmap (Haohui Mai via brandonli) |
| |
| HDFS-5407. Fix typos in DFSClientCache (Haohui Mai via brandonli) |
| |
| HDFS-5548. Use ConcurrentHashMap in portmap (Haohui Mai via brandonli) |
| |
| HDFS-5577. NFS user guide update (brandonli) |
| |
| HDFS-5563. NFS gateway should commit the buffered data when read request comes |
| after write to the same file (brandonli) |
| |
| HDFS-4997. libhdfs doesn't return correct error codes in most cases (cmccabe) |
| |
| HDFS-5587. add debug information when NFS fails to start with duplicate user |
| or group names (brandonli) |
| |
| HDFS-5590. Block ID and generation stamp may be reused when persistBlocks is |
| set to false. (jing9) |
| |
| HDFS-5353. Short circuit reads fail when dfs.encrypt.data.transfer is |
| enabled. (Colin Patrick McCabe via jing9) |
| |
| HDFS-5283. Under construction blocks only inside snapshots should not be |
| counted in safemode threshhold. (Vinay via szetszwo) |
| |
| HDFS-5257. addBlock() retry should return LocatedBlock with locations else client |
| will get AIOBE. (Vinay via jing9) |
| |
| HDFS-5427. Not able to read deleted files from snapshot directly under |
| snapshottable dir after checkpoint and NN restart. (Vinay via jing9) |
| |
| HDFS-5443. Delete 0-sized block when deleting an under-construction file that |
| is included in snapshot. (jing9) |
| |
| HDFS-5476. Snapshot: clean the blocks/files/directories under a renamed |
| file/directory while deletion. (jing9) |
| |
| HDFS-5425. Renaming underconstruction file with snapshots can make NN failure on |
| restart. (jing9 and Vinay) |
| |
| HDFS-5474. Deletesnapshot can make Namenode in safemode on NN restarts. |
| (Sathish via jing9) |
| |
| HDFS-5504. In HA mode, OP_DELETE_SNAPSHOT is not decrementing the safemode threshold, |
| leads to NN safemode. (Vinay via jing9) |
| |
| HDFS-5428. Under construction files deletion after snapshot+checkpoint+nn restart |
| leads nn safemode. (jing9) |
| |
| HDFS-5074. Allow starting up from an fsimage checkpoint in the middle of a |
| segment. (Todd Lipcon via atm) |
| |
| HDFS-4201. NPE in BPServiceActor#sendHeartBeat. (jxiang via cmccabe) |
| |
| HDFS-5666. Fix inconsistent synchronization in BPOfferService (jxiang via cmccabe) |
| |
| HDFS-5657. race condition causes writeback state error in NFS gateway (brandonli) |
| |
| HDFS-5661. Browsing FileSystem via web ui, should use datanode's fqdn instead of ip |
| address. (Benoy Antony via jing9) |
| |
| HDFS-5582. hdfs getconf -excludeFile or -includeFile always failed (sathish |
| via cmccabe) |
| |
| HDFS-5671. Fix socket leak in DFSInputStream#getBlockReader. (JamesLi via umamahesh) |
| |
| HDFS-5649. Unregister NFS and Mount service when NFS gateway is shutting down. |
| (brandonli) |
| |
| HDFS-5789. Some of snapshot APIs missing checkOperation double check in fsn. (umamahesh) |
| |
| HDFS-5343. When cat command is issued on snapshot files getting unexpected result. |
| (Sathish via umamahesh) |
| |
| HDFS-5297. Fix dead links in HDFS site documents. (Akira Ajisaka via |
| Arpit Agarwal) |
| |
| HDFS-5830. WebHdfsFileSystem.getFileBlockLocations throws |
| IllegalArgumentException when accessing another cluster. (Yongjun Zhang via |
| Colin Patrick McCabe) |
| |
| HDFS-5833. Fix SecondaryNameNode javadoc. (Bangtao Zhou via Arpit Agarwal) |
| |
| HDFS-5844. Fix broken link in WebHDFS.apt.vm. (Akira Ajisaka via |
| Arpit Agarwal) |
| |
| HDFS-5034. Remove debug prints from GetFileLinkInfo (Andrew Wang via Colin |
| Patrick McCabe) |
| |
| HDFS-4816. transitionToActive blocks if the SBN is doing checkpoint image |
| transfer. (Andrew Wang) |
| |
| HDFS-5164. deleteSnapshot should check if OperationCategory.WRITE is |
| possible before taking write lock. (Colin Patrick McCabe) |
| |
| HDFS-5170. BlockPlacementPolicyDefault uses the wrong classname when |
| alerting to enable debug logging. (Andrew Wang) |
| |
| HDFS-5266. ElasticByteBufferPool#Key does not implement equals. (cnauroth) |
| |
| HDFS-5352. Server#initLog() doesn't close InputStream in httpfs. (Ted Yu via |
| jing9) |
| |
| HDFS-4376. Fix race conditions in Balancer. (Junping Du via szetszwo) |
| |
| HDFS-5375. hdfs.cmd does not expose several snapshot commands. (cnauroth) |
| |
| HDFS-5336. DataNode should not output 'StartupProgress' metrics. |
| (Akira Ajisaka via cnauroth) |
| |
| HDFS-5400. DFS_CLIENT_MMAP_CACHE_THREAD_RUNS_PER_TIMEOUT constant is set |
| to the wrong value. (Colin Patrick McCabe) |
| |
| HDFS-5075. httpfs-config.sh calls out incorrect env script name |
| (Timothy St. Clair via stevel) |
| |
| HDFS-5438. Flaws in block report processing can cause data loss. (kihwal) |
| |
| HDFS-5502. Fix HTTPS support in HsftpFileSystem. (Haohui Mai via jing9) |
| |
| HDFS-5552. Fix wrong information of "Cluster summay" in dfshealth.html. |
| (Haohui Mai via jing9) |
| |
| HDFS-5533. Symlink delete/create should be treated as DELETE/CREATE in snapshot diff |
| report. (Binglin Chang via jing9) |
| |
| HDFS-5580. Fix infinite loop in Balancer.waitForMoveCompletion. |
| (Binglin Chang via junping_du) |
| |
| HDFS-5676. fix inconsistent synchronization of CachingStrategy (cmccabe) |
| |
| HDFS-5691. Fix typo in ShortCircuitLocalRead document. |
| (Akira Ajisaka via suresh) |
| |
| HDFS-5690. DataNode fails to start in secure mode when dfs.http.policy equals to |
| HTTP_ONLY. (Haohui Mai via jing9) |
| |
| HDFS-5449. WebHdfs compatibility broken between 2.2 and 1.x / 23.x (kihwal) |
| |
| HDFS-5756. hadoopRzOptionsSetByteBufferPool does not accept NULL argument, |
| contrary to docs. (cmccabe via wang) |
| |
| HDFS-5747. Fix NPEs in BlockManager. (Arpit Agarwal) |
| |
| HDFS-5710. FSDirectory#getFullPathName should check inodes against null. |
| (Uma Maheswara Rao G via jing9) |
| |
| HDFS-5579. Under construction files make DataNode decommission take very long |
| hours. (zhaoyunjiong via jing9) |
| |
| HDFS-5777. Update LayoutVersion for the new editlog op OP_ADD_BLOCK. (jing9) |
| |
| HDFS-5800. Fix a typo in DFSClient.renewLease(). (Kousuke Saruta |
| via szetszwo) |
| |
| HDFS-5748. Too much information shown in the dfs health page. |
| (Haohui Mai via brandonli) |
| |
| HDFS-5806. Balancer should set SoTimeout to avoid indefinite hangs. |
| (Nathan Roberts via Andrew Wang) |
| |
| HDFS-5728. Block recovery will fail if the metafile does not have crc |
| for all chunks of the block (Vinay via kihwal) |
| |
| HDFS-5719. FSImage#doRollback() should close prevState before return |
| (Ted Yu via todd) |
| |
| HDFS-5721. sharedEditsImage in Namenode#initializeSharedEdits() should be |
| closed before method returns (Ted Yu via todd) |
| |
| HDFS-5845. SecondaryNameNode dies when checkpointing with cache pools. |
| (wang) |
| |
| HDFS-5842. Cannot create hftp filesystem when using a proxy user ugi and a doAs |
| on a secure cluster. (jing9) |
| |
| HDFS-5399. Revisit SafeModeException and corresponding retry policies. |
| (Jing Zhao via todd) |
| |
| HDFS-5876. SecureDataNodeStarter does not pick up configuration in |
| hdfs-site.xml. (Haohui Mai via jing9) |
| |
| HDFS-5873. dfs.http.policy should have higher precedence over dfs.https.enable. |
| (Haohui Mai via jing9) |
| |
| HDFS-5837. dfs.namenode.replication.considerLoad should consider |
| decommissioned nodes. (Tao Luo via shv) |
| |
| HDFS-5921. Cannot browse file system via NN web UI if any directory has |
| the sticky bit set. (atm) |
| |
| BREAKDOWN OF HDFS-2832 SUBTASKS AND RELATED JIRAS |
| |
| HDFS-4985. Add storage type to the protocol and expose it in block report |
| and block locations. (Arpit Agarwal) |
| |
| HDFS-5115. Make StorageID a UUID. (Arpit Agarwal) |
| |
| HDFS-5000. DataNode configuration should allow specifying storage type. |
| (Arpit Agarwal) |
| |
| HDFS-4987. Namenode changes to track multiple storages per datanode. |
| (szetszwo) |
| |
| HDFS-5154. Fix TestBlockManager and TestDatanodeDescriptor after HDFS-4987. |
| (Junping Du via szetszwo) |
| |
| HDFS-5009. Include storage information in the LocatedBlock. (szetszwo) |
| |
| HDFS-5134. Move blockContentsStale, heartbeatedSinceFailover and |
| firstBlockReport from DatanodeDescriptor to DatanodeStorageInfo; and |
| fix a synchronization problem in DatanodeStorageInfo. (szetszwo) |
| |
| HDFS-5157. Add StorageType to FsVolume. (Junping Du via szetszwo) |
| |
| HDFS-4990. Change BlockPlacementPolicy to choose storages instead of |
| datanodes. (szetszwo) |
| |
| HDFS-5232. Protocol changes to transmit StorageUuid. (Arpit Agarwal) |
| |
| HDFS-5233. Use Datanode UUID to identify Datanodes. (Arpit Agarwal) |
| |
| HDFS-5222. Move block schedule information from DatanodeDescriptor to |
| DatanodeStorageInfo. (szetszwo) |
| |
| HDFS-4988. Datanode must support all the volumes as individual storages. |
| (Arpit Agarwal) |
| |
| HDFS-5377. Heartbeats from Datandode should include one storage report |
| per storage directory. (Arpit Agarwal) |
| |
| HDFS-5398. NameNode changes to process storage reports per storage |
| directory. (Arpit Agarwal) |
| |
| HDFS-5390. Send one incremental block report per storage directory. |
| (Arpit Agarwal) |
| |
| HDFS-5401. Fix NPE in Directory Scanner. (Arpit Agarwal) |
| |
| HDFS-5417. Fix storage IDs in PBHelper and UpgradeUtilities. (szetszwo) |
| |
| HDFS-5214. Fix NPEs in BlockManager and DirectoryScanner. (Arpit Agarwal) |
| |
| HDFS-5435. File append fails to initialize storageIDs. (Junping Du via |
| Arpit Agarwal) |
| |
| HDFS-5437. Fix TestBlockReport and TestBPOfferService failures. (Arpit |
| Agarwal) |
| |
| HDFS-5447. Fix TestJspHelper. (Arpit Agarwal) |
| |
| HDFS-5452. Fix TestReplicationPolicy and TestBlocksScheduledCounter. |
| |
| HDFS-5448. Datanode should generate its ID on first registration. (Arpit |
| Agarwal) |
| |
| HDFS-5448. Fix break caused by previous checkin for HDFS-5448. (Arpit |
| Agarwal) |
| |
| HDFS-5455. NN should update storageMap on first heartbeat. (Arpit Agarwal) |
| |
| HDFS-5457. Fix TestDatanodeRegistration, TestFsck and TestAddBlockRetry. |
| (Contributed by szetszwo) |
| |
| HDFS-5466. Update storage IDs when the pipeline is updated. (Contributed |
| by szetszwo) |
| |
| HDFS-5439. Fix TestPendingReplication. (Contributed by Junping Du, Arpit |
| Agarwal) |
| |
| HDFS-5470. Add back trunk's reportDiff algorithm to the branch. |
| (Contributed by szetszwo) |
| |
| HDFS-5472. Fix TestDatanodeManager, TestSafeMode and |
| TestNNThroughputBenchmark (Contributed by szetszwo) |
| |
| HDFS-5475. NN incorrectly tracks more than one replica per DN. (Arpit |
| Agarwal) |
| |
| HDFS-5481. Fix TestDataNodeVolumeFailure in branch HDFS-2832. (Contributed |
| by Junping Du) |
| |
| HDFS-5480. Update Balancer for HDFS-2832. (Contributed by szetszwo) |
| |
| HDFS-5486. Fix TestNameNodeMetrics for HDFS-2832. (Arpit Agarwal) |
| |
| HDFS-5491. Update editsStored for HDFS-2832. (Arpit Agarwal) |
| |
| HDFS-5494. Fix findbugs warnings for HDFS-2832. (Arpit Agarwal) |
| |
| HDFS-5508. Fix compilation error after merge. (Contributed by szetszwo) |
| |
| HDFS-5501. Fix pendingReceivedRequests tracking in BPServiceActor. (Arpit |
| Agarwal) |
| |
| HDFS-5510. Fix a findbug warning in DataStorage.java on HDFS-2832 branch. |
| (Junping Du via Arpit Agarwal) |
| |
| HDFS-5515. Fix TestDFSStartupVersions for HDFS-2832. (Arpit Agarwal) |
| |
| HDFS-5527. Fix TestUnderReplicatedBlocks on branch HDFS-2832. (Arpit |
| Agarwal) |
| |
| HDFS-5547. Fix build break after merge from trunk to HDFS-2832. (Arpit |
| Agarwal) |
| |
| HDFS-5542. Fix TODO and clean up the code in HDFS-2832. (Contributed by |
| szetszwo) |
| |
| HDFS-5559. Fix TestDatanodeConfig in HDFS-2832. (Contributed by szetszwo) |
| |
| HDFS-5484. StorageType and State in DatanodeStorageInfo in NameNode is |
| not accurate. (Eric Sirianni via Arpit Agarwal) |
| |
| HDFS-5648. Get rid of FsDatasetImpl#perVolumeReplicaMap. (Arpit Agarwal) |
| |
| HDFS-5454. DataNode UUID should be assigned prior to FsDataset |
| initialization. (Arpit Agarwal) |
| |
| HDFS-5406. Send incremental block reports for all storages in a |
| single call. (Arpit Agarwal) |
| |
| HDFS-5667. Include DatanodeStorage in StorageReport. (Arpit Agarwal) |
| |
| BREAKDOWN OF HDFS-4949 SUBTASKS AND RELATED JIRAS |
| |
| HDFS-5049. Add JNI mlock support. (Andrew Wang via Colin Patrick McCabe) |
| |
| HDFS-5051. Propagate cache status information from the DataNode to the |
| NameNode (Andrew Wang via Colin Patrick McCabe) |
| |
| HDFS-5052. Add cacheRequest/uncacheRequest support to NameNode. |
| (Contributed by Colin Patrick McCabe.) |
| |
| HDFS-5050. Add DataNode support for mlock and munlock (contributed by |
| Andrew Wang) |
| |
| HDFS-5141. Add cache status information to datanode heartbeat. (Contributed |
| by Andrew Wang) |
| |
| HDFS-5121. Add RPCs for creating and manipulating cache pools. |
| (Contributed by Colin Patrick McCabe) |
| |
| HDFS-5163. Miscellaneous cache pool RPC fixes (Contributed by Colin Patrick |
| McCabe) |
| |
| HDFS-5169. hdfs.c: translateZCRException: null pointer deref when |
| translating some exceptions (Contributed by Colin Patrick McCabe) |
| |
| HDFS-5120. Add command-line support for manipulating cache pools. (cmccabe) |
| |
| HDFS-5158. Add command-line support for manipulating cache directives. |
| (cmccabe) |
| |
| HDFS-5198. NameNodeRpcServer must not send back DNA_FINALIZE in reply to a |
| cache report. (cmccabe) |
| |
| HDFS-5195. Prevent passing null pointer to mlock and munlock. Contributed |
| by Chris Nauroth. |
| |
| HDFS-5053. NameNode should invoke DataNode APIs to coordinate caching. |
| (Andrew Wang) |
| |
| HDFS-5201. NativeIO: consolidate getrlimit into NativeIO#getMemlockLimit. |
| (Contributed by Colin Patrick McCabe) |
| |
| HDFS-5197. Document dfs.cachereport.intervalMsec in hdfs-default.xml. |
| Contributed by Chris Nauroth. |
| |
| HDFS-5210. Fix some failing unit tests on HDFS-4949 branch. (Contributed by |
| Andrew Wang) |
| |
| HDFS-5213. Separate PathBasedCacheEntry and PathBasedCacheDirectiveWithId. |
| Contributed by Colin Patrick McCabe. |
| |
| HDFS-5236. Change PathBasedCacheDirective APIs to be a single value rather |
| than batch. (Contributed by Andrew Wang) |
| |
| HDFS-5119. Persist CacheManager state in the edit log. (Contributed by |
| Andrew Wang) |
| |
| HDFS-5190. Move cache pool related CLI commands to CacheAdmin. (Contributed |
| by Andrew Wang) |
| |
| HDFS-5309. Fix failing caching unit tests. (Andrew Wang) |
| |
| HDFS-5314. Do not expose CachePool type in AddCachePoolOp (Colin Patrick |
| McCabe) |
| |
| HDFS-5304. Expose if a block replica is cached in getFileBlockLocations. |
| (Contributed by Andrew Wang) |
| |
| HDFS-5224. Refactor PathBasedCache* methods to use a Path rather than a |
| String. Contributed by Chris Nauroth. |
| |
| HDFS-5348. Fix error message when dfs.datanode.max.locked.memory is |
| improperly configured. (Contributed by Colin Patrick McCabe) |
| |
| HDFS-5349. DNA_CACHE and DNA_UNCACHE should be by blockId only (cmccabe) |
| |
| HDFS-5358. Add replication field to PathBasedCacheDirective. (Contributed |
| by Colin Patrick McCabe) |
| |
| HDFS-5359. Allow LightWeightGSet#Iterator to remove elements. (Contributed |
| by Colin Patrick McCabe) |
| |
| HDFS-5373. hdfs cacheadmin -addDirective short usage does not mention |
| -replication parameter. Contributed by Chris Nauroth. |
| |
| HDFS-5096. Automatically cache new data added to a cached path (contributed |
| by Colin Patrick McCabe) |
| |
| HDFS-5383. fix broken caching unit tests (Andrew Wang) |
| |
| HDFS-5388. Loading fsimage fails to find cache pools during namenode |
| startup (Chris Nauroth via Colin Patrick McCabe) |
| |
| HDFS-5203. Concurrent clients that add a cache directive on the same path |
| may prematurely uncache each other. (Chris Nauroth via Colin Patrick McCabe) |
| |
| HDFS-5378. In CacheReport, don't send genstamp and length on the wire |
| (Contributed by Colin Patrick McCabe) |
| |
| HDFS-5385. Caching RPCs are AtMostOnce, but do not persist client ID and |
| call ID to edit log. (Chris Nauroth via Colin Patrick McCabe) |
| |
| HDFS-5404 Resolve regressions in Windows compatibility on HDFS-4949 branch. |
| Contributed by Chris Nauroth. |
| |
| HDFS-5405. Fix possible RetryCache hang for caching RPC handlers in |
| FSNamesystem. (Contributed by Andrew Wang) |
| |
| HDFS-5419. Fixup test-patch.sh warnings on HDFS-4949 branch. (wang) |
| |
| HDFS-5386. Add feature documentation for datanode caching. Contributed by |
| Colin Patrick McCabe. |
| |
| HDFS-5468. CacheAdmin help command does not recognize commands (Stephen |
| Chu via Colin Patrick McCabe) |
| |
| HDFS-5326. add modifyDirective to cacheAdmin (cmccabe) |
| |
| HDFS-5394: Fix race conditions in DN caching and uncaching (cmccabe) |
| |
| HDFS-5320. Add datanode caching metrics. Contributed by Andrew Wang. |
| |
| HDFS-5482. DistributedFileSystem#listPathBasedCacheDirectives must support |
| relative paths. Contributed by Colin Patrick McCabe. |
| |
| HDFS-5471. CacheAdmin -listPools fails when user lacks permissions to view |
| all pools (Andrew Wang via Colin Patrick McCabe) |
| |
| HDFS-5450. better API for getting the cached blocks locations. Contributed |
| by Andrew Wang. |
| |
| HDFS-5485. add command-line support for modifyDirective (cmccabe) |
| |
| HDFS-5366. recaching improvements (cmccabe) |
| |
| HDFS-5520. loading cache path directives from edit log doesnt update |
| nextEntryId (cmccabe) |
| |
| HDFS-5512. CacheAdmin -listPools fails with NPE when user lacks permissions |
| to view all pools (awang via cmccabe) |
| |
| HDFS-5513. CacheAdmin commands fail when using . as the path. Contributed |
| by Andrew Wang. |
| |
| HDFS-5511. improve CacheManipulator interface to allow better unit testing |
| (cmccabe) |
| |
| HDFS-5451. Add byte and file statistics to PathBasedCacheEntry. Contributed |
| by Colin Patrick McCabe. |
| |
| HDFS-5473. Consistent naming of user-visible caching classes and methods |
| (cmccabe) |
| |
| HDFS-5543. Fix narrow race condition in TestPathBasedCacheRequests |
| (cmccabe) |
| |
| HDFS-5565. CacheAdmin help should match against non-dashed commands (wang |
| via cmccabe) |
| |
| HDFS-5556. Add some more NameNode cache statistics, cache pool stats |
| (cmccabe) |
| |
| HDFS-5562. TestCacheDirectives and TestFsDatasetCache should stub out |
| native mlock. Contributed by Colin Patrick McCabe and Akira Ajisaka. |
| |
| HDFS-5430. Support TTL on CacheDirectives. Contributed by Andrew Wang. |
| |
| HDFS-5555. CacheAdmin commands fail when first listed NameNode is in |
| Standby (jxiang via cmccabe) |
| |
| HDFS-5626. dfsadmin report shows incorrect values (cmccabe) |
| |
| HDFS-5630. Hook up cache directive and pool usage statistics. (wang) |
| |
| HDFS-5665. Remove the unnecessary writeLock while initializing CacheManager |
| in FsNameSystem Ctor. (Uma Maheswara Rao G via Andrew Wang) |
| |
| HDFS-5431. Support cachepool-based limit management in path-based caching. |
| (awang via cmccabe) |
| |
| HDFS-5679. TestCacheDirectives should handle the case where native code is |
| not available. (wang) |
| |
| HDFS-5636. Enforce a max TTL per cache pool (awang via cmccabe) |
| |
| HDFS-5701. Fix the CacheAdmin -addPool -maxTtl option name. Contributed by |
| Stephen Chu. |
| |
| HDFS-5708. The CacheManager throws a NPE in the DataNode logs when |
| processing cache reports that refer to a block not known to the BlockManager. |
| Contributed by Colin Patrick McCabe. |
| |
| HDFS-5659. dfsadmin -report doesn't output cache information properly. |
| Contributed by Andrew Wang. |
| |
| HDFS-5651. Remove dfs.namenode.caching.enabled and improve CRM locking. |
| Contributed by Colin Patrick McCabe. |
| |
| HDFS-5589. Namenode loops caching and uncaching when data should be |
| uncached. (awang via cmccabe) |
| |
| HDFS-5724. modifyCacheDirective logging audit log command wrongly as |
| addCacheDirective (Uma Maheswara Rao G via Colin Patrick McCabe) |
| |
| Release 2.2.0 - 2013-10-13 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| HDFS-4817. Make HDFS advisory caching configurable on a per-file basis. |
| (Colin Patrick McCabe) |
| |
| HDFS-5230. Introduce RpcInfo to decouple XDR classes from the RPC API. |
| (Haohui Mai via brandonli) |
| |
| IMPROVEMENTS |
| |
| HDFS-5246. Make Hadoop nfs server port and mount daemon port |
| configurable. (Jinghui Wang via brandonli) |
| |
| HDFS-5256. Use guava LoadingCache to implement DFSClientCache. (Haohui Mai |
| via brandonli) |
| |
| HDFS-5308. Replace HttpConfig#getSchemePrefix with implicit schemes in HDFS |
| JSP. (Haohui Mai via jing9) |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| HDFS-5139. Remove redundant -R option from setrep. |
| |
| HDFS-5251. Race between the initialization of NameNode and the http |
| server. (Haohui Mai via suresh) |
| |
| HDFS-5258. Skip tests in TestHDFSCLI that are not applicable on Windows. |
| (Chuan Liu via cnauroth) |
| |
| HDFS-5186. TestFileJournalManager fails on Windows due to file handle leaks. |
| (Chuan Liu via cnauroth) |
| |
| HDFS-5031. BlockScanner scans the block multiple times. (Vinay via Arpit |
| Agarwal) |
| |
| HDFS-5268. NFS write commit verifier is not set in a few places (brandonli) |
| |
| HDFS-5265. Namenode fails to start when dfs.https.port is unspecified. |
| (Haohui Mai via jing9) |
| |
| HDFS-5255. Distcp job fails with hsftp when https is enabled in insecure |
| cluster. (Arpit Agarwal) |
| |
| HDFS-5279. Guard against NullPointerException in NameNode JSP pages before |
| initialization of FSNamesystem. (cnauroth) |
| |
| HDFS-5289. Race condition in TestRetryCacheWithHA#testCreateSymlink causes |
| spurious test failure. (atm) |
| |
| HDFS-5300. FSNameSystem#deleteSnapshot() should not check owner in case of |
| permissions disabled. (Vinay via jing9) |
| |
| HDFS-5306. Datanode https port is not available at the namenode. (Suresh |
| Srinivas via brandonli) |
| |
| HDFS-5299. DFS client hangs in updatePipeline RPC when failover happened. |
| (Vinay via jing9) |
| |
| HDFS-5259. Support client which combines appended data with old data |
| before sends it to NFS server. (brandonli) |
| |
| Release 2.1.1-beta - 2013-09-23 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| HDFS-4962 Use enum for nfs constants (Nicholas SZE via jing9) |
| |
| HDFS-5071 Change hdfs-nfs parent project to hadoop-project (brandonli) |
| |
| HDFS-4763 Add script changes/utility for starting NFS gateway (brandonli) |
| |
| HDFS-5076 Add MXBean methods to query NN's transaction information and |
| JournalNode's journal status. (jing9) |
| |
| HDFS-5104 Support dotdot name in NFS LOOKUP operation (brandonli) |
| |
| HDFS-5107 Fix array copy error in Readdir and Readdirplus responses |
| (brandonli) |
| |
| HDFS-5110 Change FSDataOutputStream to HdfsDataOutputStream for opened |
| streams to fix type cast error. (brandonli) |
| |
| HDFS-5069 Include hadoop-nfs and hadoop-hdfs-nfs into hadoop dist for |
| NFS deployment (brandonli) |
| |
| HDFS-4947 Add NFS server export table to control export by hostname or |
| IP range (Jing Zhao via brandonli) |
| |
| HDFS-5078 Support file append in NFSv3 gateway to enable data streaming |
| to HDFS (brandonli) |
| |
| HDFS-5136 MNT EXPORT should give the full group list which can mount the |
| exports (brandonli) |
| |
| HDFS-5118. Provide testing support for DFSClient to drop RPC responses. |
| (jing9) |
| |
| HDFS-5085. Refactor o.a.h.nfs to support different types of |
| authentications. (jing9) |
| |
| HDFS-5067 Support symlink operations in NFS gateway. (brandonli) |
| |
| HDFS-5199 Add more debug trace for NFS READ and WRITE. (brandonli) |
| |
| HDFS-5234 Move RpcFrameDecoder out of the public API. |
| (Haohui Mai via brandonli) |
| |
| IMPROVEMENTS |
| |
| HDFS-4513. Clarify in the WebHDFS REST API that all JSON respsonses may |
| contain additional properties. (szetszwo) |
| |
| HDFS-5061. Make FSNameSystem#auditLoggers an unmodifiable list. |
| (Arpit Agarwal via suresh) |
| |
| HDFS-4905. Add appendToFile command to "hdfs dfs". (Arpit Agarwal via |
| cnauroth) |
| |
| HDFS-4926. Namenode webserver's page has a tooltip that is inconsistent |
| with the datanode HTML link. (Vivek Ganesan via jing9) |
| |
| HDFS-5047. Supress logging of full stack trace of quota and lease |
| exceptions. (Robert Parker via kihwal) |
| |
| HDFS-2933. Improve DataNode Web UI Index Page. (Vivek Ganesan via |
| Arpit Agarwal) |
| |
| HDFS-5111. Remove duplicated error message for snapshot commands when |
| processing invalid arguments. (jing9) |
| |
| HDFS-5045. Add more unit tests for retry cache to cover all AtMostOnce |
| methods. (jing9) |
| |
| HDFS-3245. Add metrics and web UI for cluster version summary. (Ravi |
| Prakash via kihwal) |
| |
| HDFS-5128. Allow multiple net interfaces to be used with HA namenode RPC |
| server. (kihwal) |
| |
| HDFS-5150. Allow per NN SPN for internal SPNEGO. (kihwal) |
| |
| HDFS-4680. Audit logging of delegation tokens for MR tracing. (Andrew Wang) |
| |
| HDFS-5212. Refactor RpcMessage and NFS3Response to support different |
| types of authentication information. (jing9) |
| |
| HDFS-4971. Move IO operations out of locking in OpenFileCtx. (brandonli and |
| jing9) |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| HDFS-5028. LeaseRenewer throws ConcurrentModificationException when timeout. |
| (zhaoyunjiong via szetszwo) |
| |
| HDFS-5043. For HdfsFileStatus, set default value of childrenNum to -1 |
| instead of 0 to avoid confusing applications. (brandonli) |
| |
| HDFS-4993. Fsck can fail if a file is renamed or deleted. (Robert Parker |
| via kihwal) |
| |
| HDFS-5091. Support for spnego keytab separate from the JournalNode keytab |
| for secure HA. (jing9) |
| |
| HDFS-5055. nn fails to download checkpointed image from snn in some |
| setups. (Vinay and suresh via suresh) |
| |
| HDFS-4898. BlockPlacementPolicyWithNodeGroup.chooseRemoteRack() fails to |
| properly fallback to local rack. (szetszwo) |
| |
| HDFS-4632. globStatus using backslash for escaping does not work on Windows. |
| (Chuan Liu via cnauroth) |
| |
| HDFS-5093. TestGlobPaths should re-use the MiniDFSCluster to avoid failure |
| on Windows. (Chuan Liu via cnauroth) |
| |
| HDFS-5080. BootstrapStandby not working with QJM when the existing NN is |
| active. (jing9) |
| |
| HDFS-5099. Namenode#copyEditLogSegmentsToSharedDir should close |
| EditLogInputStreams upon finishing. (Chuan Liu via cnauroth) |
| |
| HDFS-2994. If lease soft limit is recovered successfully |
| the append can fail. (Tao Luo via shv) |
| |
| HDFS-5100. TestNamenodeRetryCache fails on Windows due to incorrect cleanup. |
| (Chuan Liu via cnauroth) |
| |
| HDFS-5103. TestDirectoryScanner fails on Windows. (Chuan Liu via cnauroth) |
| |
| HDFS-5102. Snapshot names should not be allowed to contain slash characters. |
| (jing9) |
| |
| HDFS-5105. TestFsck fails on Windows. (Chuan Liu via arp) |
| |
| HDFS-5106. TestDatanodeBlockScanner fails on Windows due to incorrect path |
| format. (Chuan Liu via cnauroth) |
| |
| HDFS-4594. WebHDFS open sets Content-Length header to what is specified by |
| length parameter rather than how much data is actually returned. (cnauroth) |
| |
| HDFS-5124. DelegationTokenSecretManager#retrievePassword can cause deadlock |
| in NameNode. (Daryn Sharp via jing9) |
| |
| HDFS-5132. Deadlock in NameNode between SafeModeMonitor#run and |
| DatanodeManager#handleHeartbeat. (kihwal) |
| |
| HDFS-5077. NPE in FSNamesystem.commitBlockSynchronization(). |
| (Plamen Jeliazkov via shv) |
| |
| HDFS-5140. Too many safemode monitor threads being created in the standby |
| namenode causing it to fail with out of memory error. (jing9) |
| |
| HDFS-5159. Secondary NameNode fails to checkpoint if error occurs |
| downloading edits on first checkpoint. (atm) |
| |
| HDFS-5192. NameNode may fail to start when |
| dfs.client.test.drop.namenode.response.number is set. (jing9) |
| |
| HDFS-5219. Add configuration keys for retry policy in WebHDFSFileSystem. |
| (Haohui Mai via jing9) |
| |
| HDFS-5231. Fix broken links in the document of HDFS Federation. (Haohui Mai |
| via jing9) |
| |
| HDFS-5249. Fix dumper thread which may die silently. (brandonli) |
| |
| Release 2.1.0-beta - 2013-08-22 |
| |
| INCOMPATIBLE CHANGES |
| |
| HDFS-4053. Increase the default block size. (eli) |
| |
| HDFS-4305. Add a configurable limit on number of blocks per file, and min |
| block size. (Andrew Wang via atm) |
| |
| HDFS-4434. Provide a mapping from INodeId to INode. (suresh) |
| |
| HDFS-2802. Add HDFS Snapshot feature. (See breakdown of tasks below for |
| subtasks and contributors) |
| |
| HDFS-4866. Protocol buffer support cannot compile under C. (Arpit Agarwal via |
| cnauroth) |
| |
| HDFS-5083. Update the HDFS compatibility version range. (kihwal) |
| |
| NEW FEATURES |
| |
| HDFS-1804. Add a new block-volume device choosing policy that looks at |
| free space. (atm) |
| |
| HDFS-4296. Reserve layout version for release 1.2.0. (suresh) |
| |
| HDFS-4334. Add a unique id to INode. (Brandon Li via szetszwo) |
| |
| HDFS-4339. Persist inode id in fsimage and editlog. (Brandon Li via |
| suresh) |
| |
| HDFS-4340. Update addBlock() to inculde inode id as additional argument. |
| (Brandon Li via suresh) |
| |
| HDFS-4502. JsonUtil.toFileStatus(..) should check if the fileId property |
| exists. (Brandon Li via suresh) |
| |
| HDFS-2576. Enhances the DistributedFileSystem's create API so that clients |
| can specify favored datanodes for a file's blocks. (ddas) |
| |
| HDFS-347. DFS read performance suboptimal when client co-located on nodes |
| with data. (Colin Patrick McCabe via todd and atm) |
| |
| HADOOP-8562. Enhancements to support Hadoop on Windows Server and Windows |
| Azure environments. (See breakdown of tasks below for subtasks and |
| contributors) |
| |
| HDFS-3601. Add BlockPlacementPolicyWithNodeGroup to support block placement |
| with 4-layer network topology. (Junping Du via szetszwo) |
| |
| HDFS-3495. Update Balancer to support new NetworkTopology with NodeGroup. |
| (Junping Du via szetszwo) |
| |
| HDFS-4659 Support setting execution bit for regular files (Brandon Li via sanjay) |
| |
| HDFS-4762 Provide HDFS based NFSv3 and Mountd implementation (brandonli) |
| |
| HDFS-4372. Track NameNode startup progress. (cnauroth) |
| |
| HDFS-4373. Add HTTP API for querying NameNode startup progress. (cnauroth) |
| |
| HDFS-4374. Display NameNode startup progress in UI. (cnauroth) |
| |
| HDFS-4974. Add Idempotent and AtMostOnce annotations to namenode |
| protocol methods. (suresh) |
| |
| HDFS-4979. Implement retry cache on Namenode. (suresh) |
| |
| HDFS-5025. Record ClientId and CallId in EditLog to enable rebuilding |
| retry cache in case of HA failover. (Jing Zhao via suresh) |
| |
| IMPROVEMENTS |
| |
| HDFS-4461. DirectoryScanner: volume path prefix takes up memory for every |
| block that is scanned (Colin Patrick McCabe) |
| |
| HDFS-4222. NN is unresponsive and loses heartbeats from DNs when |
| configured to use LDAP and LDAP has issues. (Xiaobo Peng, suresh) |
| |
| HDFS-4304. Make FSEditLogOp.MAX_OP_SIZE configurable. (Colin Patrick |
| McCabe via atm) |
| |
| HDFS-4518. Finer grained metrics for HDFS capacity. |
| (Arpit Agarwal via suresh) |
| |
| HDFS-4519. Support overriding jsvc binary and log file locations |
| when launching secure datanode. (Chris Nauroth via suresh) |
| |
| HDFS-4569. Small image transfer related cleanups. |
| (Andrew Wang via suresh) |
| |
| HDFS-4521. Invalid network toploogies should not be cached. (Colin Patrick |
| McCabe via atm) |
| |
| HDFS-4246. The exclude node list should be more forgiving, for each output |
| stream. (harsh via atm) |
| |
| HDFS-4635. Move BlockManager#computeCapacity to LightWeightGSet. (suresh) |
| |
| HDFS-4621. Additional logging to help diagnose slow QJM syncs. (todd) |
| |
| HDFS-4618. Default transaction interval for checkpoints is too low. (todd) |
| |
| HDFS-4525. Provide an API for knowing that whether file is closed or not. |
| (SreeHari via umamahesh) |
| |
| HDFS-3940. Add Gset#clear method and clear the block map when namenode is |
| shutdown. (suresh) |
| |
| HDFS-4679. Namenode operation checks should be done in a consistent |
| manner. (suresh) |
| |
| HDFS-4693. Some test cases in TestCheckpoint do not clean up after |
| themselves. (Arpit Agarwal, suresh via suresh) |
| |
| HDFS-3817. Avoid printing SafeModeException stack trace. |
| (Brandon Li via suresh) |
| |
| HDFS-4124. Refactor INodeDirectory#getExistingPathINodes() to enable |
| returning more than INode array. (Jing Zhao via suresh) |
| |
| HDFS-4151. Change the methods in FSDirectory to pass INodesInPath instead |
| of INode[] as a parameter. (szetszwo) |
| |
| HDFS-4129. Add utility methods to dump NameNode in memory tree for |
| testing. (szetszwo via suresh) |
| |
| HDFS-4152. Add a new class BlocksMapUpdateInfo for the parameter in |
| INode.collectSubtreeBlocksAndClear(..). (Jing Zhao via szetszwo) |
| |
| HDFS-4206. Change the fields in INode and its subclasses to private. |
| (szetszwo) |
| |
| HDFS-4215. Remove locking from addToParent(..) since it is used in image |
| loading, and add INode.isFile(). (szetszwo) |
| |
| HDFS-4243. When replacing an INodeDirectory, the parent pointers of the |
| children of the child have to be updated to the new child. (Jing Zhao |
| via szetszwo) |
| |
| HDFS-4209. Clean up the addNode/addChild/addChildNoQuotaCheck methods in |
| FSDirectory and INodeDirectory. (szetszwo) |
| |
| HDFS-4346. Add SequentialNumber as a base class for INodeId and |
| GenerationStamp. (szetszwo) |
| |
| HDFS-4721. Speed up lease recovery by avoiding stale datanodes and choosing |
| the datanode with the most recent heartbeat as the primary. (Varun Sharma |
| via szetszwo) |
| |
| HDFS-4804. WARN when users set the block balanced preference percent below |
| 0.5 or above 1.0. (Stephen Chu via atm) |
| |
| HDFS-4698. Provide client-side metrics for remote reads, local reads, and |
| short-circuit reads. (Colin Patrick McCabe via atm) |
| |
| HDFS-3498. Support replica removal in BlockPlacementPolicy and make |
| BlockPlacementPolicyDefault extensible for reusing code in subclasses. |
| (Junping Du via szetszwo) |
| |
| HDFS-4234. Use generic code for choosing datanode in Balancer. (szetszwo) |
| |
| HDFS-4880. Print the image and edits file loaded by the namenode in the |
| logs. (Arpit Agarwal via suresh) |
| |
| HDFS-2572. Remove unnecessary double-check in DN#getHostName. (harsh) |
| |
| HDFS-2857. Cleanup BlockInfo class. (suresh) |
| |
| HDFS-3009. Remove duplicate code in DFSClient#isLocalAddress by using |
| NetUtils. (Hari Mankude via suresh) |
| |
| HDFS-4914. Use DFSClient.Conf instead of Configuration. (szetszwo) |
| |
| HDFS-4883. complete() should verify fileId. (Tao Luo via shv) |
| |
| HDFS-4772. Add number of children in HdfsFileStatus. (brandonli) |
| |
| HDFS-4932. Avoid a wide line on the name node webUI if we have more Journal |
| nodes. (Fengdong Yu via cnauroth) |
| |
| HDFS-4908. Reduce snapshot inode memory usage. (szetszwo) |
| |
| HDFS-4645. Move from randomly generated block ID to sequentially generated |
| block ID. (Arpit Agarwal via szetszwo) |
| |
| HDFS-4912. Cleanup FSNamesystem#startFileInternal. (suresh) |
| |
| HDFS-4903. Print trash configuration and trash emptier state in |
| namenode log. (Arpit Agarwal via suresh) |
| |
| HDFS-4992. Make balancer's mover thread count and dispatcher thread count |
| configurable. (Max Lapan via szetszwo) |
| |
| HDFS-4996. ClientProtocol#metaSave can be made idempotent by overwriting the |
| output file instead of appending to it. (cnauroth) |
| |
| HADOOP-9418. Add symlink support to DistributedFileSystem (Andrew Wang via |
| Colin Patrick McCabe) |
| |
| HDFS-5007. Replace hard-coded property keys with DFSConfigKeys fields. |
| (Kousuke Saruta via jing9) |
| |
| HDFS-5008. Make ClientProtocol#abandonBlock() idempotent. (jing9) |
| |
| HADOOP-9760. Move GSet and related classes to common from HDFS. |
| (suresh) |
| |
| HDFS-5020. Make DatanodeProtocol#blockReceivedAndDeleted idempotent. |
| (jing9) |
| |
| HDFS-5024. Make DatanodeProtocol#commitBlockSynchronization idempotent. |
| (Arpit Agarwal via jing9) |
| |
| HDFS-3880. Use Builder to build RPC server in HDFS. |
| (Brandon Li and Junping Du via szetszwo) |
| |
| OPTIMIZATIONS |
| |
| HDFS-4465. Optimize datanode ReplicasMap and ReplicaInfo. (atm) |
| |
| HDFS-5027. On startup, DN should scan volumes in parallel. (atm) |
| |
| BUG FIXES |
| |
| HDFS-4626. ClientProtocol#getLinkTarget should throw an exception for |
| non-symlink and non-existent paths. (Andrew Wang via cmccabe) |
| |
| HDFS-3934. duplicative dfs_hosts entries handled wrong. (Colin Patrick |
| McCabe) |
| |
| HDFS-4470. Several HDFS tests attempt file operations on invalid HDFS |
| paths when running on Windows. (Chris Nauroth via suresh) |
| |
| HDFS-4471. Namenode WebUI file browsing does not work with wildcard |
| addresses configured. (Andrew Wang via atm) |
| |
| HDFS-4342. Directories configured in dfs.namenode.edits.dir.required |
| but not in dfs.namenode.edits.dir are silently ignored. (Arpit Agarwal |
| via szetszwo) |
| |
| HDFS-4482. ReplicationMonitor thread can exit with NPE due to the race |
| between delete and replication of same file. (umamahesh) |
| |
| HDFS-4269. Datanode rejects all datanode registrations from localhost |
| in single-node developer setup on Windows. (Chris Nauroth via suresh) |
| |
| HDFS-4235. When outputting XML, OfflineEditsViewer can't handle some edits |
| containing non-ASCII strings. (Colin Patrick McCabe via atm) |
| |
| HDFS-4541. Set hadoop.log.dir and hadoop.id.str when starting secure |
| datanode to write the logs to right dir by default. (Arpit Gupta via |
| suresh) |
| |
| HDFS-4540. Namenode http server should use the web authentication |
| keytab for spnego principal. (Arpit Gupta via suresh) |
| |
| HDFS-4544. Error in deleting blocks should not do check disk, for |
| all types of errors. (Arpit Agarwal via suresh) |
| |
| HDFS-4565. Use DFSUtil.getSpnegoKeytabKey() to get the spnego keytab key |
| in secondary namenode and namenode http server. (Arpit Gupta via suresh) |
| |
| HDFS-4571. WebHDFS should not set the service hostname on the server side. |
| (tucu) |
| |
| HDFS-4013. TestHftpURLTimeouts throws NPE. (Chao Shi via suresh) |
| |
| HDFS-4592. Default values for access time precision are out of sync between |
| hdfs-default.xml and the code. (atm) |
| |
| HDFS-4522. LightWeightGSet expects incrementing a volatile to be atomic. |
| (Colin Patrick McCabe via atm) |
| |
| HDFS-4484. libwebhdfs compilation broken with gcc 4.6.2. (Colin Patrick |
| McCabe via atm) |
| |
| HDFS-4595. When short circuit read is fails, DFSClient does not fallback |
| to regular reads. (suresh) |
| |
| HDFS-4583. TestNodeCount fails. (Ivan Mitic via suresh) |
| |
| HDFS-4591. HA clients can fail to fail over while Standby NN is performing |
| long checkpoint. (atm) |
| |
| HDFS-3277. fail over to loading a different FSImage if the first one we |
| try to load is corrupt. (Colin Patrick McCabe and Andrew Wang via atm) |
| |
| HDFS-4596. Shutting down namenode during checkpointing can lead to md5sum |
| error. (Andrew Wang via atm) |
| |
| HDFS-4614. FSNamesystem#getContentSummary should use getPermissionChecker |
| helper method. (atm) |
| |
| HDFS-4620. Documentation for dfs.namenode.rpc-address specifies wrong |
| format. (Sandy Ryza via atm) |
| |
| HDFS-4609. TestAuditLogs should release log handles between tests. |
| (Ivan Mitic via szetszwo) |
| |
| HDFS-4598. Fix the default value of ConcatSourcesParam and the WebHDFS doc. |
| (szetszwo) |
| |
| HDFS-4655. DNA_FINALIZE is logged as being an unknown command by the DN |
| when received from the standby NN. (atm) |
| |
| HDFS-4656. DN heartbeat loop can be briefly tight. (atm) |
| |
| HDFS-4658. Standby NN will log that it has received a block report "after |
| becoming active" (atm) |
| |
| HDFS-4646. createNNProxyWithClientProtocol ignores configured timeout |
| value (Jagane Sundar via cos) |
| |
| HDFS-3981. Fix handling of FSN lock in getBlockLocations. (Xiaobo Peng |
| and todd via todd) |
| |
| HDFS-4676. TestHDFSFileSystemContract should set MiniDFSCluster variable |
| to null to free up memory. (suresh) |
| |
| HDFS-4669. TestBlockPoolManager fails using IBM java. (Tian Hong Wang via |
| suresh) |
| |
| HDFS-4643. Fix flakiness in TestQuorumJournalManager. (todd) |
| |
| HDFS-4639. startFileInternal() should not increment generation stamp. |
| (Plamen Jeliazkov via shv) |
| |
| HDFS-4695. TestEditLog leaks open file handles between tests. |
| (Ivan Mitic via suresh) |
| |
| HDFS-4737. JVM path embedded in fuse binaries. (Sean Mackrory via atm) |
| |
| HDFS-4739. NN can miscalculate the number of extra edit log segments to |
| retain. (atm) |
| |
| HDFS-4745. TestDataTransferKeepalive#testSlowReader has race condition that |
| causes sporadic failure. (Chris Nauroth via suresh) |
| |
| HDFS-4768. File handle leak in datanode when a block pool is removed. |
| (Chris Nauroth via suresh) |
| |
| HDFS-4748. MiniJournalCluster#restartJournalNode leaks resources, which |
| causes sporadic test failures. (Chris Nauroth via suresh) |
| |
| HDFS-4733. Make HttpFS username pattern configurable. (tucu via atm) |
| |
| HDFS-4778. Fixes some issues that the first patch on HDFS-2576 missed. |
| (ddas) |
| |
| HDFS-4785. Concat operation does not remove concatenated files from |
| InodeMap. (suresh) |
| |
| HDFS-4784. NPE in FSDirectory.resolvePath(). (Brandon Li via suresh) |
| |
| HDFS-4810. several HDFS HA tests have timeouts that are too short. (Chris |
| Nauroth via atm) |
| |
| HDFS-4799. Corrupt replica can be prematurely removed from |
| corruptReplicas map. (todd via kihwal) |
| |
| HDFS-4751. TestLeaseRenewer#testThreadName flakes. (Andrew Wang via atm) |
| |
| HDFS-4533. start-dfs.sh ignores additional parameters besides -upgrade. |
| (Fengdong Yu via suresh) |
| |
| HDFS-4765. Permission check of symlink deletion incorrectly throws |
| UnresolvedLinkException. (Andrew Wang via atm) |
| |
| HDFS-4300. TransferFsImage.downloadEditsToStorage should use a tmp file for |
| destination. (Andrew Wang via atm) |
| |
| HDFS-4813. Add volatile to BlocksMap.blocks so that the replication thread |
| can see the updated value. (Jing Zhao via szetszwo) |
| |
| HDFS-3180. Add socket timeouts to WebHdfsFileSystem. (Chris Nauroth via |
| szetszwo) |
| |
| HDFS-4787. Create a new HdfsConfiguration before each TestDFSClientRetries |
| testcases. (Tian Hong Wang via atm) |
| |
| HDFS-4830. Typo in config settings for AvailableSpaceVolumeChoosingPolicy |
| in hdfs-default.xml. (atm) |
| |
| HDFS-4824. FileInputStreamCache.close leaves dangling reference to |
| FileInputStreamCache.cacheCleaner. (Colin Patrick McCabe via todd) |
| |
| HDFS-4298. StorageRetentionManager spews warnings when used with QJM. (atm) |
| |
| HDFS-4725. Fix HDFS file handle leaks in FSEditLog, NameNode, |
| OfflineEditsBinaryLoader and some tests. (Chris Nauroth via szetszwo) |
| |
| HDFS-4825. webhdfs / httpfs tests broken because of min block size change. |
| (Andrew Wang via suresh) |
| |
| HDFS-4780. Use the correct relogin method for services. (Robert Parker via |
| kihwal) |
| |
| HDFS-4827. Slight update to the implementation of API for handling favored |
| nodes in DFSClient (ddas) |
| |
| HDFS-4865. Remove sub resource warning from httpfs log at startup time. |
| (ywskycn via tucu) |
| |
| HDFS-4240. For nodegroup-aware block placement, when a node is excluded, |
| the nodes in the same nodegroup should also be excluded. (Junping Du |
| via szetszwo) |
| |
| HDFS-4261. Fix bugs in Balaner causing infinite loop and |
| TestBalancerWithNodeGroup timeing out. (Junping Du via szetszwo) |
| |
| HDFS-4382. Fix typo MAX_NOT_CHANGED_INTERATIONS. (Ted Yu via suresh) |
| |
| HDFS-4840. ReplicationMonitor gets NPE during shutdown. (kihwal) |
| |
| HDFS-4815. TestRBWBlockInvalidation: Double call countReplicas() to fetch |
| corruptReplicas and liveReplicas is not needed. (Tian Hong Wang via atm) |
| |
| HADOOP-8957 HDFS tests for AbstractFileSystem#IsValidName should be overridden for |
| embedded file systems like ViewFs (Chris Nauroth via Sanjay Radia) |
| |
| HDFS-4586. TestDataDirs.testGetDataDirsFromURIs fails with all directories |
| in dfs.datanode.data.dir are invalid. (Ivan Mitic via atm) |
| |
| HDFS-3792. Fix two findbugs introduced by HDFS-3695 (todd) |
| |
| HADOOP-9635 Fix potential Stack Overflow in DomainSocket.c (V. Karthik Kumar |
| via cmccabe) |
| |
| HDFS-3163. TestHDFSCLI.testAll fails if the user name is not all lowercase. |
| (Brandon Li via atm) |
| |
| HDFS-4845. FSNamesystem.deleteInternal should acquire write-lock before |
| changing the inode map. (Arpit Agarwal via szetszwo) |
| |
| HDFS-4910. TestPermission failed in branch-2. (Chuan Liu via cnauroth) |
| |
| HDFS-4906. HDFS Output streams should not accept writes after being |
| closed. (atm) |
| |
| HDFS-4917. Start-dfs.sh cannot pass the parameters correctly. |
| (Fengdong Yu via suresh) |
| |
| HDFS-4205. fsck fails with symlinks. (jlowe) |
| |
| HDFS-4927. CreateEditsLog creates inodes with an invalid inode ID, which then |
| cannot be loaded by a namenode. (cnauroth) |
| |
| HDFS-4944. WebHDFS cannot create a file path containing characters that must |
| be URI-encoded, such as space. (cnauroth) |
| |
| HDFS-4888. Refactor and fix FSNamesystem.getTurnOffTip. (Ravi Prakash via |
| kihwal) |
| |
| HDFS-4943. WebHdfsFileSystem does not work when original file path has |
| encoded chars. (Jerry He via szetszwo) |
| |
| HDFS-4954. In nfs, OpenFileCtx.getFlushedOffset() should handle IOException. |
| (Brandon Li via szetszwo) |
| |
| HDFS-4948. mvn site for hadoop-hdfs-nfs fails. (brandonli) |
| |
| HDFS-4887. TestNNThroughputBenchmark exits abruptly. (kihwal) |
| |
| HDFS-4980. Incorrect logging.properties file for hadoop-httpfs. |
| (Mark Grover via suresh) |
| |
| HDFS-4999. Fix TestShortCircuitLocalRead on branch-2. (cmccabe via kihwal) |
| |
| HDFS-4687. TestDelegationTokenForProxyUser#testWebHdfsDoAs is flaky with |
| JDK7. (Andrew Wang via atm) |
| |
| HDFS-5003. TestNNThroughputBenchmark failed caused by existing directories. |
| (Xi Fang via cnauroth) |
| |
| HDFS-5018. Misspelled DFSConfigKeys#DFS_NAMENODE_STALE_DATANODE_INTERVAL_DEFAULT |
| in javadoc of DatanodeInfo#isStale(). (Ted Yu via jing9) |
| |
| HDFS-4602. TestBookKeeperHACheckpoints fails. (umamahesh) |
| |
| HDFS-5016. Deadlock in pipeline recovery causes Datanode to be marked dead. |
| (suresh) |
| |
| HDFS-5228. The RemoteIterator returned by DistributedFileSystem.listFiles |
| may throw NullPointerException. (szetszwo and cnauroth via szetszwo) |
| |
| BREAKDOWN OF HDFS-347 SUBTASKS AND RELATED JIRAS |
| |
| HDFS-4353. Encapsulate connections to peers in Peer and PeerServer classes. |
| (Colin Patrick McCabe via todd) |
| |
| HDFS-4354. Create DomainSocket and DomainPeer and associated unit tests. |
| (Colin Patrick McCabe via todd) |
| |
| HDFS-4356. BlockReaderLocal should use passed file descriptors rather than paths. |
| (Colin Patrick McCabe via todd) |
| |
| HDFS-4388. DomainSocket should throw AsynchronousCloseException when appropriate. |
| (Colin Patrick McCabe via todd) |
| |
| HDFS-4390. Bypass UNIX domain socket unit tests when they cannot be run. |
| (Colin Patrick McCabe via todd) |
| |
| HDFS-4400. DFSInputStream#getBlockReader: last retries should ignore the cache |
| (Colin Patrick McCabe via todd) |
| |
| HDFS-4401. Fix bug in DomainSocket path validation |
| (Colin Patrick McCabe via todd) |
| |
| HDFS-4402. Some small DomainSocket fixes: avoid findbugs warning, change |
| log level, etc. (Colin Patrick McCabe via todd) |
| |
| HDFS-4418. increase default FileInputStreamCache size (todd) |
| |
| HDFS-4416. Rename dfs.datanode.domain.socket.path to dfs.domain.socket.path |
| (Colin Patrick McCabe via todd) |
| |
| HDFS-4417. Fix case where local reads get disabled incorrectly |
| (Colin Patrick McCabe and todd via todd) |
| |
| HDFS-4433. Make TestPeerCache not flaky (Colin Patrick McCabe via todd) |
| |
| HDFS-4438. TestDomainSocket fails when system umask is set to 0002. (Colin |
| Patrick McCabe via atm) |
| |
| HDFS-4440. Avoid annoying log message when dfs.domain.socket.path is not |
| set. (Colin Patrick McCabe via atm) |
| |
| HDFS-4473. Don't create domain socket unless we need it. (Colin Patrick McCabe via atm) |
| |
| HDFS-4485. DN should chmod socket path a+w. (Colin Patrick McCabe via atm) |
| |
| HDFS-4453. Make a simple doc to describe the usage and design of the |
| shortcircuit read feature. (Colin Patrick McCabe via atm) |
| |
| HDFS-4496. DFSClient: don't create a domain socket unless we need it (Colin |
| Patrick McCabe via todd) |
| |
| HDFS-347: style cleanups (Colin Patrick McCabe via atm) |
| |
| HDFS-4538. Allow use of legacy blockreader (Colin Patrick McCabe via todd) |
| |
| HDFS-4661. A few little code cleanups of some HDFS-347-related code. (Colin |
| Patrick McCabe via atm) |
| |
| BREAKDOWN OF HADOOP-8562 and HDFS-3602 SUBTASKS AND RELATED JIRAS |
| |
| HDFS-4145. Merge hdfs cmd line scripts from branch-1-win. (David Lao, |
| Bikas Saha, Lauren Yang, Chuan Liu, Thejas M Nair and Ivan Mitic via suresh) |
| |
| HDFS-4163. HDFS distribution build fails on Windows. (Chris Nauroth via |
| suresh) |
| |
| HDFS-4316. branch-trunk-win contains test code accidentally added during |
| work on fixing tests on Windows. (Chris Nauroth via suresh) |
| |
| HDFS-4297. Fix issues related to datanode concurrent reading and writing on |
| Windows. (Arpit Agarwal, Chuan Liu via suresh) |
| |
| HDFS-4573. Fix TestINodeFile on Windows. (Arpit Agarwal via suresh) |
| |
| HDFS-4572. Fix TestJournal failures on Windows. (Arpit Agarwal via suresh) |
| |
| HDFS-4287. HTTPFS tests fail on Windows. (Chris Nauroth via suresh) |
| |
| HDFS-4593. TestSaveNamespace fails on Windows. (Arpit Agarwal via suresh) |
| |
| HDFS-4582. TestHostsFiles fails on Windows. (Ivan Mitic via suresh) |
| |
| HDFS-4603. TestMiniDFSCluster fails on Windows. (Ivan Mitic via suresh) |
| |
| HDFS-4604. TestJournalNode fails on Windows. (Ivan Mitic via suresh) |
| |
| HDFS-4607. In TestGetConf.testGetSpecificKey(), use a platform-specific |
| line separator; otherwise, it fails on Windows. (Ivan Mitic via szetszwo) |
| |
| HDFS-4625. Make TestNNWithQJM#testNewNamenodeTakesOverWriter work on |
| Windows. (Ivan Mitic via suresh) |
| |
| HDFS-4674. TestBPOfferService fails on Windows due to failure parsing |
| datanode data directory as URI. (Chris Nauroth via suresh) |
| |
| HDFS-4615. Fix TestDFSShell failures on Windows. (Arpit Agarwal |
| via szetszwo) |
| |
| HDFS-4584. Skip TestNNWithQJM.testNewNamenodeTakesOverWriter() on Windows. |
| (Arpit Agarwal via szetszwo) |
| |
| HDFS-4732. Fix TestDFSUpgradeFromImage which fails on Windows due to |
| failure to unpack old image tarball that contains hard links. |
| (Chris Nauroth via szetszwo) |
| |
| HDFS-4741. TestStorageRestore#testStorageRestoreFailure fails on Windows. |
| (Arpit Agarwal via suresh) |
| |
| HDFS-4743. TestNNStorageRetentionManager fails on Windows. |
| (Chris Nauroth via suresh) |
| |
| HDFS-4740. Fixes for a few test failures on Windows. |
| (Arpit Agarwal via suresh) |
| |
| HDFS-4722. TestGetConf#testFederation times out on Windows. |
| (Ivan Mitic via suresh) |
| |
| HDFS-4705. Address HDFS test failures on Windows because of invalid |
| dfs.namenode.name.dir. (Ivan Mitic via suresh) |
| |
| HDFS-4734. HDFS Tests that use ShellCommandFencer are broken on Windows. |
| (Arpit Agarwal via suresh) |
| |
| HDFS-4610. Use common utils FileUtil#setReadable/Writable/Executable and |
| FileUtil#canRead/Write/Execute. (Ivan Mitic via suresh) |
| |
| HDFS-4677. Editlog should support synchronous writes. (ivanmi) |
| |
| HDFS-4752. TestRBWBlockInvalidation fails on Windows due to file locking. |
| (Chris Nauroth via suresh) |
| |
| HDFS-4783. TestDelegationTokensWithHA#testHAUtilClonesDelegationTokens fails |
| on Windows. (cnauroth) |
| |
| HDFS-4818. Several HDFS tests that attempt to make directories unusable do |
| not work correctly on Windows. (cnauroth) |
| |
| BREAKDOWN OF HDFS-2802 HDFS SNAPSHOT SUBTASKS AND RELATED JIRAS |
| |
| HDFS-4076. Support snapshot of single files. (szetszwo) |
| |
| HDFS-4082. Add editlog opcodes for snapshot create and delete operations. |
| (suresh via szetszwo) |
| |
| HDFS-4086. Add editlog opcodes to allow and disallow snapshots on a |
| directory. (Brandon Li via suresh) |
| |
| HDFS-4083. Protocol changes for snapshots. (suresh) |
| |
| HDFS-4077. Add support for Snapshottable Directory. (szetszwo via suresh) |
| |
| HDFS-4087. Protocol changes for listSnapshots functionality. |
| (Brandon Li via suresh) |
| |
| HDFS-4079. Add SnapshotManager which maintains a list for all the |
| snapshottable directories and supports snapshot methods such as setting a |
| directory to snapshottable and creating a snapshot. (szetszwo) |
| |
| HDFS-4078. Handle replication in snapshots. (szetszwo) |
| |
| HDFS-4084. Provide CLI support to allow and disallow snapshot |
| on a directory. (Brondon Li via suresh) |
| |
| HDFS-4091. Add snapshot quota to limit the number of snapshots allowed. |
| (szetszwo) |
| |
| HDFS-4097. Provide CLI support for createSnapshot. (Brandon Li via suresh) |
| |
| HDFS-4092. Update file deletion logic for snapshot so that the current inode |
| is removed from the circular linked list; and if some blocks at the end of |
| the block list no longer belong to any other inode, collect them and update |
| the block list. (szetszwo) |
| |
| HDFS-4111. Support snapshot of subtrees. (szetszwo via suresh) |
| |
| HDFS-4119. Complete the allowSnapshot code and add a test for it. (szetszwo) |
| |
| HDFS-4133. Add testcases for testing basic snapshot functionalities. |
| (Jing Zhao via suresh) |
| |
| HDFS-4116. Add auditlog for some snapshot operations. (Jing Zhao via suresh) |
| |
| HDFS-4095. Add some snapshot related metrics. (Jing Zhao via suresh) |
| |
| HDFS-4141. Support directory diff - the difference between the current state |
| and a previous snapshot of an INodeDirectory. (szetszwo) |
| |
| HDFS-4146. Use getter and setter in INodeFileWithLink to access blocks and |
| initialize root directory as snapshottable. (szetszwo) |
| |
| HDFS-4149. Implement the disallowSnapshot(..) in FSNamesystem and add |
| resetSnapshottable(..) to SnapshotManager. (szetszwo) |
| |
| HDFS-4147. When there is a snapshot in a subtree, deletion of the subtree |
| should fail. (Jing Zhao via szetszwo) |
| |
| HDFS-4150. Update the inode in the block map when a snapshotted file or a |
| snapshot file is deleted. (Jing Zhao via szetszwo) |
| |
| HDFS-4159. Rename should fail when the destination directory is |
| snapshottable and has snapshots. (Jing Zhao via szetszwo) |
| |
| HDFS-4170. Add snapshot information to INodesInPath. (szetszwo) |
| |
| HDFS-4177. Add a snapshot parameter to INodeDirectory.getChildrenList() for |
| selecting particular snapshot children list views. (szetszwo) |
| |
| HDFS-4148. Disallow write/modify operations on files and directories in a |
| snapshot. (Brandon Li via suresh) |
| |
| HDFS-4188. Add Snapshot.ID_COMPARATOR for comparing IDs and fix a bug in |
| ReadOnlyList.Util.binarySearch(..). (szetszwo) |
| |
| HDFS-4187. Add tests for replication handling in snapshots. (Jing Zhao via |
| szetszwo) |
| |
| HDFS-4196. Support renaming of snapshots. (Jing Zhao via szetszwo) |
| |
| HDFS-4175. Additional snapshot tests for more complicated directory |
| structure and modifications. (Jing Zhao via suresh) |
| |
| HDFS-4293. Fix TestSnapshot failure. (Jing Zhao via suresh) |
| |
| HDFS-4317. Change INode and its subclasses to support HDFS-4103. (szetszwo) |
| |
| HDFS-4103. Support O(1) snapshot creation. (szetszwo) |
| |
| HDFS-4330. Support snapshots up to the snapshot limit. (szetszwo) |
| |
| HDFS-4357. Fix a bug that if an inode is replaced, further INode operations |
| should apply to the new inode. (Jing Zhao via szetszwo) |
| |
| HDFS-4230. Support listing of all the snapshottable directories. (Jing Zhao |
| via szetszwo) |
| |
| HDFS-4244. Support snapshot deletion. (Jing Zhao via szetszwo) |
| |
| HDFS-4245. Include snapshot related operations in TestOfflineEditsViewer. |
| (Jing Zhao via szetszwo) |
| |
| HDFS-4395. In INodeDirectorySnapshottable's constructor, the passed-in dir |
| could be an INodeDirectoryWithSnapshot. (Jing Zhao via szetszwo) |
| |
| HDFS-4397. Fix a bug in INodeDirectoryWithSnapshot.Diff.combinePostDiff(..) |
| that it may put the wrong node into the deleted list. (szetszwo) |
| |
| HDFS-4407. Change INodeDirectoryWithSnapshot.Diff.combinePostDiff(..) to |
| merge-sort like and keep the postDiff parameter unmodified. (szetszwo) |
| |
| HDFS-4098. Add FileWithSnapshot, INodeFileUnderConstructionWithSnapshot and |
| INodeFileUnderConstructionSnapshot for supporting append to snapshotted |
| files. (szetszwo) |
| |
| HDFS-4126. Add reading/writing snapshot information to FSImage. |
| (Jing Zhao via suresh) |
| |
| HDFS-4436. Change INode.recordModification(..) to return only the current |
| inode and remove the updateCircularList parameter from some methods in |
| INodeDirectoryWithSnapshot.Diff. (szetszwo) |
| |
| HDFS-4429. When the latest snapshot exists, INodeFileUnderConstruction |
| should be replaced with INodeFileWithSnapshot but not INodeFile. |
| (Jing Zhao via szetszwo) |
| |
| HDFS-4441. Move INodeDirectoryWithSnapshot.Diff and the related classes to a |
| package. (szetszwo) |
| |
| HDFS-4432. Support INodeFileUnderConstructionWithSnapshot in FSImage |
| saving/loading. (Jing Zhao via suresh) |
| |
| HDFS-4131. Add capability to namenode to get snapshot diff. (Jing Zhao via |
| suresh) |
| |
| HDFS-4447. Refactor INodeDirectoryWithSnapshot for supporting general INode |
| diff lists. (szetszwo) |
| |
| HDFS-4189. Renames the getMutableXxx methods to getXxx4Write and fix a bug |
| that some getExistingPathINodes calls should be getINodesInPath4Write. |
| (szetszwo) |
| |
| HDFS-4361. When listing snapshottable directories, only return those |
| where the user has permission to take snapshots. (Jing Zhao via szetszwo) |
| |
| HDFS-4464. Combine collectSubtreeBlocksAndClear with deleteDiffsForSnapshot |
| and rename it to destroySubtreeAndCollectBlocks. (szetszwo) |
| |
| HDFS-4414. Add support for getting snapshot diff from DistributedFileSystem. |
| (Jing Zhao via suresh) |
| |
| HDFS-4446. Support file snapshots with diff lists. (szetszwo) |
| |
| HDFS-4480. Eliminate the file snapshot circular linked list. (szetszwo) |
| |
| HDFS-4481. Change fsimage to support snapshot file diffs. (szetszwo) |
| |
| HDFS-4500. Refactor snapshot INode methods. (szetszwo) |
| |
| HDFS-4487. Fix snapshot diff report for HDFS-4446. (Jing Zhao via szetszwo) |
| |
| HDFS-4431. Support snapshot in OfflineImageViewer. (Jing Zhao via szetszwo) |
| |
| HDFS-4503. Update computeContentSummary(..), spaceConsumedInTree(..) and |
| diskspaceConsumed(..) in INode for snapshot. (szetszwo) |
| |
| HDFS-4499. Fix file/directory/snapshot deletion for file diff. (Jing Zhao |
| via szetszwo) |
| |
| HDFS-4524. Update SnapshotManager#snapshottables when loading fsimage. |
| (Jing Zhao via szetszwo) |
| |
| HDFS-4520. Support listing snapshots under a snapshottable directory using |
| ls. (Jing Zhao via szetszwo) |
| |
| HDFS-4514. Add CLI for supporting snapshot rename, diff report, and |
| snapshottable directory listing. (Jing Zhao via szetszwo) |
| |
| HDFS-4523. Fix INodeFile replacement, TestQuota and javac errors from trunk |
| merge. (szetszwo) |
| |
| HDFS-4507. Update quota verification for snapshots. (szetszwo) |
| |
| HDFS-4545. With snapshots, FSDirectory.unprotectedSetReplication(..) always |
| changes file replication but it may or may not changes block replication. |
| (szetszwo) |
| |
| HDFS-4557. Fix FSDirectory#delete when INode#cleanSubtree returns 0. |
| (Jing Zhao via szetszwo) |
| |
| HDFS-4579. Annotate snapshot tests. (Arpit Agarwal via suresh) |
| |
| HDFS-4574. Move Diff to the util package. (szetszwo) |
| |
| HDFS-4563. Update namespace/diskspace usage after deleting snapshots. |
| (Jing Zhao via szetszwo) |
| |
| HDFS-4144. Create test for all snapshot-related metrics. |
| (Jing Zhao via suresh) |
| |
| HDFS-4556. Add snapshotdiff and LsSnapshottableDir tools to hdfs script. |
| (Arpit Agarwal via szetszwo) |
| |
| HDFS-4534. Add INodeReference in order to support rename with snapshots. |
| (szetszwo) |
| |
| HDFS-4616. Update the FilesDeleted metric while deleting file/dir in the |
| current tree. (Jing Zhao via szetszwo) |
| |
| HDFS-4627. Fix FSImageFormat#Loader NPE and synchronization issues. |
| (Jing Zhao via suresh) |
| |
| HDFS-4612. Not to use INode.getParent() when generating snapshot diff |
| report. (Jing Zhao via szetszwo) |
| |
| HDFS-4636. Update quota usage when deleting files/dirs that were created |
| after taking the latest snapshot. (Jing Zhao via szetszwo) |
| |
| HDFS-4648. For snapshot deletion, when merging the diff from to-delete |
| snapshot to the prior snapshot, make sure files/directories created after |
| the prior snapshot get deleted. (Jing Zhao via szetszwo) |
| |
| HDFS-4637. INodeDirectory#replaceSelf4Quota may incorrectly convert a newly |
| created directory to an INodeDirectoryWithSnapshot. (Jing Zhao via szetszwo) |
| |
| HDFS-4611. Update FSImage for INodeReference. (szetszwo) |
| |
| HDFS-4647. Rename should call setLocalName after an inode is removed from |
| snapshots. (Arpit Agarwal via szetszwo) |
| |
| HDFS-4684. Use INode id for image serialization when writing INodeReference. |
| (szetszwo) |
| |
| HDFS-4675. Fix rename across snapshottable directories. (Jing Zhao via |
| szetszwo) |
| |
| HDFS-4692. Use timestamp as default snapshot names. (szetszwo) |
| |
| HDFS-4666. Define ".snapshot" as a reserved inode name so that users cannot |
| create a file/directory with ".snapshot" as the name. If ".snapshot" is |
| used in a previous version of HDFS, it must be renamed before upgrade; |
| otherwise, upgrade will fail. (szetszwo) |
| |
| HDFS-4700. Fix the undo section of rename with snapshots. (Jing Zhao via |
| szetszwo) |
| |
| HDFS-4529. Disallow concat when one of the src files is in some snapshot. |
| (szetszwo) |
| |
| HDFS-4550. Refactor INodeDirectory.INodesInPath to a standalone class. |
| (szetszwo) |
| |
| HDFS-4707. Add snapshot methods to FilterFileSystem and fix findbugs |
| warnings. (szetszwo) |
| |
| HDFS-4706. Do not replace root inode for disallowSnapshot. (szetszwo) |
| |
| HDFS-4717. Change the path parameter type of the snapshot methods in |
| HdfsAdmin from String to Path. (szetszwo) |
| |
| HDFS-4708. Add snapshot user documentation. (szetszwo) |
| |
| HDFS-4726. Fix test failures after merging the INodeId-INode mapping |
| from trunk. (Jing Zhao via szetszwo) |
| |
| HDFS-4727. Update inodeMap after deleting files/directories/snapshots. |
| (Jing Zhao via szetszwo) |
| |
| HDFS-4719. Remove AbstractINodeDiff.Factory and move its methods to |
| AbstractINodeDiffList. (Arpit Agarwal via szetszwo) |
| |
| HDFS-4735. DisallowSnapshot throws IllegalStateException for nested |
| snapshottable directories. (Jing Zhao via szetszwo) |
| |
| HDFS-4738. Changes AbstractINodeDiff to implement Comparable<Integer>, and |
| fix javadoc and other warnings. (szetszwo) |
| |
| HDFS-4686. Update quota computation for rename and INodeReference. |
| (Jing Zhao via szetszwo) |
| |
| HDFS-4729. Fix OfflineImageViewer and permission checking for snapshot |
| operations. (Jing Zhao via szetszwo) |
| |
| HDFS-4749. Use INodeId to identify the corresponding directory node in |
| FSImage saving/loading. (Jing Zhao via szetszwo) |
| |
| HDFS-4742. Fix appending to a renamed file with snapshot. (Jing Zhao via |
| szetszwo) |
| |
| HDFS-4755. Fix AccessControlException message and moves "implements |
| LinkedElement" from INode to INodeWithAdditionalFields. (szetszwo) |
| |
| HDFS-4650. Fix a bug in FSDirectory and add more unit tests for rename with |
| existence of snapshottable directories and snapshots. (Jing Zhao via |
| szetszwo) |
| |
| HDFS-4650. When passing two non-existing snapshot names to snapshotDiff, it |
| returns success if the names are the same. (Jing Zhao via szetszwo) |
| |
| HDFS-4767. If a directory is snapshottable, do not replace it when clearing |
| quota. (Jing Zhao via szetszwo) |
| |
| HDFS-4578. Restrict snapshot IDs to 24-bit wide. (Arpit Agarwal via |
| szetszwo) |
| |
| HDFS-4773. Fix bugs in quota usage computation and OfflineImageViewer. |
| (Jing Zhao via szetszwo) |
| |
| HDFS-4760. Update inodeMap after node replacement. (Jing Zhao via szetszwo) |
| |
| HDFS-4758. Disallow nested snapshottable directories and unwrap |
| RemoteException. (szetszwo) |
| |
| HDFS-4781. Fix a NullPointerException when listing .snapshot under |
| a non-existing directory. (szetszwo) |
| |
| HDFS-4791. Update and fix deletion of reference inode. (Jing Zhao via |
| szetszwo) |
| |
| HDFS-4798. Update computeContentSummary() for the reference nodes in |
| snapshots. (szetszwo) |
| |
| HDFS-4800. Fix INodeDirectoryWithSnapshot#cleanDeletedINode. (Jing Zhao via |
| szetszwo) |
| |
| HDFS-4801. lsSnapshottableDir throws IllegalArgumentException when root is |
| snapshottable. (Jing Zhao via szetszwo) |
| |
| HDFS-4802. Disallowing snapshot on / twice should throw SnapshotException |
| but not IllegalStateException. (Jing Zhao via szetszwo) |
| |
| HDFS-4806. In INodeDirectoryWithSnapshot, use isInLatestSnapshot() to |
| determine if an added/removed child should be recorded in the snapshot diff. |
| (Jing Zhao via szetszwo) |
| |
| HDFS-4809. When a QuotaExceededException is thrown during rename, the quota |
| usage should be subtracted back. (Jing Zhao via szetszwo) |
| |
| HDFS-4842. Identify the correct prior snapshot when deleting a |
| snapshot under a renamed subtree. (jing9) |
| |
| HDFS-4846. Clean up snapshot CLI commands output stacktrace for invalid |
| arguments. (Jing Zhao via brandonli) |
| |
| HDFS-4857. Snapshot.Root and AbstractINodeDiff#snapshotINode should not be |
| put into INodeMap when loading FSImage. (jing9) |
| |
| HDFS-4863. The root directory should be added to the snapshottable |
| directory list while loading fsimage. (jing9) |
| |
| HDFS-4848. copyFromLocal and renaming a file to ".snapshot" should output |
| that ".snapshot" is a reserved name. (Jing Zhao via brandonli) |
| |
| HDFS-4826. TestNestedSnapshots times out due to repeated slow edit log |
| flushes when running on virtualized disk. (Chris Nauroth via szetszwo) |
| |
| HDFS-4876. Fix the javadoc of FileWithSnapshot and move FileDiffList to |
| FileWithSnapshot. (szetszwo) |
| |
| HDFS-4850. Fix OfflineImageViewer to work on fsimages with empty files or |
| snapshots. (jing9) |
| |
| HDFS-4877. Snapshot: fix the scenario where a directory is renamed under |
| its prior descendant. (jing9) |
| |
| HDFS-4873. callGetBlockLocations returns incorrect number of blocks for |
| snapshotted files. (jing9) |
| |
| HDFS-4819. Update Snapshot doc to clarify that nested snapshots are not |
| allowed. (szetszwo) |
| |
| HDFS-4902. DFSClient.getSnapshotDiffReport should use string path rather |
| than o.a.h.fs.Path. (Binglin Chang via jing9) |
| |
| HDFS-4875. Add a test for testing snapshot file length. |
| (Arpit Agarwal via jing9) |
| |
| HDFS-4841. FsShell commands using secure webhfds fail ClientFinalizer |
| shutdown hook. (rkanter via tucu) |
| |
| HDFS-4951. FsShell commands using secure httpfs throw exceptions due |
| to missing TokenRenewer. (rknater via tucu) |
| |
| HDFS-4969. WebhdfsFileSystem expects non-standard WEBHDFS Json element. |
| (rkanter via tucu) |
| |
| HDFS-4797. BlockScanInfo does not override equals(..) and hashCode() |
| consistently. (szetszwo) |
| |
| HDFS-4978. Make disallowSnapshot idempotent. (jing9) |
| |
| HDFS-5005. Move SnapshotException and SnapshotAccessControlException |
| to o.a.h.hdfs.protocol. (jing9) |
| |
| HDFS-4982. JournalNode should relogin from keytab before fetching logs |
| from other JNs (todd) |
| |
| Release 2.0.6-alpha - 08/22/2013 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| OPTIMIZATIONS |
| |
| Release 2.0.5-alpha - 06/06/2013 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| Release 2.0.4-alpha - 2013-04-25 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| Release 2.0.3-alpha - 2013-02-06 |
| |
| INCOMPATIBLE CHANGES |
| |
| HDFS-4122. Cleanup HDFS logs and reduce the size of logged messages. |
| (suresh) |
| |
| HDFS-4362. GetDelegationTokenResponseProto does not handle null token. |
| (suresh) |
| |
| HDFS-4367. GetDataEncryptionKeyResponseProto does not handle null |
| response. (suresh) |
| |
| HDFS-4364. GetLinkTargetResponseProto does not handle null path. (suresh) |
| |
| HDFS-4369. GetBlockKeysResponseProto does not handle null response. |
| (suresh) |
| |
| HDFS-4451. hdfs balancer command returns exit code 1 on success instead |
| of 0. (Joshua Blatt via suresh) |
| |
| HDFS-4350. Make enabling of stale marking on read and write paths |
| independent. (Andrew Wang via suresh) |
| |
| |
| NEW FEATURES |
| |
| HDFS-2656. Add libwebhdfs, a pure C client based on WebHDFS. |
| (Jaimin D Jetly and Jing Zhao via szetszwo) |
| |
| HDFS-3912. Detect and avoid stale datanodes for writes. |
| (Jing Zhao via suresh) |
| |
| HDFS-4059. Add number of stale DataNodes to metrics. (Jing Zhao via suresh) |
| |
| HDFS-4155. libhdfs implementation of hsync API (Liang Xie via todd) |
| |
| HDFS-4213. Add an API to hsync for updating the last block length at the |
| namenode. (Jing Zhao via szetszwo) |
| |
| HDFS-3077. Implement QuorumJournalManager, a distributed mechanism for |
| reliably storing HDFS edit logs. See dedicated section below for breakdown |
| of subtasks. |
| |
| IMPROVEMENTS |
| |
| HDFS-3925. Prettify PipelineAck#toString() for printing to a log |
| (Andrew Wang via todd) |
| |
| HDFS-3939. NN RPC address cleanup. (eli) |
| |
| HDFS-3373. Change DFSClient input stream socket cache to global static and |
| add a thread to cleanup expired cache entries. (John George via szetszwo) |
| |
| HDFS-3896. Add descriptions for dfs.namenode.rpc-address and |
| dfs.namenode.servicerpc-address to hdfs-default.xml. (Jeff Lord via atm) |
| |
| HDFS-3996. Add debug log removed in HDFS-3873 back. (eli) |
| |
| HDFS-3916. libwebhdfs (C client) code cleanups. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-3813. Log error message if security and WebHDFS are enabled but |
| principal/keytab are not configured. (Stephen Chu via atm) |
| |
| HDFS-3483. Better error message when hdfs fsck is run against a ViewFS |
| config. (Stephen Fritz via atm) |
| |
| HDFS-3682. MiniDFSCluster#init should provide more info when it fails. |
| (todd via eli) |
| |
| HDFS-4008. TestBalancerWithEncryptedTransfer needs a timeout. (eli) |
| |
| HDFS-4007. Rehabilitate bit-rotted unit tests under |
| hadoop-hdfs-project/hadoop-hdfs/src/test/unit/ |
| (Colin Patrick McCabe via todd) |
| |
| HDFS-4041. Hadoop HDFS Maven protoc calls must not depend on external |
| sh script. (Chris Nauroth via suresh) |
| |
| HADOOP-8911. CRLF characters in source and text files. |
| (Raja Aluri via suresh) |
| |
| HDFS-4037. Rename the getReplication() method in BlockCollection to |
| getBlockReplication(). (szetszwo) |
| |
| HDFS-4036. Remove "throws UnresolvedLinkException" from |
| FSDirectory.unprotectedAddFile(..). (Jing Zhao via szetszwo) |
| |
| HDFS-2946. HA: Put a cap on the number of completed edits files retained |
| by the NN. (atm) |
| |
| HDFS-4029. GenerationStamp should use an AtomicLong. (eli) |
| |
| HDFS-4068. DatanodeID and DatanodeInfo member should be private. (eli) |
| |
| HDFS-4073. Two minor improvements to FSDirectory. (Jing Zhao via szetszwo) |
| |
| HDFS-4074. Remove the unused default constructor from INode. (Brandon Li |
| via szetszwo) |
| |
| HDFS-4088. Remove "throws QuotaExceededException" from an |
| INodeDirectoryWithQuota constructor. (szetszwo) |
| |
| HDFS-4099. Clean up replication code and add more javadoc. (szetszwo) |
| |
| HDFS-4107. Add utility methods for casting INode to INodeFile and |
| INodeFileUnderConstruction. (szetszwo) |
| |
| HDFS-4112. A few improvements on INodeDirectory include adding a utility |
| method for casting; avoiding creation of new empty lists; cleaning up |
| some code and rewriting some javadoc. (szetszwo) |
| |
| HDFS-4121. Add namespace declarations in hdfs .proto files for languages |
| other than java. (Binglin Chang via suresh) |
| |
| HDFS-3573. Supply NamespaceInfo when instantiating JournalManagers. |
| (todd and ivank via umamahesh) |
| |
| HDFS-3695. Genericize format() to non-file JournalManagers. |
| (todd via umamahesh) |
| |
| HDFS-3789. JournalManager#format() should be able to throw IOException. |
| (Ivan Kelly via umamahesh) |
| |
| HDFS-3809. Make BKJM use protobufs for all serialization with ZK. |
| (Ivan Kelly via umamhesh) |
| |
| HDFS-3916. libwebhdfs testing code cleanup. (Jing Zhao via suresh) |
| |
| HDFS-4143. Change blocks to private in INodeFile and renames isLink() to |
| isSymlink() in INode. (szetszwo) |
| |
| HDFS-4046. Rename ChecksumTypeProto enum NULL since it is illegal in |
| C/C++. (Binglin Chang via suresh) |
| |
| HDFS-4048. Use ERROR instead of INFO for volume failure logs. |
| (Stephen Chu via eli) |
| |
| HDFS-1322. Document umask in DistributedFileSystem#mkdirs javadocs. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-4038. Override toString() for BookKeeperEditLogInputStream. |
| (Vinay via umamahesh) |
| |
| HDFS-4214. OfflineEditsViewer should print out the offset at which it |
| encountered an error. (Colin Patrick McCabe via atm) |
| |
| HDFS-4199. Provide test for HdfsVolumeId. (Ivan A. Veselovsky via atm) |
| |
| HDFS-3049. During the normal NN startup process, fall back on a different |
| edit log if we see one that is corrupt (Colin Patrick McCabe via todd) |
| |
| HDFS-3571. Allow EditLogFileInputStream to read from a remote URL (todd) |
| |
| HDFS-4110. Refine a log printed in JNStorage. (Liang Xie via suresh) |
| |
| HDFS-4153. Add START_MSG/SHUTDOWN_MSG for JournalNode. (liang xie via atm) |
| |
| HDFS-3935. Add JournalNode to the start/stop scripts (Andy Isaacson via todd) |
| |
| HDFS-4268. Remove redundant enum NNHAStatusHeartbeat.State. (shv) |
| |
| HDFS-3680. Allow customized audit logging in HDFS FSNamesystem. (Marcelo |
| Vanzin via atm) |
| |
| HDFS-4130. BKJM: The reading for editlog at NN starting using bkjm is not efficient. |
| (Han Xiao via umamahesh) |
| |
| HDFS-4326. bump up Tomcat version for HttpFS to 6.0.36. (tucu via acmurthy) |
| |
| HDFS-4270. Introduce soft and hard limits for max replication so that |
| replications of the highest priority are allowed to choose a source datanode |
| that has reached its soft limit but not the hard limit. (Derek Dagit via |
| szetszwo) |
| |
| HADOOP-9173. Add security token protobuf definition to common and |
| use it in hdfs. (suresh) |
| |
| HDFS-4030. BlockManager excessBlocksCount and |
| postponedMisreplicatedBlocksCount should be AtomicLongs. (eli) |
| |
| HDFS-4031. Update findbugsExcludeFile.xml to include findbugs 2 |
| exclusions. (eli) |
| |
| HDFS-4033. Miscellaneous findbugs 2 fixes. (eli) |
| |
| HDFS-4034. Remove redundant null checks. (eli) |
| |
| HDFS-4035. LightWeightGSet and LightWeightHashSet increment a |
| volatile without synchronization. (eli) |
| |
| HDFS-4032. Specify the charset explicitly rather than rely on the |
| default. (eli) |
| |
| HDFS-4363. Combine PBHelper and HdfsProtoUtil and remove redundant |
| methods. (suresh) |
| |
| HDFS-4377. Some trivial DN comment cleanup. (eli) |
| |
| HDFS-4381. Document fsimage format details in FSImageFormat class javadoc. |
| (Jing Zhao via suresh) |
| |
| HDFS-4375. Use token request messages defined in hadoop common. |
| (suresh) |
| |
| HDFS-4392. Use NetUtils#getFreeSocketPort in MiniDFSCluster. |
| (Andrew Purtell via suresh) |
| |
| HDFS-4393. Make empty request and responses in protocol translators can be |
| static final members. (Brandon Li via suresh) |
| |
| HDFS-4403. DFSClient can infer checksum type when not provided by reading |
| first byte (todd) |
| |
| HDFS-4259. Improve pipeline DN replacement failure message (harsh) |
| |
| HDFS-3598. WebHDFS support for file concat. (Plamen Jeliazkov via shv) |
| |
| HDFS-4456. Add concat to HttpFS and WebHDFS REST API docs. (plamenj2003 via tucu) |
| |
| HDFS-3131. Improve TestStorageRestore. (Brandon Li via atm) |
| |
| OPTIMIZATIONS |
| |
| HDFS-3429. DataNode reads checksums even if client does not need them (todd) |
| |
| BUG FIXES |
| |
| HDFS-3919. MiniDFSCluster:waitClusterUp can hang forever. |
| (Andy Isaacson via eli) |
| |
| HDFS-3924. Multi-byte id in HdfsVolumeId. (Andrew Wang via atm) |
| |
| HDFS-3936. MiniDFSCluster shutdown races with BlocksMap usage. (eli) |
| |
| HDFS-3951. datanode web ui does not work over HTTPS when datanode is started in secure mode. (tucu) |
| |
| HDFS-3949. NameNodeRpcServer#join should join on both client and |
| server RPC servers. (eli) |
| |
| HDFS-3932. NameNode Web UI broken if the rpc-address is set to the wildcard. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-3931. TestDatanodeBlockScanner#testBlockCorruptionPolicy2 is broken. |
| (Andy Isaacson via eli) |
| |
| HDFS-3964. Make NN log of fs.defaultFS debug rather than info. (eli) |
| |
| HDFS-3992. Method org.apache.hadoop.hdfs.TestHftpFileSystem.tearDown() |
| sometimes throws NPEs. (Ivan A. Veselovsky via atm) |
| |
| HDFS-3753. Tests don't run with native libraries. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-4000. TestParallelLocalRead fails with "input ByteBuffers |
| must be direct buffers". (Colin Patrick McCabe via eli) |
| |
| HDFS-3999. HttpFS OPEN operation expects len parameter, it should be length. (tucu) |
| |
| HDFS-4006. TestCheckpoint#testSecondaryHasVeryOutOfDateImage |
| occasionally fails due to unexpected exit. (todd via eli) |
| |
| HDFS-4018. testMiniDFSClusterWithMultipleNN is missing some |
| cluster cleanup. (eli) |
| |
| HDFS-4020. TestRBWBlockInvalidation may time out. (eli) |
| |
| HDFS-4021. Misleading error message when resources are low on the NameNode. |
| (Christopher Conner via atm) |
| |
| HDFS-4044. Duplicate ChecksumType definition in HDFS .proto files. |
| (Binglin Chang via suresh) |
| |
| HDFS-4049. Fix hflush performance regression due to nagling delays |
| (todd) |
| |
| HDFS-3678. Edit log files are never being purged from 2NN. (atm) |
| |
| HDFS-4058. DirectoryScanner may fail with IOOB if the directory |
| scanning threads return out of volume order. (eli) |
| |
| HDFS-3985. Add timeouts to TestMulitipleNNDataBlockScanner. (todd via eli) |
| |
| HDFS-4061. TestBalancer and TestUnderReplicatedBlocks need timeouts. (eli) |
| |
| HDFS-3997. OfflineImageViewer incorrectly passes value of imageVersion when |
| visiting IS_COMPRESSED element. (Mithun Radhakrishnan via atm) |
| |
| HDFS-4055. TestAuditLogs is flaky. (Binglin Chang via eli) |
| |
| HDFS-4072. On file deletion remove corresponding blocks pending |
| replications. (Jing Zhao via suresh) |
| |
| HDFS-4022. Replication not happening for appended block. |
| (Vinay via umamahesh) |
| |
| HDFS-3948. Do not use hflush in TestWebHDFS.testNamenodeRestart() since the |
| out stream returned by WebHdfsFileSystem does not support it. (Jing Zhao |
| via szetszwo) |
| |
| HDFS-3616. Fix a ConcurrentModificationException bug that BP actor threads |
| may not be shutdown properly in DataNode. (Jing Zhao via szetszwo) |
| |
| HDFS-4127. Log message is not correct in case of short of replica. |
| (Junping Du via suresh) |
| |
| HADOOP-8994. TestDFSShell creates file named "noFileHere", making further |
| tests hard to understand (Andy Isaacson via daryn) |
| |
| HDFS-3804. TestHftpFileSystem fails intermittently with JDK7 |
| (Trevor Robinson via daryn) |
| |
| HDFS-4132. When libwebhdfs is not enabled, nativeMiniDfsClient frees |
| uninitialized memory (Colin Patrick McCabe via todd) |
| |
| HDFS-1331. dfs -test should work like /bin/test (Andy Isaacson via daryn) |
| |
| HDFS-3979. For hsync, datanode should wait for the local sync to complete |
| before sending ack. (Lars Hofhansl via szetszwo) |
| |
| HDFS-3810. Implement format() for BKJM (Ivan Kelly via umamahesh) |
| |
| HDFS-3625. Fix TestBackupNode by properly initializing edit log during |
| startup. (Junping Du via todd) |
| |
| HDFS-4138. BackupNode startup fails due to uninitialized edit log. |
| (Kihwal Lee via shv) |
| |
| HDFS-4162. Some malformed and unquoted HTML strings are returned from |
| datanode web ui. (Darek Dagit via suresh) |
| |
| HDFS-4164. fuse_dfs: add -lrt to the compiler command line on Linux. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-3921. NN will prematurely consider blocks missing when entering active |
| state while still in safe mode. (atm) |
| |
| HDFS-4106. BPServiceActor#lastHeartbeat, lastBlockReport and |
| lastDeletedReport should be volatile. (Jing Zhao via suresh) |
| |
| HDFS-4139. fuse-dfs RO mode still allows file truncation. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-4104. dfs -test -d prints inappropriate error on nonexistent directory |
| (Andy Isaacson via daryn) |
| |
| HDFS-3623. BKJM: zkLatchWaitTimeout hard coded to 6000. Make use of ZKSessionTimeout instead. |
| (umamahesh) |
| |
| HDFS-4100. Fix all findbug security warings. (Liang Xie via eli) |
| |
| HDFS-3507. DFS#isInSafeMode needs to execute only on Active NameNode. |
| (Vinay via atm) |
| |
| HDFS-4105. The SPNEGO user for secondary namenode should use the web |
| keytab. (Arpit Gupta via jitendra) |
| |
| HDFS-4156. Seeking to a negative position should throw an IOE. |
| (Eli Reisman via eli) |
| |
| HDFS-4171. WebHDFS and HttpFs should accept only valid Unix user |
| names. (tucu) |
| |
| HDFS-4178. Shell scripts should not close stderr (Andy Isaacson via daryn) |
| |
| HDFS-4179. BackupNode: allow reads, fix checkpointing, safeMode. (shv) |
| |
| HDFS-4216. Do not ignore QuotaExceededException when adding symlinks. |
| (szetszwo) |
| |
| HDFS-4242. Map.Entry is incorrectly used in LeaseManager since the behavior |
| of it is undefined after the iteration or modifications of the map. |
| (szetszwo) |
| |
| HDFS-4231. BackupNode: Introduce BackupState. (shv) |
| |
| HDFS-4238. Standby namenode should not do purging of shared |
| storage edits. (todd) |
| |
| HDFS-4282. TestEditLog.testFuzzSequences FAILED in all pre-commit test |
| (todd) |
| |
| HDFS-4236. Remove artificial limit on username length introduced in |
| HDFS-4171. (tucu via suresh) |
| |
| HDFS-4279. NameNode does not initialize generic conf keys when started |
| with -recover. (Colin Patrick McCabe via atm) |
| |
| HDFS-4291. edit log unit tests leave stray test_edit_log_file around |
| (Colin Patrick McCabe via todd) |
| |
| HDFS-4292. Sanity check not correct in RemoteBlockReader2.newBlockReader |
| (Binglin Chang via todd) |
| |
| HDFS-4295. Using port 1023 should be valid when starting Secure DataNode |
| (Stephen Chu via todd) |
| |
| HDFS-4294. Backwards compatibility is not maintained for TestVolumeId. |
| (Ivan A. Veselovsky and Robert Parker via atm) |
| |
| HDFS-2264. NamenodeProtocol has the wrong value for clientPrincipal in |
| KerberosInfo annotation. (atm) |
| |
| HDFS-4307. SocketCache should use monotonic time. (Colin Patrick McCabe |
| via atm) |
| |
| HDFS-4315. DNs with multiple BPs can have BPOfferServices fail to start |
| due to unsynchronized map access. (atm) |
| |
| HDFS-4140. fuse-dfs handles open(O_TRUNC) poorly. (Colin Patrick McCabe |
| via atm) |
| |
| HDFS-4308. addBlock() should persist file blocks once. |
| (Plamen Jeliazkov via shv) |
| |
| HDFS-4347. Avoid infinite waiting checkpoint to complete in TestBackupNode. |
| (Plamen Jeliazkov via shv) |
| |
| HDFS-4349. Add test for reading files from BackupNode. (shv) |
| |
| HDFS-4302. Fix fatal exception when starting NameNode with DEBUG logs |
| (Eugene Koontz via todd) |
| |
| HDFS-3970. Fix bug causing rollback of HDFS upgrade to result in bad |
| VERSION file. (Vinay and Andrew Wang via atm) |
| |
| HDFS-4306. PBHelper.convertLocatedBlock miss convert BlockToken. (Binglin |
| Chang via atm) |
| |
| HDFS-4384. test_libhdfs_threaded gets SEGV if JNIEnv cannot be |
| initialized. (Colin Patrick McCabe via eli) |
| |
| HDFS-4328. TestLargeBlock#testLargeBlockSize is timing out. (Chris Nauroth |
| via atm) |
| |
| HDFS-4274. BlockPoolSliceScanner does not close verification log during |
| shutdown. (Chris Nauroth via suresh) |
| |
| HDFS-1245. Pluggable block id generation. (shv) |
| |
| HDFS-4415. HostnameFilter should handle hostname resolution failures and |
| continue processing. (Robert Kanter via atm) |
| |
| HDFS-4359. Slow RPC responses from NN can prevent metrics collection on |
| DNs. (liang xie via atm) |
| |
| HDFS-4444. Add space between total transaction time and number of |
| transactions in FSEditLog#printStatistics. (Stephen Chu via suresh) |
| |
| HDFS-4428. FsDatasetImpl should disclose what the error is when a rename |
| fails. (Colin Patrick McCabe via atm) |
| |
| HDFS-4452. getAdditionalBlock() can create multiple blocks if the client |
| times out and retries. (shv) |
| |
| HDFS-4445. All BKJM ledgers are not checked while tailing, So failover will fail. |
| (Vinay via umamahesh) |
| |
| HDFS-4462. 2NN will fail to checkpoint after an HDFS upgrade from a |
| pre-federation version of HDFS. (atm) |
| |
| HDFS-4404. Create file failure when the machine of first attempted NameNode |
| is down. (Todd Lipcon via atm) |
| |
| HDFS-4344. dfshealth.jsp throws NumberFormatException when |
| dfs.hosts/dfs.hosts.exclude includes port number. (Andy Isaacson via atm) |
| |
| HDFS-4468. Use the new StringUtils methods added by HADOOP-9252 and fix |
| TestHDFSCLI and TestQuota. (szetszwo) |
| |
| HDFS-4458. In DFSUtil.getNameServiceUris(..), convert default fs URI using |
| NetUtils.createSocketAddr(..) for being consistent with other addresses. |
| (Binglin Chang via szetszwo) |
| |
| BREAKDOWN OF HDFS-3077 SUBTASKS |
| |
| HDFS-3077. Quorum-based protocol for reading and writing edit logs. |
| (todd, Brandon Li, and Hari Mankude via todd) |
| |
| HDFS-3694. Fix getEditLogManifest to fetch httpPort if necessary (todd) |
| |
| HDFS-3692. Support purgeEditLogs() call to remotely purge logs on JNs |
| (todd) |
| |
| HDFS-3693. JNStorage should read its storage info even before a writer |
| becomes active (todd) |
| |
| HDFS-3725. Fix QJM startup when individual JNs have gaps (todd) |
| |
| HDFS-3741. Exhaustive failure injection test for skipped RPCs (todd) |
| |
| HDFS-3773. TestNNWithQJM fails after HDFS-3741. (atm) |
| |
| HDFS-3793. Implement genericized format() in QJM (todd) |
| |
| HDFS-3795. QJM: validate journal dir at startup (todd) |
| |
| HDFS-3798. Avoid throwing NPE when finalizeSegment() is called on invalid |
| segment (todd) |
| |
| HDFS-3799. QJM: handle empty log segments during recovery (todd) |
| |
| HDFS-3797. QJM: add segment txid as a parameter to journal() RPC (todd) |
| |
| HDFS-3800. improvements to QJM fault testing (todd) |
| |
| HDFS-3823. QJM: TestQJMWithFaults fails occasionally because of missed |
| setting of HTTP port. (todd and atm) |
| |
| HDFS-3826. QJM: Some trivial logging / exception text improvements. (todd |
| and atm) |
| |
| HDFS-3839. QJM: hadoop-daemon.sh should be updated to accept "journalnode" |
| (eli) |
| |
| HDFS-3845. Fixes for edge cases in QJM recovery protocol (todd) |
| |
| HDFS-3877. QJM: Provide defaults for dfs.journalnode.*address (eli) |
| |
| HDFS-3863. Track last "committed" txid in QJM (todd) |
| |
| HDFS-3869. Expose non-file journal manager details in web UI (todd) |
| |
| HDFS-3884. Journal format() should reset cached values (todd) |
| |
| HDFS-3870. Add metrics to JournalNode (todd) |
| |
| HDFS-3891. Make selectInputStreams throw IOE instead of RTE (todd) |
| |
| HDFS-3726. If a logger misses an RPC, don't retry that logger until next |
| segment (todd) |
| |
| HDFS-3893. QJM: Make QJM work with security enabled. (atm) |
| |
| HDFS-3897. QJM: TestBlockToken fails after HDFS-3893. (atm) |
| |
| HDFS-3898. QJM: enable TCP_NODELAY for IPC (todd) |
| |
| HDFS-3885. QJM: optimize log sync when JN is lagging behind (todd) |
| |
| HDFS-3900. QJM: avoid validating log segments on log rolls (todd) |
| |
| HDFS-3901. QJM: send 'heartbeat' messages to JNs even when they are |
| out-of-sync (todd) |
| |
| HDFS-3899. QJM: Add client-side metrics (todd) |
| |
| HDFS-3914. QJM: acceptRecovery should abort current segment (todd) |
| |
| HDFS-3915. QJM: Failover fails with auth error in secure cluster (todd) |
| |
| HDFS-3906. QJM: quorum timeout on failover with large log segment (todd) |
| |
| HDFS-3840. JournalNodes log JournalNotFormattedException backtrace error |
| before being formatted (todd) |
| |
| HDFS-3894. QJM: testRecoverAfterDoubleFailures can be flaky due to IPC |
| client caching (todd) |
| |
| HDFS-3926. QJM: Add user documentation for QJM. (atm) |
| |
| HDFS-3943. QJM: remove currently-unused md5sum field (todd) |
| |
| HDFS-3950. QJM: misc TODO cleanup, improved log messages, etc. (todd) |
| |
| HDFS-3955. QJM: Make acceptRecovery() atomic. (todd) |
| |
| HDFS-3956. QJM: purge temporary files when no longer within retention |
| period (todd) |
| |
| HDFS-4004. TestJournalNode#testJournal fails because of test case execution |
| order (Chao Shi via todd) |
| |
| HDFS-4017. Unclosed FileInputStream in GetJournalEditServlet |
| (Chao Shi via todd) |
| |
| HDFS-4351. In BlockPlacementPolicyDefault.chooseTarget(..), numOfReplicas |
| needs to be updated when avoiding stale nodes. (Andrew Wang via szetszwo) |
| |
| HDFS-2908. Add apache license header for StorageReport.java. (Brandon Li |
| via tgraves) |
| |
| HDFS-4399. Fix RAT warnings by excluding images sub-dir in docs. (Thomas |
| Graves via acmurthy) |
| |
| Release 2.0.2-alpha - 2012-09-07 |
| |
| INCOMPATIBLE CHANGES |
| |
| HDFS-3446. HostsFileReader silently ignores bad includes/excludes |
| (Matthew Jacobs via todd) |
| |
| HDFS-3755. Creating an already-open-for-write file with overwrite=true fails |
| (todd) |
| |
| NEW FEATURES |
| |
| HDFS-744. Support hsync in HDFS. (Lars Hofhansl via szetszwo) |
| |
| HDFS-3042. Automatic failover support for NameNode HA (todd) |
| (see dedicated section below for breakdown of subtasks) |
| |
| HDFS-3518. Add a utility method HdfsUtils.isHealthy(uri) for checking if |
| the given HDFS is healthy. (szetszwo) |
| |
| HDFS-3113. httpfs does not support delegation tokens. (tucu) |
| |
| HDFS-3513. HttpFS should cache filesystems. (tucu) |
| |
| HDFS-3637. Add support for encrypting the DataTransferProtocol. (atm) |
| |
| HDFS-3150. Add option for clients to contact DNs via hostname. (eli) |
| |
| HDFS-2793. Add an admin command to trigger an edit log roll. (todd) |
| |
| HDFS-3703. Datanodes are marked stale if heartbeat is not received in |
| configured timeout and are selected as the last location to read from. |
| (Jing Zhao via suresh) |
| |
| IMPROVEMENTS |
| |
| HDFS-3040. TestMulitipleNNDataBlockScanner is misspelled. (Madhukara Phatak |
| via atm) |
| |
| HDFS-3390. DFSAdmin should print full stack traces of errors when DEBUG |
| logging is enabled. (atm) |
| |
| HDFS-3341. Change minimum RPC versions to respective SNAPSHOTs instead of |
| final releases. (todd) |
| |
| HDFS-3369. Rename {get|set|add}INode(..) methods in BlockManager and |
| BlocksMap to {get|set|add}BlockCollection(..). (John George via szetszwo) |
| |
| HDFS-3134. harden edit log loader against malformed or malicious input. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-3230. Cleanup DatanodeID creation in the tests. (eli) |
| |
| HDFS-3401. Cleanup DatanodeDescriptor creation in the tests. (eli) |
| |
| HDFS-3400. DNs should be able start with jsvc even if security is disabled. |
| (atm via eli) |
| |
| HDFS-3404. Make putImage in GetImageServlet infer remote address to fetch |
| from request. (atm) |
| |
| HDFS-3335. check for edit log corruption at the end of the log |
| (Colin Patrick McCabe via todd) |
| |
| HDFS-3417. Rename BalancerDatanode#getName to getDisplayName to be |
| consistent with Datanode. (eli) |
| |
| HDFS-3416. Cleanup DatanodeID and DatanodeRegistration |
| constructors used by testing. (eli) |
| |
| HDFS-3419. Cleanup LocatedBlock. (eli) |
| |
| HDFS-3440. More effectively limit stream memory consumption when reading |
| corrupt edit logs (Colin Patrick McCabe via todd) |
| |
| HDFS-3438. BootstrapStandby should not require a rollEdits on active node |
| (todd) |
| |
| HDFS-2885. Remove "federation" from the nameservice config options. |
| (Tsz Wo (Nicholas) Sze via eli) |
| |
| HDFS-3394. Do not use generic in INodeFile.getLastBlock(): the run-time |
| ClassCastException check is useless since generic type information is only |
| available in compile-time. (szetszwo) |
| |
| HDFS-3454. Balancer unconditionally logs InterruptedException at |
| INFO level on shutdown if security is enabled. (eli) |
| |
| HDFS-1013. Miscellaneous improvements to HTML markup for web UIs |
| (Eugene Koontz via todd) |
| |
| HDFS-3052. Change INodeFile and INodeFileUnderConstruction to package |
| private. (szetszwo) |
| |
| HDFS-3520. Add transfer rate logging to TransferFsImage. (eli) |
| |
| HDFS-3504. Support configurable retry policy in DFSClient for RPC |
| connections and RPC calls, and add MultipleLinearRandomRetry, a new retry |
| policy. (szetszwo) |
| |
| HDFS-3372. offlineEditsViewer should be able to read a binary |
| edits file with recovery mode. (Colin Patrick McCabe via eli) |
| |
| HDFS-3516. Check content-type in WebHdfsFileSystem. (szetszwo) |
| |
| HDFS-3535. Audit logging should log denied accesses. (Andy Isaacson via eli) |
| |
| HDFS-3481. Refactor HttpFS handling of JAX-RS query string parameters (tucu) |
| |
| HDFS-3572. Cleanup code which inits SPNEGO in HttpServer (todd) |
| |
| HDFS-3475. Make the replication monitor multipliers configurable. |
| (harsh via eli) |
| |
| HDFS-3343. Improve metrics for DN read latency (Andrew Wang via todd) |
| |
| HDFS-3170. Add more useful metrics for write latency (Matthew Jacobs via |
| todd) |
| |
| HDFS-3604. Add dfs.webhdfs.enabled to hdfs-default.xml. (eli) |
| |
| HDFS-2988. Improve error message when storage directory lock fails |
| (Miomir Boljanovic via harsh) |
| |
| HDFS-2391. Newly set BalancerBandwidth value is not displayed anywhere. |
| (harsh) |
| |
| HDFS-3067. NPE in DFSInputStream.readBuffer if read is repeated on |
| corrupted block. (Henry Robinson via atm) |
| |
| HDFS-3555. idle client socket triggers DN ERROR log |
| (should be INFO or DEBUG). (Andy Isaacson via harsh) |
| |
| HDFS-3568. fuse_dfs: add support for security. (Colin McCabe via atm) |
| |
| HDFS-3629. Fix the typo in the error message about inconsistent |
| storage layout version. (Brandon Li via harsh) |
| |
| HDFS-3613. GSet prints some INFO level values, which aren't |
| really very useful to all (Andrew Wang via harsh) |
| |
| HDFS-3611. NameNode prints unnecessary WARNs about edit log normally skipping |
| a few bytes. (Colin Patrick McCabe via harsh) |
| |
| HDFS-3582. Hook daemon process exit for testing. (eli) |
| |
| HDFS-3641. Move server Util time methods to common and use now |
| instead of System#currentTimeMillis. (eli) |
| |
| HDFS-3633. libhdfs: hdfsDelete should pass JNI_FALSE or JNI_TRUE. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-799. libhdfs must call DetachCurrentThread when a thread is destroyed. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-3306. fuse_dfs: don't lock release operations. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-3612. Single namenode image directory config warning can |
| be improved. (Andy Isaacson via harsh) |
| |
| HDFS-3606. libhdfs: create self-contained unit test. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-3539. libhdfs code cleanups. (Colin Patrick McCabe via eli) |
| |
| HDFS-3610. fuse_dfs: Provide a way to use the default (configured) NN URI. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-3663. MiniDFSCluster should capture the code path that led to |
| the first ExitException. (eli) |
| |
| HDFS-3659. Add missing @Override to methods across the hadoop-hdfs |
| project. (Brandon Li via harsh) |
| |
| HDFS-3537. Move libhdfs and fuse-dfs source to native subdirectories. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-3665. Add a test for renaming across file systems via a symlink. (eli) |
| |
| HDFS-3666. Plumb more exception messages to terminate. (eli) |
| |
| HDFS-3673. libhdfs: fix some compiler warnings. (Colin Patrick McCabe via eli) |
| |
| HDFS-3675. libhdfs: follow documented return codes. (Colin Patrick McCabe via eli) |
| |
| HDFS-1249. With fuse-dfs, chown which only has owner (or only group) |
| argument fails with Input/output error. (Colin Patrick McCabe via eli) |
| |
| HDFS-3583. Convert remaining tests to Junit4. (Andrew Wang via atm) |
| |
| HDFS-3711. Manually convert remaining tests to JUnit4. (Andrew Wang via atm) |
| |
| HDFS-3650. Use MutableQuantiles to provide latency histograms for various |
| operations. (Andrew Wang via atm) |
| |
| HDFS-3667. Add retry support to WebHdfsFileSystem. (szetszwo) |
| |
| HDFS-3291. add test that covers HttpFS working w/ a non-HDFS Hadoop |
| filesystem (tucu) |
| |
| HDFS-3634. Add self-contained, mavenized fuse_dfs test. (Colin Patrick |
| McCabe via atm) |
| |
| HDFS-3190. Simple refactors in existing NN code to assist |
| QuorumJournalManager extension. (todd) |
| |
| HDFS-3276. initializeSharedEdits should have a -nonInteractive flag (todd) |
| |
| HDFS-3765. namenode -initializeSharedEdits should be able to initialize |
| all shared storages. (Vinay and todd via todd) |
| |
| HDFS-3723. Add support -h, -help to all the commands. (Jing Zhao via |
| suresh) |
| |
| HDFS-3803. Change BlockPoolSliceScanner chatty INFO log to DEBUG. |
| (Andrew Purtell via suresh) |
| |
| HDFS-3802. StartupOption.name in HdfsServerConstants should be final. |
| (Jing Zhao via szetszwo) |
| |
| HDFS-3796. Speed up edit log tests by avoiding fsync() (todd) |
| |
| HDFS-2963. Console Output is confusing while executing metasave |
| (dfsadmin command). (Andrew Wang via eli) |
| |
| HDFS-3672. Expose disk-location information for blocks to enable better |
| scheduling. (Andrew Wang via atm) |
| |
| HDFS-2727. libhdfs should get the default block size from the server. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-3832. Remove protocol methods related to DistributedUpgrade. (suresh) |
| |
| HDFS-3819. Should check whether invalidate work percentage default value is |
| not greater than 1.0f. (Jing Zhao via jitendra) |
| |
| HDFS-3177. Update DFSClient and DataXceiver to handle different checkum |
| types in file checksum computation. (Kihwal Lee via szetszwo) |
| |
| HDFS-3844. Add @Override and remove {@inheritdoc} and unnecessary |
| imports. (Jing Zhao via suresh) |
| |
| HDFS-3853. Port MiniDFSCluster enableManagedDfsDirsRedundancy |
| option to branch-2. (Colin Patrick McCabe via eli) |
| |
| HDFS-3871. Change NameNodeProxies to use RetryUtils. (Arun C Murthy |
| via szetszwo) |
| |
| HDFS-3887. Remove redundant chooseTarget methods in BlockPlacementPolicy. |
| (Jing Zhao via szetszwo) |
| |
| HDFS-3888. Clean up BlockPlacementPolicyDefault. (Jing Zhao via szetszwo) |
| |
| HDFS-3907. Allow multiple users for local block readers. (eli) |
| |
| HDFS-3510. Editlog pre-allocation is performed prior to writing edits |
| to avoid partial edits case disk out of space. (Colin McCabe via todd) |
| |
| HDFS-3910. DFSTestUtil#waitReplication should timeout. (eli) |
| |
| HDFS-3920. libwebdhfs string processing and using strerror consistently |
| to handle all errors. (Jing Zhao via suresh) |
| |
| OPTIMIZATIONS |
| |
| HDFS-2982. Startup performance suffers when there are many edit log |
| segments. (Colin Patrick McCabe via todd) |
| |
| HDFS-2834. Add a ByteBuffer-based read API to DFSInputStream. |
| (Henry Robinson via todd) |
| |
| HDFS-3110. Use directRead API to reduce the number of buffer copies in |
| libhdfs (Henry Robinson via todd) |
| |
| HDFS-3697. Enable fadvise readahead by default. (todd) |
| |
| HDFS-2421. Improve the concurrency of SerialNumberMap in NameNode. |
| (Jing Zhao and Weiyan Wang via szetszwo) |
| |
| HDFS-3866. HttpFS POM should have property where to download tomcat from (zero45 via tucu) |
| |
| BUG FIXES |
| |
| HDFS-3385. The last block of INodeFileUnderConstruction is not |
| necessarily a BlockInfoUnderConstruction, so do not cast it in |
| FSNamesystem.recoverLeaseInternal(..). (szetszwo) |
| |
| HDFS-3414. Balancer does not find NameNode if rpc-address or |
| servicerpc-address are not set in client configs. (atm) |
| |
| HDFS-3031. Fix complete() and getAdditionalBlock() RPCs to be idempotent |
| (todd) |
| |
| HDFS-2759. Pre-allocate HDFS edit log files after writing version number. |
| (atm) |
| |
| HDFS-3413. TestFailureToReadEdits timing out. (atm) |
| |
| HDFS-3422. TestStandbyIsHot timeouts too aggressive (todd) |
| |
| HDFS-3433. GetImageServlet should allow administrative requestors when |
| security is enabled. (atm) |
| |
| HDFS-1153. dfsnodelist.jsp should handle invalid input parameters. |
| (Ravi Phulari via eli) |
| |
| HDFS-3434. InvalidProtocolBufferException when visiting DN |
| browseDirectory.jsp (eli) |
| |
| HDFS-2800. Fix cancellation of checkpoints in the standby node to be more |
| reliable. (todd) |
| |
| HDFS-3391. Fix InvalidateBlocks to compare blocks including their |
| generation stamps. (todd) |
| |
| HDFS-3444. hdfs groups command doesn't work with security enabled. (atm) |
| |
| HDFS-2717. BookKeeper Journal output stream doesn't check addComplete rc. |
| (Ivan Kelly via umamahesh) |
| |
| HDFS-3415. Make sure all layout versions are the same for all storage |
| directories in the Namenode. (Brandon Li via szetszwo) |
| |
| HDFS-3436. In DataNode.transferReplicaForPipelineRecovery(..), it should |
| use the stored generation stamp to check if the block is valid. (Vinay |
| via szetszwo) |
| |
| HDFS-3460. HttpFS proxyuser validation with Kerberos ON uses full |
| principal name. (tucu) |
| |
| HDFS-3058. HA: Bring BookKeeperJournalManager up to date with HA changes. |
| (Ivan Kelly via umamahesh) |
| |
| HDFS-3368. Missing blocks due to bad DataNodes coming up and down. (shv) |
| |
| HDFS-3452. BKJM:Switch from standby to active fails and NN gets shut down |
| due to delay in clearing of lock. (umamahesh) |
| |
| HDFS-3398. Client will not retry when primaryDN is down once it's just got pipeline. |
| (Amith D K via umamahesh) |
| |
| HDFS-3474. Cleanup Exception handling in BookKeeper journal manager. |
| (Ivan Kelly via umamahesh) |
| |
| HDFS-3468. Make BKJM-ZK session timeout configurable. (umamahesh) |
| |
| HDFS-3423. BKJM: NN startup is failing, when tries to recoverUnfinalizedSegments() |
| a bad inProgress_ ZNodes. (Ivan Kelly and Uma via umamahesh) |
| |
| HDFS-3441. Race condition between rolling logs at active NN and purging at standby. |
| (Rakesh R via umamahesh) |
| |
| HDFS-3484. hdfs fsck doesn't work if NN HTTP address is set to |
| 0.0.0.0 even if NN RPC address is configured. (atm via eli) |
| |
| HDFS-3486. offlineimageviewer can't read fsimage files that contain |
| persistent delegation tokens. (Colin Patrick McCabe via eli) |
| |
| HDFS-3487. offlineimageviewer should give byte offset information |
| when it encounters an exception. (Colin Patrick McCabe via eli) |
| |
| HDFS-3442. Incorrect count for Missing Replicas in FSCK report. (Andrew |
| Wang via atm) |
| |
| HDFS-2025. Go Back to File View link is not working in tail.jsp. |
| (Ashish and Sravan via umamahesh) |
| |
| HDFS-3501. Checkpointing with security enabled will stop working |
| after ticket lifetime expires. (atm via eli) |
| |
| HDFS-3266. DFSTestUtil#waitCorruptReplicas doesn't sleep between checks. |
| (Madhukara Phatak via atm) |
| |
| HDFS-3505. DirectoryScanner does not join all threads in shutdown. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-3485. DataTransferThrottler will over-throttle when currentTimeMillis |
| jumps (Andy Isaacson via todd) |
| |
| HDFS-2914. HA: Standby should not enter safemode when resources are low. |
| (Vinay via atm) |
| |
| HDFS-3235. MiniDFSClusterManager doesn't correctly support -format option. |
| (Henry Robinson via atm) |
| |
| HDFS-3514. Add missing TestParallelLocalRead. (Henry Robinson via atm) |
| |
| HDFS-3243. TestParallelRead timing out on jenkins. (Henry Robinson via todd) |
| |
| HDFS-3490. DatanodeWebHdfsMethods throws NullPointerException if |
| NamenodeRpcAddressParam is not set. (szetszwo) |
| |
| HDFS-2797. Fix misuses of InputStream#skip in the edit log code. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-3517. TestStartup should bind ephemeral ports. (eli) |
| |
| HDFS-3522. If a namenode is in safemode, it should throw SafeModeException |
| when getBlockLocations has zero locations. (Brandon Li via szetszwo) |
| |
| HDFS-3408. BKJM : Namenode format fails, if there is no BK root. (Rakesh R via umamahesh) |
| |
| HDFS-3389. Document the BKJM usage in Namenode HA. (umamahesh and Ivan Kelly via umamahesh) |
| |
| HDFS-3531. EditLogFileOutputStream#preallocate should check for |
| incomplete writes. (Colin Patrick McCabe via eli) |
| |
| HDFS-766. Error message not clear for set space quota out of boundary |
| values. (Jon Zuanich via atm) |
| |
| HDFS-3480. Multiple SLF4J binding warning. (Vinay via eli) |
| |
| HDFS-3524. Update TestFileLengthOnClusterRestart for HDFS-3522. (Brandon |
| Li via szetszwo) |
| |
| HDFS-3559. DFSTestUtil: use Builder class to construct DFSTestUtil |
| instances. (Colin Patrick McCabe via atm) |
| |
| HDFS-3551. WebHDFS CREATE should use client location for HTTP redirection. |
| (szetszwo) |
| |
| HDFS-3157. Fix a bug in the case that the generation stamps of the stored |
| block in a namenode and the reported block from a datanode do not match. |
| (Ashish Singhi via szetszwo) |
| |
| HDFS-3575. HttpFS does not log Exception Stacktraces (brocknoland via tucu) |
| |
| HDFS-3574. Fix small race and do some cleanup in GetImageServlet (todd) |
| |
| HDFS-3581. FSPermissionChecker#checkPermission sticky bit check |
| missing range check. (eli) |
| |
| HDFS-3541. Deadlock between recovery, xceiver and packet responder. |
| (Vinay via umamahesh) |
| |
| HDFS-3428. Move DelegationTokenRenewer to common (tucu) |
| |
| HDFS-3491. HttpFs does not set permissions correctly (tucu) |
| |
| HDFS-3580. incompatible types; no instance(s) of type variable(s) V exist |
| so that V conforms to boolean compiling HttpFSServer.java with OpenJDK |
| (adi2 via tucu) |
| |
| HDFS-3603. Decouple TestHDFSTrash from TestTrash. (Jason Lowe via eli) |
| |
| HDFS-711. hdfsUtime does not handle atime = 0 or mtime = 0 correctly. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-3548. NamenodeFsck.copyBlock fails to create a Block Reader. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-3615. Two BlockTokenSecretManager findbugs warnings. (atm) |
| |
| HDFS-470. libhdfs should handle 0-length reads from FSInputStream |
| correctly. (Colin Patrick McCabe via eli) |
| |
| HDFS-3492. fix some misuses of InputStream#skip. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-3609. libhdfs: don't force the URI to look like hdfs://hostname:port. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-2966 TestNameNodeMetrics tests can fail under load. (stevel) |
| |
| HDFS-3605. Block mistakenly marked corrupt during edit log catchup |
| phase of failover. (todd and Brahma Reddy Battula via todd) |
| |
| HDFS-3690. BlockPlacementPolicyDefault incorrectly casts LOG. (eli) |
| |
| HDFS-3597. SNN fails to start after DFS upgrade. (Andy Isaacson via todd) |
| |
| HDFS-3608. fuse_dfs: detect changes in UID ticket cache. (Colin Patrick |
| McCabe via atm) |
| |
| HDFS-3709. TestStartup tests still binding to the ephemeral port. (eli) |
| |
| HDFS-3720. hdfs.h must get packaged. (Colin Patrick McCabe via atm) |
| |
| HDFS-3626. Creating file with invalid path can corrupt edit log (todd) |
| |
| HDFS-3679. fuse_dfs notrash option sets usetrash. (Conrad Meyer via suresh) |
| |
| HDFS-3732. fuse_dfs: incorrect configuration value checked for connection |
| expiry timer period. (Colin Patrick McCabe via atm) |
| |
| HDFS-3738. TestDFSClientRetries#testFailuresArePerOperation sets incorrect |
| timeout config. (atm) |
| |
| HDFS-3756. DelegationTokenFetcher creates 2 HTTP connections, the second |
| one not properly configured. (tucu) |
| |
| HDFS-3579. libhdfs: fix exception handling. (Colin Patrick McCabe via atm) |
| |
| HDFS-3754. BlockSender doesn't shutdown ReadaheadPool threads. (eli) |
| |
| HDFS-3760. primitiveCreate is a write, not a read. (Andy Isaacson via atm) |
| |
| HDFS-3710. libhdfs misuses O_RDONLY/WRONLY/RDWR. (Andy Isaacson via atm) |
| |
| HDFS-3721. hsync support broke wire compatibility. (todd and atm) |
| |
| HDFS-3758. TestFuseDFS test failing. (Colin Patrick McCabe via eli) |
| |
| HDFS-2330. In NNStorage.java, IOExceptions of stream closures can mask |
| root exceptions. (umamahesh via todd) |
| |
| HDFS-3790. test_fuse_dfs.c doesn't compile on centos 5. (Colin Patrick |
| McCabe via atm) |
| |
| HDFS-3658. Fix bugs in TestDFSClientRetries and add more tests. (szetszwo) |
| |
| HDFS-3794. WebHDFS OPEN returns the incorrect Content-Length in the HTTP |
| header when offset is specified and length is omitted. |
| (Ravi Prakash via szetszwo) |
| |
| HDFS-3048. Small race in BlockManager#close. (Andy Isaacson via eli) |
| |
| HDFS-3194. DataNode block scanner is running too frequently. |
| (Andy Isaacson via eli) |
| |
| HDFS-3808. fuse_dfs: postpone libhdfs intialization until after fork. |
| (Colin Patrick McCabe via atm) |
| |
| HDFS-3788. ByteRangeInputStream should not expect HTTP Content-Length header |
| when chunked transfer-encoding is used. (szetszwo) |
| |
| HDFS-3816. Invalidate work percentage default value should be 0.32f |
| instead of 32. (Jing Zhao via suresh) |
| |
| HDFS-3707. TestFSInputChecker: improper use of skip. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-3830. test_libhdfs_threaded: use forceNewInstance. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-3835. Long-lived 2NN cannot perform a checkpoint if security is |
| enabled and the NN restarts with outstanding delegation tokens. (atm) |
| |
| HDFS-3715. Fix TestFileCreation#testFileCreationNamenodeRestart. |
| (Andrew Whang via eli) |
| |
| HDFS-3683. Edit log replay progress indicator shows >100% complete. (Plamen |
| Jeliazkov via atm) |
| |
| HDFS-3731. Release upgrade must handle blocks being written from 1.0. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-3856. TestHDFSServerPorts failure is causing surefire fork failure. |
| (eli) |
| |
| HDFS-3860. HeartbeatManager#Monitor may wrongly hold the writelock of |
| namesystem. (Jing Zhao via atm) |
| |
| HDFS-3849. When re-loading the FSImage, we should clear the existing |
| genStamp and leases. (Colin Patrick McCabe via atm) |
| |
| HDFS-3864. NN does not update internal file mtime for OP_CLOSE when reading |
| from the edit log. (atm) |
| |
| HDFS-3837. Fix DataNode.recoverBlock findbugs warning. (eli) |
| |
| HDFS-3733. Audit logs should include WebHDFS access. (Andy Isaacson via |
| eli) |
| |
| HDFS-2686. Remove DistributedUpgrade related code. (suresh) |
| |
| HDFS-3833. TestDFSShell fails on windows due to concurrent file |
| read/write. (Brandon Li via suresh) |
| |
| HDFS-3466. Get HTTP kerberos principal from the web authentication keytab. |
| (omalley) |
| |
| HDFS-3879. Fix findbugs warning in TransferFsImage on branch-2. (eli) |
| |
| HDFS-3469. start-dfs.sh will start zkfc, but stop-dfs.sh will not stop zkfc similarly. |
| (Vinay via umamahesh) |
| |
| HDFS-1490. TransferFSImage should timeout (Dmytro Molkov and Vinay via todd) |
| |
| HDFS-3828. Block Scanner rescans blocks too frequently. |
| (Andy Isaacson via eli) |
| |
| HDFS-3895. hadoop-client must include commons-cli (tucu) |
| |
| HDFS-2757. Cannot read a local block that's being written to when |
| using the local read short circuit. (Jean-Daniel Cryans via eli) |
| |
| HDFS-3664. BlockManager race when stopping active services. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-3928. MiniDFSCluster should reset the first ExitException on shutdown. (eli) |
| |
| HDFS-3938. remove current limitations from HttpFS docs. (tucu) |
| |
| HDFS-3944. Httpfs resolveAuthority() is not resolving host correctly. (tucu) |
| |
| HDFS-3972. Trash emptier fails in secure HA cluster. (todd via eli) |
| |
| HDFS-4443. Remove a trailing '`' character from the HTML code generated by |
| NamenodeJspHelper.generateNodeData(..). (Christian Rohling via szetszwo) |
| |
| BREAKDOWN OF HDFS-3042 SUBTASKS |
| |
| HDFS-2185. HDFS portion of ZK-based FailoverController (todd) |
| |
| HDFS-3200. Scope all ZKFC configurations by nameservice (todd) |
| |
| HDFS-3223. add zkfc to hadoop-daemon.sh script (todd) |
| |
| HDFS-3261. TestHASafeMode fails on HDFS-3042 branch (todd) |
| |
| HDFS-3159. Document NN auto-failover setup and configuration (todd) |
| |
| HDFS-3412. Fix findbugs warnings in auto-HA branch (todd) |
| |
| HDFS-3432. TestDFSZKFailoverController tries to fail over too early (todd) |
| |
| HDFS-3902. TestDatanodeBlockScanner#testBlockCorruptionPolicy is broken. |
| (Andy Isaacson via eli) |
| |
| Release 2.0.0-alpha - 05-23-2012 |
| |
| INCOMPATIBLE CHANGES |
| |
| HDFS-2676. Remove Avro RPC. (suresh) |
| |
| HDFS-2303. Unbundle jsvc. (Roman Shaposhnik and Mingjie Lai via eli) |
| |
| HDFS-3137. Bump LAST_UPGRADABLE_LAYOUT_VERSION to -16. (eli) |
| |
| HDFS-3138. Move DatanodeInfo#ipcPort to DatanodeID. (eli) |
| |
| HDFS-3164. Move DatanodeInfo#hostName to DatanodeID. (eli) |
| |
| NEW FEATURES |
| |
| HDFS-2978. The NameNode should expose name dir statuses via JMX. (atm) |
| |
| HDFS-395. DFS Scalability: Incremental block reports. (Tomasz Nykiel |
| via hairong) |
| |
| HDFS-2517. Add protobuf service for JounralProtocol. (suresh) |
| |
| HDFS-2518. Add protobuf service for NamenodeProtocol. (suresh) |
| |
| HDFS-2520. Add protobuf service for InterDatanodeProtocol. (suresh) |
| |
| HDFS-2519. Add protobuf service for DatanodeProtocol. (suresh) |
| |
| HDFS-2581. Implement protobuf service for JournalProtocol. (suresh) |
| |
| HDFS-2618. Implement protobuf service for NamenodeProtocol. (suresh) |
| |
| HDFS-2629. Implement protobuf service for InterDatanodeProtocol. (suresh) |
| |
| HDFS-2636. Implement protobuf service for ClientDatanodeProtocol. (suresh) |
| |
| HDFS-2642. Protobuf translators for DatanodeProtocol. (jitendra) |
| |
| HDFS-2647. Used protobuf based RPC for InterDatanodeProtocol, |
| ClientDatanodeProtocol, JournalProtocol, NamenodeProtocol. (suresh) |
| |
| HDFS-2661. Enable protobuf RPC for DatanodeProtocol. (jitendra) |
| |
| HDFS-2697. Move RefreshAuthPolicy, RefreshUserMappings, GetUserMappings |
| protocol to protocol buffers. (jitendra) |
| |
| HDFS-2880. Protobuf changes in DatanodeProtocol to add multiple storages. |
| (suresh) |
| |
| HDFS-2899. Service protocol changes in DatanodeProtocol to add multiple |
| storages. (suresh) |
| |
| HDFS-2430. The number of failed or low-resource volumes the NN can tolerate |
| should be configurable. (atm) |
| |
| HDFS-1623. High Availability Framework for HDFS NN. Contributed by Todd |
| Lipcon, Aaron T. Myers, Eli Collins, Uma Maheswara Rao G, Bikas Saha, |
| Suresh Srinivas, Jitendra Nath Pandey, Hari Mankude, Brandon Li, Sanjay |
| Radia, Mingjie Lai, and Gregory Chanan |
| |
| HDFS-2941. Add an administrative command to download a copy of the fsimage |
| from the NN. (atm) |
| |
| HDFS-2413. Add an API DistributedFileSystem.isInSafeMode() and change |
| DistributedFileSystem to @InterfaceAudience.LimitedPrivate. |
| (harsh via szetszwo) |
| |
| HDFS-3167. CLI-based driver for MiniDFSCluster. (Henry Robinson via atm) |
| |
| HDFS-3148. The client should be able to use multiple local interfaces |
| for data transfer. (eli) |
| |
| HDFS-3000. Add a public API for setting quotas. (atm) |
| |
| HDFS-3102. Add CLI tool to initialize the shared-edits dir. (atm) |
| |
| HDFS-3004. Implement Recovery Mode. (Colin Patrick McCabe via eli) |
| |
| HDFS-3282. Add HdfsDataInputStream as a public API. (umamahesh) |
| |
| HDFS-3298. Add HdfsDataOutputStream as a public API. (szetszwo) |
| |
| HDFS-234. Integration with BookKeeper logging system. (Ivan Kelly |
| via umamahesh) |
| |
| IMPROVEMENTS |
| |
| HDFS-2018. Move all journal stream management code into one place. |
| (Ivan Kelly via jitendra) |
| |
| HDFS-2223. Untangle depencencies between NN components (todd) |
| |
| HDFS-2351. Change Namenode and Datanode to register each of their protocols |
| seperately (sanjay) |
| |
| HDFS-2337. DFSClient shouldn't keep multiple RPC proxy references (atm) |
| |
| HDFS-2181. Separate HDFS Client wire protocol data types (sanjay) |
| |
| HDFS-2459. Separate datatypes for Journal Protocol. (suresh) |
| |
| HDFS-2480. Separate datatypes for NamenodeProtocol. (suresh) |
| |
| HDFS-2489. Move Finalize and Register to separate file out of |
| DatanodeCommand.java. (suresh) |
| |
| HDFS-2488. Separate datatypes for InterDatanodeProtocol. (suresh) |
| |
| HDFS-2496. Separate datatypes for DatanodeProtocol. (suresh) |
| |
| HDFS-2479. HDFS Client Data Types in Protocol Buffers (sanjay) |
| |
| HADOOP-7862. Hdfs changes to work with HADOOP-7862: Move the support for |
| multiple protocols to lower layer so that Writable, PB and Avro can all |
| use it. (sanjay) |
| |
| HDFS-2597. ClientNameNodeProtocol in Protocol Buffers. (sanjay) |
| |
| HDFS-2651. ClientNameNodeProtocol Translators for Protocol Buffers. (sanjay) |
| |
| HDFS-2650. Replace @inheritDoc with @Override. (Hari Mankude via suresh). |
| |
| HDFS-2669. Enable protobuf rpc for ClientNamenodeProtocol. (sanjay) |
| |
| HDFS-2801. Provide a method in client side translators to check for a |
| methods supported in underlying protocol. (jitendra) |
| |
| HDFS-2895. Remove Writable wire protocol types and translators to |
| complete transition to protocol buffers. (suresh) |
| |
| HDFS-2992. Edit log failure trace should include transaction ID of |
| error. (Colin Patrick McCabe via eli) |
| |
| HDFS-3030. Remove getProtocolVersion and getProtocolSignature from |
| translators. (jitendra) |
| |
| HDFS-2158. Add JournalSet to manage the set of journals. (jitendra) |
| |
| HDFS-2334. Add Closeable to JournalManager. (Ivan Kelly via jitendra) |
| |
| HDFS-1580. Add interface for generic Write Ahead Logging mechanisms. |
| (Ivan Kelly via jitendra) |
| |
| HDFS-3060. Bump TestDistributedUpgrade#testDistributedUpgrade timeout (eli) |
| |
| HDFS-2410. Further cleanup of hardcoded configuration keys and values. |
| (suresh) |
| |
| HDFS-2878. Fix TestBlockRecovery and move it back into main test directory. |
| (todd) |
| |
| HDFS-3003. Remove getHostPortString() from NameNode, replace it with |
| NetUtils.getHostPortString(). (Brandon Li via atm) |
| |
| HDFS-3014. FSEditLogOp and its subclasses should have toString() method. |
| (Sho Shimauchi via atm) |
| |
| HDFS-3021. Use generic type to declare FSDatasetInterface. (szetszwo) |
| |
| HDFS-3056. Add a new interface RollingLogs for DataBlockScanner logging. |
| (szetszwo) |
| |
| HDFS-2731. Add command to bootstrap the Standby Node's name directories |
| from the Active NameNode. (todd) |
| |
| HDFS-3082. Clean up FSDatasetInterface and change DataNode.data to package |
| private. (szetszwo) |
| |
| HDFS-3057. httpfs and hdfs launcher scripts should honor CATALINA_HOME |
| and HADOOP_LIBEXEC_DIR (rvs via tucu) |
| |
| HDFS-3088. Move FSDatasetInterface inner classes to a package. (szetszwo) |
| |
| HDFS-3111. Missing license headers in trunk. (umamahesh) |
| |
| HDFS-3091. Update the usage limitations of ReplaceDatanodeOnFailure policy in |
| the config description for the smaller clusters. (szetszwo via umamahesh) |
| |
| HDFS-3105. Add DatanodeStorage information to block recovery. (szetszwo) |
| |
| HDFS-3086. Change Datanode not to send storage list in registration. |
| (szetszwo) |
| |
| HDFS-309. FSEditLog should log progress during replay. (Sho Shimauchi |
| via todd) |
| |
| HDFS-3044. fsck move should be non-destructive by default. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-3071. haadmin failover command does not provide enough detail when |
| target NN is not ready to be active. (todd) |
| |
| HDFS-3089. Move FSDatasetInterface and the related classes to a package. |
| (szetszwo) |
| |
| HDFS-3129. NetworkTopology: add test that getLeaf should check for |
| invalid topologies. (Colin Patrick McCabe via eli) |
| |
| HDFS-3155. Clean up FSDataset implemenation related code. (szetszwo) |
| |
| HDFS-3158. LiveNodes member of NameNodeMXBean should list non-DFS used |
| space and capacity per DN. (atm) |
| |
| HDFS-3172. dfs.upgrade.permission is dead code. (eli) |
| |
| HDFS-3171. The DatanodeID "name" field is overloaded. (eli) |
| |
| HDFS-3144. Refactor DatanodeID#getName by use. (eli) |
| |
| HDFS-3130. Move fsdataset implementation to a package. (szetszwo) |
| |
| HDFS-3120. Enable hsync and hflush by default. (eli) |
| |
| HDFS-3187. Upgrade guava to 11.0.2 (todd) |
| |
| HDFS-3168. Remove unnecessary "throw IOException" and change fields to |
| final in FSNamesystem and BlockManager. (szetszwo) |
| |
| HDFS-2564. Cleanup unnecessary exceptions thrown and unnecessary casts. |
| (Hari Mankude via eli) |
| |
| HDFS-3084. FenceMethod.tryFence() and ShellCommandFencer should pass |
| namenodeId as well as host:port (todd) |
| |
| HDFS-3050. rework OEV to share more code with the NameNode. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-3204. Minor modification to JournalProtocol.proto to make |
| it generic. (suresh) |
| |
| HDFS-3226. Allow GetConf tool to print arbitrary keys (todd) |
| |
| HDFS-3240. Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG |
| (todd) |
| |
| HDFS-3238. ServerCommand and friends don't need to be writables. (eli) |
| |
| HDFS-3094. add -nonInteractive and -force option to namenode -format |
| command (Arpit Gupta via todd) |
| |
| HDFS-3244. Remove dead writable code from hdfs/protocol. (eli) |
| |
| HDFS-3247. Improve bootstrapStandby behavior when original NN is not active |
| (todd) |
| |
| HDFS-3249. Use ToolRunner.confirmPrompt in NameNode (todd) |
| |
| HDFS-3179. Improve the exception message thrown by DataStreamer when |
| it failed to add a datanode. (szetszwo) |
| |
| HDFS-2983. Relax the build version check to permit rolling upgrades within a release. (atm) |
| |
| HDFS-3259. NameNode#initializeSharedEdits should populate shared edits dir |
| with edit log segments. (atm) |
| |
| HDFS-2708. Stats for the # of blocks per DN. (atm) |
| |
| HDFS-3279. Move the FSEditLog constructor with @VisibleForTesting to |
| TestEditLog. (Arpit Gupta via szetszwo) |
| |
| HDFS-3294. Fix code indentation in NamenodeWebHdfsMethods and |
| DatanodeWebHdfsMethods. (szetszwo) |
| |
| HDFS-3263. HttpFS should read HDFS config from Hadoop site.xml files (tucu) |
| |
| HDFS-3206. Miscellaneous xml cleanups for OEV. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-3169. TestFsck should test multiple -move operations in a row. |
| (Colin Patrick McCabe via eli) |
| |
| HDFS-3258. Test for HADOOP-8144 (pseudoSortByDistance in |
| NetworkTopology for first rack local node). (Junping Du via eli) |
| |
| HDFS-3322. Use HdfsDataInputStream and HdfsDataOutputStream in Hdfs. |
| (szetszwo) |
| |
| HDFS-3339. Change INode to package private. (John George via szetszwo) |
| |
| HDFS-3303. Remove Writable implementation from RemoteEditLogManifest. |
| (Brandon Li via szetszwo) |
| |
| HDFS-2617. Replaced Kerberized SSL for image transfer and fsck |
| with SPNEGO-based solution. (jghoman, omalley, tucu, and atm via eli) |
| |
| HDFS-3365. Enable users to disable socket caching in DFS client |
| configuration (todd) |
| |
| HDFS-3375. Put client name in DataXceiver thread name for readBlock |
| and keepalive (todd) |
| |
| HDFS-3363. Define BlockCollection and MutableBlockCollection interfaces |
| so that INodeFile and INodeFileUnderConstruction do not have to be used in |
| block management. (John George via szetszwo) |
| |
| HDFS-3211. Add fence(..) and replace NamenodeRegistration with JournalInfo |
| and epoch in JournalProtocol. (suresh via szetszwo) |
| |
| HADOOP-8285 HDFS changes for Use ProtoBuf for RpcPayLoadHeader (sanjay radia) |
| |
| HDFS-3418. Rename BlockWithLocationsProto datanodeIDs field to storageIDs. |
| (eli) |
| |
| OPTIMIZATIONS |
| |
| HDFS-2477. Optimize computing the diff between a block report and the |
| namenode state. (Tomasz Nykiel via hairong) |
| |
| HDFS-2495. Increase granularity of write operations in ReplicationMonitor |
| thus reducing contention for write lock. (Tomasz Nykiel via hairong) |
| |
| HDFS-2476. More CPU efficient data structure for under-replicated, |
| over-replicated, and invalidated blocks. (Tomasz Nykiel via todd) |
| |
| HDFS-3036. Remove unused method DFSUtil#isDefaultNamenodeAddress. (atm) |
| |
| HDFS-3378. Remove DFS_NAMENODE_SECONDARY_HTTPS_PORT_KEY and DEFAULT. (eli) |
| |
| BUG FIXES |
| |
| HDFS-2481. Unknown protocol: org.apache.hadoop.hdfs.protocol.ClientProtocol. |
| (sanjay) |
| |
| HDFS-2497. Fix TestBackupNode failure. (suresh) |
| |
| HDFS-2499. RPC client is created incorrectly introduced in HDFS-2459. |
| (suresh) |
| |
| HDFS-2526. (Client)NamenodeProtocolTranslatorR23 do not need to keep a |
| reference to rpcProxyWithoutRetry (atm) |
| |
| HDFS-2532. TestDfsOverAvroRpc timing out in trunk (Uma Maheswara Rao G |
| via todd) |
| |
| HDFS-2666. Fix TestBackupNode failure. (suresh) |
| |
| HDFS-2663. Optional protobuf parameters are not handled correctly. (suresh) |
| |
| HDFS-2694. Removal of Avro broke non-PB NN services. (atm) |
| |
| HDFS-2687. Tests failing with ClassCastException post protobuf RPC |
| changes. (suresh) |
| |
| HDFS-2700. Fix failing TestDataNodeMultipleRegistrations in trunk |
| (Uma Maheswara Rao G via todd) |
| |
| HDFS-2739. SecondaryNameNode doesn't start up. (jitendra) |
| |
| HDFS-2768. BackupNode stop can not close proxy connections because |
| it is not a proxy instance. (Uma Maheswara Rao G via eli) |
| |
| HDFS-2968. Protocol translator for BlockRecoveryCommand broken when |
| multiple blocks need recovery. (todd) |
| |
| HDFS-3020. Fix editlog to automatically sync when buffer is full. (todd) |
| |
| HDFS-3038. Add FSEditLog.metrics to findbugs exclude list. (todd via atm) |
| |
| HDFS-2188. Make FSEditLog create its journals from a list of URIs rather |
| than NNStorage. (Ivan Kelly via jitendra) |
| |
| HDFS-1765. Block Replication should respect under-replication |
| block priority. (Uma Maheswara Rao G via eli) |
| |
| HDFS-2285. BackupNode should reject requests to modify namespace. |
| (shv and Uma Maheswara Rao) |
| |
| HDFS-2764. TestBackupNode is racy. (atm) |
| |
| HDFS-3093. Fix bug where namenode -format interpreted the -force flag in |
| reverse. (todd) |
| |
| HDFS-3005. FSVolume.decDfsUsed(..) should be synchronized. (szetszwo) |
| |
| HDFS-3099. SecondaryNameNode does not properly initialize metrics system. |
| (atm) |
| |
| HDFS-3062. Fix bug which prevented MR job submission from creating |
| delegation tokens on an HA cluster. (Mingjie Lai via todd) |
| |
| HDFS-3083. Cannot run an MR job with HA and security enabled when |
| second-listed NN active. (atm) |
| |
| HDFS-3100. In BlockSender, throw an exception when it needs to verify |
| checksum but the meta data does not exist. (Brandon Li via szetszwo) |
| |
| HDFS-3132. Fix findbugs warning on HDFS trunk. (todd) |
| |
| HDFS-3156. TestDFSHAAdmin is failing post HADOOP-8202. (atm) |
| |
| HDFS-3143. TestGetBlocks.testGetBlocks is failing. (Arpit Gupta via atm) |
| |
| HDFS-3142. TestHDFSCLI.testAll is failing. (Brandon Li via atm) |
| |
| HDFS-3070. HDFS balancer doesn't ensure that hdfs-site.xml is loaded. (atm) |
| |
| HDFS-2995. start-dfs.sh should only start the 2NN for namenodes |
| with dfs.namenode.secondary.http-address configured. (eli) |
| |
| HDFS-3174. Fix assert in TestPendingDataNodeMessages. (eli) |
| |
| HDFS-3199. TestValidateConfigurationSettings is failing. (todd via eli) |
| |
| HDFS-3202. NamespaceInfo PB translation drops build version. (atm) |
| |
| HDFS-3109. Remove hsqldf exclusions from pom.xml. (Ravi Prakash |
| via suresh) |
| |
| HDFS-3210. JsonUtil#toJsonMap for for a DatanodeInfo should use |
| "ipAddr" instead of "name". (eli) |
| |
| HDFS-3208. Bogus entries in hosts files are incorrectly displayed |
| in the report. (eli) |
| |
| HDFS-3136. Remove SLF4J dependency as HDFS does not need it to fix |
| unnecessary warnings. (Jason Lowe via suresh) |
| |
| HDFS-3214. InterDatanodeProtocolServerSideTranslatorPB doesn't handle |
| null response from initReplicaRecovery (todd) |
| |
| HDFS-3119. Overreplicated block is not deleted even after the replication |
| factor is reduced after sync follwed by closing that file. (Ashish Singhi |
| via umamahesh) |
| |
| HDFS-3234. Accidentally left log message in GetConf after HDFS-3226 (todd) |
| |
| HDFS-3236. NameNode does not initialize generic conf keys when started |
| with -initializeSharedEditsDir (atm) |
| |
| HDFS-3248. bootstrapStandby repeated twice in hdfs namenode usage message |
| (Colin Patrick McCabe via todd) |
| |
| HDFS-2696. Fix the fuse-fds build. (Bruno Mahé via eli) |
| |
| HDFS-3254. Branch-2 build broken due to wrong version number in |
| fuse-dfs' pom.xml. (Anupam Seth via eli) |
| |
| HDFS-3260. TestDatanodeRegistration should set minimum DN version in |
| addition to minimum NN version. (atm) |
| |
| HDFS-3255. HA DFS returns wrong token service (Daryn Sharp via todd) |
| |
| HDFS-3256. HDFS considers blocks under-replicated if topology script is |
| configured with only 1 rack. (atm) |
| |
| HDFS-2799. Trim fs.checkpoint.dir values. (Amith D K via eli) |
| |
| HDFS-2765. TestNameEditsConfigs is incorrectly swallowing IOE. (atm) |
| |
| HDFS-3280. DFSOutputStream.sync should not be synchronized (todd) |
| |
| HDFS-3268. FileContext API mishandles token service and incompatible with |
| HA (Daryn Sharp via todd) |
| |
| HDFS-3284. bootstrapStandby fails in secure cluster (todd) |
| |
| HDFS-3165. HDFS Balancer scripts are refering to wrong path of |
| hadoop-daemon.sh (Amith D K via eli) |
| |
| HDFS-891. DataNode no longer needs to check for dfs.network.script. |
| (harsh via eli) |
| |
| HDFS-3305. GetImageServlet should consider SBN a valid requestor in a |
| secure HA setup. (atm) |
| |
| HDFS-3314. HttpFS operation for getHomeDirectory is incorrect. (tucu) |
| |
| HDFS-3319. Change DFSOutputStream to not to start a thread in constructors. |
| (szetszwo) |
| |
| HDFS-3222. DFSInputStream#openInfo should not silently get the length as 0 |
| when locations length is zero for last partial block. (umamahesh) |
| |
| HDFS-3181. Fix a test case in TestLeaseRecovery2. (szetszwo) |
| |
| HDFS-3309. HttpFS (Hoop) chmod not supporting octal and sticky bit |
| permissions. (tucu) |
| |
| HDFS-3326. Append enabled log message uses the wrong variable. |
| (Matthew Jacobs via eli) |
| |
| HDFS-3275. Skip format for non-file based directories. |
| (Amith D K via umamahesh) |
| |
| HDFS-3286. When the threshold value for balancer is zero, unexpected output is displayed. |
| (Ashish Singhi via umamahesh) |
| |
| HDFS-3336. hdfs launcher script will be better off not special casing |
| namenode command with regards to hadoop.security.logger (rvs via tucu) |
| |
| HDFS-3330. If GetImageServlet throws an Error or RTE, response should not |
| have HTTP "OK" status. (todd) |
| |
| HDFS-3351. NameNode#initializeGenericKeys should always set fs.defaultFS |
| regardless of whether HA or Federation is enabled. (atm) |
| |
| HDFS-3332. NullPointerException in DN when directoryscanner is trying to |
| report bad blocks. (Amith D K via umamahesh) |
| |
| HDFS-3359. DFSClient.close should close cached sockets. (todd) |
| |
| HDFS-3350. In INode, add final to compareTo(..), equals(..) and hashCode(), |
| and remove synchronized from updatePermissionStatus(..). (szetszwo) |
| |
| HDFS-3357. DataXceiver reads from client socket with incorrect/no timeout |
| (todd) |
| |
| HDFS-3376. DFSClient fails to make connection to DN if there are many |
| unusable cached sockets (todd) |
| |
| HDFS-3328. NPE in DataNode.getIpcPort. (eli) |
| |
| HDFS-3396. FUSE build fails on Ubuntu 12.04. (Colin Patrick McCabe via eli) |
| |
| HDFS-3395. NN doesn't start with HA+security enabled and HTTP address set to 0.0.0.0. (atm) |
| |
| HDFS-3026. HA: Handle failure during HA state transition. (atm) |
| |
| HDFS-860. fuse-dfs truncate behavior causes issues with scp. |
| (Brian Bockelman via eli) |
| |
| BREAKDOWN OF HDFS-1623 SUBTASKS |
| |
| HDFS-2179. Add fencing framework and mechanisms for NameNode HA. (todd) |
| |
| HDFS-1974. Introduce active and standy states to the namenode. (suresh) |
| |
| HDFS-2407. getServerDefaults and getStats don't check operation category (atm) |
| |
| HDFS-1973. HA: HDFS clients must handle namenode failover and switch over to |
| the new active namenode. (atm) |
| |
| HDFS-2301. Start/stop appropriate namenode services when transition to active |
| and standby states. (suresh) |
| |
| HDFS-2231. Configuration changes for HA namenode. (suresh) |
| |
| HDFS-2418. Change ConfiguredFailoverProxyProvider to take advantage of |
| HDFS-2231. (atm) |
| |
| HDFS-2393. Mark appropriate methods of ClientProtocol with the idempotent |
| annotation. (atm) |
| |
| HDFS-2523. Small NN fixes to include HAServiceProtocol and prevent NPE on |
| shutdown. (todd) |
| |
| HDFS-2577. NN fails to start since it tries to start secret manager in |
| safemode. (todd) |
| |
| HDFS-2582. Scope dfs.ha.namenodes config by nameservice (todd) |
| |
| HDFS-2591. MiniDFSCluster support to mix and match federation with HA (todd) |
| |
| HDFS-1975. Support for sharing the namenode state from active to standby. |
| (jitendra, atm, todd) |
| |
| HDFS-1971. Send block report from datanode to both active and standby |
| namenodes. (sanjay, todd via suresh) |
| |
| HDFS-2616. Change DatanodeProtocol#sendHeartbeat() to return HeartbeatResponse. |
| (suresh) |
| |
| HDFS-2622. Fix TestDFSUpgrade in HA branch. (todd) |
| |
| HDFS-2612. Handle refreshNameNodes in federated HA clusters (todd) |
| |
| HDFS-2623. Add test case for hot standby capability (todd) |
| |
| HDFS-2626. BPOfferService.verifyAndSetNamespaceInfo needs to be synchronized |
| (todd) |
| |
| HDFS-2624. ConfiguredFailoverProxyProvider doesn't correctly stop |
| ProtocolTranslators (todd) |
| |
| HDFS-2625. TestDfsOverAvroRpc failing after introduction of HeartbeatResponse |
| type (todd) |
| |
| HDFS-2627. Determine DN's view of which NN is active based on heartbeat |
| responses (todd) |
| |
| HDFS-2634. Standby needs to ingest latest edit logs before transitioning to |
| active (todd) |
| |
| HDFS-2671. NN should throw StandbyException in response to RPCs in STANDBY |
| state (todd) |
| |
| HDFS-2680. DFSClient should construct failover proxy with exponential backoff |
| (todd) |
| |
| HDFS-2683. Authority-based lookup of proxy provider fails if path becomes |
| canonicalized (todd) |
| |
| HDFS-2689. HA: BookKeeperEditLogInputStream doesn't implement isInProgress() |
| (atm) |
| |
| HDFS-2602. NN should log newly-allocated blocks without losing BlockInfo (atm) |
| |
| HDFS-2667. Fix transition from active to standby (todd) |
| |
| HDFS-2684. Fix up some failing unit tests on HA branch (todd) |
| |
| HDFS-2679. Add interface to query current state to HAServiceProtocol (eli via |
| todd) |
| |
| HDFS-2677. Web UI should indicate the NN state. (eli via todd) |
| |
| HDFS-2678. When a FailoverProxyProvider is used, DFSClient should not retry |
| connection ten times before failing over (atm via todd) |
| |
| HDFS-2682. When a FailoverProxyProvider is used, Client should not retry for 45 |
| times if it is timing out to connect to server. (Uma Maheswara Rao G via todd) |
| |
| HDFS-2693. Fix synchronization issues around state transition (todd) |
| |
| HDFS-1972. Fencing mechanism for block invalidations and replications (todd) |
| |
| HDFS-2714. Fix test cases which use standalone FSNamesystems (todd) |
| |
| HDFS-2692. Fix bugs related to failover from/into safe mode. (todd) |
| |
| HDFS-2716. Configuration needs to allow different dfs.http.addresses for each |
| HA NN (todd) |
| |
| HDFS-2720. Fix MiniDFSCluster HA support to work properly on Windows. (Uma |
| Maheswara Rao G via todd) |
| |
| HDFS-2291. Allow the StandbyNode to make checkpoints in an HA setup. (todd) |
| |
| HDFS-2709. Appropriately handle error conditions in EditLogTailer (atm via |
| todd) |
| |
| HDFS-2730. Refactor shared HA-related test code into HATestUtil class (todd) |
| |
| HDFS-2762. Fix TestCheckpoint timing out on HA branch. (Uma Maheswara Rao G via |
| todd) |
| |
| HDFS-2724. NN web UI can throw NPE after startup, before standby state is |
| entered. (todd) |
| |
| HDFS-2753. Fix standby getting stuck in safemode when blocks are written while |
| SBN is down. (Hari Mankude and todd via todd) |
| |
| HDFS-2773. Reading edit logs from an earlier version should not leave blocks in |
| under-construction state. (todd) |
| |
| HDFS-2775. Fix TestStandbyCheckpoints.testBothNodesInStandbyState failing |
| intermittently. (todd) |
| |
| HDFS-2766. Test for case where standby partially reads log and then performs |
| checkpoint. (atm) |
| |
| HDFS-2738. FSEditLog.selectinputStreams is reading through in-progress streams |
| even when non-in-progress are requested. (atm) |
| |
| HDFS-2789. TestHAAdmin.testFailover is failing (eli) |
| |
| HDFS-2747. Entering safe mode after starting SBN can NPE. (Uma Maheswara Rao G |
| via todd) |
| |
| HDFS-2772. On transition to active, standby should not swallow ELIE. (atm) |
| |
| HDFS-2767. ConfiguredFailoverProxyProvider should support NameNodeProtocol. |
| (Uma Maheswara Rao G via todd) |
| |
| HDFS-2795. Standby NN takes a long time to recover from a dead DN starting up. |
| (todd) |
| |
| HDFS-2592. Balancer support for HA namenodes. (Uma Maheswara Rao G via todd) |
| |
| HDFS-2367. Enable the configuration of multiple HA cluster addresses. (atm) |
| |
| HDFS-2812. When becoming active, the NN should treat all leases as freshly |
| renewed. (todd) |
| |
| HDFS-2737. Automatically trigger log rolls periodically on the active NN. (todd |
| and atm) |
| |
| HDFS-2820. Add a simple sanity check for HA config (todd) |
| |
| HDFS-2688. Add tests for quota tracking in an HA cluster. (todd) |
| |
| HDFS-2804. Should not mark blocks under-replicated when exiting safemode (todd) |
| |
| HDFS-2807. Service level authorizartion for HAServiceProtocol. (jitendra) |
| |
| HDFS-2809. Add test to verify that delegation tokens are honored after |
| failover. (jitendra and atm) |
| |
| HDFS-2838. NPE in FSNamesystem when in safe mode. (Gregory Chanan via eli) |
| |
| HDFS-2805. Add a test for a federated cluster with HA NNs. (Brandon Li via |
| jitendra) |
| |
| HDFS-2841. HAAdmin does not work if security is enabled. (atm) |
| |
| HDFS-2691. Fixes for pipeline recovery in an HA cluster: report RBW replicas |
| immediately upon pipeline creation. (todd) |
| |
| HDFS-2824. Fix failover when prior NN died just after creating an edit log |
| segment. (atm via todd) |
| |
| HDFS-2853. HA: NN fails to start if the shared edits dir is marked required |
| (atm via eli) |
| |
| HDFS-2845. SBN should not allow browsing of the file system via web UI. (Bikas |
| Saha via atm) |
| |
| HDFS-2742. HA: observed dataloss in replication stress test. (todd via eli) |
| |
| HDFS-2870. Fix log level for block debug info in processMisReplicatedBlocks |
| (todd) |
| |
| HDFS-2859. LOCAL_ADDRESS_MATCHER.match has NPE when called from |
| DFSUtil.getSuffixIDs when the host is incorrect (Bikas Saha via todd) |
| |
| HDFS-2861. checkpointing should verify that the dfs.http.address has been |
| configured to a non-loopback for peer NN (todd) |
| |
| HDFS-2860. TestDFSRollback#testRollback is failing. (atm) |
| |
| HDFS-2769. HA: When HA is enabled with a shared edits dir, that dir should be |
| marked required. (atm via eli) |
| |
| HDFS-2863. Failures observed if dfs.edits.dir and shared.edits.dir have same |
| directories. (Bikas Saha via atm) |
| |
| HDFS-2874. Edit log should log to shared dirs before local dirs. (todd) |
| |
| HDFS-2890. DFSUtil#getSuffixIDs should skip unset configurations. (atm) |
| |
| HDFS-2792. Make fsck work. (atm) |
| |
| HDFS-2808. HA: haadmin should use namenode ids. (eli) |
| |
| HDFS-2819. Document new HA-related configs in hdfs-default.xml. (eli) |
| |
| HDFS-2752. HA: exit if multiple shared dirs are configured. (eli) |
| |
| HDFS-2894. HA: automatically determine the nameservice Id if only one |
| nameservice is configured. (eli) |
| |
| HDFS-2733. Document HA configuration and CLI. (atm) |
| |
| HDFS-2794. Active NN may purge edit log files before standby NN has a chance to |
| read them (todd) |
| |
| HDFS-2901. Improvements for SBN web UI - not show under-replicated/missing |
| blocks. (Brandon Li via jitendra) |
| |
| HDFS-2905. HA: Standby NN NPE when shared edits dir is deleted. (Bikas Saha via |
| jitendra) |
| |
| HDFS-2579. Starting delegation token manager during safemode fails. (todd) |
| |
| HDFS-2510. Add HA-related metrics. (atm) |
| |
| HDFS-2924. Standby checkpointing fails to authenticate in secure cluster. |
| (todd) |
| |
| HDFS-2915. HA: TestFailureOfSharedDir.testFailureOfSharedDir() has race |
| condition. (Bikas Saha via jitendra) |
| |
| HDFS-2912. Namenode not shutting down when shared edits dir is inaccessible. |
| (Bikas Saha via atm) |
| |
| HDFS-2917. HA: haadmin should not work if run by regular user (eli) |
| |
| HDFS-2939. TestHAStateTransitions fails on Windows. (Uma Maheswara Rao G via |
| atm) |
| |
| HDFS-2947. On startup NN throws an NPE in the metrics system. (atm) |
| |
| HDFS-2942. TestActiveStandbyElectorRealZK fails if build dir does not exist. |
| (atm) |
| |
| HDFS-2948. NN throws NPE during shutdown if it fails to startup (todd) |
| |
| HDFS-2909. HA: Inaccessible shared edits dir not getting removed from FSImage |
| storage dirs upon error. (Bikas Saha via jitendra) |
| |
| HDFS-2934. Allow configs to be scoped to all NNs in the nameservice. (todd) |
| |
| HDFS-2935. Shared edits dir property should be suffixed with nameservice and |
| namenodeID (todd) |
| |
| HDFS-2928. ConfiguredFailoverProxyProvider should not create a NameNode proxy |
| with an underlying retry proxy. (Uma Maheswara Rao G via atm) |
| |
| HDFS-2955. IllegalStateException during standby startup in getCurSegmentTxId. |
| (Hari Mankude via atm) |
| |
| HDFS-2937. TestDFSHAAdmin needs tests with MiniDFSCluster. (Brandon Li via |
| suresh) |
| |
| HDFS-2586. Add protobuf service and implementation for HAServiceProtocol. |
| (suresh via atm) |
| |
| HDFS-2952. NN should not start with upgrade option or with a pending an |
| unfinalized upgrade. (atm) |
| |
| HDFS-2974. MiniDFSCluster does not delete standby NN name dirs during format. |
| (atm) |
| |
| HDFS-2929. Stress test and fixes for block synchronization (todd) |
| |
| HDFS-2972. Small optimization building incremental block report (todd) |
| |
| HDFS-2973. Re-enable NO_ACK optimization for block deletion. (todd) |
| |
| HDFS-2922. HA: close out operation categories (eli) |
| |
| HDFS-2993. HA: BackupNode#checkOperation should permit CHECKPOINT operations |
| (eli) |
| |
| HDFS-2904. Client support for getting delegation tokens. (todd) |
| |
| HDFS-3013. HA: NameNode format doesn't pick up |
| dfs.namenode.name.dir.NameServiceId configuration (Mingjie Lai via todd) |
| |
| HDFS-3019. Fix silent failure of TestEditLogJournalFailures (todd) |
| |
| HDFS-2958. Sweep for remaining proxy construction which doesn't go through |
| failover path. (atm) |
| |
| HDFS-2920. fix remaining TODO items. (atm and todd) |
| |
| HDFS-3027. Implement a simple NN health check. (atm) |
| |
| HDFS-3023. Optimize entries in edits log for persistBlocks call. (todd) |
| |
| HDFS-2979. Balancer should use logical uri for creating failover proxy with HA |
| enabled. (atm) |
| |
| HDFS-3035. Fix failure of TestFileAppendRestart due to OP_UPDATE_BLOCKS (todd) |
| |
| HDFS-3039. Address findbugs and javadoc warnings on branch. (todd via atm) |
| |
| Release 0.23.10 - UNRELEASED |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| HDFS-5010. Reduce the frequency of getCurrentUser() calls from namenode |
| (kihwal) |
| |
| HDFS-5346. Avoid unnecessary call to getNumLiveDataNodes() for each block |
| during IBR processing (Ravi Prakash via kihwal) |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| HDFS-4998. TestUnderReplicatedBlocks fails intermittently (kihwal) |
| |
| HDFS-4329. DFSShell issues with directories with spaces in name (Cristina |
| L. Abad via jeagles) |
| |
| HDFS-5526. Datanode cannot roll back to previous layout version (kihwal) |
| |
| HDFS-5557. Write pipeline recovery for the last packet in the block may |
| cause rejection of valid replicas. (kihwal) |
| |
| HDFS-5558. LeaseManager monitor thread can crash if the last block is |
| complete but another block is not. (kihwal) |
| |
| Release 0.23.9 - 2013-07-08 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| HDFS-4867. metaSave NPEs when there are invalid blocks in repl queue. |
| (Plamen Jeliazkov and Ravi Prakash via shv) |
| |
| HDFS-4862. SafeModeInfo.isManual() returns true when resources are low even |
| if it wasn't entered into manually (Ravi Prakash via kihwal) |
| |
| HDFS-4832. Namenode doesn't change the number of missing blocks in |
| safemode when DNs rejoin or leave (Ravi Prakash via kihwal) |
| |
| HDFS-4878. On Remove Block, block is not removed from neededReplications |
| queue. (Tao Luo via shv) |
| |
| HDFS-4205. fsck fails with symlinks. (jlowe) |
| |
| Release 0.23.8 - 2013-06-05 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| HDFS-4714. Log short messages in Namenode RPC server for exceptions |
| meant for clients. (kihwal) |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| HDFS-4477. Secondary namenode may retain old tokens (daryn via kihwal) |
| |
| HDFS-4699. TestPipelinesFailover#testPipelineRecoveryStress fails |
| sporadically (Chris Nauroth via kihwal) |
| |
| HDFS-4805. Webhdfs client is fragile to token renewal errors |
| (daryn via kihwal) |
| |
| HDFS-3875. Issue handling checksum errors in write pipeline. (kihwal) |
| |
| HDFS-4807. createSocketForPipeline() should not include timeout extension |
| on connect. (Cristina L. Abad via kihwal) |
| |
| Release 0.23.7 - 2013-04-08 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| OPTIMIZATIONS |
| |
| HDFS-4532. RPC call queue may fill due to current user lookup (daryn) |
| |
| BUG FIXES |
| |
| HDFS-4288. NN accepts incremental BR as IBR in safemode (daryn via kihwal) |
| |
| HDFS-4495. Allow client-side lease renewal to be retried beyond soft-limit |
| (kihwal) |
| |
| HDFS-4128. 2NN gets stuck in inconsistent state if edit log replay fails |
| in the middle (kihwal via daryn) |
| |
| HDFS-4542. Webhdfs doesn't support secure proxy users (Daryn Sharp via |
| kihwal) |
| |
| HDFS-4560. Webhdfs cannot use tokens obtained by another user (daryn) |
| |
| HDFS-4566. Webdhfs token cancelation should use authentication (daryn) |
| |
| HDFS-4567. Webhdfs does not need a token for token operations (daryn via |
| kihwal) |
| |
| HDFS-4577. Webhdfs operations should declare if authentication is required |
| (daryn via kihwal) |
| |
| HDFS-3344. Unreliable corrupt blocks counting in TestProcessCorruptBlocks |
| (kihwal) |
| |
| HDFS-3367. WebHDFS doesn't use the logged in user when opening |
| connections (daryn) |
| |
| HDFS-4581. checkDiskError should not be called on network errors (Rohit |
| Kochar via kihwal) |
| |
| HDFS-4649. Webhdfs cannot list large directories (daryn via kihwal) |
| |
| HDFS-4548. Webhdfs doesn't renegotiate SPNEGO token (daryn via kihwal) |
| |
| Release 0.23.6 - 2013-02-06 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| HDFS-4247. saveNamespace should be tolerant of dangling lease (daryn) |
| |
| HDFS-4248. Renaming directories may incorrectly remove the paths in leases |
| under the tree. (daryn via szetszwo) |
| |
| HDFS-4385. Maven RAT plugin is not checking all source files (tgraves) |
| |
| HDFS-4426. Secondary namenode shuts down immediately after startup. |
| (Arpit Agarwal via suresh) |
| |
| Release 0.23.5 - 2012-11-28 |
| |
| INCOMPATIBLE CHANGES |
| |
| HDFS-4080. Add a separate logger for block state change logs to enable turning |
| off those logs. (Kihwal Lee via suresh) |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| OPTIMIZATIONS |
| |
| HDFS-4075. Reduce recommissioning overhead (Kihwal Lee via daryn) |
| |
| HDFS-3990. NN's health report has severe performance problems (daryn) |
| |
| HDFS-4181. LeaseManager tries to double remove and prints extra messages |
| (Kihwal Lee via daryn) |
| |
| BUG FIXES |
| |
| HDFS-3829. TestHftpURLTimeouts fails intermittently with JDK7 (Trevor |
| Robinson via tgraves) |
| |
| HDFS-3824. TestHftpDelegationToken fails intermittently with JDK7 (Trevor |
| Robinson via tgraves) |
| |
| HDFS-3224. Bug in check for DN re-registration with different storage ID |
| (jlowe) |
| |
| HDFS-4090. getFileChecksum() result incompatible when called against |
| zero-byte files. (Kihwal Lee via daryn) |
| |
| HDFS-4172. namenode does not URI-encode parameters when building URI for |
| datanode request (Derek Dagit via bobby) |
| |
| HDFS-4182. SecondaryNameNode leaks NameCache entries (bobby) |
| |
| HDFS-4186. logSync() is called with the write lock held while releasing |
| lease (Kihwal Lee via daryn) |
| |
| Release 0.23.4 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| HDFS-3831. Failure to renew tokens due to test-sources left in classpath |
| (jlowe via bobby) |
| |
| Release 0.23.3 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| HDFS-2505. Add a test to verify getFileChecksum(..) with ViewFS. (Ravi |
| Prakash via szetszwo) |
| |
| OPTIMIZATIONS |
| |
| BUG FIXES |
| |
| HDFS-3166. Add timeout to Hftp connections. (Daryn Sharp via szetszwo) |
| |
| HDFS-3176. Use MD5MD5CRC32FileChecksum.readFields() in JsonUtil . (Kihwal |
| Lee via szetszwo) |
| |
| HDFS-2652. Add support for host-based delegation tokens. (Daryn Sharp via |
| szetszwo) |
| |
| HDFS-3308. Uses canonical URI to select delegation tokens in HftpFileSystem |
| and WebHdfsFileSystem. (Daryn Sharp via szetszwo) |
| |
| HDFS-3312. In HftpFileSystem, the namenode URI is non-secure but the |
| delegation tokens have to use secure URI. (Daryn Sharp via szetszwo) |
| |
| HDFS-3318. Use BoundedInputStream in ByteRangeInputStream, otherwise, it |
| hangs on transfers >2 GB. (Daryn Sharp via szetszwo) |
| |
| HDFS-3321. Fix safe mode turn off tip message. (Ravi Prakash via szetszwo) |
| |
| HDFS-3334. Fix ByteRangeInputStream stream leakage. (Daryn Sharp via |
| szetszwo) |
| |
| HDFS-3331. In namenode, check superuser privilege for setBalancerBandwidth |
| and acquire the write lock for finalizeUpgrade. (szetszwo) |
| |
| HDFS-3037. TestMulitipleNNDataBlockScanner#testBlockScannerAfterRestart is |
| racy. (atm) |
| |
| HDFS-3577. In DatanodeWebHdfsMethods, use MessageBodyWriter instead of |
| StreamingOutput, otherwise, it will fail to transfer large files. |
| (szetszwo) |
| |
| HDFS-3646. LeaseRenewer can hold reference to inactive DFSClient |
| instances forever. (Kihwal Lee via daryn) |
| |
| HDFS-3696. Set chunked streaming mode in WebHdfsFileSystem write operations |
| to get around a Java library bug causing OutOfMemoryError. (szetszwo) |
| |
| HDFS-3553. Hftp proxy tokens are broken (daryn) |
| |
| HDFS-3718. Datanode won't shutdown because of runaway DataBlockScanner |
| thread (Kihwal Lee via daryn) |
| |
| HDFS-3861. Deadlock in DFSClient (Kihwal Lee via daryn) |
| |
| HDFS-3873. Hftp assumes security is disabled if token fetch fails (daryn) |
| |
| HDFS-3852. TestHftpDelegationToken is broken after HADOOP-8225 (daryn) |
| |
| HDFS-3890. filecontext mkdirs doesn't apply umask as expected |
| (Tom Graves via daryn) |
| |
| Release 0.23.2 - UNRELEASED |
| |
| INCOMPATIBLE CHANGES |
| |
| HDFS-2887. FSVolume, is a part of FSDatasetInterface implementation, should |
| not be referred outside FSDataset. A new FSVolumeInterface is defined. |
| The BlockVolumeChoosingPolicy.chooseVolume(..) method signature is also |
| updated. (szetszwo) |
| |
| NEW FEATURES |
| |
| HDFS-2943. Expose last checkpoint time and transaction stats as JMX |
| metrics. (atm) |
| |
| IMPROVEMENTS |
| |
| HDFS-2931. Switch DataNode's BlockVolumeChoosingPolicy to private-audience. |
| (harsh via szetszwo) |
| |
| HDFS-2655. BlockReaderLocal#skip performs unnecessary IO. (Brandon Li |
| via jitendra) |
| |
| HDFS-2725. hdfs script usage information is missing the information |
| about "dfs" command (Prashant Sharma via stevel) |
| |
| HDFS-2907. Add a conf property dfs.datanode.fsdataset.factory to make |
| FSDataset in Datanode pluggable. (szetszwo) |
| |
| HDFS-2985. Improve logging when replicas are marked as corrupt. (todd) |
| |
| HDFS-3098. Update and add tests for HADOOP-8173. (Daryn Sharp via szetszwo) |
| |
| HDFS-3104. Add tests for HADOOP-8175. (Daryn Sharp via szetszwo) |
| |
| HDFS-3066. Cap space usage of default log4j rolling policy. |
| (Patrick Hunt via eli) |
| |
| OPTIMIZATIONS |
| |
| HDFS-3024. Improve performance of stringification in addStoredBlock (todd) |
| |
| BUG FIXES |
| HDFS-2923. Namenode IPC handler count uses the wrong configuration key |
| (todd) |
| |
| HDFS-2764. TestBackupNode is racy. (atm) |
| |
| HDFS-2869. Fix an error in the webhdfs docs for the mkdir op (harsh) |
| |
| HDFS-776. Fix exception handling in Balancer. (Uma Maheswara Rao G |
| via szetszwo) |
| |
| HDFS-2815. Namenode sometimes oes not come out of safemode during |
| NN crash + restart. (Uma Maheswara Rao via suresh) |
| |
| HDFS-2950. Secondary NN HTTPS address should be listed as a |
| NAMESERVICE_SPECIFIC_KEY. (todd) |
| |
| HDFS-2525. Race between BlockPoolSliceScanner and append. (Brandon Li |
| via jitendra) |
| |
| HDFS-2938. Recursive delete of a large directory make namenode |
| unresponsive. (Hari Mankude via suresh) |
| |
| HDFS-2969. ExtendedBlock.equals is incorrectly implemented (todd) |
| |
| HDFS-2944. Typo in hdfs-default.xml causes |
| dfs.client.block.write.replace-datanode-on-failure.enable to be mistakenly |
| disabled. (atm) |
| |
| HDFS-2981. In hdfs-default.xml, the default value of |
| dfs.client.block.write.replace-datanode-on-failure.enable should be true. |
| (szetszwo) |
| |
| HDFS-3008. Negative caching of local addrs doesn't work. (eli) |
| |
| HDFS-3006. In WebHDFS, when the return body is empty, set the Content-Type |
| to application/octet-stream instead of application/json. (szetszwo) |
| |
| HDFS-2991. Fix case where OP_ADD would not be logged in append(). (todd) |
| |
| HDFS-3012. Exception while renewing delegation token. (Bobby Evans via |
| jitendra) |
| |
| HDFS-3032. Change DFSClient.renewLease() so that it only retries up to the |
| lease soft-limit. (Kihwal Lee via szetszwo) |
| |
| HDFS-2038. Update TestHDFSCLI to handle relative paths with globs. |
| (Kihwal Lee via szetszwo) |
| |
| HDFS-3101. Cannot read empty file using WebHDFS. (szetszwo) |
| |
| HDFS-3160. httpfs should exec catalina instead of forking it. |
| (Roman Shaposhnik via eli) |
| |
| Release 0.23.1 - 2012-02-17 |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| HDFS-2316. [umbrella] WebHDFS: a complete FileSystem implementation for |
| accessing HDFS over HTTP (szetszwo) |
| |
| HDFS-2594. Support getDelegationTokens and createSymlink in WebHDFS. |
| (szetszwo) |
| |
| HDFS-2545. Change WebHDFS to support multiple namenodes in federation. |
| (szetszwo) |
| |
| HDFS-2178. Contributing Hoop to HDFS, replacement for HDFS proxy |
| with read/write capabilities. (tucu) |
| |
| |
| IMPROVEMENTS |
| HDFS-2560. Refactor BPOfferService to be a static inner class (todd) |
| |
| HDFS-2544. Hadoop scripts unconditionally source |
| "$bin"/../libexec/hadoop-config.sh. (Bruno Mahé via tomwhite) |
| |
| HDFS-2543. HADOOP_PREFIX cannot be overridden. (Bruno Mahé via tomwhite) |
| |
| HDFS-2562. Refactor DN configuration variables out of DataNode class |
| (todd) |
| |
| HDFS-2563. Some cleanup in BPOfferService. (todd) |
| |
| HDFS-2568. Use a set to manage child sockets in XceiverServer. |
| (harsh via eli) |
| |
| HDFS-2454. Move maxXceiverCount check to before starting the |
| thread in dataXceiver. (harsh via eli) |
| |
| HDFS-2570. Add descriptions for dfs.*.https.address in hdfs-default.xml. |
| (eli) |
| |
| HDFS-2536. Remove unused imports. (harsh via eli) |
| |
| HDFS-2566. Move BPOfferService to be a non-inner class. (todd) |
| |
| HDFS-2552. Add Forrest doc for WebHDFS REST API. (szetszwo) |
| |
| HDFS-2587. Add apt doc for WebHDFS REST API. (szetszwo) |
| |
| HDFS-2604. Add a log message to show if WebHDFS is enabled and a |
| configuration section in the forrest doc. (szetszwo) |
| |
| HDFS-2511. Add dev script to generate HDFS protobufs. (tucu) |
| |
| HDFS-2654. Make BlockReaderLocal not extend RemoteBlockReader2. (eli) |
| |
| HDFS-2675. Reduce warning verbosity when double-closing edit logs |
| (todd) |
| |
| HDFS-2335. DataNodeCluster and NNStorage always pull fresh entropy. |
| (Uma Maheswara Rao G via eli) |
| |
| HDFS-2574. Remove references to some deprecated properties in conf |
| templates and defaults files. (Joe Crobak via harsh) |
| |
| HDFS-2722. HttpFs should not be using an int for block size. (harsh) |
| |
| HDFS-2710. Add HDFS tests related to HADOOP-7933. (Siddarth Seth via |
| suresh) |
| |
| HDFS-2349. Corruption detected during block transfers between DNs |
| should log a WARN instead of INFO. (harsh) |
| |
| HDFS-2729. Update BlockManager's comments regarding the invalid block |
| set (harsh) |
| |
| HDFS-2726. Fix a logging issue under DFSClient's createBlockOutputStream |
| method (harsh) |
| |
| HDFS-554. Use System.arraycopy in BlockInfo.ensureCapacity. (harsh) |
| |
| HDFS-1314. Make dfs.blocksize accept size-indicating prefixes. |
| (Sho Shimauchi via harsh) |
| |
| HDFS-69. Improve the 'dfsadmin' commandline help. (harsh) |
| |
| HDFS-2788. HdfsServerConstants#DN_KEEPALIVE_TIMEOUT is dead code. (eli) |
| |
| HDFS-362. FSEditLog should not writes long and short as UTF8, and should |
| not use ArrayWritable for writing non-array items. (Uma Maheswara Rao G |
| via szetszwo) |
| |
| HDFS-2803. Add logging to LeaseRenewer for better lease expiration debugging. |
| (Jimmy Xiang via todd) |
| |
| HDFS-2817. Combine the two TestSafeMode test suites. (todd) |
| |
| HDFS-2818. Fix a missing space issue in HDFS webapps' title tags. (Devaraj K via harsh) |
| |
| HDFS-2397. Undeprecate SecondaryNameNode. (eli) |
| |
| HDFS-2814. NamenodeMXBean does not account for svn revision in the version |
| information. (Hitesh Shah via jitendra) |
| |
| HDFS-2784. Update hftp and hdfs for host-based token support. |
| (Kihwal Lee via jitendra) |
| |
| HDFS-2785. Update webhdfs and httpfs for host-based token support. |
| (Robert Joseph Evans via jitendra) |
| |
| HDFS-2868. Expose xceiver counts via the DataNode MXBean. (harsh) |
| |
| HDFS-2786. Fix host-based token incompatibilities in DFSUtil. (Kihwal |
| Lee via jitendra) |
| |
| HDFS-208. name node should warn if only one dir is listed in dfs.name.dir. |
| (Uma Maheswara Rao G via eli) |
| |
| HDFS-3139. Minor Datanode logging improvement. (eli) |
| |
| OPTIMIZATIONS |
| |
| HDFS-2130. Switch default checksum to CRC32C. (todd) |
| |
| HDFS-2533. Remove needless synchronization on some FSDataSet methods. |
| (todd) |
| |
| HDFS-2129. Simplify BlockReader to not inherit from FSInputChecker. |
| (todd) |
| |
| HDFS-2246. Enable reading a block directly from local file system |
| for a client on the same node as the block file. (Andrew Purtell, |
| Suresh Srinivas and Jitendra Nath Pandey via szetszwo) |
| |
| HDFS-2825. Add test hook to turn off the writer preferring its local |
| DN. (todd) |
| |
| HDFS-2826. Add test case for HDFS-1476 (safemode can initialize |
| replication queues before exiting) (todd) |
| |
| HDFS-2864. Remove some redundant methods and the constant METADATA_VERSION |
| from FSDataset. (szetszwo) |
| |
| HDFS-2879. Change FSDataset to package private. (szetszwo) |
| |
| BUG FIXES |
| |
| HDFS-2541. For a sufficiently large value of blocks, the DN Scanner |
| may request a random number with a negative seed value. (harsh via eli) |
| |
| HDFS-2502. hdfs-default.xml should include dfs.name.dir.restore. |
| (harsh via eli) |
| |
| HDFS-2567. When 0 DNs are available, show a proper error when |
| trying to browse DFS via web UI. (harsh via eli) |
| |
| HDFS-2575. DFSTestUtil may create empty files (todd) |
| |
| HDFS-2588. hdfs jsp pages missing DOCTYPE. (Dave Vronay via mattf) |
| |
| HDFS-2590. Fix the missing links in the WebHDFS forrest doc. (szetszwo) |
| |
| HDFS-2596. TestDirectoryScanner doesn't test parallel scans. (eli) |
| |
| HDFS-2606. webhdfs client filesystem impl must set the content-type |
| header for create/append. (tucu) |
| |
| HDFS-2614. hadoop dist tarball is missing hdfs headers. (tucu) |
| |
| HDFS-2653. DFSClient should cache whether addrs are non-local when |
| short-circuiting is enabled. (eli) |
| |
| HDFS-2649. eclipse:eclipse build fails for hadoop-hdfs-httpfs. |
| (Jason Lowe via eli) |
| |
| HDFS-2640. Javadoc generation hangs. (tomwhite) |
| |
| HDFS-2553. Fix BlockPoolSliceScanner spinning in a tight loop (Uma |
| Maheswara Rao G via todd) |
| |
| HDFS-2658. HttpFS introduced 70 javadoc warnings. (tucu) |
| |
| HDFS-2706. Use configuration for blockInvalidateLimit if it is set. |
| (szetszwo) |
| |
| HDFS-2646. Hadoop HttpFS introduced 4 findbug warnings. (tucu) |
| |
| HDFS-2657. TestHttpFSServer and TestServerWebApp are failing on trunk. (tucu) |
| |
| HDFS-2705. HttpFS server should check that upload requests have correct |
| content-type. (tucu) |
| |
| HDFS-2707. HttpFS should read the hadoop-auth secret from a file instead |
| inline from the configuration. (tucu) |
| |
| HDFS-2790. FSNamesystem.setTimes throws exception with wrong |
| configuration name in the message. (Arpit Gupta via eli) |
| |
| HDFS-2810. Leases not getting renewed properly by clients (todd) |
| |
| HDFS-2751. Datanode may incorrectly drop OS cache behind reads |
| even for short reads. (todd) |
| |
| HDFS-2816. Fix missing license header in httpfs findbugsExcludeFile.xml. |
| (hitesh via tucu) |
| |
| HDFS-2822. processMisReplicatedBlock incorrectly identifies |
| under-construction blocks as under-replicated. (todd) |
| |
| HDFS-442. dfsthroughput in test jar throws NPE (harsh) |
| |
| HDFS-2836. HttpFSServer still has 2 javadoc warnings in trunk. |
| (revans2 via tucu) |
| |
| HDFS-2837. mvn javadoc:javadoc not seeing LimitedPrivate class. |
| (revans2 via tucu) |
| |
| HDFS-2840. TestHostnameFilter should work with localhost or |
| localhost.localdomain (tucu) |
| |
| HDFS-2791. If block report races with closing of file, replica is |
| incorrectly marked corrupt. (todd) |
| |
| HDFS-2827. When the parent of a directory is the root, renaming the |
| directory results in leases updated incorrectly. (Uma Maheswara Rao G |
| via szetszwo) |
| |
| HDFS-2835. Fix findbugs and javadoc issue with GetConf.java. |
| (suresh) |
| |
| HDFS-2889. getNumCurrentReplicas is package private but should be public on |
| 0.23 (see HDFS-2408). (Gregory Chanan via atm) |
| |
| HDFS-2893. The start/stop scripts don't start/stop the 2NN when |
| using the default configuration. (eli) |
| |
| Release 0.23.0 - 2011-11-01 |
| |
| INCOMPATIBLE CHANGES |
| |
| HDFS-1526. Dfs client name for a map/reduce task should be unique |
| among threads. (hairong) |
| |
| HDFS-1536. Improve HDFS WebUI. (hairong) |
| |
| HDFS-2210. Remove hdfsproxy. (eli) |
| |
| HDFS-1073. Redesign the NameNode's storage layout for image checkpoints |
| and edit logs to introduce transaction IDs and be more robust. |
| Please see HDFS-1073 section below for breakout of individual patches. |
| |
| NEW FEATURES |
| |
| HDFS-1359. Add BlockPoolID to Block. (suresh) |
| |
| HDFS-1365. Federation: propose ClusterID and BlockPoolID format |
| (Tanping via boryas) |
| |
| HDFS-1394. Federation: modify -format option for namenode to generated |
| new blockpool id and accept newcluster (boryas) |
| |
| HDFS-1400. Federation: DataTransferProtocol uses ExtendedBlockPool to |
| include BlockPoolID in the protocol. (suresh) |
| |
| HDFS-1428. Federation : add cluster ID and block pool ID into |
| Name node web UI(Tanping via boryas) |
| |
| HDFS-1450. Federation: Introduce block pool ID into FSDatasetInterface. |
| (suresh) |
| |
| HDFS-1632. Federation: data node storage structure changes and |
| introduce block pool storage. (Tanping via suresh) |
| |
| HDFS-1634. Federation: Convert single threaded DataNode into |
| per BlockPool thread model.(boryas) |
| |
| HDFS-1637. Federation: FSDataset in Datanode should be created after |
| initial handshake with namenode. (boryas and jitendra) |
| |
| HDFS-1653. Federation: Block received message from datanode sends invalid |
| DatanodeRegistration. (Tanping via suresh) |
| |
| HDFS-1645. Federation: DatanodeCommond.Finalize needs to include |
| BlockPoolId. (suresh) |
| |
| HDFS-1638. Federation: DataNode.handleDiskError needs to inform |
| ALL namenodes if a disk failed (boryas) |
| |
| HDFS-1647. Federation: Multiple namenode configuration. (jitendra) |
| |
| HDFS-1639. Federation: Add block pool management to FSDataset. (suresh) |
| |
| HDFS-1648. Federation: Only DataStorage must be locked using in_use.lock |
| and no locks must be associated with BlockPoolStorage. (Tanping via suresh) |
| |
| HDFS-1641. Federation: Datanode fields that are no longer used should |
| be removed (boryas) |
| |
| HDFS-1642. Federation: add Datanode.getDNRegistration(String bpid) |
| method (boryas) |
| |
| HDFS-1643. Federation: remove namenode argument from DataNode |
| constructor (boryas) |
| |
| HDFS-1657. Federation: Tests that corrupt block files fail due to changed |
| file path in federation. (suresh) |
| |
| HDFS-1661. Federation: Remove unnecessary TODO:FEDERATION comments. |
| (jitendra) |
| |
| HDFS-1660. Federation: Datanode doesn't start with two namenodes (boryas) |
| |
| HDFS-1650. Federation: TestReplication fails. (Tanping via suresh) |
| |
| HDFS-1651. Federation: Tests fail due to null pointer exception in |
| Datnode#shutdown() method. (Tanping via suresh) |
| |
| HDFS-1649. Federation: Datanode command to refresh namenode list at |
| the datanode. (jitendra) |
| |
| HDFS-1646. Federation: MiniDFSClsuter#waitActive() waits for ever |
| with the introduction of BPOfferService in datanode. (suresh) |
| |
| HDFS-1659. Federation: BPOfferService exits after one iteration |
| incorrectly. (Tanping via suresh) |
| |
| HDFS-1654. Federation: Fix TestDFSUpgrade and TestDFSRollback failures. |
| (suresh) |
| |
| HDFS-1668. Federation: Datanodes sends block pool usage information |
| to the namenode in heartbeat. (suresh) |
| |
| HDFS-1669. Federation: Fix TestHftpFileSystem failure. (suresh) |
| |
| HDFS-1670. Federation: remove dnRegistration from Datanode (boryas) |
| |
| HDFS-1662. Federation: fix unit test case, TestCheckpoint |
| and TestDataNodeMXBean (tanping via boryas) |
| |
| HDFS-1671. Federation: shutdown in DataNode should be able to |
| shutdown individual BP threads as well as the whole DN (boryas). |
| |
| HDFS-1663. Federation: Rename getPoolId() everywhere to |
| getBlockPoolId() (tanping via boryas) |
| |
| HDFS-1652. FederationL Add support for multiple namenodes in |
| MiniDFSCluster. (suresh) |
| |
| HDFS-1672. Federation: refactor stopDatanode(name) to work |
| with multiple Block Pools (boryas) |
| |
| HDFS-1687. Federation: DirectoryScanner changes for |
| federation (Matt Foley via boryas) |
| |
| HDFS-1626. Make BLOCK_INVALIDATE_LIMIT configurable. (szetszwo) |
| |
| HDFS-1655. Federation: DatablockScanner should scan blocks for |
| all the block pools. (jitendra) |
| |
| HDFS-1664. Federation: Add block pool storage usage to Namenode WebUI. |
| (Tanping via suresh) |
| |
| HDFS-1674. Federation: Rename BlockPool class to BlockPoolSlice. |
| (jghoman, Tanping via suresh) |
| |
| HDFS-1673. Federation: Datanode changes to track block token secret per |
| namenode. (suresh) |
| |
| HDFS-1677. Federation: Fix TestFsck and TestListCorruptFileBlocks |
| failures. (Tanping via suresh) |
| |
| HDFS-1678. Federation: Remove unnecessary #getBlockpool() |
| for NameNodeMXBean in FSNameSystem. (Tanping via Suresh) |
| |
| HDFS-1688. Federation: Fix failures in fault injection tests, |
| TestDiskError, TestDatanodeRestart and TestDFSTartupVersions. (suresh) |
| |
| HDFS-1696. Federation: when build version doesn't match - |
| datanode should wait (keep connecting) untill NN comes up |
| with the right version (boryas) |
| |
| HDFS-1681. Balancer: support per pool and per node policies. (szetszwo) |
| |
| HDFS-1695. Federation: Fix testOIV and TestDatanodeUtils |
| (jhoman and tanping via boryas) |
| |
| HDFS-1699. Federation: Fix failure of TestBlockReport. |
| (Matt Foley via suresh) |
| |
| HDFS-1698. Federation: TestOverReplicatedBlocks and TestWriteToReplica |
| failing. (jhoman and jitendra) |
| |
| HDFS-1701. Federation: Fix TestHeartbeathandling. |
| (Erik Steffl and Tanping Wang via suresh) |
| |
| HDFS-1693. Federation: Fix TestDFSStorageStateRecovery failure. (suresh) |
| |
| HDFS-1694. Federation: SimulatedFSDataset changes to work with |
| federation and multiple block pools. (suresh) |
| |
| HDFS-1689. Federation: Configuration for namenodes. (suresh and jitendra) |
| |
| HDFS-1682. Change Balancer CLI for multiple namenodes and balancing |
| policy. (szetszwo) |
| |
| HDFS-1697. Federation: fix TestBlockRecovery (boryas) |
| |
| HDFS-1702. Federation: fix TestBackupNode and TestRefreshNamendoes |
| failures. (suresh) |
| |
| HDFS-1706. Federation: TestFileAppend2, TestFileAppend3 and |
| TestBlockTokenWithDFS failing. (jitendra) |
| |
| HDFS-1704. Federation: Add a tool that lists namenodes, secondary and |
| backup from configuration file. (suresh) |
| |
| HDFS-1711. Federation: create method for updating machine name in |
| DataNode.java (boryas) |
| |
| HDFS-1712. Federation: when looking up datanode we should use machineNmae |
| (in testOverReplicatedBlocks) (boryas) |
| |
| HDFS-1709. Federation: Error "nnaddr url param is null" when clicking on a |
| node from NN Live Node Link. (jitendra) |
| |
| HDFS-1714. Federation: refactor upgrade object in DataNode (boryas) |
| |
| HDFS-1715. Federation: warning/error not generated when datanode sees |
| inconsistent/different Cluster ID between namenodes (boryas) |
| |
| HDFS-1715. Federation: warning/error not generated when datanode sees |
| inconsistent/different Cluster ID between namenodes (boryas) |
| |
| HDFS-1716. Federation: Add decommission tests for federated namenodes. |
| (suresh) |
| |
| HDFS-1713. Federation: Prevent DataBlockScanner from running in tight loop. |
| (jitendra) |
| |
| HDFS-1721. Federation: Configuration for principal names should not be |
| namenode specific. (jitendra) |
| |
| HDFS-1717. Federation: FSDataset volumeMap access is not synchronized |
| correctly. (suresh) |
| |
| HDFS-1722. Federation: Add flag to MiniDFSCluser to differentiate between |
| federation and non-federation modes. (boryas via suresh) |
| |
| HDFS-1718. Federation: MiniDFSCluster#waitActive() bug causes some tests |
| to fail. (suresh) |
| |
| HDFS-1719. Federation: Fix TestDFSRemove that fails intermittently. |
| (suresh) |
| |
| HDFS-1720. Federation: FSVolumeSet volumes is not synchronized correctly. |
| (suresh) |
| |
| HDFS-1700. Federation: fsck needs to work with federation changes. |
| (Matt Foley via suresh) |
| |
| HDFS-1482. Add listCorruptFileBlocks to DistributedFileSystem. |
| (Patrick Kling via hairong) |
| |
| HDFS-1448. Add a new tool Offline Edits Viewer (oev). (Erik Steffl |
| via szetszwo) |
| |
| HDFS-1735. Federation: merge FSImage change in federation to |
| FSImage+NNStorage refactoring in trunk. (suresh) |
| |
| HDFS-1737. Federation: Update the layout version for federation |
| changes. (suresh) |
| |
| HDFS-1744. Federation: Add new layout version to offline image viewer |
| and edits viewer. (suresh) |
| |
| HDFS-1745. Federation: Fix fault injection test failures. (suresh) |
| |
| HDFS-1746. Federation: TestFileAppend3 fails intermittently. (jitendra) |
| |
| HDFS-1703. Improve start/stop scripts and add decommission tool for |
| federation. (Tanping Wang, Erik Steffl via suresh) |
| |
| HDFS-1749. Federation: TestListCorruptFileBlocks failing in federation |
| branch. (jitendra) |
| |
| HDFS-1754. Federation: testFsck fails. (boryas) |
| |
| HDFS-1755. Federation: The BPOfferService must always connect to namenode |
| as the login user. (jitendra) |
| |
| HDFS-1675. Support transferring RBW between datanodes. (szetszwo) |
| |
| HDFS-1791. Federation: Add command to delete block pool directories |
| from a datanode. (jitendra) |
| |
| HDFS-1761. Add a new DataTransferProtocol operation, Op.TRANSFER_BLOCK, |
| for transferring RBW/Finalized with acknowledgement and without using RPC. |
| (szetszwo) |
| |
| HDFS-1813. Federation: Authentication using BlockToken in RPC to datanode |
| fails. (jitendra) |
| |
| HDFS-1630. Support fsedits checksum. (hairong) |
| |
| HDFS-1606. Provide a stronger data guarantee in the write pipeline by |
| adding a new datanode when an existing datanode failed. (szetszwo) |
| |
| HDFS-1442. Api to get delegation token in Hdfs class. (jitendra) |
| |
| HDFS-1070. Speedup namenode image loading and saving by storing only |
| local file names. (hairong) |
| |
| HDFS-1751. Intrinsic limits for HDFS files, directories (daryn via boryas). |
| |
| HDFS-1873. Federation: Add cluster management web console. |
| (Tanping Wang via suresh) |
| |
| HDFS-1911 HDFS tests for the newly added viewfs |
| |
| HDFS-1814. Add "hdfs groups" command to query the server-side groups |
| resolved for a user. (Aaron T. Myers via todd) |
| |
| HDFS-1914. Federation: namenode storage directories must be configurable |
| specific to name service. (suresh) |
| |
| HDFS-1963. Create RPM and Debian packages for HDFS. Changes deployment |
| layout to be consistent across the binary tgz, rpm, and deb. |
| (Eric Yang via omalley) |
| |
| HDFS-2058. Change Data Transfer wire protocol to use protocol buffers. |
| (todd) |
| |
| HDFS-2055. Add hflush support to libhdfs. (Travis Crawford via eli) |
| |
| HDFS-2083. Query JMX statistics over http via JMXJsonServlet. (tanping) |
| |
| HDFS-2156. Make hdfs and mapreduce rpm only depend on the same major |
| version for common and hdfs. (eyang via omalley) |
| |
| HDFS-2202. Add a new DFSAdmin command to set balancer bandwidth of |
| datanodes without restarting. (Eric Payne via szetszwo) |
| |
| HDFS-2284. Add a new FileSystem, webhdfs://, for supporting write Http |
| access to HDFS. (szetszwo) |
| |
| HDFS-2317. Support read access to HDFS in WebHDFS. (szetszwo) |
| |
| HDFS-2338. Add configuration option to enable/disable WebHDFS. |
| (jitendra via szetszwo) |
| |
| HDFS-2318. Provide authentication to WebHDFS using SPNEGO and delegation |
| tokens. (szetszwo) |
| |
| HDFS-2340. Support getFileBlockLocations and getDelegationToken in WebHDFS. |
| (szetszwo) |
| |
| HDFS-2348. Support getContentSummary and getFileChecksum in WebHDFS. |
| (szetszwo) |
| |
| HDFS-2385. Support renew and cancel delegation tokens in WebHDFS. |
| (szetszwo) |
| |
| HDFS-2539. Support doAs and GETHOMEDIRECTORY in WebHDFS. |
| (szetszwo) |
| |
| IMPROVEMENTS |
| |
| HDFS-1875. MiniDFSCluster hard-codes dfs.datanode.address to localhost |
| (Eric Payne via mattf) |
| |
| HDFS-2019. Fix all the places where Java method File.list is used with |
| FileUtil.list API (Bharath Mundlapudi via mattf) |
| |
| HDFS-1934. Fix NullPointerException when certain File APIs return null |
| (Bharath Mundlapudi via mattf) |
| |
| HDFS-1510. Added test-patch.properties required by test-patch.sh (nigel) |
| |
| HDFS-1628. Display full path in AccessControlException. (John George |
| via szetszwo) |
| |
| HDFS-1707. Federation: Failure in browsing data on new namenodes. |
| (jitendra) |
| |
| HDFS-1683. Test Balancer with multiple NameNodes. (szetszwo) |
| |
| HDFS-1547. Improve decommission mechanism. (suresh) |
| |
| HDFS-2143. Federation: In cluster web console, add link to namenode page |
| that displays live and dead datanodes. (Ravi Prakash via suresh) |
| |
| HDFS-1588. Remove hardcoded strings for configuration keys, "dfs.hosts" |
| and "dfs.hosts.exlude". (Erik Steffl via suresh) |
| |
| HDFS-1481. NameNode should validate fsimage before rolling. (hairong) |
| |
| HDFS-1506. Refactor fsimage loading code. (hairong) |
| |
| HDFS-1533. A more elegant FileSystem#listCorruptFileBlocks API |
| (HDFS portion) (Patrick Kling via hairong) |
| |
| HDFS-1476. listCorruptFileBlocks should be functional while the |
| name node is in safe mode. (Patrick Kling via hairong) |
| |
| HDFS-1534. Fix some incorrect logs in FSDirectory. (eli) |
| |
| HDFS-1539. A config option for the datanode to fsycn a block file |
| when block is completely written. (dhruba) |
| |
| HDFS-1335. HDFS side change of HADDOP-6904: RPC compatibility. (hairong) |
| |
| HDFS-1557. Separate Storage from FSImage. (Ivan Kelly via jitendra) |
| |
| HDFS-560 Enhancements/tuning to hadoop-hdfs/build.xml |
| |
| HDFS-1629. Add a method to BlockPlacementPolicy for keeping the chosen |
| nodes in the output array. (szetszwo) |
| |
| HDFS-1731. Allow using a file to exclude certain tests from build (todd) |
| |
| HDFS-1736. Remove the dependency from DatanodeJspHelper to FsShell. |
| (Daryn Sharp via szetszwo) |
| |
| HDFS-780. Revive TestFuseDFS. (eli) |
| |
| HDFS-1445. Batch the calls in DataStorage to FileUtil.createHardLink(). |
| (Matt Foley via jghoman) |
| |
| HDFS-1763. Replace hard-coded option strings with variables from |
| DFSConfigKeys. (eli) |
| |
| HDFS-1541. Not marking datanodes dead when namenode in safemode. |
| (hairong) |
| |
| HDFS-1120. Make DataNode's block-to-device placement policy pluggable |
| (Harsh J Chouraria via todd) |
| |
| HDFS-1785. In BlockReceiver and DataXceiver, clientName.length() is used |
| multiple times for determining whether the source is a client or a |
| datanode. (szetszwo) |
| |
| HDFS-1789. Refactor frequently used codes from DFSOutputStream and |
| DataXceiver. (szetszwo) |
| |
| HDFS-1767. Namenode ignores non-initial block report from datanodes |
| when in safemode during startup. (Matt Foley via suresh) |
| |
| HDFS-1817. Move pipeline_Fi_[39-51] from TestFiDataTransferProtocol |
| to TestFiPipelineClose. (szetszwo) |
| |
| HDFS-1760. In FSDirectory.getFullPathName(..), it is better to return "/" |
| for root directory instead of an empty string. (Daryn Sharp via szetszwo) |
| |
| HDFS-1833. Reduce repeated string constructions and unnecessary fields, |
| and fix comments in BlockReceiver.PacketResponder. (szetszwo) |
| |
| HDFS-1486. Generalize CLITest structure and interfaces to faciliate |
| upstream adoption (e.g. for web testing). (cos) |
| |
| HDFS-1844. Move "fs -help" shell command tests from HDFS to COMMOM; see |
| also HADOOP-7230. (Daryn Sharp via szetszwo) |
| |
| HDFS-1840. In DFSClient, terminate the lease renewing thread when all files |
| being written are closed for a grace period, and start a new thread when |
| new files are opened for write. (szetszwo) |
| |
| HDFS-1854. make failure message more useful in |
| DFSTestUtil.waitReplication(). (Matt Foley via eli) |
| |
| HDFS-1562. Add rack policy tests. (eli) |
| |
| HDFS-1856. TestDatanodeBlockScanner waits forever, errs without giving |
| information. (Matt Foley via eli) |
| |
| HDFS-1295. Improve namenode restart times by short-circuiting the |
| first block reports from datanodes. (Matt Foley via suresh) |
| Corrected merge error in DataNode.java. (Matt Foley) |
| |
| HDFS-1843. Discover file not found early for file append. |
| (Bharath Mundlapudi via jitendra) |
| |
| HDFS-1862. Improve test reliability of HDFS-1594. (Aaron T. Myers via eli) |
| |
| HDFS-1846. Preallocate edit log with OP_INVALID instead of zero bytes |
| to ensure blocks are actually allocated. (Aaron T. Myers via todd) |
| |
| HDFS-1741. Provide a minimal pom file to allow integration of HDFS into Sonar |
| analysis (cos) |
| |
| HDFS-1870. Move and rename DFSClient.LeaseChecker to a seperated class |
| LeaseRenewer. (szetszwo) |
| |
| HDFS-1866. Document dfs.datanode.max.transfer.threads in hdfs-default.xml |
| (Harsh J Chouraria via todd) |
| |
| HDFS-1890. Improve the name, class and value type of the map |
| LeaseRenewer.pendingCreates. (szetszwo) |
| |
| HDFS-1865. Share LeaseRenewer among DFSClients so that there is only a |
| LeaseRenewer thread per namenode per user. (szetszwo) |
| |
| HDFS-1906. Remove logging exception stack trace in client logs when one of |
| the datanode targets to read from is not reachable. (suresh) |
| |
| HDFS-1378. Edit log replay should track and report file offsets in case of |
| errors. (Aaron T. Myers and Todd Lipcon via todd) |
| |
| HDFS-1917. Separate hdfs jars from common in ivy configuration. (Eric Yang |
| via szetszwo) |
| |
| HDFS-1899. GenericTestUtils.formatNamenode should be moved to DFSTestUtil |
| (Ted Yu via todd) |
| |
| HDFS-1117. Metrics 2.0 HDFS instrumentation. (Luke Lu via suresh) |
| |
| HDFS-1946. HDFS part of HADOOP-7291. (eli) |
| |
| HDFS-1945. Removed the deprecated fields in DataTransferProtocol. |
| (szetszwo) |
| |
| HDFS-1730. Use DaemonFactory from common and delete it from HDFS. |
| (Tanping via suresh) |
| |
| HDFS-1573. Add useful tracing information to Lease Renewer thread names |
| (todd) |
| |
| HDFS-1939. In ivy.xml, test conf should not extend common conf. |
| (Eric Yang via szetszwo) |
| |
| HDFS-1332 Include more information in exceptions and debug messages |
| when BlockPlacementPolicy cannot be satisfied. (Ted Yu via szetszwo) |
| |
| HDFS-1958. Confirmation should be more lenient of user input when |
| formatting the NameNode. (todd) |
| |
| HDFS-1905. Improve namenode -format command by not making -clusterId |
| parameter mandatory. (Bharath Mundlapudi via suresh) |
| |
| HDFS-1877. Add a new test for concurrent read and write. (CW Chung |
| via szetszwo) |
| |
| HDFS-1959. Better error message for missing namenode directory. (eli) |
| |
| HDFS-1996. ivy: hdfs test jar should be independent to common test jar. |
| (Eric Yang via szetszwo) |
| |
| HDFS-1812. TestHDFSCLI should clean up cluster in teardown method. |
| (Uma Maheswara Rao G via todd) |
| |
| HDFS-1884. Improve TestDFSStorageStateRecovery to properly throw in the |
| case of errors. (Aaron T. Myers via todd) |
| |
| HDFS-1727. fsck command should display command usage if user passes any |
| illegal argument. (Sravan Kumar via todd) |
| |
| HDFS-1636. If dfs.name.dir points to an empty dir, namenode format |
| shouldn't require confirmation. (Harsh J Chouraria via todd) |
| |
| HDFS-1966. Encapsulate individual DataTransferProtocol op headers. |
| (szetszwo) |
| |
| HDFS-2024. Format TestWriteRead source codes. (CW Chung via szetszwo) |
| |
| HDFS-1968. Enhance TestWriteRead to support position/sequential read, |
| append, truncate and verbose options. (CW Chung via szetszwo) |
| |
| HDFS-1986. Add option to get http/https address from |
| DFSUtil#getInfoServer(). (Tanping via suresh) |
| |
| HDFS-2029. In TestWriteRead, check visible length immediately after |
| openning the file and fix code style. (John George via szetszwo) |
| |
| HDFS-2040. Only build libhdfs if a flag is passed. (eli) |
| |
| HDFS-1586. Add InterfaceAudience and InterfaceStability annotations to |
| MiniDFSCluster. (suresh) |
| |
| HDFS-2003. Separate FSEditLog reading logic from edit log memory state |
| building logic. (Ivan Kelly via todd) |
| |
| HDFS-2066. Create a package and individual class files for |
| DataTransferProtocol. (szetszwo) |
| |
| HADOOP-7106. Reorganize project SVN layout to "unsplit" the projects. |
| (todd, nigel) |
| |
| HDFS-2046. Force entropy to come from non-true random for tests. (todd) |
| |
| HDFS-2073. Add @Override annotation to NameNode. (suresh) |
| |
| HDFS-420. Fuse-dfs should cache fs handles. (Brian Bockelman and eli) |
| |
| HDFS-1568. Improve the log messages in DataXceiver. (Joey Echeverria via |
| szetszwo) |
| |
| HDFS-2100. Improve TestStorageRestore. (atm) |
| |
| HDFS-2092. Remove some object references to Configuration in DFSClient. |
| (Bharath Mundlapudi via szetszwo) |
| |
| HDFS-2087. Declare methods in DataTransferProtocol interface, and change |
| Sender and Receiver to implement the interface. (szetszwo) |
| |
| HDFS-1723. quota errors messages should use the same scale. (Jim Plush via |
| atm) |
| |
| HDFS-2110. StreamFile and ByteRangeInputStream cleanup. (eli) |
| |
| HDFS-2107. Move block management code from o.a.h.h.s.namenode to a new |
| package o.a.h.h.s.blockmanagement. (szetszwo) |
| |
| HDFS-2109. Store uMask as member variable to DFSClient.Conf. (Bharath |
| Mundlapudi via szetszwo) |
| |
| HDFS-2111. Add tests for ensuring that the DN will start with a few bad |
| data directories. (Harsh J Chouraria via todd) |
| |
| HDFS-2134. Move DecommissionManager to the blockmanagement package. |
| (szetszwo) |
| |
| HDFS-1977. Stop using StringUtils.stringifyException(). |
| (Bharath Mundlapudi via jitendra) |
| |
| HDFS-2131. Add new tests for the -overwrite/-f option in put and |
| copyFromLocal by HADOOP-7361. (Uma Maheswara Rao G via szetszwo) |
| |
| HDFS-2140. Move Host2NodesMap to the blockmanagement package. (szetszwo) |
| |
| HDFS-2154. In TestDFSShell, use TEST_ROOT_DIR and fix some deprecated |
| warnings. (szetszwo) |
| |
| HDFS-2153. Move DFSClientAdapter to test and fix some javac warnings in |
| OfflineEditsViewerHelper. (szetszwo) |
| |
| HDFS-2159. Deprecate DistributedFileSystem.getClient() and fixed the |
| deprecated warnings in DFSAdmin. (szetszwo) |
| |
| HDFS-2157. Improve header comment in o.a.h.hdfs.server.namenode.NameNode. |
| (atm via eli) |
| |
| HDFS-2147. Move cluster network topology to block management and fix some |
| javac warnings. (szetszwo) |
| |
| HDFS-2141. Remove NameNode roles Active and Standby (they become |
| states of the namenode). (suresh) |
| |
| HDFS-2161. Move createNamenode(..), createClientDatanodeProtocolProxy(..) |
| and Random object creation to DFSUtil; move DFSClient.stringifyToken(..) |
| to DelegationTokenIdentifier. (szetszwo) |
| |
| HDFS-1774. Small optimization to FSDataset. (Uma Maheswara Rao G via eli) |
| |
| HDFS-2167. Move dnsToSwitchMapping and hostsReader from FSNamesystem to |
| DatanodeManager. (szetszwo) |
| |
| HDFS-2116. Use Mokito in TestStreamFile and TestByteRangeInputStream. |
| (Plamen Jeliazkov via shv) |
| |
| HDFS-2112. Move ReplicationMonitor to block management. (Uma Maheswara |
| Rao G via szetszwo) |
| |
| HDFS-1739. Add available volume size to the error message when datanode |
| throws DiskOutOfSpaceException. (Uma Maheswara Rao G via szetszwo) |
| |
| HDFS-2144. If SNN shuts down during initialization it does not log the |
| cause. (Ravi Prakash via atm) |
| |
| HDFS-2180. Refactor NameNode HTTP server into new class. (todd) |
| |
| HDFS-2198. Remove hardcoded configuration keys. (suresh) |
| |
| HDFS-2149. Move EditLogOp serialization formats into FsEditLogOp |
| implementations. (Ivan Kelly via todd) |
| |
| HDFS-2191. Move datanodeMap from FSNamesystem to DatanodeManager. |
| (szetszwo) |
| |
| HDFS-2200. Change FSNamesystem.LOG to package private. (szetszwo) |
| |
| HDFS-2195. Refactor StorageDirectory to not be an non-static inner class. |
| (todd via eli) |
| |
| HDFS-2212. Refactor double-buffering code out of EditLogOutputStreams. |
| (todd via eli) |
| |
| HDFS-2199. Move blockTokenSecretManager from FSNamesystem to BlockManager. |
| (Uma Maheswara Rao G via szetszwo) |
| |
| HDFS-2187. Make EditLogInputStream act like an iterator over FSEditLogOps |
| (Ivan Kelly and todd via todd) |
| |
| HDFS-2225. Refactor edit log file management so it's not in classes |
| which should be generic to the type of edit log storage. (Ivan Kelly |
| via todd) |
| |
| HDFS-2108. Move datanode heartbeat handling from namenode package to |
| blockmanagement package. (szetszwo) |
| |
| HDFS-2226. Clean up counting of operations in FSEditLogLoader (todd) |
| |
| HDFS-2228. Move block and datanode code from FSNamesystem to |
| BlockManager and DatanodeManager. (szetszwo) |
| |
| HDFS-2238. In NamenodeFsck.toString(), uses StringBuilder.(..) instead of |
| string concatenation. (Uma Maheswara Rao G via szetszwo) |
| |
| HDFS-2230. ivy to resolve/retrieve latest common-tests jar published by |
| hadoop common maven build. (gkesavan) |
| |
| HDFS-2227. getRemoteEditLogManifest should pull its information from |
| FileJournalManager during checkpoint process (Ivan Kelly and Todd Lipcon |
| via todd) |
| |
| HDFS-2239. Reduce access levels of the fields and methods in FSNamesystem. |
| (szetszwo) |
| |
| HDFS-2241. Remove implementing FSConstants interface to just get the |
| constants from the interface. (suresh) |
| |
| HDFS-2237. Change UnderReplicatedBlocks from public to package private. |
| (szetszwo) |
| |
| HDFS-2233. Add WebUI tests with URI reserved chars. (eli) |
| |
| HDFS-2265. Remove unnecessary BlockTokenSecretManager fields/methods from |
| BlockManager. (szetszwo) |
| |
| HDFS-2260. Refactor BlockReader into an interface and implementation. |
| (todd) |
| |
| HDFS-2096. Mavenization of hadoop-hdfs (Alejandro Abdelnur via tomwhite) |
| |
| HDFS-2273. Refactor BlockManager.recentInvalidateSets to a new class. |
| (szetszwo) |
| |
| HDFS-2266. Add Namesystem and SafeMode interfaces to avoid directly |
| referring to FSNamesystem in BlockManager. (szetszwo) |
| |
| HDFS-1217. Change some NameNode methods from public to package private. |
| (Laxman via szetszwo) |
| |
| HDFS-1620. Rename HdfsConstants -> HdfsServerConstants, FSConstants -> |
| HdfsConstants. (Harsh J Chouraria via atm) |
| |
| HDFS-2197. Refactor RPC call implementations out of NameNode class (todd) |
| |
| HDFS-2332. Add test for HADOOP-7629 (using an immutable FsPermission |
| object as an RPC parameter fails). (todd) |
| |
| HDFS-2363. Move datanodes size printing from FSNamesystem.metasave(..) |
| to BlockManager. (Uma Maheswara Rao G via szetszwo) |
| |
| HDFS-2209. Make MiniDFS easier to embed in other apps. (stevel) |
| |
| HDFS-2205. Log message for failed connection to datanode is not |
| followed by a success message. (Ravi Prakash via stevel) |
| |
| HDFS-2401. Running a set of methods in a Single Test Class. |
| (Jonathan Eagles via mahadev) |
| |
| HDFS-2471. Add federation documentation. (suresh) |
| |
| HDFS-2485. Improve code layout and constants in UnderReplicatedBlocks |
| (stevel) |
| |
| HDFS-2356. Support case insensitive query parameter names in WebHDFS. |
| (szetszwo) |
| |
| HDFS-2368. Move SPNEGO conf properties from hdfs-default.xml to |
| hdfs-site.xml. (szetszwo) |
| |
| HDFS-2395. Add a root element in the JSON responses of WebHDFS. |
| (szetszwo) |
| |
| HDFS-2427. Change the default permission in WebHDFS to 755 and add range |
| check/validation for all parameters. (szetszwo) |
| |
| HDFS-2501. Add version prefix and root methods to WebHDFS. (szetszwo) |
| |
| HDFS-1869. mkdirs should use the supplied permission for all of the created |
| directories. (Daryn Sharp via szetszwo) |
| |
| HDFS-2355. Federation: enable using the same configuration file across |
| all the nodes in the cluster. (suresh) |
| |
| HDFS-2371. Refactor BlockSender.java for better readability. (suresh) |
| |
| HDFS-2493. Remove reference to FSNamesystem in blockmanagement classes. |
| (szetszwo) |
| |
| HDFS-2294. Download of commons-daemon TAR should not be under target (tucu) |
| |
| HDFS-2322. the build fails in Windows because commons-daemon TAR cannot be |
| fetched. (tucu) |
| |
| HDFS-2436. Change FSNamesystem.setTimes(..) for allowing setting times on |
| directories. (Uma Maheswara Rao G via szetszwo) |
| |
| HDFS-2512. Add textual error message to data transfer protocol responses |
| (todd) |
| |
| HDFS-2521. Remove custom checksum headers from data transfer protocol |
| (todd) |
| |
| HDFS-2308. NamenodeProtocol.endCheckpoint is vestigial and can be removed. |
| (eli) |
| |
| HDFS-2507. Allow saveNamespace operations to be canceled. (todd) |
| |
| OPTIMIZATIONS |
| |
| HDFS-1458. Improve checkpoint performance by avoiding unnecessary image |
| downloads and loading. (hairong) |
| |
| HDFS-1601. Pipeline ACKs are sent as lots of tiny TCP packets (todd) |
| |
| HDFS-1826. NameNode should save image to name directories in parallel |
| during upgrade. (Matt Foley via hairong) |
| |
| HDFS-2030. Improve usability of namenode -upgrade command. |
| (Bharath Mundlapudi via suresh) |
| |
| HDFS-2056. Update fetchdt usage. (Tanping Wang via jitendra) |
| |
| HDFS-2118. Couple dfs data dir improvements. (eli) |
| |
| HDFS-2500. Avoid file system operations in BPOfferService thread while |
| processing deletes. (todd) |
| |
| HDFS-2465. Add HDFS support for fadvise readahead and drop-behind. (todd) |
| |
| BUG FIXES |
| |
| HDFS-2344. Fix the TestOfflineEditsViewer test failure in 0.23 branch. |
| (Uma Maheswara Rao G via mattf) |
| |
| HDFS-2347. Fix checkpointTxnCount's comment about editlog size. |
| (Uma Maheswara Rao G via mattf) |
| |
| HDFS-2011. Removal and restoration of storage directories on checkpointing |
| failure doesn't work properly. (Ravi Prakash via mattf) |
| |
| HDFS-1955. FSImage.doUpgrade() was made too fault-tolerant by HDFS-1826. |
| (mattf) |
| |
| HDFS-2061. Two minor bugs in BlockManager block report processing. (mattf) |
| |
| HDFS-1449. Fix test failures - ExtendedBlock must return |
| block file name in #getBlockName(). (suresh) |
| |
| HDFS-1680. Fix TestBalancer. (szetszwo) |
| |
| HDFS-1705. Balancer command throws NullPointerException. (suresh via |
| szetszwo) |
| |
| HDFS-1559. Add missing UGM overrides to TestRefreshUserMappings |
| (Todd Lipcon via eli) |
| |
| HDFS-1585. Fix build after HDFS-1547 (todd) |
| |
| HDFS-1684. Balancer cannot start with with multiple namenodes. (szetszwo) |
| |
| HDFS-1516. mvn-install is broken after 0.22 branch creation. (cos) |
| |
| HDFS-1360. TestBlockRecovery should bind ephemeral ports. |
| (Todd Lipcon via hairong) |
| |
| HDFS-1551. Fix pom templates dependency list (gkesavan) |
| |
| HDFS-1509. A savenamespace command writes the fsimage and edits into |
| all configured directories. (dhruba) |
| |
| HDFS-1540. Make Datanode handle errors from RPC calls to namenode |
| more elegantly. (dhruba) |
| |
| HDFS-1463. Accesstime of a file is not updated in safeMode. (dhruba) |
| |
| HDFS-863. Potential deadlock in TestOverReplicatedBlocks. |
| (Ken Goodhope via jghoman) |
| |
| HDFS-1607. Fix referenced to misspelled method name getProtocolSigature |
| (todd) |
| |
| HDFS-1610. Fix TestClientProtocolWithDelegationToken and TestBlockToken |
| on trunk after HADOOP-6904 (todd) |
| |
| HDFS-1600. Fix release audit warnings on trunk. (todd) |
| |
| HDFS-1691. Remove a duplicated static initializer for reading default |
| configurations in DFSck. (Alexey Diomin via szetszwo) |
| |
| HDFS-1748. Balancer utilization classification is incomplete. (szetszwo) |
| |
| HDFS-1738. change hdfs jmxget to return an empty string instead of |
| null when an attribute value is not available (tanping vi boryas) |
| |
| HDFS-1757. Don't compile fuse-dfs by default. (eli) |
| |
| HDFS-1770. TestFiRename fails due to invalid block size. (eli) |
| |
| HDFS-1797. Fix new findbugs warning introduced by HDFS-1120 (todd) |
| |
| HDFS-1611. Fix up some log messages in DFSClient and MBean registration |
| (Uma Maheswara Rao G via todd) |
| |
| HDFS-1543. Reduce dev. cycle time by moving system testing artifacts from |
| default build and push to maven for HDFS (Luke Lu via cos) |
| |
| HDFS-1818. TestHDFSCLI is failing on trunk after HADOOP-7202. |
| (Aaron T. Myers via todd) |
| |
| HDFS-1828. TestBlocksWithNotEnoughRacks intermittently fails assert. |
| (Matt Foley via eli) |
| |
| HDFS-1824. delay instantiation of file system object until it is |
| needed (linked to HADOOP-7207) (boryas) |
| |
| HDFS-1831. Fix append bug in FileContext and implement CreateFlag |
| check (related to HADOOP-7223). (suresh) |
| |
| HDFS-1594. When the disk becomes full Namenode is getting shutdown and |
| not able to recover. (Aaron T. Myers via eli) |
| |
| HDFS-1822. Handle editlog opcode conflict with 0.20.203 during upgrade, |
| by throwing an error to indicate the editlog needs to be empty. |
| (suresh) |
| |
| HDFS-1808. TestBalancer waits forever, errs without giving information. |
| (Matt Foley via eli) |
| |
| HDFS-1829. TestNodeCount waits forever, errs without giving information. |
| (Matt Foley via eli) |
| |
| HDFS-1860. when renewing/canceling DelegationToken over http we need to |
| pass exception information back to the caller.(boryas) |
| |
| HDFS-1871. Mapreduce build fails due to MiniDFSCluster change from |
| HDFS-1052. (suresh) |
| |
| HDFS-1876. One MiniDFSCluster constructor ignores numDataNodes parameter |
| (todd) |
| |
| HDFS-1773. Do not show decommissioned datanodes, which are not in both |
| include and exclude lists, on web and JMX interfaces. |
| (Tanping Wang via szetszwo) |
| |
| HDFS-1888. MiniDFSCluster#corruptBlockOnDatanodes() access must be |
| public. (suresh) |
| |
| HDFS-1889. incorrect path in start/stop dfs script. (John George via eli) |
| |
| HDFS-1891. Disable IPV6 for junit tests to fix TestBackupNode failure. |
| (suresh) |
| |
| HDFS-1898. Tests failing on trunk due to use of NameNode.format. |
| (todd via eli) |
| |
| HDFS-1902. Fix setrep path display for TestHDFSCLI. (Daryn Sharp |
| via szetszwo) |
| |
| HDFS-1827. Fix timeout problem in TestBlockReplacement. (Matt Foley |
| via szetszwo) |
| |
| HDFS-1908. Fix a NullPointerException in fi.DataTransferTestUtil. |
| (szetszwo) |
| |
| HDFS-1912. Update tests for FsShell standardized error messages. |
| (Daryn Sharp via szetszwo) |
| |
| HDFS-1903. Fix path display for rm/rmr in TestHDFSCLI and TestDFSShell. |
| (Daryn Sharp via szetszwo) |
| |
| HDFS-1627. Fix NullPointerException in Secondary NameNode. (hairong) |
| |
| HDFS-1928. Fix path display for touchz in TestHDFSCLI. |
| (Daryn Sharp via todd) |
| |
| HDFS-1938. Fix ivy-retrieve-hdfs dependence in build.xml and aop.xml. |
| (Eric Yang via szetszwo) |
| |
| HDFS-1929. TestEditLogFileOutputStream fails if running on same host as NN |
| (Aaron T. Myers via todd) |
| |
| HDFS-1933. Update TestDFSShell for improved "test" shell command. (Daryn |
| Sharp via todd) |
| |
| HDFS-1931. Update TestDFSShell for improved "du" shell command. (Daryn |
| Sharp via todd) |
| |
| HDFS-1439. HDFS Federation: Fix compilation error in TestFiHftp. (suresh) |
| |
| HDFS-1881. Federation: after taking snapshot the current directory |
| of datanode is empty. (Tanping Wang via suresh) |
| |
| HDFS-1927. Fix a bug which causes ip=null in NameNode audit log. |
| (John George via szetszwo) |
| |
| HDFS-1953. Federation: Change name node mxbean name in cluster web |
| console. (Tanping Wang via suresh) |
| |
| HDFS-1922. Fix recurring failure of TestJMXGet (Luke Lu via todd) |
| |
| HDFS-1371. One bad node can incorrectly flag many files as corrupt. |
| (Tanping Wang via jitendra) |
| |
| HDFS-1943. Fail to start datanode while start-dfs.sh is executed by |
| root user. (Wei Yongjun via jghoman) |
| |
| HDFS-1983. Fix path display for copy and rm commands in TestHDFSCLI and |
| TestDFSShell. (Daryn Sharp via todd) |
| |
| HDFS-1999. Tests use deprecated configs. (Aaron T. Myers via eli) |
| |
| HDFS-1592. Datanode startup doesn't honor volumes.tolerated. |
| (Bharath Mundlapudi via jitendra) |
| |
| HDFS-1920. libhdfs does not build for ARM processors. |
| (Trevor Robinson via eli) |
| |
| HDFS-1936. Layout version change from HDFS-1822 causes upgrade failure. |
| (suresh) |
| |
| HDFS-2021. Update numBytesAcked before sending the ack in PacketResponder. |
| (John George via szetszwo) |
| |
| HDFS-2020. Fix TestDFSUpgradeFromImage by removing the use of DataNode |
| as a singleton. (suresh via todd) |
| |
| HDFS-2022. ant binary should build libhdfs. (Eric Yang via eli) |
| |
| HDFS-2014. Change HDFS scripts to work in developer enviroment post |
| RPM packaging changes. (Eric Yang via suresh) |
| |
| HDFS-1995. Federation: Minor bug fixes and modification cluster web UI. |
| (Tanping Wang via suresh) |
| |
| HDFS-1907. Fix position read for reading still-being-written file in |
| DFSInputStream. (John George via szetszwo) |
| |
| HDFS-1923. In TestFiDataTransferProtocol2, reduce random sleep time period |
| and increase the number of datanodes. (szetszwo) |
| |
| HDFS-1149. Lease reassignment should be persisted to the edit log. |
| (Aaron T. Myers via todd) |
| |
| HDFS-1998. Federation: Make refresh-namenodes.sh refresh all the |
| namenode. (Tanping Wang via suresh) |
| |
| HDFS-2041. OP_CONCAT_DELETE doesn't properly restore modification time |
| of the concatenated file when edit logs are replayed. (todd) |
| |
| HDFS-2063. libhdfs test is broken. (Eric Yang via eli) |
| |
| HDFS-2067. Bump DATA_TRANSFER_VERSION constant in trunk after introduction |
| of protocol buffers in the protocol. (szetszwo via todd) |
| |
| HDFS-2069. Incorrect default trash interval value in the docs. |
| (Harsh J Chouraria via eli) |
| |
| HDFS-1942. Datanode must exist when all the block pool service threads |
| exit. (Bharath Mundlapudi via suresh) |
| |
| HDFS-1656. Fixes an issue to do with fetching of delegation tokens in |
| HftpFileSystem. Contributed by Kan Zhang. |
| |
| HDFS-1692. In secure mode, Datanode process doesn't exit when disks |
| fail. (Bharath Mundlapudi via suresh) |
| |
| HDFS-1734. 'Chunk size to view' option is not working in Name Node UI. |
| (Uma Maheswara Rao G via jitendra) |
| |
| HDFS-2086. If the include hosts list contains host names, after restarting |
| namenode, data nodes registration is denied. Contributed by Tanping Wang. |
| |
| HDFS-2082. SecondaryNameNode web interface doesn't show the right info. (atm) |
| |
| HDFS-1321. If service port and main port are the same, there is no clear |
| log message explaining the issue. (Jim Plush via atm) |
| |
| HDFS-1381. HDFS javadocs hard-code references to dfs.namenode.name.dir and |
| dfs.datanode.data.dir parameters (Jim Plush via atm) |
| |
| HDFS-2053. Bug in INodeDirectory#computeContentSummary warning. |
| (Michael Noll via eli) |
| |
| HDFS-1990. Fix resource leaks in BlockReceiver.close(). (Uma Maheswara |
| Rao G via szetszwo) |
| |
| HDFS-2034. Length in DFSInputStream.getBlockRange(..) becomes -ve when |
| reading only from a currently being written block. (John George via |
| szetszwo) |
| |
| HDFS-2132. Potential resource leak in EditLogFileOutputStream.close. (atm) |
| |
| HDFS-2120. on reconnect, DN can connect to NN even with different source |
| versions. (John George via atm) |
| |
| HDFS-2152. TestWriteConfigurationToDFS causing the random failures. (Uma |
| Maheswara Rao G via atm) |
| |
| HDFS-2114. re-commission of a decommissioned node does not delete |
| excess replicas. (John George via mattf) |
| |
| HDFS-1776. Bug in Concat code. (Bharath Mundlapudi via Dmytro Molkov) |
| |
| HDFS-2196. Make ant build system work with hadoop-common JAR generated |
| by Maven. (Alejandro Abdelnur via tomwhite) |
| |
| HDFS-2245. Fix a NullPointerException in BlockManager.chooseTarget(..). |
| (szetszwo) |
| |
| HDFS-2229. Fix a deadlock in namenode by enforcing lock acquisition |
| ordering. (szetszwo) |
| |
| HDFS-2235. Encode servlet paths. (eli) |
| |
| HDFS-2186. DN volume failures on startup are not counted. (eli) |
| |
| HDFS-2240. Fix a deadlock in LeaseRenewer by enforcing lock acquisition |
| ordering. (szetszwo) |
| |
| HDFS-73. DFSOutputStream does not close all the sockets. |
| (Uma Maheswara Rao G via eli) |
| |
| HDFS-1257. Fix a race condition on BlockManager.recentInvalidateSets. |
| (Eric Payne via szetszwo) |
| |
| HDFS-2267. DataXceiver thread name incorrect while waiting on op during |
| keepalive. (todd) |
| |
| HDFS-1480. All replicas of a block can end up on the same rack when |
| some datanodes are decommissioning. (todd) |
| |
| HDFS-2286. DataXceiverServer logs AsynchronousCloseException at shutdown |
| (todd) |
| |
| HDFS-2289. Ensure jsvc is bundled with the HDFS distribution artifact. |
| (Alejandro Abdelnur via acmurthy) |
| |
| HDFS-2314. MRV1 test compilation broken after HDFS-2197 (todd) |
| |
| HDFS-2323. start-dfs.sh script fails for tarball install (tomwhite) |
| |
| HDFS-2346. TestHost2NodesMap & TestReplicasMap will fail depending upon |
| execution order of test methods (Laxman via atm) |
| |
| HDFS-2345. TestLeaseRecovery2 fails on 0.23 branch (Uma Maheswara Rao G |
| via atm) |
| |
| HDFS-2412. Add backwards-compatibility layer for renamed FSConstants |
| class (todd) |
| |
| HDFS-2414. Fix TestDFSRollback to avoid spurious failures. (todd) |
| |
| HDFS-2422. The NN should tolerate the same number of low-resource volumes |
| as failed volumes (atm) |
| |
| HDFS-2467. HftpFileSystem uses incorrect compare for finding delegation |
| tokens. (omalley) |
| |
| HDFS-2331. Fix WebHdfsFileSystem compilation problems for a bug in JDK |
| version < 1.6.0_26. (Abhijit Suresh Shingate via szetszwo) |
| |
| HDFS-2333. Change DFSOutputStream back to package private, otherwise, |
| there are two SC_START_IN_CTOR findbugs warnings. (szetszwo) |
| |
| HDFS-2366. Initialize WebHdfsFileSystem.ugi in object construction. |
| (szetszwo) |
| |
| HDFS-2361. hftp is broken, fixed username checks in JspHelper. (jitendra) |
| |
| HDFS-2403. NamenodeWebHdfsMethods.generateDelegationToken(..) does not use |
| the renewer parameter. (szetszwo) |
| |
| HDFS-2409. _HOST in dfs.web.authentication.kerberos.principal. (jitendra) |
| |
| HDFS-2404. WebHDFS liststatus json response is not correct. (suresh) |
| |
| HDFS-2441. Remove the Content-Type set by HttpServer.QuotingInputFilter in |
| WebHDFS responses. (szetszwo) |
| |
| HDFS-2428. Convert com.sun.jersey.api.ParamException$QueryParamException |
| to IllegalArgumentException and response it as http BAD_REQUEST in WebHDFS. |
| (szetszwo) |
| |
| HDFS-2424. Added a root element "HdfsFileStatuses" for the response |
| of WebHDFS listStatus. (szetszwo) |
| |
| MAPREDUCE-2764. Fix renewal of dfs delegation tokens. (Owen via jitendra) |
| |
| HDFS-2439. Fix NullPointerException in WebHDFS when opening a non-existing |
| file or creating a file without specifying the replication parameter. |
| (szetszwo) |
| |
| HDFS-2453. Fix http response code for partial content in WebHDFS, added |
| getDefaultBlockSize() and getDefaultReplication() in WebHdfsFileSystem |
| and cleared content type in ExceptionHandler. (szetszwo) |
| |
| HDFS-2411. The the auth to local mappings are not being respected, with |
| WebHDFS enabled. (jitendra) |
| |
| HDFS-2494. Close the streams and DFSClient in DatanodeWebHdfsMethods. |
| (Uma Maheswara Rao G via szetszwo) |
| |
| HDFS-2298. Fix TestDfsOverAvroRpc by changing ClientProtocol to |
| not include multiple methods of the same name. (cutting) |
| |
| HDFS-2432. WebHDFS: response FORBIDDEN when setReplication on non-files; |
| clear umask before creating a flie; throw IllegalArgumentException if |
| setOwner with both owner and group empty; throw FileNotFoundException if |
| getFileStatus on non-existing files; fix bugs in getBlockLocations; and |
| changed getFileChecksum json response root to "FileChecksum". (szetszwo) |
| |
| HDFS-2065. Add null checks in DFSClient.getFileChecksum(..). (Uma |
| Maheswara Rao G via szetszwo) |
| |
| HDFS-2522. Disable TestDfsOverAvroRpc test. (suresh) |
| |
| HDFS-2416. distcp with a WebHDFS uri on a secure cluster fails. (jitendra) |
| |
| HDFS-2527. WebHDFS: remove the use of "Range" header in Open; use ugi |
| username if renewer parameter is null in GetDelegationToken; response OK |
| when setting replication for non-files; rename GETFILEBLOCKLOCATIONS to |
| GET_BLOCK_LOCATIONS and state that it is a private unstable API; replace |
| isDirectory and isSymlink with enum {FILE, DIRECTORY, SYMLINK} in |
| HdfsFileStatus JSON object. (szetszwo) |
| |
| HDFS-2528. WebHDFS: set delegation kind to WEBHDFS and add a HDFS token |
| when http requests are redirected to datanode. (szetszwo) |
| |
| HDFS-2540. WebHDFS: change "Expect: 100-continue" to two-step write; change |
| "HdfsFileStatus" and "localName" respectively to "FileStatus" and |
| "pathSuffix" in JSON response. (szetszwo) |
| |
| BREAKDOWN OF HDFS-1073 SUBTASKS |
| |
| HDFS-1521. Persist transaction ID on disk between NN restarts. |
| (Ivan Kelly and Todd Lipcon via todd) |
| |
| HDFS-1538. Refactor more startup and image loading code out of FSImage. |
| (todd) |
| |
| HDFS-1729. Add code to detect valid length of an edits file. (todd) |
| |
| HDFS-1793. Add code to inspect a storage directory with txid-based |
| filenames (todd) |
| |
| HDFS-1794. Add code to list which edit logs are available on a remote NN |
| (todd) |
| |
| HDFS-1858. Add state management variables to FSEditLog (Ivan Kelly and Todd |
| Lipcon via todd) |
| |
| HDFS-1859. Add some convenience functions to iterate over edit log streams |
| (Ivan Kelly and Todd Lipcon via todd) |
| |
| HDFS-1894. Add constants for LAYOUT_VERSIONs in edits log branch (todd) |
| |
| HDFS-1892. Fix EditLogFileInputStream.getValidLength to be aware of |
| OP_INVALID filler (todd) |
| |
| HDFS-1799. Refactor log rolling and filename management out of FSEditLog |
| (Ivan Kelly and Todd Lipcon via todd) |
| |
| HDFS-1801. Remove use of timestamps to identify checkpoints and logs (todd) |
| |
| HDFS-1930. TestDFSUpgrade failing in HDFS-1073 branch (todd) |
| |
| HDFS-1800. Extend image checksumming to function with multiple fsimage |
| files per directory. (todd) |
| |
| HDFS-1725. Set storage directories only at FSImage construction (Ivan Kelly |
| via todd) |
| |
| HDFS-1926. Remove references to StorageDirectory from JournalManager |
| interface (Ivan Kelly via todd) |
| |
| HDFS-1893. Change edit logs and images to be named based on txid (todd) |
| |
| HDFS-1985. Clean up image transfer servlet (todd) |
| |
| HDFS-1984. Enable multiple secondary namenodes to run simultaneously (todd) |
| |
| HDFS-1987. Re-enable TestCheckpoint.testSecondaryImageDownload which was |
| not running previously. (todd) |
| |
| HDFS-1993. TestCheckpoint needs to clean up between cases (todd) |
| |
| HDFS-1992. Remove vestiges of NNStorageListener. (todd) |
| |
| HDFS-1991. Some refactoring of Secondary NameNode to be able to share more |
| code with the BackupNode or CheckpointNode. (todd) |
| |
| HDFS-1994. Fix race conditions when running two rapidly checkpointing |
| Secondary NameNodes. (todd) |
| |
| HDFS-2001. Remove use of previous.checkpoint and lastcheckpoint.tmp |
| directories (todd) |
| |
| HDFS-2015. Remove checkpointTxId from VERSION file. (todd) |
| |
| HDFS-2016. Add infrastructure to remove or archive old and unneeded storage |
| files within the name directories. (todd) |
| |
| HDFS-2047. Improve TestNamespace and TestEditLog in HDFS-1073 branch. |
| (todd) |
| |
| HDFS-2048. Add upgrade tests and fix upgrade from 0.22 with corrupt image. |
| (todd) |
| |
| HDFS-2027. Image inspector should return finalized logs before unfinalized |
| logs. (todd) |
| |
| HDFS-2074. Determine edit log validity by truly reading and validating |
| transactions. (todd) |
| |
| HDFS-2085. Finalize in-progress edit logs at startup. (todd) |
| |
| HDFS-2026. SecondaryNameNode should properly handle the case where the |
| NameNode is reformatted. (todd) |
| |
| HDFS-2077. Address checkpoint upload when one of the storage dirs is failed |
| (todd) |
| |
| HDFS-2078. NameNode should not clear directory when restoring removed |
| storage. (todd) |
| |
| HDFS-2088. Move edits log archiving logic into FSEditLog/JournalManager |
| (todd) |
| |
| HDFS-2093. Handle case where an entirely empty log is left during NN crash |
| (todd) |
| |
| HDFS-2102. Zero-pad edits filename to make them lexically sortable. (Ivan |
| Kelly via todd) |
| |
| HDFS-2010. Fix NameNode to exit if all edit streams become inaccessible. |
| (atm via todd) |
| |
| HDFS-2123. Checkpoint interval should be based on txn count, not size. |
| (todd) |
| |
| HDFS-1979. Fix backupnode for new edits/image layout. (todd) |
| |
| HDFS-2101. Fix remaining unit tests for new storage filenames. (todd) |
| |
| HDFS-2133. Address remaining TODOs and pre-merge cleanup on HDFS-1073 |
| branch. (todd) |
| |
| HDFS-1780. Reduce need to rewrite FSImage on startup. (todd) |
| |
| HDFS-2104. Add a flag to the 2NN to format its checkpoint dirs on startup. |
| (todd) |
| |
| HDFS-2135. Fix regression of HDFS-1955 in HDFS-1073 branch. (todd) |
| |
| HDFS-2160. Fix CreateEditsLog test tool in HDFS-1073 branch. (todd) |
| |
| HDFS-2168. Reenable TestEditLog.testFailedOpen and fix exposed bug. (todd) |
| |
| HDFS-2169. Clean up TestCheckpoint and remove TODOs (todd) |
| |
| HDFS-2170. Address remaining TODOs in HDFS-1073 branch. (todd) |
| |
| HDFS-2172. Address findbugs and javadoc warnings in HDFS-1073 branch. |
| (todd) |
| |
| HDFS-2445. Ensure failed tests exit with proper error code. (Jonathan |
| Eagles via acmurthy) |
| |
| Release 0.22.1 - Unreleased |
| |
| INCOMPATIBLE CHANGES |
| |
| NEW FEATURES |
| |
| IMPROVEMENTS |
| |
| OPTIMIZATIONS |
| |
| HDFS-2718. Optimize OP_ADD in edits loading. (shv) |
| |
| HDFS-2886. CreateEditLogs should generate a realistic edit log. (shv) |
| |
| BUG FIXES |
| |
| HDFS-2877. If locking of a storage dir fails, it will remove the other |
| NN's lock file on exit. (todd) |
| |
| Release 0.22.0 - 2011-11-29 |
| |
| INCOMPATIBLE CHANGES |
| |
| HDFS-1825. Remove thriftfs contrib. (nigel via eli) |
| |
| NEW FEATURES |
| |
| HDFS-992. Re-factor block access token implementation to conform to the |
| generic Token interface in Common (Kan Zhang and Jitendra Pandey via jghoman) |
| |
| HDFS-599. Allow NameNode to have a seprate port for service requests from |
| client requests. (Dmytro Molkov via hairong) |
| |
| HDFS-1004. Update NN to support Kerberized SSL from HADOOP-6584. |
| (jghoman and Kan Zhang via jghoman) |
| |
| HDFS-1005. Fsck security. (borya and Kan Zhang via jghoman) |
| |
| HDFS-1006. getImage/putImage http requests should be https for the case |
| of security enabled. (borya and jghoman via jghoman) |
| |
| HDFS-1033. In secure clusters, NN and SNN should verify that the remote |
| principal during image and edits transfer. (jghoman) |
| |
| HDFS-1023. Allow http server to start as regular principal if https |
| principal not defined. (jghoman) |
| |
| HDFS-1150. Verify datanodes' identities to clients in secure clusters. |
| (jghoman) |
| |
| HDFS-1330. Make RPCs to DataNodes timeout. (hairong) |
| Added additional unit tests per HADOOP-6889. (John George via mattf) |
| |
| HDFS-202. HDFS support of listLocatedStatus introduced in HADOOP-6870. |
| HDFS piggyback block locations to each file status when listing a |
| directory. (hairong) |
| |
| HDFS-1361. Add -fileStatus operation to NNThroughputBenchmark. (shv) |
| |
| HDFS-1435. Provide an option to store fsimage compressed. (hairong) |
| |
| HDFS-903. Support fsimage validation through MD5 checksum. (hairong) |
| |
| HDFS-1457. Provide an option to throttle image transmission between |
| pimary and secondary NameNodes. (Yifei Lu and hairong via hairong) |
| |
| HDFS-1164. TestHdfsProxy is failing. (Todd Lipcon via cos) |
| |
| HDFS-811. Add metrics, failure reporting and additional tests for HDFS-457. |
| (eli) |
| |
| HDFS-895. Allow hflush/sync to occur in parallel with new writes |
| to the file. (Todd Lipcon via hairong) |
| |
| HDFS-528. Add ability for safemode to wait for a minimum number of |
| live datanodes (Todd Lipcon via eli) |
| |
| HDFS-1753. Resource Leak in StreamFile. (Uma Maheswara Rao G via eli) |
| |
| IMPROVEMENTS |
| |
| HDFS-1304. Add a new unit test for HftpFileSystem.open(..). (szetszwo) |
| |
| HDFS-1096. fix for prev. commit. (boryas) |
| |
| HDFS-1096. allow dfsadmin/mradmin refresh of superuser proxy group |
| mappings (boryas) |
| |
| HDFS-1146. Javadoc for getDelegationTokenSecretManager in FSNamesystem (jnp via boryas) |
| |
| HDFS-1132. Refactor TestFileStatus (Eli Collins via cos) |
| |
| HDFS-1163. normalize property names for JT/NN kerberos principal |
| names in configuration (from HADOOP 6633) (boryas) |
| |
| HDFS-1003. authorization checks for inter-server protocol |
| (based on HADOOP-6600) (boryas) |
| |
| HDFS-1061. Memory footprint optimization for INodeFile object. |
| (Bharath Mundlapudi via jghoman) |
| |
| HDFS-1079. Throw exceptions as specified by the AbstractFileSystem |
| in HDFS implemenation and protocols. (suresh) |
| |
| HDFS-1112. Edit log buffer should not grow unfoundedly. (hairong) |
| |
| HDFS-1119. Introduce a GSet interface to BlocksMap. (szetszwo) |
| |
| HDFS-1184. Replace tabs in code with spaces. (Jeff Ames via jghoman) |
| |
| HDFS-1185. Remove duplicate now() functions in DataNode, FSNamesysetm. |
| (Jeff Ames via jghoman) |
| |
| HDFS-1183. Remove some duplicate code in NamenodeJspHelper.java. |
| (Jeff Ames via jghoman) |
| |
| HDFS-1190. Remove unused getNamenode() method from DataNode. |
| (Jeff Ames via jghoman) |
| |
| HDFS-1110. Reuses objects for commonly used file names in namenode to |
| reduce the heap usage. (suresh) |
| |
| HDFS-752. Add interfaces classification to to HDFS source code. (suresh) |
| |
| HDFS-947. An Hftp read request is redirected to a datanode that has |
| the most replicas of the blocks in the file. (Dmytro Molkov via dhruba) |
| |
| HDFS-1272. Fixes to take care of the changes in HADOOP-6845. |
| (Jitendra Pandey via ddas) |
| |
| HDFS-1298 - Add support in HDFS for new statistics added in FileSystem |
| to track the file system operations. (suresh) |
| |
| HDFS-1201. The HDFS component for HADOOP-6632. |
| (Kan Zhang & Jitendra Pandey via ddas) |
| |
| HDFS-1307 Add start time, end time and total time taken for FSCK to |
| FSCK report (suresh) |
| |
| HDFS-1302. The HDFS side of the changes corresponding to HADOOP-6861. |
| (Jitendra Pandey & Owen O'Malley via ddas) |
| |
| HDFS-1315. Add fsck event to audit log and remove other audit log events |
| corresponding to FSCK listStatus and open calls. (suresh) |
| |
| HDFS-1178. The NameNode servlets should not use RPC to connect to the |
| NameNode. (Kan Zhang via jghoman) |
| |
| HDFS-1130. Adds dfs.cluster.administrator ACL configuration that can |
| be used to control who can view the default hdfs servlets. (ddas) |
| |
| HDFS-1297. Fix some comments. (Jeff Ames via jghoman) |
| |
| HDFS-330. Datanode Web UIs should provide robots.txt. |
| (Allen Wittenauer via jghoman) |
| |
| HDFS-881. Refactor DataNode Packet header into DataTransferProtocol. |
| (Todd Lipcon via jghoman) |
| |
| HDFS-1036. docs for fetchdt |
| |
| HDFS-1318. Add JMX interface for read access to namenode and datanode |
| web UI information. (Tanping Wang via suresh). |
| |
| HDFS-1356. Provide information as to whether or not security is |
| enabled on web interface for NameNode (boryas) |
| |
| HDFS-1205. FSDatasetAsyncDiskService should name its threads. |
| (Todd Lipcon via eli) |
| |
| HDFS-1111. Introduce getCorruptFileBlocks() for fsck. (Sriram Rao via shv) |
| |
| HDFS-1395. Add @Override to FSDataset methods that implement |
| FSDatasetInterface methods. (suresh) |
| |
| HDFS-1383. Improve the error messages when using hftp://. (szetszwo) |
| |
| HDFS-1093. Change the FSNamesystem lock to a read/write lock. (dhruba) |
| |
| HDFS-1407. Change DataTransferProtocol methods to use Block instead |
| of individual elements of Block. (suresh) |
| |
| HDFS-1417. Add @Override to SimulatedFSDataset methods that implement |
| FSDatasetInterface methods. (suresh) |
| |
| HDFS-1426. Remove unused method BlockInfo#listCount. (hairong) |
| |
| HDFS-1472. Allow programmatic access to fsck output. |
| (Ramkumar Vadali via dhruba) |
| |
| HADOOP-7007. Update the hudson-test-patch ant target to work with the |
| latest test-patch.sh script (gkesavan) |
| |
| HDFS-1462. Refactor edit log loading to a separate class from edit log writing. |
| (Todd Lipcon via eli) |
| |
| HDFS-1485. Fix typo in BlockPlacementPolicy. (Jingguo Yao via shv) |
| |
| HDFS-1035. Generate Eclipse's .classpath file from Ivy config. (nigel) |
| |
| HDFS-1408. Herriot NN and DN clients should vend statistics. (cos) |
| |
| HDFS-1491 Update Hdfs to match the change of methods from protected to public |
| in AbstractFileSystem (Hadoop-6903) (sanjay) |
| |
| HDFS-1160. Improve some FSDataset warnings and comments. (eli) |
| |
| HDFS-556. Provide info on failed volumes in the web ui. (eli) |
| |
| HDFS-697. Enable asserts for tests by default. (eli) |
| |
| HDFS-1187. Modify fetchdt to allow renewing and canceling token. |
| (Owen O'Malley and Kan Zhang via jghoman) |
| |
| HDFS-1387. Update HDFS permissions guide for security. (Todd Lipcon via eli) |
| |
| HDFS-455. Make NN and DN handle in a intuitive way comma-separated |
| configuration strings. (Michele Catasta via eli) |
| |
| HDFS-1071. savenamespace should write the fsimage to all configured |
| fs.name.dir in parallel (Dmytro Molkov via jghoman) |
| |
| HDFS-1055. Improve thread naming for DataXceivers. |
| (Todd Lipcon and Ramkumar Vadali via eli). |
| |
| HDFS-718. Configuration parameter to prevent accidental formatting of |
| HDFS filesystem. (Andrew Ryan via jghoman) |
| |
| HDFS-1500. TestOfflineImageViewer failing on trunk. (Todd Lipcon |
| via hairong) |
| |
| HDFS-1483. DFSClient.getBlockLocations should indicate if corresponding |
| blocks are corrupt. (Patrick Kling via hairong) |
| |
| HDFS-259. Remove intentionally corrupt 0.13 directory layout creation. |
| (Todd Lipcon via eli) |
| |
| HDFS-1513. Fix a number of warnings. (eli) |
| |
| HDFS-1473. Refactor storage management into separate classes than fsimage |
| file reading/writing. (Todd Lipcon via eli) |
| |
| HDFS-1582. Remove auto-generated native build files. (rvs via eli) |
| |
| HDFS-1456. Provide builder for constructing instances of MiniDFSCluster. |
| (jghoman) |
| |
| HDFS-1861. Rename dfs.datanode.max.xcievers and bump its default value. |
| (eli) |
| |
| HDFS-1052. HDFS Federation - Merge of umbrella jira changes from |
| HDFS-1052 branch into trunk. |
| |
| HDFS-1835. DataNode should not depend on SHA1PRNG secure random generator |
| to generate a storage ID. (John Carrino via todd) |
| |
| HDFS-1947. DFSClient should use mapreduce.task.attempt.id. (eli) |
| |
| HDFS-1957. Add documentation for HFTP. (Ari Rabkin via todd) |
| |
| HDFS-1454. Update the documentation to reflect that clients don't write |
| blocks to local disk before copying to HDFS. (Harsh J Chouraria via todd) |
| |
| HDFS-1980. Move build/webapps deeper in the build directory heirarchy |
| to aid eclipse users. (todd) |
| |
| HDFS-1619. Remove AC_TYPE* from the libhdfs. (Roman Shaposhnik via eli) |
| |
| HDFS-1948 Forward port 'hdfs-1520 lightweight namenode operation to |
| trigger lease recovery' (stack) |
| |
| HDFS-1954. Improved corrupt files warning on NameNode web UI. |
| (Patrick Hunt via shv) |
| |
| HDFS-1409. BackupNode registration throwing |
| UnsupportedActionException("register") instead of "journal". |
| (Ching-Shen Chen via shv) |
| |
| HDFS-2054 BlockSender.sendChunk() prints ERROR for connection closures |
| encountered during transferToFully() (Kihwal Lee via stack) |
| |
| OPTIMIZATIONS |
| |
| HDFS-1140. Speedup INode.getPathComponents. (Dmytro Molkov via shv) |
| |
| HDFS-1081. Performance regression in |
| DistributedFileSystem::getFileBlockLocations in secure systems (jghoman) |
| |
| HDFS-1114. Implement LightWeightGSet for BlocksMap in order to reduce |
| NameNode memory footprint. (szetszwo) |
| |
| HDFS-1320. Add LOG.isDebugEnabled() guard for each LOG.debug(..). |
| (Erik Steffl via szetszwo) |
| |
| HDFS-1368. Add a block counter to DatanodeDescriptor. (hairong) |
| |
| HDFS-1434. Refactor Datanode#startDataNode method into smaller methods. |
| (suresh) |
| |
| HDFS-941. The DFS client should cache and reuse open sockets to datanodes |
| while performing reads. (bc Wong and Todd Lipcon via todd) |
| |
| BUG FIXES |
| |
| HDFS-1039. Adding test for JspHelper.getUGI(jnp via boryas) |
| |
| HDFS-1019. Incorrect default values for delegation tokens in |
| hdfs-default.xml (jnp via boryas) |
| |
| HDFS-1039. Service should be set in the token in JspHelper.getUGI(jnp via boryas) |
| |
| HDFS-1038. FIX. A test missed in a previous commit for this JIRA. (boryas) |
| |
| HDFS-1038. In nn_browsedfscontent.jsp fetch delegation token only |
| if security is enabled. (jnp via boryas) |
| |
| HDFS-1044. Cannot submit mapreduce job from secure client to |
| unsecure sever (boryas) |
| |
| HDFS-1021. specify correct server principal for RefreshAuthorizationPolicyProtocol |
| and RefreshUserToGroupMappingsProtocol protocols in DFSAdmin (for HADOOP-6612) (boryas) |
| |
| HDFS-970. fsync fsimage to disk before closing fsimage file. |
| (Todd Lipcon via dhruba) |
| |
| HDFS-1027. Update copyright year to 2010. (Ravi Phulari via jghoman) |
| |
| HDFS-1080. SecondaryNameNode image transfer should use the defined http |
| address rather than local ip address. (jghoman) |
| |
| HDFS-1198. Resolving cross-realm principals. (Jitendra Pandey via jghoman) |
| |
| HDFS-1118. Fix socketleak on DFSClient. (Zheng Shao via dhruba) |
| |
| HDFS-1192. refreshSuperUserGroupsConfiguration should use server side |
| configuration for the refresh (for HADOOP-6815) (boryas) |
| |
| HDFS-1036. in DelegationTokenFetch dfs.getURI returns no port (boryas) |
| |
| HDFS-1017. browsedfs jsp should call JspHelper.getUGI rather |
| than using createRemoteUser() (jnp via boryas) |
| |
| HDFS-1250. Namenode should reject block reports and block received |
| requests from dead datanodes (suresh) |
| |
| HDFS-1145. When NameNode is shutdown it does not try to exit |
| safemode anymore. (dhruba) |
| |
| HDFS-1202. DataBlockScanner throws NPE when updated before |
| initialized. (Todd Lipcon via dhruba) |
| |
| HDFS-882. Datanode logs the hostname and port its listening on. |
| (Steve Loughran via dhruba) |
| |
| HDFS-1238. ant eclipse-files has drifted again, (jghoman) |
| |
| HDFS-1045. In secure clusters, re-login is necessary for https |
| clients before opening connections. (jghoman) |
| |
| HDFS-1289. Datanode secure mode is broken. (Kan Zhang via jghoman) |
| |
| HDFS-1007. HFTP needs to be updated to use delegation tokens (boryas) |
| |
| HDFS-1085. HFTP read may fail silently on the client side if there is an |
| exception on the server side. (szetszwo) |
| |
| HDFS-1308. job conf key for the services name of DelegationToken for HFTP |
| url is constructed incorrectly in HFTPFileSystem (boryas) |
| |
| HDFS-1319. Fix location of re-login for secondary namenode from HDFS-999. |
| (jghoman) |
| |
| HDFS-1317. Remove the FILEPATH_PATTERN from hdfsproxy.AuthorizationFilter. |
| (Rohini Palaniswamy via szetszwo) |
| |
| HDFS-912. sed in build.xml on Solaris fails. (Allen Wittenauer via jghoman) |
| |
| HDFS-1296. using delegation token over hftp for long running |
| clients (boryas) |
| |
| HDFS-1334. open in HftpFileSystem does not add delegation tokens to the url. |
| (Jitendra Pandey via jghoman) |
| |
| HDFS-1301. TestHDFSProxy need to use server side conf for ProxyUser |
| stuff. (boryas) |
| |
| HDFS-1340. When security is turned off, there is a potential XSS attack. |
| This patch fixes it by removing delegationtoken string from the URL, |
| before returning a response to the client. (Jitendra Pandey via ddas) |
| |
| HDFS-1347. TestDelegationToken uses mortbay.log for logging (boryas) |
| |
| HDFS-1157. Modifications introduced by HDFS-1150 are breaking aspect's |
| bindings (cos) |
| |
| HDFS-1349. Remove empty java files. (Eli Collins) |
| |
| HDFS-1340. A null delegation token is appended to the url if security |
| is disabled when browsing filesystem. (boryas) |
| |
| HDFS-1352. Fix jsvc.location. (Eli Collins via jghoman) |
| |
| HDFS-1284. TestBlockToken fails. (Kan Zhang via jghoman) |
| |
| HDFS-1355. ant veryclean (clean-cache) doesn't clean enough. |
| (Luke Lu via jghoman) |
| |
| HDFS-1353. Remove most of getBlockLocation optimization. (jghoman) |
| |
| HDFS-1369. Invalid javadoc reference in FSDatasetMBean.java (Eli Collins) |
| |
| HDFS-829. hdfsJniHelper.c: #include <error.h> is not portable. |
| (Allen Wittenauer via jghoman) |
| |
| HDFS-1310. The ClientDatanodeProtocol proxy should be stopped in |
| DFSInputStream.readBlockLength(..). (sam rash via szetszwo) |
| |
| HDFS-1357. HFTP traffic served by DataNode shouldn't use service port |
| on NameNode. (Kan Zhang via jghoman) |
| |
| HDFS-1419. HDFS Federation: Three test cases need minor modification after |
| the new block id change (Tanping Wang via suresh) |
| |
| HDFS-96. HDFS supports blocks larger than 2 GB. |
| (Patrick Kling via dhruba) |
| |
| HDFS-1433. Fix test failures - TestPread and TestFileLimit. (suresh) |
| |
| HDFS-1364. Makes long running HFTP-based applications do relogins |
| if necessary. (Jitendra Pandey via ddas) |
| |
| HDFS-1399. Distinct minicluster services (e.g. NN and JT) overwrite each |
| other's service policies. (Aaron T. Myers via tomwhite) |
| |
| HDFS-1440. Fix TestComputeInvalidateWork failure. (suresh) |
| |
| HDFS-1498. FSDirectory#unprotectedConcat calls setModificationTime |
| on a file. (eli) |
| |
| HDFS-1625. Ignore disk space values in TestDataNodeMXBean. (szetszwo) |
| |
| HDFS-1850. DN should transmit absolute failed volume count rather than |
| increments to the NN. (eli) |
| |
| HDFS-671. Documentation change for updated configuration keys. |
| (tomwhite via eli) |
| |
| HDFS-1544. Ivy resolve force mode should be turned off by default. |
| (Luke Lu via tomwhite) |
| |
| HDFS-1615. seek() on closed DFS input stream throws NullPointerException |
| (Scott Carey via todd) |
| |
| HDFS-1897. Documentation refers to removed option dfs.network.script |
| (Andrew Whang via todd) |
| |
| HDFS-1621. Fix references to hadoop-common-${version} in build.xml |
| (Jolly Chen via todd) |
| |
| HDFS-1505. saveNamespace appears to succeed even if all directories fail |
| to save. (Aaron T. Myers via todd) |
| |
| HDFS-1921. saveNamespace can cause NN to be unable to come up on restart |
| (Matt Foley via todd) |
| |
| HDFS-1925. SafeModeInfo should use the correct constant instead of a |
| hard-coded value for its default. (Joey Echeverria via todd) |
| |
| HDFS-1575. Viewing block from web UI is broken. (Aaron T. Myers via todd) |
| |
| HDFS-1932. Ensure that HDFS configuration deprecations are set up in every |
| spot that HDFS configurations are loaded. (Jolly Chen via todd) |
| |
| HDFS-1952. FSEditLog.open() appears to succeed even if all EDITS |
| directories fail. (Andrew Wang via todd) |
| |
| HDFS-1965. IPCs done using block token-based tickets can't reuse |
| connections (todd) |
| |
| HDFS-1978. All but first option in LIBHDFS_OPTS is ignored. (eli) |
| |
| HDFS-1964. Fix incorrect HTML unescaping in DatanodeJspHelper |
| (Aaron T. Myers via todd) |
| |
| HDFS-1997. Image transfer process misreports client side exceptions. |
| (todd via eli) |
| |
| HDFS-2000. Missing deprecation for io.bytes.per.checksum. |
| (Aaron T. Myers vie eli) |
| |
| HDFS-977. DataNode.createInterDataNodeProtocolProxy() guards a log |
| at the wrong level. (Harsh J Chouraria via todd) |
| |
| HDFS-1969. Running rollback on new-version namenode destroys the |
| namespace. (todd) |
| |
| HDFS-2039. TestNameNodeMetrics uses a bad test root path, preventing it |
| from running inside Eclipse. (todd) |
| |
| HDFS-988. saveNamespace race can corrupt the edits log. (eli) |
| |
| HDFS-2071. Use of isConnected() in DataXceiver is invalid. (Kihwal Lee |
| via todd) |
| |
| HDFS-1981. NameNode does not saveNamespace() when editsNew is empty. |
| (Uma Maheswara Rao G via shv) |
| |
| HDFS-2232. Generalize regular expressions in TestHDFSCLI. |
| (Plamen Jeliazkov via shv) |
| |
| HDFS-2290. Block with corrupt replica is not getting replicated. |
| (Benoy Antony via shv) |
| |
| HDFS-2452. OutOfMemoryError in DataXceiverServer takes down the DataNode |
| (Uma Maheswara Rao via cos) |
| |
| HDFS-2002. Incorrect computation of needed blocks in getTurnOffTip(). |
| (Plamen Jeliazkov via shv) |
| |
| HDFS-2514. Link resolution bug for intermediate symlinks with |
| relative targets. (eli) |
| |
| Release 0.21.1 - Unreleased |
| |
| HDFS-1466. TestFcHdfsSymlink relies on /tmp/test not existing. (eli) |
| |
| HDFS-874. TestHDFSFileContextMainOperations fails on weirdly |
| configured DNS hosts. (Todd Lipcon via eli) |
| |
| HDFS-1507. TestAbandonBlock should abandon a block. (eli) |
| |
| HDFS-1487. FSDirectory.removeBlock() should update diskspace count |
| of the block owner node (Zhong Wang via eli). |
| |
| HDFS-1467. Append pipeline never succeeds with more than one replica. |
| (Todd Lipcon via eli) |
| |
| HDFS-1167. New property for local conf directory in system-test-hdfs.xml |
| file. (Vinay Thota via cos) |
| |
| HDFS-1503. TestSaveNamespace fails. (Todd Lipcon via cos) |
| |
| HDFS-1524. Image loader should make sure to read every byte in image file. |
| (hairong) |
| |
| HDFS-1523. TestLargeBlock is failing on trunk. (cos) |
| |
| HDFS-1502. TestBlockRecovery triggers NPE in assert. (hairong via cos) |
| |
| HDFS-1532. Exclude Findbugs warning in FSImageFormat$Saver. (Todd Lipcon |
| via cos) |
| |
| HDFS-1527. SocketOutputStream.transferToFully fails for blocks >= 2GB on |
| 32 bit JVM. (Patrick Kling via cos) |
| |
| HDFS-1531. Clean up stack traces due to duplicate MXBean registration. |
| (Todd Lipcon via cos) |
| |
| HDFS-613. TestBalancer and TestBlockTokenWithDFS fail Balancer assert. |
| (Todd Lipcon via cos) |
| |
| HDFS-1511. 98 Release Audit warnings on trunk and branch-0.22. |
| (jghoman) |
| |
| HDFS-1560. dfs.data.dir permissions should default to 700. |
| (Todd Lipcon via eli) |
| |
| HDFS-1550. NPE when listing a file with no location. (hairong) |
| |
| HDFS-1542. Add test for HADOOP-7082, a deadlock writing Configuration to |
| HDFS. (todd) |
| |
| HDFS-1504. FSImageSaver should catch all exceptions, not just IOE. (todd) |
| |
| HDFS-884. DataNode throws IOException if all data directories are |
| unavailable. (Steve Loughran and shv) |
| |
| HDFS-1591. HDFS part of HADOOP-6642. (Chris Douglas, Po Cheung via shv) |
| |
| HDFS-900. Corrupt replicas are not processed correctly in block report (shv) |
| |
| HDFS-1529. Incorrect handling of interrupts in waitForAckedSeqno can cause |
| deadlock (todd) |
| |
| HDFS-1597. Batched edit log syncs can reset synctxid and throw assertions |
| (todd) |
| |
| HDFS-1602. Fix HADOOP-4885 for it is doesn't work as expected. (boryas) |
| |
| HDFS-1618. configure files that are generated as part of the released |
| tarball need to have executable bit set (Roman Shaposhnik via cos) |
| |
| HDFS-981. test-contrib fails due to test-cactus failure (cos) |
| |
| HDFS-1001. DataXceiver and BlockReader disagree on when to send/recv |
| CHECKSUM_OK. (bc Wong via eli) |
| |
| HDFS-1781. Fix the path for jsvc in bin/hdfs. (John George via szetszwo) |
| |
| HDFS-1782. Fix an NPE in FSNamesystem.startFileInternal(..). |
| (John George via szetszwo) |
| |
| HDFS-1821. Fix username resolution in NameNode.createSymlink(..) and |
| FSDirectory.addSymlink(..). (John George via szetszwo) |
| |
| HDFS-1806. TestBlockReport.blockReport_08() and _09() are timing-dependent |
| and likely to fail on fast servers. (Matt Foley via eli) |
| |
| HDFS-1845. Symlink comes up as directory after namenode restart. |
| (John George via eli) |
| |
| HDFS-1666. Disable failing hdfsproxy test TestAuthorizationFilter (todd) |
| |
| HDFS-1823. start-dfs.sh script fails if HADOOP_HOME is not set. |
| (tomwhite via eli) |
| |
| Release 0.21.1 - Unreleased |
| |
| HDFS-1411. Correct backup node startup command in hdfs user guide. |
| (Ching-Shen Chen via shv) |
| |
| BUG FIXES |
| |
| HDFS-1363. Eliminate second synchronized sections in appendFile(). (shv) |
| |
| HDFS-1413. Fix broken links to HDFS Wiki. (shv) |
| |
| HDFS-1420. Clover build doesn't generate per-test coverage (cos) |
| |
| HDFS-1444. Test related code of build.xml is error-prone and needs to be |
| re-aligned. (cos) |
| |
| HDFS-1343. Instrumented build should be concentrated in one build area (cos) |
| |
| HDFS-1452. ant compile-contrib is broken (cos) |
| |
| HDFS-1474. ant binary-system is broken (cos) |
| |
| HDFS-1292. Allow artifacts to be published to the staging Apache Nexus |
| Maven Repository. (Giridharan Kesavan via tomwhite) |
| |
| HDFS-1552. Remove java5 dependencies from build. (cos) |
| |
| HDFS-1189. Quota counts missed between clear quota and set quota. |
| (John George via szetszwo) |
| |
| HDFS-1665. Balancer misuses dfs.heartbeat.interval as milliseconds. |
| (szetszwo) |
| |
| HDFS-1728. SecondaryNameNode.checkpointSize is in bytes but not in MB. |
| (szetszwo) |
| |
| HDFS-1206. TestFiHFlush fails intermittently. (cos) |
| |
| HDFS-1548. Fault-injection tests are executed multiple times if invoked |
| with run-test-hdfs-fault-inject target (cos) |
| |
| HDFS-1552. Remove java5 dependencies from build. (cos) |
| |
| HDFS-996. JUnit tests should never depend on anything in conf (cos) |
| |
| HDFS-1612. Update HDFS design documentation for append, quota, symlink, |
| block placement and checkpoint/backup node features. (Joe Crobak |
| via szetszwo) |
| |
| |
| HDFS-1596. Replace fs.checkpoint.* with dfs.namenode.checkpoint.* |
| in documentations. (Harsh J Chouraria via szetszwo) |
| |
| HDFS-1786. Some cli test cases expect a "null" message |
| (Uma Maheswara Rao G via todd) |
| |
| HDFS-1855. TestDatanodeBlockScanner.testBlockCorruptionRecoveryPolicy() |
| part 2 fails in two different ways. (Matt Foley via eli) |
| |
| Release 0.21.0 - 2010-08-13 |
| |
| INCOMPATIBLE CHANGES |
| |
| HDFS-538. Per the contract elucidated in HADOOP-6201, throw |
| FileNotFoundException from FileSystem::listStatus rather than returning |
| null. (Jakob Homan via cdouglas) |
| |
| HDFS-602. DistributedFileSystem mkdirs throws FileAlreadyExistsException |
| instead of FileNotFoundException. (Boris Shkolnik via suresh) |
| |
| HDFS-544. Add a "rbw" subdir to DataNode data directory. (hairong) |
| |
| HDFS-576. Block report includes under-construction replicas. (shv) |
| |
| HDFS-636. SafeMode counts complete blocks only. (shv) |
| |
| HDFS-644. Lease recovery, concurrency support. (shv) |
| |
| HDFS-570. Get last block length from a data-node when opening a file |
| being written to. (Tsz Wo (Nicholas), SZE via shv) |
| |
| HDFS-657. Remove unused legacy data-node protocol methods. (shv) |
| |
| HDFS-658. Block recovery for primary data-node. (shv) |
| |
| HDFS-660. Remove deprecated methods from InterDatanodeProtocol. (shv) |
| |
| HDFS-512. Block.equals() and compareTo() compare blocks based |
| only on block Ids, ignoring generation stamps. (shv) |
| |
| HDFS-873. Configuration specifies data-node storage directories as URIs. |
| (shv) |
| |
| HDFS-905. Use the new UserGroupInformation from HDFS-6299. |
| (jghoman via omalley) |
| |
| HDFS-984. Persistent delegation tokens. (Jitendra Pandey via shv) |
| |
| HDFS-1016. HDFS side change for HADOOP-6569. This jira changes the |
| error message on the screen when cat a directory or a |
| non-existent file. (hairong) |
| |
| NEW FEATURES |
| |
| HDFS-1134. Large-scale Automated Framework. (cos) |
| |
| HDFS-436. Introduce AspectJ framework for HDFS code and tests. |
| (Konstantin Boudnik via szetszwo) |
| |
| HDFS-447. Add LDAP lookup to hdfsproxy. (Zhiyong Zhang via cdouglas) |
| |
| HDFS-459. Introduce Job History Log Analyzer. (shv) |
| |
| HDFS-461. Tool to analyze file size distribution in HDFS. (shv) |
| |
| HDFS-492. Add two JSON JSP pages to the Namenode for providing corrupt |
| blocks/replicas information. (Bill Zeller via szetszwo) |
| |
| HDFS-578. Add support for new FileSystem method for clients to get server |
| defaults. (Kan Zhang via suresh) |
| |
| HDFS-595. umask settings in configuration may now use octal or symbolic |
| instead of decimal. (Jakob Homan via suresh) |
| |
| HADOOP-6234. Updated hadoop-core and test jars to propagate new option |
| dfs.umaskmode in configuration. (Jakob Homan via suresh) |
| |
| HDFS-235. Add support for byte ranges in HftpFileSystem to serve |
| range of bytes from a file. (Bill Zeller via suresh) |
| |
| HDFS-385. Add support for an experimental API that allows a module external |
| to HDFS to specify how HDFS blocks should be placed. (dhruba) |
| |
| HADOOP-4952. Update hadoop-core and test jars to propagate new FileContext |
| file system application interface. (Sanjay Radia via suresh). |
| |
| HDFS-567. Add block forensics contrib tool to print history of corrupt and |
| missing blocks from the HDFS logs. |
| (Bill Zeller, Jitendra Nath Pandey via suresh). |
| |
| HDFS-610. Support o.a.h.fs.FileContext. (Sanjay Radia via szetszwo) |
| |
| HDFS-536. Support hflush at DFSClient. (hairong) |
| |
| HDFS-517. Introduce BlockInfoUnderConstruction to reflect block replica |
| states while writing. (shv) |
| |
| HDFS-565. Introduce block committing logic during new block allocation |
| and file close. (shv) |
| |
| HDFS-537. DataNode exposes a replica's meta info to BlockReceiver for the |
| support of dfs writes/hflush. It also updates a replica's bytes received, |
| bytes on disk, and bytes acked after receiving a packet. (hairong) |
| |
| HDFS-585. Datanode should serve up to visible length of a replica for read |
| requests. (szetszwo) |
| |
| HDFS-604. Block report processing for append. (shv) |
| |
| HDFS-619. Support replica recovery initialization in datanode for the new |
| append design. (szetszwo) |
| |
| HDFS-592. Allow clients to fetch a new generation stamp from NameNode for |
| pipeline recovery. (hairong) |
| |
| HDFS-624. Support a new algorithm for pipeline recovery and pipeline setup |
| for append. (hairong) |
| |
| HDFS-627. Support replica update in data-node. |
| (Tsz Wo (Nicholas), SZE and Hairong Kuang via shv) |
| |
| HDFS-642. Support pipeline close and close error recovery. (hairong) |
| |
| HDFS-631. Rename configuration keys towards API standardization and |
| backward compatibility. (Jitendra Nath Pandey via suresh) |
| |
| HDFS-669. Add unit tests framework (Mockito) (cos, Eli Collins) |
| |
| HDFS-731. Support new Syncable interface in HDFS. (hairong) |
| |
| HDFS-702. Add HDFS implementation of AbstractFileSystem. |
| (Sanjay Radio via suresh) |
| |
| HDFS-758. Add decommissioning status page to Namenode Web UI. |
| (Jitendra Nath Pandey via suresh) |
| |
| HDFS-814. Add an api to get the visible length of a DFSDataInputStream. |
| (szetszwo) |
| |
| HDFS-654. Add support new atomic rename functionality in HDFS for |
| supporting rename in FileContext. (suresh) |
| |
| HDFS-222. Support for concatenating of files into a single file |
| without copying. (Boris Shkolnik via hairong) |
| |
| HDFS-933. Adds Delegation token based authentication in the NameNode. |
| (Kan Zhang via ddas) |
| |
| HDFS-935. Adds a real user component in Delegation token. |
| (Jitendra Nath Pandey via ddas) |
| |
| HDFS-245. Adds a symlink implementation to HDFS. This complements the new |
| symlink feature added in HADOOP-6421 (Eli Collins via Sanjay Radia) |
| |
| HDFS-1009. Support Kerberos authorization in HDFSProxy. (Srikanth |
| Sundarrajan via szetszwo) |
| |
| HDFS-1091. Implement listStatus that returns an iterator of FileStatus. |
| (hairong) |
| |
| IMPROVEMENTS |
| |
| HDFS-381. Remove blocks from DataNode maps when corresponding file |
| is deleted. (Suresh Srinivas via rangadi) |
| |
| HDFS-377. Separate codes which implement DataTransferProtocol. |
| (szetszwo) |
| |
| HDFS-396. NameNode image and edits directories are specified as URIs. |
| (Luca Telloli via rangadi) |
| |
| HDFS-444. Allow to change probability levels dynamically in the fault |
| injection framework. (Konstantin Boudnik via szetszwo) |
| |
| HDFS-352. Documentation for saveNamespace command. (Ravi Phulari via shv) |
| |
| HADOOP-6106. Updated hadoop-core and test jars from hudson trunk |
| build #12. (Giridharan Kesavan) |
| |
| HDFS-204. Add a new metrics FilesInGetListingOps to the Namenode. |
| (Jitendra Nath Pandey via szetszwo) |
| |
| HDFS-278. HDFS Outputstream close does not hang forever. (dhruba) |
| |
| HDFS-443. Add a new metrics numExpiredHeartbeats to the Namenode. |
| (Jitendra Nath Pandey via szetszwo) |
| |
| HDFS-475. Add new ant targets for fault injection jars and tests. |
| (Konstantin Boudnik via szetszwo) |
| |
| HDFS-458. Create a new ant target, run-commit-test. (Jakob Homan |
| via szetszwo) |
| |
| HDFS-493. Change build.xml so that the fault-injected tests are executed |
| only by the run-test-*-fault-inject targets. (Konstantin Boudnik via |
| szetszwo) |
| |
| HDFS-446. Improvements to Offline Image Viewer. (Jakob Homan via shv) |
| |
| HADOOP-6160. Fix releaseaudit target to run on specific directories. |
| (gkesavan) |
| |
| HDFS-501. Use enum to define the constants in DataTransferProtocol. |
| (szetszwo) |
| |
| HDFS-508. Factor out BlockInfo from BlocksMap. (shv) |
| |
| HDFS-510. Rename DatanodeBlockInfo to be ReplicaInfo. |
| (Jakob Homan & Hairong Kuang via shv) |
| |
| HDFS-500. Deprecate NameNode methods deprecated in NameNodeProtocol. |
| (Jakob Homan via shv) |
| |
| HDFS-514. Change DFSClient.namenode from public to private. (Bill Zeller |
| via szetszwo) |
| |
| HDFS-496. Use PureJavaCrc32 in HDFS. (Todd Lipcon via szetszwo) |
| |
| HDFS-511. Remove redundant block searches in BlockManager. (shv) |
| |
| HDFS-504. Update the modification time of a file when the file |
| is closed. (Chun Zhang via dhruba) |
| |
| HDFS-498. Add development guide and documentation for the fault injection |
| framework. (Konstantin Boudnik via szetszwo) |
| |
| HDFS-524. Further DataTransferProtocol code refactoring. (szetszwo) |
| |
| HDFS-529. Use BlockInfo instead of Block to avoid redundant block searches |
| in BlockManager. (shv) |
| |
| HDFS-530. Refactor TestFileAppend* to remove code duplication. |
| (Konstantin Boudnik via szetszwo) |
| |
| HDFS-451. Add fault injection tests for DataTransferProtocol. (szetszwo) |
| |
| HDFS-409. Add more access token tests. (Kan Zhang via szetszwo) |
| |
| HDFS-546. DatanodeDescriptor iterates blocks as BlockInfo. (shv) |
| |
| HDFS-457. Do not shutdown datanode if some, but not all, volumes fail. |
| (Boris Shkolnik via szetszwo) |
| |
| HDFS-548. TestFsck takes nearly 10 minutes to run. (hairong) |
| |
| HDFS-539. Refactor fault injeciton pipeline test util for future reuse. |
| (Konstantin Boudnik via szetszwo) |
| |
| HDFS-552. Change TestFiDataTransferProtocol to junit 4 and add a few new |
| tests. (szetszwo) |
| |
| HDFS-563. Simplify the codes in FSNamesystem.getBlockLocations(..). |
| (szetszwo) |
| |
| HDFS-581. Introduce an iterator over blocks in the block report array.(shv) |
| |
| HDFS-549. Add a new target, run-with-fault-inject-testcaseonly, which |
| allows an execution of non-FI tests in FI-enable environment. (Konstantin |
| Boudnik via szetszwo) |
| |
| HDFS-173. Namenode will not block until a large directory deletion |
| completes. It allows other operations when the deletion is in progress. |
| (suresh) |
| |
| HDFS-551. Create new functional test for a block report. (Konstantin |
| Boudnik via hairong) |
| |
| HDFS-288. Redundant computation in hashCode() implementation. |
| (szetszwo via tomwhite) |
| |
| HDFS-412. Hadoop JMX usage makes Nagios monitoring impossible. |
| (Brian Bockelman via tomwhite) |
| |
| HDFS-472. Update hdfsproxy documentation. Adds a setup guide and design |
| document. (Zhiyong Zhang via cdouglas) |
| |
| HDFS-617. Support non-recursive create(). (Kan Zhang via szetszwo) |
| |
| HDFS-618. Support non-recursive mkdir(). (Kan Zhang via szetszwo) |
| |
| HDFS-574. Split the documentation between the subprojects. |
| (Corinne Chandel via omalley) |
| |
| HDFS-598. Eclipse launch task for HDFS. (Eli Collins via tomwhite) |
| |
| HDFS-641. Move all of the components that depend on map/reduce to |
| map/reduce. (omalley) |
| |
| HDFS-509. Redesign DataNode volumeMap to include all types of Replicas. |
| (hairong) |
| |
| HDFS-562. Add a test for NameNode.getBlockLocations(..) to check read from |
| un-closed file. (szetszwo) |
| |
| HDFS-543. Break FSDatasetInterface#writToBlock() into writeToRemporary, |
| writeToRBW, ad append. (hairong) |
| |
| HDFS-603. Add a new interface, Replica, which is going to replace the use |
| of Block in datanode. (szetszwo) |
| |
| HDFS-589. Change block write protocol to support pipeline recovery. |
| (hairong) |
| |
| HDFS-652. Replace BlockInfo.isUnderConstruction() with isComplete() (shv) |
| |
| HDFS-648. Change some methods in AppendTestUtil to public. (Konstantin |
| Boudnik via szetszwo) |
| |
| HDFS-662. Unnecessary info message from DFSClient. (hairong) |
| |
| HDFS-518. Create new tests for Append's hflush. (Konstantin Boudnik |
| via szetszwo) |
| |
| HDFS-688. Add configuration resources to DFSAdmin. (shv) |
| |
| HDFS-29. Validate the consistency of the lengths of replica and its file |
| in replica recovery. (szetszwo) |
| |
| HDFS-680. Add new access method to a copy of a block's replica. (shv) |
| |
| HDFS-704. Unify build property names to facilitate cross-projects |
| modifications (cos) |
| |
| HDFS-705. Create an adapter to access some of package-private methods of |
| DataNode from tests (cos) |
| |
| HDFS-710. Add actions with constraints to the pipeline fault injection |
| tests and change SleepAction to support uniform random sleeping over an |
| interval. (szetszwo) |
| |
| HDFS-713. Need to properly check the type of the test class from an aspect |
| (cos) |
| |
| HDFS-716. Define a pointcut for pipeline close and add a few fault |
| injection tests to simulate out of memory problem. (szetszwo) |
| |
| HDFS-719. Add 6 fault injection tests for pipeline close to simulate slow |
| datanodes and disk errors. (szetszwo) |
| |
| HDFS-616. Create functional tests for new design of the block report. (cos) |
| |
| HDFS-584. Fail the fault-inject build if any advices are mis-bound. (cos) |
| |
| HDFS-730. Add 4 fault injection tests to simulate non-responsive datanode |
| and out-of-memory problem for pipeline close ack. (szetszwo) |
| |
| HDFS-728. Create a comprehensive functional test for append. (hairong) |
| |
| HDFS-736. commitBlockSynchronization() updates block GS and length |
| in-place. (shv) |
| |
| HADOOP-5107. Use Maven ant tasks to publish the subproject jars. |
| (Giridharan Kesavan via omalley) |
| |
| HDFS-521. Create new tests for pipeline (cos) |
| |
| HDFS-764. Places the Block Access token implementation in hdfs project. |
| (Kan Zhang via ddas) |
| |
| HDFS-787. Upgrade some libraries to be consistent with common and |
| mapreduce. (omalley) |
| |
| HDFS-519. Create new tests for lease recovery (cos) |
| |
| HDFS-804. New unit tests for concurrent lease recovery (cos) |
| |
| HDFS-813. Enable the append test in TestReadWhileWriting. (szetszwo) |
| |
| HDFS-145. Cleanup inconsistent block length handling code in |
| FSNameSystem#addStoredBlock. (hairong) |
| |
| HDFS-127. Reset failure count in DFSClient for each block acquiring |
| operation. (Igor Bolotin via szetszwo) |
| |
| HDFS-520. Create new tests for block recovery. (hairong) |
| |
| HDFS-1067. Create block recovery tests that handle errors. (hairong) |
| |
| HDFS-1107. Turn on append by default. (shv) |
| |
| HDFS-968. Use StringBuilder instead of StringBuffer for better |
| performance. (Kay Kay via suresh) |
| |
| HDFS-703. Replace current fault injection implementation with one |
| from (cos) |
| |
| HDFS-754. Reduce ivy console output to observable level (cos) |
| |
| HDFS-832. HDFS side of HADOOP-6222. (cos) |
| |
| HDFS-840. Change tests to use FileContext test helper introduced in |
| HADOOP-6394. (Jitendra Nath Pandey via suresh) |
| |
| HDFS-685. Use the user-to-groups mapping service in the NameNode. |
| (boryas, acmurthy) |
| |
| HDFS-755. Read multiple checksum chunks at once in DFSInputStream. |
| (Todd Lipcon via tomwhite) |
| |
| HDFS-786. Implement getContentSummary in HftpFileSystem. |
| (Tsz Wo (Nicholas), SZE via cdouglas) |
| |
| HDFS-587. Add support for specifying queue name in mapreduce tests. |
| (Erik Steffl via suresh) |
| |
| HDFS-902 Move contrib/raid to MapReduce. (Eli Collins via omalley) |
| |
| HDFS-800. The last block of a file under construction may change to the |
| COMPLETE state in response to getAdditionalBlock or completeFileInternal. |
| (hairong) |
| |
| HDFS-899. Delegation Token Implementation |
| and corresponding changes in Namenode and DFS Api to issue, |
| renew and cancel delegation tokens. (jnp via boryas) |
| |
| HDFS-844. Log the filename when file locking fails. (tomwhite) |
| |
| HDFS-914. Refactor DFSOutputStream and DFSInputStream out of DFSClient. |
| (Todd Lipcon via tomwhite) |
| |
| HDFS-949. Move DelegationToken into Common so that it can be used by |
| MapReduce. (omalley) |
| |
| HDFS-930. Better error message for DATA_TRANSFER_VERSION mismatched. |
| (Kay Kay via szetszwo) |
| |
| HDFS-986. Delegation token renewing and cancelling should provide |
| meaningful exceptions when there are failures instead of returning |
| false. (omalley) |
| |
| HADOOP-6579. Upgrade the commons-codec library to 1.4. (omalley) |
| |
| HDFS-991. Allow authentication to the web ui via a delegation token. |
| (omalley) |
| |
| HDFS-994. Allow fetching of delegation token from NameNode for hftp. |
| (Jakob Homan via acmurthy) |
| |
| HDFS-998. Quote blocks streamed through jsps. (cdouglas) |
| |
| HDFS-729. NameNode API to list files that have missing blocks. |
| (Rodrigo Schmidt via dhruba) |
| |
| HDFS-850. The WebUI display more details about namenode memory usage. |
| (Dmytro Molkov via dhruba) |
| |
| HDFS-826. The DFSOutputStream has a API that returns the number of |
| active datanode(s) in the current pipeline. (dhruba) |
| |
| HDFS-985. HDFS should issue multiple RPCs for listing a large |
| directory. (hairong) |
| |
| HDFS-1043. NNThroughputBenchmark modifications to support benchmarking of |
| server-side user group resolution. (shv) |
| |
| HDFS-892. Optionally use Avro reflection for Namenode RPC. This |
| is not a complete implementation yet, but rather a starting point. |
| (cutting) |
| |
| HDFS-854. Datanode should scan devices in parallel to generate |
| block report. (Dmytro Molkov via jhoman) |
| |
| HDFS-1032. fsck has an option to list corrupt files. |
| (Andre Oriai via dhruba) |
| |
| HDFS-1024. SecondaryNameNode verifies size of fsimage and edits file. |
| (Dmytro Molkov via dhruba) |
| |
| HDFS-1011. hdfsproxy: Improve log messages by restoring the previous |
| thread name. (Srikanth Sundarrajan via szetszwo) |
| |
| HDFS-997. Allow datanode storage directory permissions to be configurable. |
| (Luke Lu via cdouglas) |
| |
| HDFS-1012. hdfsproxy: Support for fully qualified HDFS path in addition to |
| simple unqualified path. (Srikanth Sundarrajan via szetszwo) |
| |
| HDFS-933. Namenode should issue a delegation token only for kerberos |
| authenticated clients.(jnp via boryas) |
| |
| HDFS-1087. Modify audit log to use a StringBuilder rather than a Formatter. |
| (cdouglas) |
| |
| HDFS-1083. Update TestHDFSCLI not to expect exception class name |
| in error messages. (suresh) |
| |
| HDFS-1099. Add test for umask backward compatibility. (suresh) |
| |
| HDFS-1092. Use logging rather than System.err in MiniDFSCluster. |
| (Kay Kay via jghoman) |
| |
| HDFS-1047. Install/deploy source jars to Maven repo. |
| (Patrick Angeles via jghoman) |
| |
| HDFS-666. Unit test for FsShell -text. (cdouglas via jghoman) |
| |
| HDFS-1054. Remove unnecessary sleep after failure in nextBlockOutputStream. |
| (Todd Lipcon via jghoman) |
| |
| HDFS-921. Convert TestDFSClientRetries::testNotYetReplicatedErrors |
| to Mockito. (jghoman) |
| |
| HDFS-1100. Override unwrapException in TestFcHdfsSymlink to test |
| symlink API conformance. (Eli Collins via suresh). |
| |
| HDFS-1089. Remove uses of FileContext#isFile, isDirectory, and exists. |
| (Eli Collins via hairong) |
| |
| HDFS-1028. Efficient splitting of path components reduces the time |
| to load in fsimage by 20%. (Dmytro Molkov via dhruba) |
| |
| HDFS-1109. HFTP supports filenames that contains the character "+". |
| (Dmytro Molkov via dhruba) |
| |
| HDFS-853. The HDFS webUI displays the balanced-ness of the cluster. |
| (Dmytro Molkov via dhruba) |
| |
| HDFS-1126. Change HDFS to depend on Hadoop 'common' artifacts instead |
| of 'core'. (tomwhite) |
| |
| HDFS-995. Replace usage of FileStatus#isDir(). (Eli Collins via |
| tomwhite) |
| |
| HDFS-1161. Make DN minimum valid volumes configurable. |
| (Eli Collins via tomwhite) |
| |
| HDFS-1181. Move configuration and script files post split. (tomwhite) |
| |
| HDFS-1170. Add more assertions to TestLargeDirectoryDelete. |
| (Steve Loughran via tomwhite) |
| |
| HDFS-1199. Extract a subset of tests for smoke (DOA) validation. (cos) |
| |
| HDFS-1174. New properties for suspend and resume process. (Vinay Thota via |
| cos) |
| |
| HDFS-1277. [Herriot] New property for multi user list. (Vinay Thota via |
| cos) |
| |
| HDFS-806. Add new unit tests to the 10-mins 'run-commit-test' target (cos) |
| |
| OPTIMIZATIONS |
| |
| HDFS-946. NameNode should not return full path name when lisitng a |
| diretory or getting the status of a file. (hairong) |
| |
| BUG FIXES |
| |
| HDFS-76. Better error message to users when commands fail because of |
| lack of quota. Allow quota to be set even if the limit is lower than |
| current consumption. (Boris Shkolnik via rangadi) |
| |
| HADOOP-4687. HDFS is split from Hadoop Core. It is a subproject under |
| Hadoop (Owen O'Malley) |
| |
| HADOOP-6096. Fix Eclipse project and classpath files following project |
| split. (tomwhite) |
| |
| HDFS-195. Handle expired tokens when write pipeline is reestablished. |
| (Kan Zhang via rangadi) |
| |
| HDFS-181. Validate src path in FSNamesystem.getFileInfo(..). (Todd |
| Lipcon via szetszwo) |
| |
| HDFS-441. Remove TestFTPFileSystem. (szetszwo) |
| |
| HDFS-440. Fix javadoc broken links in DFSClient. (szetszwo) |
| |
| HDFS-480. Fix a typo in the jar name in build.xml. |
| (Konstantin Shvachko via gkesavan) |
| |
| HDFS-438. Check for NULL before invoking GenericArgumentParser in |
| DataNode. (Raghu Angadi) |
| |
| HDFS-415. BlockReceiver hangs in case of certain runtime exceptions. |
| (Konstantin Boudnik via rangadi) |
| |
| HDFS-462. loadFSImage should close edits file. (Jakob Homan via shv) |
| |
| HDFS-489. Update TestHDFSCLI for the -skipTrash option in rm. (Jakob Homan |
| via szetszwo) |
| |
| HDFS-445. pread() does not pick up changes to block locations. |
| (Kan Zhang via rangadi) |
| |
| HDFS-463. CreateEditLog utility broken after HDFS-396 (URI for |
| FSImage). (Suresh Srinivas via rangadi) |
| |
| HDFS-484. Fix bin-package and package target to package jar files. |
| (gkesavan) |
| |
| HDFS-490. Eliminate the deprecated warnings introduced by H-5438. |
| (He Yongqiang via szetszwo) |
| |
| HDFS-119. Fix a bug in logSync(), which causes NameNode block forever. |
| (Suresh Srinivas via shv) |
| |
| HDFS-534. Include avro in ivy. (szetszwo) |
| |
| HDFS-532. Allow applications to know that a read request failed |
| because block is missing. (dhruba) |
| |
| HDFS-561. Fix write pipeline READ_TIMEOUT in DataTransferProtocol. |
| (Kan Zhang via szetszwo) |
| |
| HDFS-553. BlockSender reports wrong failed position in ChecksumException. |
| (hairong) |
| |
| HDFS-568. Set mapred.job.tracker.retire.jobs to false in |
| src/test/mapred-site.xml for mapreduce tests to run. (Amareshwari |
| Sriramadasu via szetszwo) |
| |
| HDFS-15. All replicas end up on 1 rack. (Jitendra Nath Pandey via hairong) |
| |
| HDFS-586. TestBlocksWithNotEnoughRacks sometimes fails. |
| (Jitendra Nath Pandey via hairong) |
| |
| HADOOP-6243. Fixed a NullPointerException in handling deprecated keys. |
| (Sreekanth Ramakrishnan via yhemanth) |
| |
| HDFS-605. Do not run fault injection tests in the run-test-hdfs-with-mr |
| target. (Konstantin Boudnik via szetszwo) |
| |
| HDFS-606. Fix ConcurrentModificationException in invalidateCorruptReplicas() |
| (shv) |
| |
| HDFS-601. TestBlockReport obtains data directories directly from |
| MiniHDFSCluster. (Konstantin Boudnik via shv) |
| |
| HDFS-614. TestDatanodeBlockScanner obtains data directories directly from |
| MiniHDFSCluster. (shv) |
| |
| HDFS-612. Remove the use of org.mortbay.log.Log in FSDataset. (szetszwo) |
| |
| HDFS-622. checkMinReplication should count live nodes only. (shv) |
| |
| HDFS-629. Remove ReplicationTargetChooser.java along with fixing |
| import warnings generated by Eclipse. (dhruba) |
| |
| HDFS-637. DataNode sends a Success ack when block write fails. (hairong) |
| |
| HDFS-640. Fixed TestHDFSFileContextMainOperations.java build failure. (suresh) |
| |
| HDFS-547. TestHDFSFileSystemContract#testOutputStreamClosedTwice |
| sometimes fails with CloseByInterruptException. (hairong) |
| |
| HDFS-588. Fix TestFiDataTransferProtocol and TestAppend2 failures. (shv) |
| |
| HDFS-550. DataNode restarts may introduce corrupt/duplicated/lost replicas |
| when handling detached replicas. (hairong) |
| |
| HDFS-659. If the the last block is not complete, update its length with |
| one of its replica's length stored in datanode. (szetszwo) |
| |
| HDFS-649. Check null pointers for DataTransferTest. (Konstantin Boudnik |
| via szetszwo) |
| |
| HDFS-661. DataNode upgrade fails on non-existant current directory. |
| (hairong) |
| |
| HDFS-597. Mofication introduced by HDFS-537 breakes an advice binding in |
| FSDatasetAspects. (Konstantin Boudnik via szetszwo) |
| |
| HDFS-665. TestFileAppend2 sometimes hangs. (hairong) |
| |
| HDFS-676. Fix NPE in FSDataset.updateReplicaUnderRecovery() (shv) |
| |
| HDFS-673. BlockReceiver#PacketResponder should not remove a packet from |
| the ack queue before its ack is sent. (hairong) |
| |
| HDFS-682. Fix bugs in TestBlockUnderConstruction. (szetszwo) |
| |
| HDFS-668. TestFileAppend3#TC7 sometimes hangs. (hairong) |
| |
| HDFS-679. Appending to a partial chunk incorrectly assumes the |
| first packet fills up the partial chunk. (hairong) |
| |
| HDFS-722. Fix callCreateBlockWriteStream pointcut in FSDatasetAspects. |
| (szetszwo) |
| |
| HDFS-690. TestAppend2#testComplexAppend failed on "Too many open files". |
| (hairong) |
| |
| HDFS-725. Support the build error fix for HADOOP-6327. (Sanjay Radia via |
| szetszwo) |
| |
| HDFS-625. Fix NullPointerException thrown from ListPathServlet. (suresh) |
| |
| HDFS-735. TestReadWhileWriting has wrong line termination symbols (cos) |
| |
| HDFS-691. Fix an overflow error in DFSClient.DFSInputStream.available(). |
| (szetszwo) |
| |
| HDFS-733. TestBlockReport fails intermittently. (cos) |
| |
| HDFS-774. Intermittent race condition in TestFiPipelines (cos) |
| |
| HDFS-741. TestHFlush test doesn't seek() past previously written part of |
| the file (cos, szetszwo) |
| |
| HDFS-706. Intermittent failures in TestFiHFlush (cos) |
| |
| HDFS-646. Fix test-patch failure by adding test-contrib ant target. |
| (gkesavan) |
| |
| HDFS-791. Build is broken after HDFS-787 patch has been applied (cos) |
| |
| HDFS-792. TestHDFSCLI is failing. (Todd Lipcon via cos) |
| |
| HDFS-781. Namenode metrics PendingDeletionBlocks is not decremented. |
| (Suresh) |
| |
| HDFS-192. Fix TestBackupNode failures. (shv) |
| |
| HDFS-797. TestHDFSCLI much slower after HDFS-265 merge. (Todd Lipcon via cos) |
| |
| HDFS-824. Stop lease checker in TestReadWhileWriting. (szetszwo) |
| |
| HDFS-823. CheckPointer should use addInternalServlet for image-fetching |
| servlet (jghoman) |
| |
| HDFS-456. Fix URI generation for windows file paths. (shv) |
| |
| HDFS-812. FSNamesystem#internalReleaseLease throws NullPointerException on |
| a single-block file's lease recovery. (cos) |
| |
| HDFS-724. Pipeline hangs if one of the block receiver is not responsive. |
| (hairong) |
| |
| HDFS-564. Adding pipeline tests 17-35. (hairong) |
| |
| HDFS-849. TestFiDataTransferProtocol2#pipeline_Fi_18 sometimes fails. |
| (hairong) |
| |
| HDFS-762. Balancer causes Null Pointer Exception. |
| (Cristian Ivascu via dhruba) |
| |
| HDFS-868. Fix link to Hadoop Upgrade Wiki. (Chris A. Mattmann via shv) |
| |
| HDFS-880. TestNNLeaseRecovery fails on windows (cos, shv) |
| |
| HDFS-699. Primary datanode should compare replicas' on disk lengths. |
| (hairong) |
| |
| HDFS-897. Fix a bug related to generation stamp comparison in |
| ReplicasMap. (suresh) |
| |
| HDFS-793. Data node should receive the whole packet ack message before it |
| constructs and sends its own ack message for the packet. (hairong) |
| |
| HDFS-101. DFS write pipeline: DFSClient sometimes does not detect second |
| datanode failure. (hairong) |
| |
| HDFS-822. Appends to already-finalized blocks can rename across volumes. |
| (hairong) |
| |
| HDFS-1046. Fix Tomcat version in hdfsproxy/build.xml. (Srikanth |
| Sundarrajan via szetszwo) |
| |
| HDFS-1072. Fix TestReadWhileWriting failure. (Erik Steffl via shv) |
| |
| HDFS-913. Rename fault injection test TestRename.java to TestFiRename.java |
| to include it in tests run by ant target run-test-hdfs-fault-inject. |
| (suresh) |
| |
| HDFS-695. RaidNode should read in configuration from hdfs-site.xml. |
| (dhruba) |
| |
| HDFS-726. Eclipse .classpath template has outdated jar files and is |
| missing some new ones. (cos) |
| |
| HDFS-750. Fix build failure due to TestRename. (suresh) |
| |
| HDFS-712. Move libhdfs from mapreduce subproject to hdfs subproject. |
| (Eli Collins via dhruba) |
| |
| HDFS-757. Enable Unit test for HDFS Raid. (dhruba) |
| |
| HDFS-611. Prevent DataNode heartbeat times from increasing even when |
| the DataNode has many blocks to delete. (Zheng Shao via dhruba) |
| |
| HDFS-751. Fix TestCrcCorruption to pick up the correct datablocks to |
| corrupt. (dhruba) |
| |
| HDFS-763. Fix slightly misleading report from DataBlockScanner |
| about corrupted scans. (dhruba) |
| |
| HDFS-727. bug setting block size hdfsOpenFile (Eli Collins via cos) |
| |
| HDFS-756. libhdfs unit tests do not run. (Eli Collins via cos) |
| |
| HDFS-783. libhdfs tests brakes code coverage runs with Clover (cos) |
| |
| HDFS-785. Add Apache license to several namenode unit tests. |
| (Ravi Phulari via jghoman) |
| |
| HDFS-802. Update Eclipse configuration to match changes to Ivy |
| configuration (Edwin Chan via cos) |
| |
| HDFS-423. Unbreak FUSE build and fuse_dfs_wrapper.sh (Eli Collins via cos) |
| |
| HDFS-825. Build fails to pull latest hadoop-core-* artifacts (cos) |
| |
| HDFS-94. The Heap Size printed in the NameNode WebUI is accurate. |
| (Dmytro Molkov via dhruba) |
| |
| HDFS-767. An improved retry policy when the DFSClient is unable to fetch a |
| block from the datanode. (Ning Zhang via dhruba) |
| |
| HDFS-775. FSDataset calls getCapacity() twice. (stevel) |
| |
| HDFS-885. Datanode toString() NPEs on null dnRegistration. (stevel) |
| |
| HDFS-877. Client-driven block verification not functioning. (Todd |
| Lipcon via hairong) |
| |
| HDFS-630. In DFSOutputStream.nextBlockOutputStream(), the client can |
| exclude specific datanodes when locating the next block. |
| (Cosmin Lehene via Stack) |
| |
| HDFS-922. Remove unnecessary semicolon added by HDFS-877 that causes |
| problems for Eclipse compilation. (jghoman) |
| |
| HDFS-927 DFSInputStream retries too many times for new block locations |
| (Todd Lipcon via Stack) |
| |
| HDFS-938. Replace calls to UGI.getUserName() with UGI.getShortUserName() |
| (jghoman) |
| |
| HDFS-894. DatanodeID.ipcPort is not updated when existing node |
| re-registers. (Todd Lipcon via tomwhite) |
| |
| HDFS-965. Split TestDelegationToken in to two parts and fix configuration |
| to allow proxy users in the test. (Jitendra Pandey via omalley) |
| |
| HDFS-999. Secondary namenode should login using kerberos if security is |
| configured (boryas) |
| |
| HDFS-856. Hardcoded replication level for new files in fuse-dfs. |
| (Brian Bockelman via tomwhite) |
| |
| HDFS-857. Incorrect type for fuse-dfs capacity can cause "df" to return |
| negative values on 32-bit machines. (Brian Bockelman via tomwhite) |
| |
| HDFS-858. Incorrect return codes for fuse-dfs. (Brian Bockelman via |
| tomwhite) |
| |
| HDFS-859. fuse-dfs utime behavior causes issues with tar. |
| (Brian Bockelman via tomwhite) |
| |
| HDFS-861. fuse-dfs does not support O_RDWR. (Brian Bockelman via tomwhite) |
| |
| HDFS-961. dfs_readdir incorrectly parses paths. (Eli Collins via tomwhite) |
| |
| HDFS-1015. Fix intermittent failure in TestSecurityTokenEditLog. |
| (Jitendra Nath Pandey via suresh) |
| |
| HDFS-939. libhdfs test is broken. (Eli Collins via tomwhite) |
| |
| HDFS-1074. hdfsproxy: Fix bugs in TestProxyUtil. (Srikanth Sundarrajan |
| via szetszwo) |
| |
| HDFS-481. hdfsproxy: Bug Fixes + HdfsProxy to use proxy user to |
| impresonate the real user. (Srikanth Sundarrajan via szetszwo) |
| |
| HDFS-482. Move HsftpFileSystem's ssl.client.do.not.authenticate.server |
| configuration setting to ssl-client.xml. (Srikanth Sundarrajan via |
| szetszwo) |
| |
| HDFS-1010. hdfsproxy: Retrieve groups from UnixUserGroupInformation |
| instead of LdapEntry. (Srikanth Sundarrajan via szetszwo) |
| |
| HDFS-466. hdfs_write infinite loop when dfs fails and cannot write |
| files > 2 GB. (Pete Wyckoff via tomwhite) |
| |
| HDFS-651. HDFS Docs - fix listing of docs in the doc menu. |
| (Corinne Chandel via tomwhite) |
| |
| HDFS-1014. Error in reading delegation tokens from edit logs. |
| (Jitendra Nath Pandey via jhoman) |
| |
| HDFS-1088. Prevent renaming a symbolik link to its target. |
| (Eli Collins via suresh) |
| |
| HDFS-966. NameNode does not recovers lease when it is in safemode. |
| (dhruba) |
| |
| HDFS-833. Datanode shutdown should log problems with Storage.unlockAll() |
| (Steve Loughran via dhruba) |
| |
| HDFS-1101. TestDiskError.testLocalDirs() fails. (cdouglas via jghoman) |
| |
| HDFS-1031. Enhance the webUi to list a few of the corrupted files in HDFS. |
| (Andre Orian via dhruba) |
| |
| HDFS-1078. Create static and dynamic versions of libhdfs. |
| (Sam Rash via dhruba) |
| |
| HDFS-1104. Fsck triggers full GC on NameNode. (hairong) |
| |
| HDFS-1141. Closing a file is successful only if the client still has a |
| valid lease. (Todd Lipcon via dhruba) |
| |
| HDFS-1138. Prevent erroneous updation of modification time of a directory |
| when fsimage loads. (Dmytro Molkov via dhruba) |
| |
| HDFS-1000. Updates libhdfs to the new API for UGI (ddas) |
| |
| HDFS-609. Create a file with the append flag does not work in HDFS. |
| (tomwhite) |
| |
| HDFS-1255. Fix failing test-libhdfs.sh test. (tomwhite) |
| |
| HDFS-1256. libhdfs is missing from the tarball. (tomwhite) |
| |
| HDFS-1057. Concurrent readers hit ChecksumExceptions if following a |
| writer to very end of file. (sam rash via hairong) |
| |
| HDFS-1212. Harmonize HDFS JAR library versions with Common. (tomwhite) |
| |
| HDFS-1159. clean-cache target removes wrong ivy cache (cos) |
| |
| HDFS-1193. -mvn-system-deploy target is broken which inturn fails the |
| mvn-deploy task leading to unstable mapreduce build (Giridharan |
| Kesavan via cos) |
| |
| HDFS-1299. 'compile-fault-inject' never should be called directly. (cos) |
| |
| HDFS-1311. Running tests with 'testcase' cause triple execution of the |
| same test case (Cos) |
| |
| HDFS-1267. fuse-dfs does not compile. (Devaraj Das via tomwhite) |
| |
| HDFS-1598. Directory listing on hftp:// does not show .*.crc files. |
| (szetszwo) |
| |
| HDFS-1750. ListPathsServlet should not use HdfsFileStatus.getLocalName() |
| to get file name since it may return an empty string. (szetszwo) |
| |
| Release 0.20.3 - Unreleased |
| |
| IMPROVEMENTS |
| |
| BUG FIXES |
| |
| HDFS-1041. DFSClient.getFileChecksum(..) should retry if connection to |
| the first datanode fails. (szetszwo) |
| |
| HDFS-909. Wait until edits syncing is finishes before purging edits. |
| (Todd Lipcon via shv) |
| |
| HDFS-1258. Clearing namespace quota on "/" corrupts fs image. |
| (Aaron T. Myers via szetszwo) |
| |
| HDFS-1406. TestCLI fails on Ubuntu with default /etc/hosts. (cos) |
| |
| Release 0.20.203.0 - 2011-5-11 |
| |
| IMPROVEMENTS |
| |
| HADOOP-7259. Contrib modules should include the build.properties from |
| the enclosing hadoop directory. (omalley) |
| |
| BUG FIXES |
| |
| HDFS-132. Fix namenode to not report files deleted metrics for deletions |
| done while replaying edits during startup. (suresh & shv) |
| |
| HDFS-955. New implementation of saveNamespace() to avoid loss of edits |
| when name-node fails during saving. (shv) |
| |
| Release 0.20.2 - 2009-09-01 |
| |
| IMPROVEMENTS |
| |
| HDFS-737. Add full path name of the file to the block information and |
| summary of total number of files, blocks, live and deadnodes to |
| metasave output. (Jitendra Nath Pandey via suresh) |
| |
| HDFS-919. Create test to validate the BlocksVerified metric (Gary Murry |
| via cos) |
| |
| HDFS-907. Add tests for getBlockLocations and totalLoad metrics. |
| (Ravi Phulari via cos) |
| |
| BUG FIXES |
| |
| HDFS-686. NullPointerException is thrown while merging edit log and image. |
| (hairong) |
| |
| HDFS-677. Rename failure when both source and destination quota exceeds |
| results in deletion of source. (suresh) |
| |
| HDFS-709. Fix TestDFSShell failure due to rename bug introduced by |
| HDFS-677. (suresh) |
| |
| HDFS-579. Fix DfsTask to follow the semantics of 0.19, regarding non-zero |
| return values as failures. (Christian Kunz via cdouglas) |
| |
| HDFS-723. Fix deadlock in DFSClient#DFSOutputStream. (hairong) |
| |
| HDFS-596. Fix memory leak in hdfsFreeFileInfo() for libhdfs. |
| (Zhang Bingjun via dhruba) |
| |
| HDFS-185. Disallow chown, chgrp, chmod, setQuota, and setSpaceQuota when |
| name-node is in safemode. (Ravi Phulari via shv) |
| |
| HDFS-187. Initialize secondary namenode http address in TestStartup. |
| (Todd Lipcon via szetszwo) |
| |
| HDFS-464. Fix memory leaks in libhdfs. (Christian Kunz via suresh) |
| |
| HDFS-1377. Quota bug for partial blocks allows quotas to be violated. (eli) |
| |
| Release 0.20.1 - 2009-09-01 |
| |
| IMPROVEMENTS |
| |
| HDFS-438. Improve help message for space quota command. (Raghu Angadi) |
| |
| BUG FIXES |
| |
| HDFS-167. Fix a bug in DFSClient that caused infinite retries on write. |
| (Bill Zeller via szetszwo) |
| |
| HDFS-527. Remove/deprecate unnecessary DFSClient constructors. (szetszwo) |
| |
| HDFS-525. The SimpleDateFormat object in ListPathsServlet is not thread |
| safe. (Suresh Srinivas and cdouglas) |
| |
| HDFS-761. Fix failure to process rename operation from edits log due to |
| quota verification. (suresh) |