| Hadoop HDFS Change Log |
| |
| Trunk (unreleased changes) |
| |
| INCOMPATIBLE CHANGES |
| |
| HDFS-538. Per the contract elucidated in HADOOP-6201, throw |
| FileNotFoundException from FileSystem::listStatus rather than returning |
| null. (Jakob Homan via cdouglas) |
| |
| HDFS-602. DistributedFileSystem mkdirs throws FileAlreadyExistsException |
| instead of FileNotFoundException. (Boris Shkolnik via suresh) |
| |
| HDFS-544. Add a "rbw" subdir to DataNode data directory. (hairong) |
| |
| HDFS-576. Block report includes under-construction replicas. (shv) |
| |
| HDFS-636. SafeMode counts complete blocks only. (shv) |
| |
| HDFS-644. Lease recovery, concurrency support. (shv) |
| |
| HDFS-570. Get last block length from a data-node when opening a file |
| being written to. (Tsz Wo (Nicholas), SZE via shv) |
| |
| HDFS-657. Remove unused legacy data-node protocol methods. (shv) |
| |
| HDFS-658. Block recovery for primary data-node. (shv) |
| |
| NEW FEATURES |
| |
| HDFS-436. Introduce AspectJ framework for HDFS code and tests. |
| (Konstantin Boudnik via szetszwo) |
| |
| HDFS-447. Add LDAP lookup to hdfsproxy. (Zhiyong Zhang via cdouglas) |
| |
| HDFS-459. Introduce Job History Log Analyzer. (shv) |
| |
| HDFS-461. Tool to analyze file size distribution in HDFS. (shv) |
| |
| HDFS-492. Add two JSON JSP pages to the Namenode for providing corrupt |
| blocks/replicas information. (Bill Zeller via szetszwo) |
| |
| HDFS-578. Add support for new FileSystem method for clients to get server |
| defaults. (Kan Zhang via suresh) |
| |
| HDFS-595. umask settings in configuration may now use octal or symbolic |
| instead of decimal. (Jakob Homan via suresh) |
| |
| HADOOP-6234. Updated hadoop-core and test jars to propagate new option |
| dfs.umaskmode in configuration. (Jakob Homan via suresh) |
| |
| HDFS-235. Add support for byte ranges in HftpFileSystem to serve |
| range of bytes from a file. (Bill Zeller via suresh) |
| |
| HDFS-385. Add support for an experimental API that allows a module external |
| to HDFS to specify how HDFS blocks should be placed. (dhruba) |
| |
| HADOOP-4952. Update hadoop-core and test jars to propagate new FileContext |
| file system application interface. (Sanjay Radia via suresh). |
| |
| HDFS-567. Add block forensics contrib tool to print history of corrupt and |
| missing blocks from the HDFS logs. |
| (Bill Zeller, Jithendra Pandey via suresh). |
| |
| HDFS-610. Support o.a.h.fs.FileContext. (Sanjay Radia via szetszwo) |
| |
| HDFS-536. Support hflush at DFSClient. (hairong) |
| |
| HDFS-517. Introduce BlockInfoUnderConstruction to reflect block replica |
| states while writing. (shv) |
| |
| HDFS-565. Introduce block committing logic during new block allocation |
| and file close. (shv) |
| |
| HDFS-537. DataNode exposes a replica's meta info to BlockReceiver for the |
| support of dfs writes/hflush. It also updates a replica's bytes received, |
| bytes on disk, and bytes acked after receiving a packet. (hairong) |
| |
| HDFS-585. Datanode should serve up to visible length of a replica for read |
| requests. (szetszwo) |
| |
| HDFS-604. Block report processing for append. (shv) |
| |
| HDFS-619. Support replica recovery initialization in datanode for the new |
| append design. (szetszwo) |
| |
| HDFS-592. Allow clients to fetch a new generation stamp from NameNode for |
| pipeline recovery. (hairong) |
| |
| HDFS-624. Support a new algorithm for pipeline recovery and pipeline setup |
| for append. (hairong) |
| |
| HDFS-627. Support replica update in data-node. |
| (Tsz Wo (Nicholas), SZE and Hairong Kuang via shv) |
| |
| HDFS-642. Support pipeline close and close error recovery. (hairong) |
| |
| IMPROVEMENTS |
| |
| HDFS-381. Remove blocks from DataNode maps when corresponding file |
| is deleted. (Suresh Srinivas via rangadi) |
| |
| HDFS-377. Separate codes which implement DataTransferProtocol. |
| (szetszwo) |
| |
| HDFS-396. NameNode image and edits directories are specified as URIs. |
| (Luca Telloli via rangadi) |
| |
| HDFS-444. Allow to change probability levels dynamically in the fault |
| injection framework. (Konstantin Boudnik via szetszwo) |
| |
| HDFS-352. Documentation for saveNamespace command. (Ravi Phulari via shv) |
| |
| HADOOP-6106. Updated hadoop-core and test jars from hudson trunk |
| build #12. (Giridharan Kesavan) |
| |
| HDFS-204. Add a new metrics FilesInGetListingOps to the Namenode. |
| (Jitendra Nath Pandey via szetszwo) |
| |
| HDFS-278. HDFS Outputstream close does not hang forever. (dhruba) |
| |
| HDFS-443. Add a new metrics numExpiredHeartbeats to the Namenode. |
| (Jitendra Nath Pandey via szetszwo) |
| |
| HDFS-475. Add new ant targets for fault injection jars and tests. |
| (Konstantin Boudnik via szetszwo) |
| |
| HDFS-458. Create a new ant target, run-commit-test. (Jakob Homan |
| via szetszwo) |
| |
| HDFS-493. Change build.xml so that the fault-injected tests are executed |
| only by the run-test-*-fault-inject targets. (Konstantin Boudnik via |
| szetszwo) |
| |
| HDFS-446. Improvements to Offline Image Viewer. (Jakob Homan via shv) |
| |
| HADOOP-6160. Fix releaseaudit target to run on specific directories. |
| (gkesavan) |
| |
| HDFS-501. Use enum to define the constants in DataTransferProtocol. |
| (szetszwo) |
| |
| HDFS-508. Factor out BlockInfo from BlocksMap. (shv) |
| |
| HDFS-510. Rename DatanodeBlockInfo to be ReplicaInfo. |
| (Jakob Homan & Hairong Kuang via shv) |
| |
| HDFS-500. Deprecate NameNode methods deprecated in NameNodeProtocol. |
| (Jakob Homan via shv) |
| |
| HDFS-514. Change DFSClient.namenode from public to private. (Bill Zeller |
| via szetszwo) |
| |
| HDFS-496. Use PureJavaCrc32 in HDFS. (Todd Lipcon via szetszwo) |
| |
| HDFS-511. Remove redundant block searches in BlockManager. (shv) |
| |
| HDFS-504. Update the modification time of a file when the file |
| is closed. (Chun Zhang via dhruba) |
| |
| HDFS-498. Add development guide and documentation for the fault injection |
| framework. (Konstantin Boudnik via szetszwo) |
| |
| HDFS-524. Further DataTransferProtocol code refactoring. (szetszwo) |
| |
| HDFS-529. Use BlockInfo instead of Block to avoid redundant block searches |
| in BlockManager. (shv) |
| |
| HDFS-530. Refactor TestFileAppend* to remove code duplication. |
| (Konstantin Boudnik via szetszwo) |
| |
| HDFS-451. Add fault injection tests for DataTransferProtocol. (szetszwo) |
| |
| HDFS-409. Add more access token tests. (Kan Zhang via szetszwo) |
| |
| HDFS-546. DatanodeDescriptor iterates blocks as BlockInfo. (shv) |
| |
| HDFS-457. Do not shutdown datanode if some, but not all, volumes fail. |
| (Boris Shkolnik via szetszwo) |
| |
| HDFS-548. TestFsck takes nearly 10 minutes to run. (hairong) |
| |
| HDFS-539. Refactor fault injeciton pipeline test util for future reuse. |
| (Konstantin Boudnik via szetszwo) |
| |
| HDFS-552. Change TestFiDataTransferProtocol to junit 4 and add a few new |
| tests. (szetszwo) |
| |
| HDFS-563. Simplify the codes in FSNamesystem.getBlockLocations(..). |
| (szetszwo) |
| |
| HDFS-581. Introduce an iterator over blocks in the block report array. (shv) |
| |
| HDFS-549. Add a new target, run-with-fault-inject-testcaseonly, which |
| allows an execution of non-FI tests in FI-enable environment. (Konstantin |
| Boudnik via szetszwo) |
| |
| HDFS-173. Namenode will not block until a large directory deletion completes. |
| It allows other operations when the deletion is in progress. (suresh) |
| |
| HDFS-551. Create new functional test for a block report. (Konstantin |
| Boudnik via hairong) |
| |
| HDFS-288. Redundant computation in hashCode() implementation. |
| (szetszwo via tomwhite) |
| |
| HDFS-412. Hadoop JMX usage makes Nagios monitoring impossible. |
| (Brian Bockelman via tomwhite) |
| |
| HDFS-472. Update hdfsproxy documentation. Adds a setup guide and design |
| document. (Zhiyong Zhang via cdouglas) |
| |
| HDFS-617. Support non-recursive create(). (Kan Zhang via szetszwo) |
| |
| HDFS-618. Support non-recursive mkdir(). (Kan Zhang via szetszwo) |
| |
| HDFS-574. Split the documentation between the subprojects. |
| (Corinne Chandel via omalley) |
| |
| HDFS-598. Eclipse launch task for HDFS. (Eli Collins via tomwhite) |
| |
| HDFS-509. Redesign DataNode volumeMap to include all types of Replicas. |
| (hairong) |
| |
| HDFS-562. Add a test for NameNode.getBlockLocations(..) to check read from |
| un-closed file. (szetszwo) |
| |
| HDFS-543. Break FSDatasetInterface#writToBlock() into writeToRemporary, |
| writeToRBW, ad append. (hairong) |
| |
| HDFS-603. Add a new interface, Replica, which is going to replace the use |
| of Block in datanode. (szetszwo) |
| |
| HDFS-589. Change block write protocol to support pipeline recovery. |
| (hairong) |
| |
| HDFS-652. Replace BlockInfo.isUnderConstruction() with isComplete() (shv) |
| |
| HDFS-648. Change some methods in AppendTestUtil to public. (Konstantin |
| Boudnik via szetszwo) |
| |
| HDFS-662. Unnecessary info message from DFSClient. (hairong) |
| |
| HDFS-518. Create new tests for Append's hflush. (Konstantin Boudnik |
| via szetszwo) |
| |
| BUG FIXES |
| |
| HDFS-76. Better error message to users when commands fail because of |
| lack of quota. Allow quota to be set even if the limit is lower than |
| current consumption. (Boris Shkolnik via rangadi) |
| |
| HADOOP-4687. HDFS is split from Hadoop Core. It is a subproject under |
| Hadoop (Owen O'Malley) |
| |
| HADOOP-6096. Fix Eclipse project and classpath files following project |
| split. (tomwhite) |
| |
| HDFS-195. Handle expired tokens when write pipeline is reestablished. |
| (Kan Zhang via rangadi) |
| |
| HDFS-181. Validate src path in FSNamesystem.getFileInfo(..). (Todd |
| Lipcon via szetszwo) |
| |
| HDFS-441. Remove TestFTPFileSystem. (szetszwo) |
| |
| HDFS-440. Fix javadoc broken links in DFSClient. (szetszwo) |
| |
| HDFS-480. Fix a typo in the jar name in build.xml. |
| (Konstantin Shvachko via gkesavan) |
| |
| HDFS-438. Check for NULL before invoking GenericArgumentParser in |
| DataNode. (Raghu Angadi) |
| |
| HDFS-415. BlockReceiver hangs in case of certain runtime exceptions. |
| (Konstantin Boudnik via rangadi) |
| |
| HDFS-462. loadFSImage should close edits file. (Jakob Homan via shv) |
| |
| HDFS-489. Update TestHDFSCLI for the -skipTrash option in rm. (Jakob Homan |
| via szetszwo) |
| |
| HDFS-445. pread() does not pick up changes to block locations. |
| (Kan Zhang via rangadi) |
| |
| HDFS-463. CreateEditLog utility broken after HDFS-396 (URI for |
| FSImage). (Suresh Srinivas via rangadi) |
| |
| HDFS-484. Fix bin-package and package target to package jar files. |
| (gkesavan) |
| |
| HDFS-490. Eliminate the deprecated warnings introduced by H-5438. |
| (He Yongqiang via szetszwo) |
| |
| HDFS-119. Fix a bug in logSync(), which causes NameNode block forever. |
| (Suresh Srinivas via shv) |
| |
| HDFS-534. Include avro in ivy. (szetszwo) |
| |
| HDFS-532. Allow applications to know that a read request failed |
| because block is missing. (dhruba) |
| |
| HDFS-561. Fix write pipeline READ_TIMEOUT in DataTransferProtocol. |
| (Kan Zhang via szetszwo) |
| |
| HDFS-553. BlockSender reports wrong failed position in ChecksumException. |
| (hairong) |
| |
| HDFS-568. Set mapred.job.tracker.retire.jobs to false in |
| src/test/mapred-site.xml for mapreduce tests to run. (Amareshwari |
| Sriramadasu via szetszwo) |
| |
| HDFS-15. All replicas end up on 1 rack. (Jitendra Nath Pandey via hairong) |
| |
| HDFS-586. TestBlocksWithNotEnoughRacks sometimes fails. |
| (Jitendra Nath Pandey via hairong) |
| |
| HADOOP-6243. Fixed a NullPointerException in handling deprecated keys. |
| (Sreekanth Ramakrishnan via yhemanth) |
| |
| HDFS-605. Do not run fault injection tests in the run-test-hdfs-with-mr |
| target. (Konstantin Boudnik via szetszwo) |
| |
| HDFS-606. Fix ConcurrentModificationException in invalidateCorruptReplicas() |
| (shv) |
| |
| HDFS-601. TestBlockReport obtains data directories directly from |
| MiniHDFSCluster. (Konstantin Boudnik via shv) |
| |
| HDFS-614. TestDatanodeBlockScanner obtains data directories directly from |
| MiniHDFSCluster. (shv) |
| |
| HDFS-612. Remove the use of org.mortbay.log.Log in FSDataset. (szetszwo) |
| |
| HDFS-622. checkMinReplication should count live nodes only. (shv) |
| |
| HDFS-629. Remove ReplicationTargetChooser.java along with fixing |
| import warnings generated by Eclipse. (dhruba) |
| |
| HDFS-640. Fix TestHDFSFileContextMainOperations.java build failure. (suresh) |
| |
| HDFS-637. DataNode sends a Success ack when block write fails. (hairong) |
| |
| HDFS-547. TestHDFSFileSystemContract#testOutputStreamClosedTwice |
| sometimes fails with CloseByInterruptException. (hairong) |
| |
| HDFS-588. Fix TestFiDataTransferProtocol and TestAppend2 failures. (shv) |
| |
| HDFS-550. DataNode restarts may introduce corrupt/duplicated/lost replicas |
| when handling detached replicas. (hairong) |
| |
| HDFS-659. If the the last block is not complete, update its length with |
| one of its replica's length stored in datanode. (szetszwo) |
| |
| HDFS-649. Check null pointers for DataTransferTest. (Konstantin Boudnik |
| via szetszwo) |
| |
| HDFS-661. DataNode upgrade fails on non-existant current directory. |
| (hairong) |
| |
| HDFS-597. Mofication introduced by HDFS-537 breakes an advice binding in |
| FSDatasetAspects. (Konstantin Boudnik via szetszwo) |
| |
| HDFS-665. TestFileAppend2 sometimes hangs. (hairong) |
| |
| Release 0.20.1 - 2009-09-01 |
| |
| IMPROVEMENTS |
| |
| HDFS-438. Improve help message for space quota command. (Raghu Angadi) |
| |
| BUG FIXES |
| |
| HDFS-167. Fix a bug in DFSClient that caused infinite retries on write. |
| (Bill Zeller via szetszwo) |
| |
| HDFS-527. Remove/deprecate unnecessary DFSClient constructors. (szetszwo) |
| |
| HDFS-525. The SimpleDateFormat object in ListPathsServlet is not thread |
| safe. (Suresh Srinivas and cdouglas) |