Hadoop HDFS Change Log

Trunk (unreleased changes)

  INCOMPATIBLE CHANGES

    HDFS-538. Per the contract elucidated in HADOOP-6201, throw
    FileNotFoundException from FileSystem::listStatus rather than returning
    null. (Jakob Homan via cdouglas)

    HDFS-602. DistributedFileSystem mkdirs throws FileAlreadyExistsException
    instead of FileNotFoundException. (Boris Shkolnik via suresh)

    HDFS-544. Add a "rbw" subdir to DataNode data directory. (hairong)

    HDFS-576. Block report includes under-construction replicas. (shv)

    HDFS-636. SafeMode counts complete blocks only. (shv)

    HDFS-644. Lease recovery, concurrency support. (shv)

    HDFS-570. Get last block length from a data-node when opening a file
    being written to. (Tsz Wo (Nicholas), SZE via shv)

    HDFS-657. Remove unused legacy data-node protocol methods. (shv)

    HDFS-658. Block recovery for primary data-node. (shv)

  NEW FEATURES

    HDFS-436. Introduce AspectJ framework for HDFS code and tests.
    (Konstantin Boudnik via szetszwo)

    HDFS-447. Add LDAP lookup to hdfsproxy. (Zhiyong Zhang via cdouglas)

    HDFS-459. Introduce Job History Log Analyzer. (shv)

    HDFS-461. Tool to analyze file size distribution in HDFS. (shv)

    HDFS-492. Add two JSON JSP pages to the Namenode for providing corrupt
    blocks/replicas information.  (Bill Zeller via szetszwo)

    HDFS-578. Add support for new FileSystem method for clients to get server
    defaults. (Kan Zhang via suresh)

    HDFS-595. umask settings in configuration may now use octal or symbolic 
    instead of decimal. (Jakob Homan via suresh)

    HADOOP-6234. Updated hadoop-core and test jars to propagate new option 
    dfs.umaskmode in configuration. (Jakob Homan via suresh)

    HDFS-235. Add support for byte ranges in HftpFileSystem to serve
    range of bytes from a file. (Bill Zeller via suresh)

    HDFS-385. Add support for an experimental API that allows a module external
    to HDFS to specify how HDFS blocks should be placed. (dhruba)

    HADOOP-4952. Update hadoop-core and test jars to propagate new FileContext
    file system application interface. (Sanjay Radia via suresh).

    HDFS-567. Add block forensics contrib tool to print history of corrupt and
    missing blocks from the HDFS logs.
    (Bill Zeller, Jithendra Pandey via suresh).

    HDFS-610. Support o.a.h.fs.FileContext.  (Sanjay Radia via szetszwo)

    HDFS-536. Support hflush at DFSClient. (hairong)

    HDFS-517. Introduce BlockInfoUnderConstruction to reflect block replica
    states while writing. (shv)

    HDFS-565. Introduce block committing logic during new block allocation
    and file close. (shv)

    HDFS-537. DataNode exposes a replica's meta info to BlockReceiver for the
    support of dfs writes/hflush. It also updates a replica's bytes received,
    bytes on disk, and bytes acked after receiving a packet. (hairong)

    HDFS-585. Datanode should serve up to visible length of a replica for read
    requests.  (szetszwo)

    HDFS-604. Block report processing for append. (shv)

    HDFS-619. Support replica recovery initialization in datanode for the new
    append design.  (szetszwo)

    HDFS-592. Allow clients to fetch a new generation stamp from NameNode for
    pipeline recovery. (hairong)

    HDFS-624. Support a new algorithm for pipeline recovery and pipeline setup
    for append. (hairong)

    HDFS-627. Support replica update in data-node.
    (Tsz Wo (Nicholas), SZE and Hairong Kuang via shv)

    HDFS-642. Support pipeline close and close error recovery. (hairong)

  IMPROVEMENTS

    HDFS-381. Remove blocks from DataNode maps when corresponding file
    is deleted. (Suresh Srinivas via rangadi)

    HDFS-377. Separate codes which implement DataTransferProtocol.
    (szetszwo)

    HDFS-396. NameNode image and edits directories are specified as URIs.
    (Luca Telloli via rangadi)

    HDFS-444. Allow to change probability levels dynamically in the fault
    injection framework.  (Konstantin Boudnik via szetszwo)

    HDFS-352. Documentation for saveNamespace command. (Ravi Phulari via shv)

    HADOOP-6106. Updated hadoop-core and test jars from hudson trunk 
    build #12. (Giridharan Kesavan)

    HDFS-204. Add a new metrics FilesInGetListingOps to the Namenode.
    (Jitendra Nath Pandey via szetszwo)

    HDFS-278. HDFS Outputstream close does not hang forever. (dhruba)

    HDFS-443. Add a new metrics numExpiredHeartbeats to the Namenode.
    (Jitendra Nath Pandey via szetszwo)

    HDFS-475. Add new ant targets for fault injection jars and tests.
    (Konstantin Boudnik via szetszwo)

    HDFS-458. Create a new ant target, run-commit-test.  (Jakob Homan
    via szetszwo)

    HDFS-493. Change build.xml so that the fault-injected tests are executed
    only by the run-test-*-fault-inject targets.  (Konstantin Boudnik via
    szetszwo)

    HDFS-446. Improvements to Offline Image Viewer. (Jakob Homan via shv)

    HADOOP-6160. Fix releaseaudit target to run on specific directories.
    (gkesavan)

    HDFS-501. Use enum to define the constants in DataTransferProtocol.
    (szetszwo)

    HDFS-508. Factor out BlockInfo from BlocksMap. (shv)

    HDFS-510. Rename DatanodeBlockInfo to be ReplicaInfo.
    (Jakob Homan & Hairong Kuang via shv)

    HDFS-500. Deprecate NameNode methods deprecated in NameNodeProtocol.
    (Jakob Homan via shv)

    HDFS-514. Change DFSClient.namenode from public to private.  (Bill Zeller
    via szetszwo)

    HDFS-496. Use PureJavaCrc32 in HDFS.  (Todd Lipcon via szetszwo)

    HDFS-511. Remove redundant block searches in BlockManager. (shv)

    HDFS-504. Update the modification time of a file when the file 
    is closed. (Chun Zhang via dhruba)

    HDFS-498. Add development guide and documentation for the fault injection
    framework.  (Konstantin Boudnik via szetszwo)

    HDFS-524. Further DataTransferProtocol code refactoring.  (szetszwo)

    HDFS-529. Use BlockInfo instead of Block to avoid redundant block searches
    in BlockManager. (shv)

    HDFS-530. Refactor TestFileAppend* to remove code duplication.
    (Konstantin Boudnik via szetszwo)

    HDFS-451. Add fault injection tests for DataTransferProtocol.  (szetszwo)

    HDFS-409. Add more access token tests.  (Kan Zhang via szetszwo)

    HDFS-546. DatanodeDescriptor iterates blocks as BlockInfo. (shv)

    HDFS-457. Do not shutdown datanode if some, but not all, volumes fail.
    (Boris Shkolnik via szetszwo)

    HDFS-548. TestFsck takes nearly 10 minutes to run. (hairong)

    HDFS-539. Refactor fault injeciton pipeline test util for future reuse.
    (Konstantin Boudnik via szetszwo)

    HDFS-552. Change TestFiDataTransferProtocol to junit 4 and add a few new
    tests.  (szetszwo)

    HDFS-563. Simplify the codes in FSNamesystem.getBlockLocations(..).
    (szetszwo)

    HDFS-581. Introduce an iterator over blocks in the block report array. (shv)

    HDFS-549. Add a new target, run-with-fault-inject-testcaseonly, which
    allows an execution of non-FI tests in FI-enable environment.  (Konstantin
    Boudnik via szetszwo)

    HDFS-173. Namenode will not block until a large directory deletion completes.
    It allows other operations when the deletion is in progress. (suresh)

    HDFS-551. Create new functional test for a block report. (Konstantin
    Boudnik via hairong)

    HDFS-288. Redundant computation in hashCode() implementation.
    (szetszwo via tomwhite)

    HDFS-412. Hadoop JMX usage makes Nagios monitoring impossible.
    (Brian Bockelman via tomwhite)

    HDFS-472. Update hdfsproxy documentation. Adds a setup guide and design
    document. (Zhiyong Zhang via cdouglas)

    HDFS-617. Support non-recursive create().  (Kan Zhang via szetszwo)

    HDFS-618. Support non-recursive mkdir().  (Kan Zhang via szetszwo)

    HDFS-574. Split the documentation between the subprojects.
    (Corinne Chandel via omalley)

    HDFS-598. Eclipse launch task for HDFS. (Eli Collins via tomwhite)

    HDFS-509. Redesign DataNode volumeMap to include all types of Replicas.
    (hairong)

    HDFS-562. Add a test for NameNode.getBlockLocations(..) to check read from
    un-closed file.  (szetszwo)

    HDFS-543. Break FSDatasetInterface#writToBlock() into writeToRemporary,
    writeToRBW, ad append. (hairong)

    HDFS-603. Add a new interface, Replica, which is going to replace the use
    of Block in datanode.  (szetszwo)

    HDFS-589. Change block write protocol to support pipeline recovery.
    (hairong)

    HDFS-652. Replace BlockInfo.isUnderConstruction() with isComplete() (shv)

    HDFS-648. Change some methods in AppendTestUtil to public.  (Konstantin
    Boudnik via szetszwo)

    HDFS-662. Unnecessary info message from DFSClient. (hairong)

    HDFS-518. Create new tests for Append's hflush. (Konstantin Boudnik
    via szetszwo)

  BUG FIXES

    HDFS-76. Better error message to users when commands fail because of 
    lack of quota. Allow quota to be set even if the limit is lower than
    current consumption. (Boris Shkolnik via rangadi)

    HADOOP-4687. HDFS is split from Hadoop Core. It is a subproject under 
    Hadoop (Owen O'Malley)

    HADOOP-6096. Fix Eclipse project and classpath files following project
    split. (tomwhite)

    HDFS-195. Handle expired tokens when write pipeline is reestablished.
    (Kan Zhang via rangadi)

    HDFS-181. Validate src path in FSNamesystem.getFileInfo(..).  (Todd
    Lipcon via szetszwo)

    HDFS-441. Remove TestFTPFileSystem.  (szetszwo)

    HDFS-440. Fix javadoc broken links in DFSClient.  (szetszwo)

    HDFS-480. Fix a typo in the jar name in build.xml.  
    (Konstantin Shvachko via gkesavan)

    HDFS-438. Check for NULL before invoking GenericArgumentParser in
    DataNode. (Raghu Angadi)

    HDFS-415. BlockReceiver hangs in case of certain runtime exceptions.
    (Konstantin Boudnik via rangadi)

    HDFS-462. loadFSImage should close edits file. (Jakob Homan via shv)

    HDFS-489. Update TestHDFSCLI for the -skipTrash option in rm. (Jakob Homan
    via szetszwo)

    HDFS-445. pread() does not pick up changes to block locations. 
    (Kan Zhang via rangadi) 

    HDFS-463. CreateEditLog utility broken after HDFS-396 (URI for
    FSImage). (Suresh Srinivas via rangadi)

    HDFS-484. Fix bin-package and package target to package jar files.
    (gkesavan)

    HDFS-490. Eliminate the deprecated warnings introduced by H-5438.
    (He Yongqiang via szetszwo)

    HDFS-119. Fix a bug in logSync(), which causes NameNode block forever.
    (Suresh Srinivas via shv)

    HDFS-534. Include avro in ivy.  (szetszwo)

    HDFS-532. Allow applications to know that a read request failed 
    because block is missing. (dhruba)

    HDFS-561. Fix write pipeline READ_TIMEOUT in DataTransferProtocol.
    (Kan Zhang via szetszwo)

    HDFS-553. BlockSender reports wrong failed position in ChecksumException.
    (hairong)

    HDFS-568. Set mapred.job.tracker.retire.jobs to false in
    src/test/mapred-site.xml for mapreduce tests to run.  (Amareshwari
    Sriramadasu via szetszwo)
 
    HDFS-15. All replicas end up on 1 rack. (Jitendra Nath Pandey via hairong)
 
    HDFS-586. TestBlocksWithNotEnoughRacks sometimes fails.
    (Jitendra Nath Pandey via hairong)

    HADOOP-6243. Fixed a NullPointerException in handling deprecated keys.
    (Sreekanth Ramakrishnan via yhemanth)

    HDFS-605. Do not run fault injection tests in the run-test-hdfs-with-mr
    target.  (Konstantin Boudnik via szetszwo)

    HDFS-606. Fix ConcurrentModificationException in invalidateCorruptReplicas()
    (shv)

    HDFS-601. TestBlockReport obtains data directories directly from
    MiniHDFSCluster. (Konstantin Boudnik via shv)

    HDFS-614. TestDatanodeBlockScanner obtains data directories directly from
    MiniHDFSCluster. (shv)

    HDFS-612. Remove the use of org.mortbay.log.Log in FSDataset.  (szetszwo)

    HDFS-622. checkMinReplication should count live nodes only. (shv)

    HDFS-629. Remove ReplicationTargetChooser.java along with fixing 
    import warnings generated by Eclipse. (dhruba)

    HDFS-640. Fix TestHDFSFileContextMainOperations.java build failure. (suresh)

    HDFS-637. DataNode sends a Success ack when block write fails. (hairong)

    HDFS-547. TestHDFSFileSystemContract#testOutputStreamClosedTwice
    sometimes fails with CloseByInterruptException. (hairong)

    HDFS-588. Fix TestFiDataTransferProtocol and TestAppend2 failures. (shv)

    HDFS-550. DataNode restarts may introduce corrupt/duplicated/lost replicas
    when handling detached replicas. (hairong)

    HDFS-659. If the the last block is not complete, update its length with
    one of its replica's length stored in datanode.  (szetszwo)

    HDFS-649. Check null pointers for DataTransferTest.  (Konstantin Boudnik
    via szetszwo)

    HDFS-661. DataNode upgrade fails on non-existant current directory.
    (hairong)

    HDFS-597. Mofication introduced by HDFS-537 breakes an advice binding in
    FSDatasetAspects.  (Konstantin Boudnik via szetszwo)

    HDFS-665. TestFileAppend2 sometimes hangs. (hairong)

Release 0.20.1 - 2009-09-01

  IMPROVEMENTS

    HDFS-438. Improve help message for space quota command. (Raghu Angadi)

  BUG FIXES

    HDFS-167. Fix a bug in DFSClient that caused infinite retries on write.
    (Bill Zeller via szetszwo)

    HDFS-527. Remove/deprecate unnecessary DFSClient constructors.  (szetszwo)

    HDFS-525. The SimpleDateFormat object in ListPathsServlet is not thread
    safe. (Suresh Srinivas and cdouglas)