hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt

  BREAKDOWN OF HDFS-7285 SUBTASKS AND RELATED JIRAS

    HDFS-7347. Configurable erasure coding policy for individual files and
    directories ( Zhe Zhang via vinayakumarb )

    HDFS-7339. Representing striped block groups in NameNode with hierarchical
    naming protocol ( Zhe Zhang )

    HDFS-7652. Process block reports for erasure coded blocks (Zhe Zhang)

    HDFS-7716. Erasure Coding: extend BlockInfo to handle EC info (Jing Zhao)

    HDFS-7749. Erasure Coding: Add striped block support in INodeFile (Jing Zhao)

    HDFS-7837. Erasure Coding: allocate and persist striped blocks in NameNode
    (Jing Zhao via Zhe Zhang)

    HDFS-7872. Erasure Coding: INodeFile.dumpTreeRecursively() supports to print
    striped blocks (Takuya Fukudome via jing9)

    HDFS-7853. Erasure coding: extend LocatedBlocks to support reading from
    striped files (Jing Zhao)

    HDFS-7826. Erasure Coding: Update INodeFile quota computation for striped
    blocks ( Kai Sasaki via jing9 )

    HDFS-7912. Erasure Coding: track BlockInfo instead of Block in
    UnderReplicatedBlocks and PendingReplicationBlocks (Jing Zhao)

    HDFS-7369. Erasure coding: distribute recovery work for striped blocks to
    DataNode (Zhe Zhang)

    HDFS-7864. Erasure Coding: Update safemode calculation for striped blocks
    (GAO Rui via jing9)

    HDFS-7827. Erasure Coding: support striped blocks in non-protobuf fsimage
    ( Hui Zheng via jing9 )

    HDFS-7616. Add a test for BlockGroup support in FSImage.
    (Takuya Fukudome via szetszwo)

    HDFS-7907. Erasure Coding: track invalid, corrupt, and under-recovery striped
    blocks in NameNode (Jing Zhao)

    HDFS-8005. Erasure Coding: simplify striped block recovery work computation
    and add tests (Jing Zhao)

    HDFS-7617. Add unit tests for editlog transactions for EC 
    (Hui Zheng via Zhe Zhang)

    HDFS-7839. Erasure coding: implement facilities in NameNode to create and
    manage EC zones (Zhe Zhang)

    HDFS-7969. Erasure coding: NameNode support for lease recovery of striped
    block groups. (Zhe Zhang)

    HDFS-7782. Erasure coding: pread from files in striped layout.
    (Zhe Zhang and Jing Zhao via Zhe Zhang)

    HDFS-8023. Erasure Coding: retrieve eraure coding schema for a file from
    NameNode (vinayakumarb)

    HDFS-8074. Define a system-wide default EC schema. (Kai Zheng)

    HDFS-8077. Erasure coding: fix bugs in EC zone and symlinks.
    (Jing Zhao and Zhe Zhang via Jing Zhao)

    HDFS-8104. Make hard-coded values consistent with the system default schema first before remove them. (Kai Zheng)

    HDFS-7889. Subclass DFSOutputStream to support writing striping layout files. (Li Bo via Kai Zheng)

    HDFS-8090. Erasure Coding: Add RPC to client-namenode to list all
    ECSchemas loaded in Namenode. (vinayakumarb)

    HDFS-8122. Erasure Coding: Support specifying ECSchema during creation of ECZone.
    (Vinayakumar B via Zhe Zhang)

    HDFS-8114. Erasure coding: Add auditlog FSNamesystem#createErasureCodingZone if this
    operation fails. (Rakesh R via Zhe Zhang)

    HDFS-8123. Erasure Coding: Better to move EC related proto messages to a
    separate erasurecoding proto file (Rakesh R via vinayakumarb)

    HDFS-7349. Support DFS command for the EC encoding (vinayakumarb)

    HDFS-8120. Erasure coding: created util class to analyze striped block groups.
    (Contributed by Zhe Zhang and Li Bo via Jing Zhao)

    HDFS-7994. Detect if resevered EC Block ID is already used during namenode
    startup. (Hui Zheng via szetszwo)

    HDFS-8167. BlockManager.addBlockCollectionWithCheck should check if the block is a striped block. (Hui Zheng via zhz).

    HDFS-8166. DFSStripedOutputStream should not create empty blocks. (Jing Zhao)

    HDFS-7937. Erasure Coding: INodeFile quota computation unit tests.
    (Kai Sasaki via Jing Zhao)

    HDFS-8145. Fix the editlog corruption exposed by failed TestAddStripedBlocks.
    (Jing Zhao)

    HDFS-8146. Protobuf changes for BlockECRecoveryCommand and its fields for
    making it ready for transfer to DN (Uma Maheswara Rao G via vinayakumarb)

    HDFS-8181. createErasureCodingZone sets retryCache state as false always
    (Uma Maheswara Rao G via vinayakumarb)

    HDFS-8190. StripedBlockUtil.getInternalBlockLength may have overflow error.
    (szetszwo)

    HDFS-8216. TestDFSStripedOutputStream should use BlockReaderTestUtil to 
    create BlockReader. (szetszwo via Zhe Zhang)

    HDFS-8212. DistributedFileSystem.createErasureCodingZone should pass schema
    in FileSystemLinkResolver. (szetszwo via Zhe Zhang)

    HDFS-8024. Erasure Coding: ECworker frame, basics, bootstraping and configuration.
    (umamahesh)

    HDFS-8156. Add/implement necessary APIs even we just have the system default 
    schema. (Kai Zheng via Zhe Zhang)

    HDFS-8136. Client gets and uses EC schema when reads and writes a stripping
    file. (Kai Sasaki via Kai Zheng)

    HDFS-8233. Fix DFSStripedOutputStream#getCurrentBlockGroupBytes when the last
    stripe is at the block group boundary. (jing9)

    HDFS-8223. Should calculate checksum for parity blocks in DFSStripedOutputStream.
    (Yi Liu via jing9)

    HDFS-8228. Erasure Coding: SequentialBlockGroupIdGenerator#nextValue may cause 
    block id conflicts (Jing Zhao via Zhe Zhang)

    HDFS-8033. Erasure coding: stateful (non-positional) read from files in 
    striped layout (Zhe Zhang)

    HDFS-8230. Erasure Coding: Ignore DatanodeProtocol#DNA_ERASURE_CODING_RECOVERY 
    commands from standbynode if any (vinayakumarb)

    HDFS-8189. ClientProtocol#createErasureCodingZone API was wrongly annotated
    as Idempotent (vinayakumarb)

    HDFS-8235. Erasure Coding: Create DFSStripedInputStream in DFSClient#open.
    (Kai Sasaki via jing9)

    HDFS-8272. Erasure Coding: simplify the retry logic in DFSStripedInputStream 
    (stateful read). (Jing Zhao via Zhe Zhang)

    HDFS-8282. Erasure coding: move striped reading logic to StripedBlockUtil.
    (Zhe Zhang)

    HDFS-8183. Erasure Coding: Improve DFSStripedOutputStream closing of 
    datastreamer threads. (Rakesh R via Zhe Zhang)

    HDFS-8308. Erasure Coding: NameNode may get blocked in waitForLoadingFSImage()
    when loading editlog. (jing9)

    HDFS-7949. WebImageViewer need support file size calculation with striped 
    blocks. (Rakesh R via Zhe Zhang)

    HDFS-8316. Erasure coding: refactor EC constants to be consistent with HDFS-8249.
    (Zhe Zhang via jing9)

    HDFS-8281. Erasure Coding: implement parallel stateful reading for striped layout.
    (jing9)

    HDFS-8137. Send the EC schema to DataNode via EC encoding/recovering command(umamahesh)

    HDFS-8242. Erasure Coding: XML based end-to-end test for ECCli commands
    (Rakesh R via vinayakumarb)

    HDFS-8324. Add trace info to DFSClient#getErasureCodingZoneInfo(..) (vinayakumarb via 
    umamahesh)

    HDFS-7672. Handle write failure for stripping blocks and refactor the
    existing code in DFSStripedOutputStream and StripedDataStreamer.  (szetszwo)

    HDFS-7348. Erasure Coding: DataNode reconstruct striped blocks. 
    (Yi Liu via Zhe Zhang)

    HADOOP-11921. Enhance tests for erasure coders. (Kai Zheng)

    HDFS-8334. Erasure coding: rename DFSStripedInputStream related test 
    classes. (Zhe Zhang)

    HDFS-8129. Erasure Coding: Maintain consistent naming for Erasure Coding related classes - EC/ErasureCoding
    (umamahesh)

    HDFS-8203. Erasure Coding: Seek and other Ops in DFSStripedInputStream.
    (Yi Liu via jing9)

    HDFS-8289. Erasure Coding: add ECSchema to HdfsFileStatus. (Yong Zhang via
    jing9)

    HDFS-8355. Erasure Coding: Refactor BlockInfo and BlockInfoUnderConstruction.
    (Tsz Wo Nicholas Sze via jing9)

    HDFS-7678. Erasure coding: DFSInputStream with decode functionality (pread).
    (Zhe Zhang)

    HDFS-8372. Erasure coding: compute storage type quotas for striped files,
    to be consistent with HDFS-8327. (Zhe Zhang via jing9)

    HDFS-8368. Erasure Coding: DFS opening a non-existent file need to be 
    handled properly (Rakesh R via zhz)

    HDFS-8363. Erasure Coding: DFSStripedInputStream#seekToNewSource. (yliu)

    HDFS-8195. Erasure coding: Fix file quota change when we complete/commit 
    the striped blocks. (Takuya Fukudome via zhz)

    HDFS-8364. Erasure coding: fix some minor bugs in EC CLI
    (Walter Su via vinayakumarb)

    HDFS-8391. NN should consider current EC tasks handling count from DN while 
    assigning new tasks. (umamahesh)

    HDFS-8367. BlockInfoStriped uses EC schema. (Kai Sasaki via Kai Zheng)

    HDFS-8352. Erasure Coding: test webhdfs read write stripe file. (waltersu4549)

    HDFS-8417. Erasure Coding: Pread failed to read data starting from not-first stripe.
    (Walter Su via jing9)

    HDFS-8418. Fix the isNeededReplication calculation for Striped block in NN.
    (Yi Liu via jing9)

    HDFS-8320. Erasure coding: consolidate striping-related terminologies. (zhz)

    HDFS-8366. Erasure Coding: Make the timeout parameter of polling blocking queue 
    configurable in DFSStripedOutputStream. (Li Bo)

    HDFS-8378. Erasure Coding: Few improvements for the erasure coding worker.
    (Rakesh R via waltersu4549)

    HDFS-8375. Add cellSize as an XAttr to ECZone. ( Vinayakumar B via zhz).

    HDFS-8428. Erasure Coding: Fix the NullPointerException when deleting file.
    (Yi Liu via zhz).

    HDFS-8323. Bump GenerationStamp for write faliure in DFSStripedOutputStream.
    (Tsz Wo Nicholas Sze via jing9)

    HDFS-8427. Remove dataBlockNum and parityBlockNum from BlockInfoStriped.
    (Kai Sasaki via jing9)

    HDFS-8186. Erasure coding: Make block placement policy for EC file configurable.
    (Walter Su via zhz)

    HDFS-8294. Erasure Coding: Fix Findbug warnings present in erasure coding.
    (Rakesh R via zhz)

    HDFS-8441. Erasure Coding: make condition check earlier for setReplication.
    (waltersu4549)

    HDFS-7768. Change fsck to support EC files.  (Takanobu Asanuma via szetszwo)

    HDFS-8382. Remove chunkSize and initialize from erasure coder. (Kai Zheng)

    HDFS-8408. Revisit and refactor ErasureCodingInfo (vinayakumarb)

    HDFS-8479. Erasure coding: fix striping related logic in FSDirWriteFileOp to
    sync with HDFS-8421. (Zhe Zhang via jing9)

    HDFS-8481. Erasure coding: remove workarounds in client side stripped blocks 
    recovering. (zhz)

    HDFS-8336. Expose some administrative erasure coding operations to HdfsAdmin
    (Uma Maheswara Rao G via vinayakumarb)

    HDFS-8444. Erasure Coding: fix cannot rename a zone dir
    (Walter Su via vinayakumarb)

    HDFS-8517. Fix a decoding issue in stripped block recovering in client side.
    (Kai Zheng via jing9)

    HDFS-8453. Erasure coding: properly handle start offset for internal blocks
    in a block group. (Zhe Zhang via jing9)

    HDFS-7621. Erasure Coding: update the Balancer/Mover data migration logic.
    (Walter Su via zhz)

    HDFS-8328. Follow-on to update decode for DataNode striped blocks
    reconstruction. (yliu)

    HDFS-8319. Erasure Coding: support decoding for stateful read.
    (Jing Zhao via zhz)

    HDFS-8460. Erasure Coding: stateful read result doesn't match data 
    occasionally because of flawed test. (Walter Su via zhz)

    HDFS-8556. Erasure Coding: Fix usage of 'createZone' (vinayakumarb)

    HDFS-8571. Fix TestErasureCodingCli test (Vinayakumar B via waltersu4549)

    HDFS-8450. Erasure Coding: Consolidate erasure coding zone related
    implementation into a single class (Rakesh R via vinayakumarb)

    HDFS-8585. Erasure Coding: Remove dataBlockNum and parityBlockNum from
    StripedBlockProto. (Yi Liu via jing9)

    HDFS-8559. Erasure Coding: fix non-protobuf fsimage for striped blocks.
    (Jing Zhao via yliu)

    HDFS-8580. Erasure coding: Persist cellSize in BlockInfoStriped and
    StripedBlocksFeature. (Walter Su via jing9)

    HDFS-8466. Refactor BlockInfoContiguous and fix NPE in
    TestBlockInfo#testCopyConstructor() (vinayakumarb)

    HDFS-8254. Avoid assigning a leading streamer in StripedDataStreamer to
    tolerate datanode failure. (Tsz Wo Nicholas Sze via jing9)

    HDFS-8543. Erasure Coding: processOverReplicatedBlock() handles striped block.
    (Walter Su via jing9)

    HDFS-8602. Erasure Coding: Client can't read(decode) the EC files which have
    corrupt blocks. (jing9 and Kai Sasaki)

    HDFS-8567. Erasure Coding: SafeMode handles file smaller than a full stripe.
    (Walter Su via jing9)

    HDFS-8253. DFSStripedOutputStream.closeThreads releases cellBuffers
    multiple times.  (Kai Sasaki via szetszwo)

    HDFS-8468. 2 RPC calls for every file read in DFSClient#open(..) resulting in
    double Audit log entries (vinayakumarb)

    HDFS-8684. Erasure Coding: fix some block number calculation for striped
    block. (yliu)

    HDFS-8461. Erasure coding: fix priority level of UnderReplicatedBlocks for
    striped block. (Walter Su via jing9)

    HDFS-8719. Erasure Coding: client generates too many small packets when
    writing parity data. (Li Bo via waltersu4549)

    HDFS-8563. Erasure Coding: fsck handles file smaller than a full stripe.
    (Walter Su via jing9)

    HDFS-8484. Erasure coding: Two contiguous blocks occupy IDs belong to same
    striped group. (Walter Su via jing9)

    HDFS-8744. Erasure Coding: the number of chunks in packet is not updated
    when writing parity data. (Li Bo)

    HDFS-8669. Erasure Coding: handle missing internal block locations in
    DFSStripedInputStream. (jing9)

    HDFS-8702. Erasure coding: update BlockManager.blockHasEnoughRacks(..) logic
    for striped block. (Kai Sasaki via jing9)

    HDFS-8734. Erasure Coding: fix one cell need two packets. (Walter Su via
    jing9)

    HDFS-8619. Erasure Coding: revisit replica counting for striped blocks.
    (Jing Zhao via yliu)

    HDFS-8058. Erasure coding: use BlockInfo[] for both striped and contiguous
    blocks in INodeFile. (Zhe Zhang and Yi Liu via zhz)

    HDFS-8787. Erasure coding: rename BlockInfoContiguousUC and BlockInfoStripedUC
    to be consistent with trunk. (zhz)

    HDFS-8433. Erasure coding: set blockToken in LocatedStripedBlock.(waltersu4549)

    HDFS-8760. Erasure Coding: reuse BlockReader when reading the same block in pread.
    (jing9)

    HDFS-8781. Erasure Coding: Correctly handle BlockManager#InvalidateBlocks for
    striped block. (Yi Liu via jing9)

    HDFS-8813. Erasure Coding: Client no need to decode missing parity blocks.
    (Walter Su via jing9)

    HDFS-8798. Erasure Coding: fix DFSStripedInputStream/DFSStripedOutputStream
    re-fetch token when expired. (Walter Su via jing9)

    HDFS-8769. Erasure coding: unit test for SequentialBlockGroupIdGenerator.
    (Rakesh R via waltersu4549)

    HDFS-8202. Improve end to end stirpping file test to add erasure recovering
    test. (Xinwei Qin via zhz)

    HDFS-8804. Erasure Coding: use DirectBufferPool in DFSStripedInputStream for
    buffer allocation. (jing9)

    HDFS-8399. Erasure Coding: unit test the behaviour of BlockManager recovery
    work for the deleted blocks. (Rakesh R via zhz)

    HDFS-8857. Erasure Coding: Fix ArrayIndexOutOfBoundsException in
    TestWriteStripedFileWithFailure. (Li Bo)

    HDFS-8827. Erasure Coding: Fix NPE when NameNode processes over-replicated
    striped blocks. (Walter Su and Takuya Fukudome via jing9)

    HDFS-8854. Erasure coding: add ECPolicy to replace schema+cellSize in
    hadoop-hdfs. (Walter Su via zhz)

    HDFS-8220. Erasure Coding: StripedDataStreamer fails to handle the
    blocklocations which doesn't satisfy BlockGroupSize. (Rakesh R via zhz)