tree 9eeeb60a054876dde73f4cde395bfe0a08868c86
parent 4a729e10f75ea3c5b99540e089b9dfb204bb84f5
author haomarch <marchpure@126.com> 1602601858 +0800
committer QiangCai <qiangcai@qq.com> 1604394072 +0800

[CARBONDATA-4044] Fix dirty data in index file while IUD with stale data in segment folder

Why is this PR needed?
XX.mergecarbonindex and XX..segment records the indexfiles list of a segment. now, we generate xx.mergeindexfile and xx.segment  based on filter out all indexfiles(including carbonindex and mergecarbonindex), which will leading dirty data when there is stale data in segment folder.
For example, there are a stale index file in segment_0 folder, "0_1603763776.carbonindex".
While loading, a new carbonindex "0_16037752342.carbonindex" is wrote, when merge carbonindex files, we expect to only merge 0_16037752342.carbonindex, But If we filter out all carbonindex in segment folder, both "0_1603763776.carbonindex" and 0_16037752342.carbonindex will be merged and recorded into segment file.
While updating, there has same problem.

What changes were proposed in this PR?
1. IUD: merge file based on UUID(timestamp).
2. IUD: write segment file based on UUID(timestamp).
3. Update: update will generate a new segment to avoid rewrite segment file. also will rollback the new segment when update fails
4. We delete horizotal compaction processing of update delta(both carbondata files and carbonindex).
5. Clean the dead code when update write into new segment
6. Fix an update issue: The wrong update result after drop partition.

Does this PR introduce any user interface change?
Yes

Is any new testcase added?
Yes

This closes #3999