commit d4b0709cf8c4f20ad61114d69022f15663b84915
author Vikram Ahuja <vikramahuja8803@gmail.com> Wed Jun 24 12:16:42 2020 +0530
committer akashrn5 <akashnilugal@gmail.com> Fri Jul 10 22:55:49 2020 +0530
tree c850007044293a611b2a58f9343525729e88743f
parent 65d9813652d76bd6b0c37e360618459224a08f21

[CARBONDATA-3885] [CARBONDATA-3884] Fix for load failures with isSIenabled = false and fix for concurrent load failure

Why is this PR needed?
Issue 1:
In some concurrent scenarios it was found that certain segment folders were getting deleted for the SI table. When
the segment was being inserted again back to the SI table during a new load to the main table by doing isSIenabled = false,
the load to the SI table failed.

Root Cause: It was found that when the segment folder gets deleted and it's entry from table status file is manually
deleted then, during new load to SI when table status is changed to InsertInProgress during the loading previously
deleted segments the stale .segmentfile still exists which was pointing to the mergeindexfile which does exist
anymore(since the segment folder is deleted). Since that mergeindexfile does not exist anymore the load to SI table failed

Issue 2:
In some concurrent load scenarios in main table with SI table with isSITableEnabled = false , load to SI table fails.

Root Cause: When concurrent loading is done with isSITableEnabled = false, maintable details are taken earlier in the
code and SItable details are taken later. It was found that there were some scenarios where another concurrent load to
main table finishes and can change the number of segments in the main table due to which there is mismatch in number
of segments in the SI table and in the main table. Due to this mismatch when the validmaintableload method is called,
there can be scenarios where the load fails.

What changes were proposed in this PR?
Solution for Issue 1: Whenever isSIenabled is set to false and it tries to load previously deleted SI segments to the
SI table we go and delete the stale .segment file from the metadata folder if it exists. Due to this, while loading a
new .segment file will be created thus having a new index file path.

Solution for Issue 2: Whenever isSIenabled is set to false and it tries to load previously deleted SI segments,
just before adding segments in SIFailedSegmentList checking again if the loadname is present in the main table. Only
adding in SIFailedSegmenList if the loadname is present in the main table and it's in SUCCESS state.

This closes #3802