tree d9fbe05bc54e7ddb4fc109beb3e09e90b6f10fda
parent 3115974570bc431d5f00dfaa2be4823c91fbc634
author John Yang <johnyangk@gmail.com> 1550734734 +0900
committer Sanha Lee <sanhaleehana@gmail.com> 1550734734 +0900

[NEMO-338] SkewSamplingPass (#193)

JIRA: [NEMO-338: SkewSamplingPass](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-338)

**Major changes:**
- SamplingSkewReshapingPass: Inserts SkewSampling and MessageBarrier vertices
- SamplingVertex: Instantiated with (originalVertex, desiredSampleRate)
- IRDAG: Automatically inserts IREdges from/to SamplingVertex objects, similar to other insert() methods
- PhysicalPlanGenerator: Handles SamplingVertex objects appropriately
- Stage: Uses getTaskIndices(), returns a subset of tasks if consists of SamplingVertex objects, to determine the tasks to execute

**Minor changes to note:**
- Refactors other insert() methods to share code as much as possible

**Tests for the changes:**
- PerKeyMedianITCase#testLargeShuffleSamplingSkew (combines large shuffle + skew handling optimizations)

**Other comments:**
- Sanha(@sanha) wrote the original code. I refactored the code and added comments to create this PR.

Closes #193