tree 86a8b558a77e762e765bfd2688c2279ad171a481
parent 988c4ecbfab6f9096c22a3afef7685f45f6a5de0
author Ekta Khanna <ekhanna@pivotal.io> 1571963752 -0700
committer Ekta Khanna <ekhanna@pivotal.io> 1573079395 -0800

DL: Using distribution_key instead of gp_segment_id

Since `gp_segment_id` is not the actual distribution key column, the
optimizer/planner generates a plan with `Redistribute Motion`, creating
multiple slices on each segment. For DL, since GPU memory allocation is
tied to the process where it is initialized, we want to minimize
creating any additional slices per segment. This is mainly to avoid any
GPU memory allocation failures which can occur when a newly created
slice(process) tries allocating GPU memory which is already allocated by
a previously created slice(process).
Since the minibatch preprocessor evenly distributes the data with
`__dist_key__` as the table's distribution key, using it wherever
possible will avoid creation of unnecessary slices(processes).

Co-authored-by: Nikhil Kak <nkak@pivotal.io>