tree 575b5eb19d7a28fe23ed0f752c5c419857d33c16
parent 687473ea1153b2e7649fd3223933f46ff0e818f8
author Domino Valdano <domino@apache.org> 1578622016 -0800
committer GitHub <noreply@github.com> 1578622016 -0800
gpgsig -----BEGIN PGP SIGNATURE-----
 
 wsBcBAABCAAQBQJeF9xACRBK7hj4Ov3rIwAAdHIIAK3TwzwSkX2HclTL4NGVvyKZ
 x515ME8LSyIXwkUdYZx7/dRh+MLhZlKAYGjdC6XZtEC9Gf8ljz5rXfaOcxb1mvFF
 Yn3cFKK/O1oF+00j++4YTw8/enCCD6Lxur+tYSGwHXpdUGBWslvj/OFG9BkJjWXu
 eGrVWlbznugejOKLa+aCDtjaoo4MWhA7dLylpuwja9U9iXvVIq61uGrpgyTbv1T6
 P1gfXJKTqX2edWS6FARqI66qVi/NZkkHh8iL3IuiCkZPlC7nmt56j67jGJc+xHgr
 B4KON+HIEQ438n3E5FS+J43hFBPR2gnFO3MchtkN4q4hB+N/IPjpyJ5/iTL2fHY=
 =vGJI
 -----END PGP SIGNATURE-----
 

DL: Improve performance of mini-batch preprocessor (#467)

JIRA MADLIB-1334

This commit adds the following optimizations to minibatch preprocessor:

- Skip normalization if normalizing contant is 1.0

- Split the batching query to generate buffer_id's (based of row_id)
without moving around any data. Previously, calling `ROW_NUMBER()
OVER()` to add row_id's to the table was causing the data to be gathered
on the master node and then numbering the rows, which for large datasets
would be taking most of the time.

- Separate out the JOIN (called for even distribution) as well as
converting to bytea from the batching query. This avoids any VMEM limit
issues.

- num_buffers gets rounded up to the nearest multiple of num_segments
for even distribution across buffers on segments.

- Add new C function `array_to_bytea()` to convert array to bytea, and
some tests for it.  This is much faster than the python version we were
using, speeding up the query significantly.

Additionally, this commit adds a new function `plpy_execute_debug()`
in the utilities module that prints EXPLAIN plans and execution time
for debugging a specific query.

Co-authored-by: Ekta Khanna <ekhanna@pivotal.io>

Co-authored-by: Ekta Khanna <ekhanna@pivotal.io>