tree 0ceb1e7c7716017bedb42fd3f4395880f00f4ea6
parent 610cf6de306499e170d8e03dab85e1ee5559ccaa
author kaknikhil <kaknikhil@users.noreply.github.com> 1571166808 -0700
committer GitHub <noreply@github.com> 1571166808 -0700
gpgsig -----BEGIN PGP SIGNATURE-----
 
 wsBcBAABCAAQBQJdphpYCRBK7hj4Ov3rIwAAdHIIAKRmuGE/CK4w/EcIW1SsqXMw
 3H3P/DXtzLBCITrvjEp4+ITYkNrlyoGNAjyZDI/3SLDqW+AnKo77Sh5c6pkhU4Mo
 MsDwTcavFwJa/jwzdZuT6E1p1ng0gcQeJoTzskDcWoA7brNVUEK/bsKsXpM5J131
 GZSYu/yj4S2EvmvhzkudVmsb9f7hCNH6WACz6fgaKeN/OLBNjD/PHUpd5K0WSl8c
 yRi5UFioUh0cRGrq825iwAzX38LT/ujAn8Y9Mu+yY7zD6+di8f28kr6wuP3A8Olm
 WezcI/NQid7azV8NinJiMzGAfgbn2Vt3w+hvb9+LE2t4tC1YJorZy5PVIVVp3xc=
 =Y10Q
 -----END PGP SIGNATURE-----
 

DL: Use REAL[] instead of anyarray for aggregating arrays (#449)

* DL: Use REAL[] instead of anyarray for aggregating arrays

JIRA: MADLIB-1334

Using real[] instead of anyarray for agg_array_concat solves the scaling
problem i.e. running input preprocessor on same dataset with different
buffer sizes results in comparable runtimes. This is because the plan
for the agg with real[] aggregates on the segments and then gathers on
the master but the plan for the agg with anyarray aggregate first
gathers all the data on the master and then runs the aggregate function
on the master.

Here are the results from my local mac with gpdb 5.21 3 segments

```
mnist 10k rows anyarray agg
madlib=# select madlib.training_preprocessor_dl('mnist_train_10k', 'mnist_batch_10k_anyarray', 'y', 'x');
Time: 109790.096 ms

mnist 10k rows real_array_agg
madlib=# select madlib.training_preprocessor_dl('mnist_train_10k', 'mnist_batch_10k_real_array', 'y', 'x');
Time: 14086.535 ms

mnist 60k rows with anyarray agg
madlib=# select madlib.training_preprocessor_dl('mnist_train', 'mnist_batch_anyarray', 'y', 'x');
Had to cancel after 55 mins

mnist 60k rows with real_array agg
madlib=# select madlib.training_preprocessor_dl('mnist_train', 'mnist_batch_real_array', 'y', 'x');
Time: 580822.559 ms
```

Co-authored-by: Domino Valdano <dvaldano@pivotal.io>