tree 719311c57ff5da8f36a47bee0c892c23fce0b3b9
parent b97128edc645be45a09fccdc449cae2fd2225681
author Jeremy Nelson <jeremy@digitalminion.com> 1591213454 -0500
committer Jeremy Nelson <jeremy@digitalminion.com> 1591213454 -0500

(SPOT-286) Do not call org.apache.spark.sql.functions.broadcast() on large sets

https://issues.apache.org/jira/projects/SPOT/issues/SPOT-286
identifies an issue with calling org.apache.spark.sql.functions.broadcast()
on large datasets which may exceed a limit and cause an exception.

It was determined wrapping a dataset in this way is a performance
optimization, and the justification for this optimization is not clear.

It was decided it was preferable to undo the optimization than allow
it to fail for some situations.