tree 3f6c43e8a5376a705e336aa1e92781369e0aa21a
parent 1d4103a09ec63dec50a64d9a31e76b8f203b601a
author Hossein <hossein@databricks.com> 1406361896 -0700
committer Matei Zaharia <matei@databricks.com> 1406361905 -0700

[SPARK-2696] Reduce default value of spark.serializer.objectStreamReset

The current default value of spark.serializer.objectStreamReset is 10,000.
When trying to re-partition (e.g., to 64 partitions) a large file (e.g., 500MB), containing 1MB records, the serializer will cache 10000 x 1MB x 64 ~= 640 GB which will cause out of memory errors.

This patch sets the default value to a more reasonable default value (100).

Author: Hossein <hossein@databricks.com>

Closes #1595 from falaki/objectStreamReset and squashes the following commits:

650a935 [Hossein] Updated documentation
1aa0df8 [Hossein] Reduce default value of spark.serializer.objectStreamReset

(cherry picked from commit 66f26a4610aede57322cb7e193a50aecb6c57d22)
Signed-off-by: Matei Zaharia <matei@databricks.com>