commit 9caf3a9659821a3b4fd4394c9f4134adff9caf88
author Hossein <hossein@databricks.com> Sat Jul 26 01:04:56 2014 -0700
committer Matei Zaharia <matei@databricks.com> Sat Jul 26 01:05:05 2014 -0700
tree 3f6c43e8a5376a705e336aa1e92781369e0aa21a
parent 1d4103a09ec63dec50a64d9a31e76b8f203b601a

[SPARK-2696] Reduce default value of spark.serializer.objectStreamReset

The current default value of spark.serializer.objectStreamReset is 10,000.
When trying to re-partition (e.g., to 64 partitions) a large file (e.g., 500MB), containing 1MB records, the serializer will cache 10000 x 1MB x 64 ~= 640 GB which will cause out of memory errors.

This patch sets the default value to a more reasonable default value (100).

Author: Hossein <hossein@databricks.com>

Closes #1595 from falaki/objectStreamReset and squashes the following commits:

650a935 [Hossein] Updated documentation
1aa0df8 [Hossein] Reduce default value of spark.serializer.objectStreamReset

(cherry picked from commit 66f26a4610aede57322cb7e193a50aecb6c57d22)
Signed-off-by: Matei Zaharia <matei@databricks.com>

core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala

diff --git a/core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala b/core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala
index 0a7e1ec..a7fa057 100644
--- a/core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala
+++ b/core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala
@@ -108,7 +108,7 @@
  */
 @DeveloperApi
 class JavaSerializer(conf: SparkConf) extends Serializer with Externalizable {
-  private var counterReset = conf.getInt("spark.serializer.objectStreamReset", 10000)
+  private var counterReset = conf.getInt("spark.serializer.objectStreamReset", 100)
 
   def newInstance(): SerializerInstance = new JavaSerializerInstance(counterReset)
 

docs/configuration.md

diff --git a/docs/configuration.md b/docs/configuration.md
index 71fafa5..44766ef 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -362,13 +362,13 @@
 </tr>
 <tr>
   <td><code>spark.serializer.objectStreamReset</code></td>
-  <td>10000</td>
+  <td>100</td>
   <td>
     When serializing using org.apache.spark.serializer.JavaSerializer, the serializer caches
     objects to prevent writing redundant data, however that stops garbage collection of those
     objects. By calling 'reset' you flush that info from the serializer, and allow old
     objects to be collected. To turn off this periodic reset set it to a value &lt;= 0.
-    By default it will reset the serializer every 10,000 objects.
+    By default it will reset the serializer every 100 objects.
   </td>
 </tr>
 <tr>