DATAFU-166 Drop support for Spark 2.1.x and advance our version to 1.7.0-SNAPSHOT
diff --git a/datafu-spark/README.md b/datafu-spark/README.md
index b72221e..e8125e5 100644
--- a/datafu-spark/README.md
+++ b/datafu-spark/README.md
@@ -12,7 +12,7 @@
* [Count distinct up to](https://github.com/apache/datafu/blob/spark-tmp/datafu-spark/src/main/scala/datafu/spark/SparkUDAFs.scala#L224) - an efficient implementation when you just want to verify that a certain minimum of distinct rows appear in a table
-It has been tested on Spark releases from 2.1.0 to 2.4.3, using Scala 2.10, 2.11 and 2.12. You can check if your Spark/Scala version combination has been tested by looking [here.](https://github.com/apache/datafu/blob/spark-tmp/datafu-spark/build_and_test_spark.sh#L20)
+It has been tested on Spark releases from 2.2.0 to 2.4.3, using Scala 2.11 and 2.12. You can check if your Spark/Scala version combination has been tested by looking [here.](https://github.com/apache/datafu/blob/master/datafu-spark/build_and_test_spark.sh#L20)
-----------
@@ -21,9 +21,9 @@
First, call pyspark with the following parameters
```bash
-export PYTHONPATH=datafu-spark_2.11_2.3.0-1.5.0-SNAPSHOT.jar
+export PYTHONPATH=datafu-spark_2.11_2.3.0-1.6.1.jar
-pyspark --jars datafu-spark_2.11_2.3.0-1.5.0-SNAPSHOT.jar --conf spark.executorEnv.PYTHONPATH=datafu-spark_2.11_2.3.0-1.5.0-SNAPSHOT.jar
+pyspark --jars datafu-spark_2.11_2.3.0-1.6.1.jar --conf spark.executorEnv.PYTHONPATH=datafu-spark_2.11_2.3.0-1.6.1.jar
```
The following is an example of calling the Spark version of the datafu _dedup_ method
diff --git a/datafu-spark/build_and_test_spark.sh b/datafu-spark/build_and_test_spark.sh
index 703818f..744ecd3 100755
--- a/datafu-spark/build_and_test_spark.sh
+++ b/datafu-spark/build_and_test_spark.sh
@@ -17,10 +17,10 @@
#!/bin/bash
-export SPARK_VERSIONS_FOR_SCALA_211="2.1.0 2.1.1 2.1.2 2.1.3 2.2.0 2.2.1 2.2.2 2.3.0 2.3.1 2.3.2 2.4.0 2.4.1 2.4.2 2.4.3"
+export SPARK_VERSIONS_FOR_SCALA_211="2.2.0 2.2.1 2.2.2 2.3.0 2.3.1 2.3.2 2.4.0 2.4.1 2.4.2 2.4.3"
export SPARK_VERSIONS_FOR_SCALA_212="2.4.0 2.4.1 2.4.2 2.4.3"
-export LATEST_SPARK_VERSIONS_FOR_SCALA_211="2.1.3 2.2.2 2.3.2 2.4.3"
+export LATEST_SPARK_VERSIONS_FOR_SCALA_211="2.2.2 2.3.2 2.4.3"
export LATEST_SPARK_VERSIONS_FOR_SCALA_212="2.4.3"
STARTTIME=$(date +%s)
diff --git a/gradle.properties b/gradle.properties
index 89e1a3f..b520f86 100644
--- a/gradle.properties
+++ b/gradle.properties
@@ -16,9 +16,8 @@
# under the License.
group=org.apache.datafu
-version=1.6.1
+version=1.7.0-SNAPSHOT
gradleVersion=5.6.4
-org.gradle.jvmargs="-XX:MaxPermSize=512m"
scalaCompatVersion=2.11
sparkCompatVersion=2.4
sparkVersion=2.4.3
