Like other modules of Apache Griffin, measure
module is also built using Maven build tool. Building measure
module requires Maven version 3.5+ and Java 8.
Starting from Apache Griffin 0.7, the measure
module will be (scala-spark) cross version compatible. Since both Scala and Spark are dependencies for Apache Griffin, details of Spark-Scala cross version compatibility is mentioned below,
Spark 2.3.x | Spark 2.4.x | Spark 3.0.x | |
---|---|---|---|
Scala 2.11 | ✓ | ✓ | x |
Scala 2.12 | x | ✓ | ✓ |
Execute the below commands to build measure
with desired version of Spark and Scala,
By default, the build is compiled with Scala 2.11 and Apache Spark 2.4.x.
# For measure module with Scala 2.11 and Spark 2.4.x mvn clean package
To change Scala or Spark version you can use the commands below,
# For measure module with Scala 2.12 and Spark 2.4.x mvn clean package -Dscala-2.12
# For measure module with Scala 2.11 and Spark 2.3.x mvn clean package -Dspark-2.3
# For measure module with Scala 2.12 and Spark 3.0.x mvn clean package -Dscala-2.12 -Dspark-3.0
Note:
-Dscala-2.12
and -Dspark-2.3
option together will cause build failure due to missing dependencies as it is not cross compiled, see details here-Dspark-3.0
option without -Dscala-2.12
option will cause build failure due to missing dependencies as it is not cross compiled, see details hereStarting Spark 2.4.x, AVRO source (spark-avro
package) was migrated from com.databricks
group to org.apache.spark
. Additionally, the older dependency does not support scala 2.12.
All builds of measure
module will contain AVRO source support by default.