[GRIFFIN-369] Bug fix for avro format in Spark 2.3.x environment
**What changes were proposed in this pull request?**
Built in Avro format is released in Spark 2.4.0,https://issues.apache.org/jira/browse/SPARK-24768
For Griffin, we still need to convert the Avro to com.databricks.spark.avro in Spark 2.3.x environment.
**Does this PR introduce any user-facing change?**
No.
**How was this patch tested?**
Unit Tests
Closes #598 from lipzhu/GRIFFIN-369.
Lead-authored-by: Lipeng Zhu <lipzhu@ebay.com>
Co-authored-by: Chitral Verma <chitralverma@gmail.com>
Co-authored-by: lipzhu <lipzhu@ebay.com>
Signed-off-by: chitralverma <chitralverma@gmail.com>
diff --git a/measure/src/main/scala/org/apache/griffin/measure/datasource/connector/batch/FileBasedDataConnector.scala b/measure/src/main/scala/org/apache/griffin/measure/datasource/connector/batch/FileBasedDataConnector.scala
index 2e4f482..4f75812 100644
--- a/measure/src/main/scala/org/apache/griffin/measure/datasource/connector/batch/FileBasedDataConnector.scala
+++ b/measure/src/main/scala/org/apache/griffin/measure/datasource/connector/batch/FileBasedDataConnector.scala
@@ -79,7 +79,8 @@
SupportedFormats.contains(format),
s"Invalid format '$format' specified. Must be one of ${SupportedFormats.mkString("['", "', '", "']")}")
- if (format.equalsIgnoreCase("avro") && sparkSession.version < "2.3.0") {
+ // Use old implementation for AVRO format if current spark version is not 2.4.x and above
+ if ("avro".equalsIgnoreCase(format) && sparkSession.version < "2.4.0") {
format = "com.databricks.spark.avro"
}
