[KYUUBI #6441] Kyuubi Spark TPC-DS/H Connector cross version test
# :mag: Description
## Issue References 🔗
This pull request adds cross-version tests for Kyuubi Spark TPC-DS Connector and TPC-H Connector.
## Describe Your Solution 🔧
Add TPC-DS Connector and TPC-H Connector into GitHub Actions job `spark-connector-cross-version-test`.
## Types of changes :bookmark:
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request :coffin:
#### Behavior With This Pull Request :tada:
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6441 from zhouyifan279/tcp-ds/h-cross-version.
Closes #6441
c2abc468a [zhouyifan279] Kyuubi Spark TPC-DS/H Connector cross version test
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
diff --git a/.github/workflows/master.yml b/.github/workflows/master.yml
index 6b4cb55..41394a5 100644
--- a/.github/workflows/master.yml
+++ b/.github/workflows/master.yml
@@ -190,6 +190,9 @@
comment: [ "normal" ]
env:
SPARK_LOCAL_IP: localhost
+ TEST_MODULES: "extensions/spark/kyuubi-spark-connector-hive,\
+ extensions/spark/kyuubi-spark-connector-tpcds,\
+ extensions/spark/kyuubi-spark-connector-tpch"
steps:
- uses: actions/checkout@v4
- name: Free up disk space
@@ -205,16 +208,14 @@
check-latest: false
- name: Setup Maven
uses: ./.github/actions/setup-maven
- - name: Build Kyuubi Spark Hive Connector with Spark-${{ matrix.spark-compile }}
+ - name: Build Kyuubi Spark Connectors with Spark-${{ matrix.spark-compile }}
run: |
- ./build/mvn clean install ${MVN_OPT} -pl extensions/spark/kyuubi-spark-connector-hive -am \
- -Pjava-${{ matrix.java }} -Pscala-${{ matrix.scala }} -Pspark-${{ matrix.spark-compile }} \
- -DskipTests
- - name: Test Kyuubi Spark Hive Connector with Spark-${{ matrix.spark-runtime }}
+ ./build/mvn clean install ${MVN_OPT} -pl ${TEST_MODULES} -am -Pjava-${{ matrix.java }} \
+ -Pscala-${{ matrix.scala }} -Pspark-${{ matrix.spark-compile }} -DskipTests
+ - name: Test Kyuubi Spark Connectors with Spark-${{ matrix.spark-runtime }}
run: |
- ./build/mvn test ${MVN_OPT} -pl extensions/spark/kyuubi-spark-connector-hive \
- -Pjava-${{ matrix.java }} -Pscala-${{ matrix.scala }} -Pspark-${{ matrix.spark-runtime }} \
- -Pcross-version-test
+ ./build/mvn test ${MVN_OPT} -pl ${TEST_MODULES} -Pjava-${{ matrix.java }} \
+ -Pscala-${{ matrix.scala }} -Pspark-${{ matrix.spark-runtime }} -Pcross-version-test
- name: Upload test logs
if: failure()
uses: actions/upload-artifact@v3
diff --git a/extensions/spark/kyuubi-spark-connector-tpcds/pom.xml b/extensions/spark/kyuubi-spark-connector-tpcds/pom.xml
index 53e229d..6d27d01 100644
--- a/extensions/spark/kyuubi-spark-connector-tpcds/pom.xml
+++ b/extensions/spark/kyuubi-spark-connector-tpcds/pom.xml
@@ -202,4 +202,45 @@
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
</build>
+
+ <profiles>
+ <profile>
+ <id>cross-version-test</id>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.kyuubi</groupId>
+ <artifactId>kyuubi-spark-connector-tpcds-local_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ <scope>system</scope>
+ <systemPath>${project.basedir}/target/kyuubi-spark-connector-tpcds_${scala.binary.version}-${project.version}.jar</systemPath>
+ </dependency>
+ </dependencies>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-clean-plugin</artifactId>
+ <configuration>
+ <excludeDefaultDirectories>true</excludeDefaultDirectories>
+ <filesets>
+ <fileset>
+ <directory>target/scala-${scala.binary.version}/classes</directory>
+ <includes>**/*.*</includes>
+ </fileset>
+ </filesets>
+ </configuration>
+ <executions>
+ <execution>
+ <id>clean target/scala-${scala.binary.version}/classes</id>
+ <goals>
+ <goal>clean</goal>
+ </goals>
+ <phase>process-test-classes</phase>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
</project>
diff --git a/extensions/spark/kyuubi-spark-connector-tpch/pom.xml b/extensions/spark/kyuubi-spark-connector-tpch/pom.xml
index 33f7c7e..458df4b 100644
--- a/extensions/spark/kyuubi-spark-connector-tpch/pom.xml
+++ b/extensions/spark/kyuubi-spark-connector-tpch/pom.xml
@@ -208,4 +208,45 @@
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
</build>
+
+ <profiles>
+ <profile>
+ <id>cross-version-test</id>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.kyuubi</groupId>
+ <artifactId>kyuubi-spark-connector-tpch-local_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ <scope>system</scope>
+ <systemPath>${project.basedir}/target/kyuubi-spark-connector-tpch_${scala.binary.version}-${project.version}.jar</systemPath>
+ </dependency>
+ </dependencies>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-clean-plugin</artifactId>
+ <configuration>
+ <excludeDefaultDirectories>true</excludeDefaultDirectories>
+ <filesets>
+ <fileset>
+ <directory>target/scala-${scala.binary.version}/classes</directory>
+ <includes>**/*.*</includes>
+ </fileset>
+ </filesets>
+ </configuration>
+ <executions>
+ <execution>
+ <id>clean target/scala-${scala.binary.version}/classes</id>
+ <goals>
+ <goal>clean</goal>
+ </goals>
+ <phase>process-test-classes</phase>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
</project>
