For Griffin developers, you need to build an entire environment for development, it wastes time. You can directly use the Griffin docker image, to the functional test during your development. Please follow this griffin docker guide.
Follow this griffin docker guide.
Usually, you can run the docker image in batch mode(always the first choice) or streaming mode(only if you need a streaming environment like kafka data source), choose a mode to start. For Batch mode and Streaming mode, follow step 1 and 2, you can start the docker container.
If you need to develop the service module, you need to modify some configuration in the following files. Docker host is your machine running the docker containers, which means if you install docker and run docker containers on 192.168.100.100, then the <docker host ip>
is 192.168.100.100.
In service/src/main/resources/application.properties
spring.datasource.url = jdbc:postgresql://<docker host ip>:35432/quartz?autoReconnect=true&useSSL=false hive.metastore.uris = thrift://<docker host ip>:39083 elasticsearch.host = <docker host ip> elasticsearch.port = 39200
In service/src/main/resources/sparkJob.properties
livy.uri=http://<docker host ip>:38998/batches spark.uri=http://<docker host ip>:38088
Now you can start the service module in your local IDE, by running or debugging org.apache.griffin.core.GriffinWebApplication.
If you need to develop the ui module only, you need to modify some configuration.
In ui/angular/src/app/service/service.service.ts
// public BACKEND_SERVER = ""; public BACKEND_SERVER = 'http://<docker host ip>:38080';
After this, you can test your ui module by using remote service.
However, in most conditions, you need to develop the ui module with some modification in service module. Then you need to follow the steps above for service module first, and In ui/angular/src/app/service/service.service.ts
// public BACKEND_SERVER = ""; public BACKEND_SERVER = 'http://localhost:8080';
After this, you can start service module at local, and test your ui module by using local service.
If you need to develop the measure module only, you can ignore any of the service or ui module. You can test your built measure JAR in the docker container, using the existed spark environment.
For debug phase, you'd better install hadoop, spark, hive at local, and test your program at local for fast.
First, in the incubator-griffin directory, build you packages at once.
mvn clean install
docker exec -it <griffin docker container id> bash cd ~/service ps -ef | grep service.jar kill -9 <pid of service.jar>
service/target/service-<version>.jar
, copy it into your docker container.docker cp service-<version>.jar <griffin docker container id>:/root/service/service.jar
cd ~/service nohup java -jar service.jar > service.log &
Now you can follow the service log by tail -f service.log
.
measure/target/measure-<version>.jar
, copy it into your docker container.docker cp measure-<version>.jar <griffin docker container id>:/root/measure/griffin-measure.jar
docker exec -it <griffin docker container id> bash hadoop fs -rm /griffin/griffin-measure.jar hadoop fs -put /root/measure/griffin-measure.jar /griffin/griffin-measure.jar
Now the griffin service will submit jobs by using this new griffin-measure.jar.
For end2end test, you will need to build a new griffin docker image, for more convenient test.
git clone https://github.com/bhlx3lyx7/griffin-docker.git
cp service-<version>.jar <path to>/griffin-docker/griffin_spark2/prep/service/service.jar cp measure-<version>.jar <path to>/griffin-docker/griffin_spark2/prep/measure/griffin-measure.jar
cd <path to>/griffin-docker/griffin_spark2 docker build -t <image name>[:<image version>] .
griffin: image: <image name>[:<image version>]
docker-compose -f <docker-compose file> up -d