We have pre-built Apache Griffin docker images for Apache Griffin developers. You can use those images directly, which set up a ready development environment for you much faster than building the environment locally.
Here are step-by-step instructions of how to pull Docker images from the repository and run containers using the images.
If you need to develop the service module, you need to modify some configuration in the following files. Docker host is your machine running the docker containers, which means if you install docker and run docker containers on 192.168.100.100, then the <docker host ip>
is 192.168.100.100.
In service/src/main/resources/application.properties
spring.datasource.url = jdbc:postgresql://<docker host ip>:35432/quartz?autoReconnect=true&useSSL=false hive.metastore.uris = thrift://<docker host ip>:39083 elasticsearch.host = <docker host ip> elasticsearch.port = 39200
In service/src/main/resources/sparkJob.properties
livy.uri=http://<docker host ip>:38998/batches yarn.uri=http://<docker host ip>:38088
Now you can start the service module in your local IDE, running or debugging org.apache.griffin.core.GriffinWebApplication.
If you only wanna develop the UI module, you just need to modify some configuration.
In ui/angular/src/app/service/service.service.ts
// public BACKEND_SERVER = ""; public BACKEND_SERVER = 'http://<docker host ip>:38080';
Making the change above, you can test your UI module by using remote service.
However, on the most of conditions, you need to develop the UI module with some modification in service module. Then you need to follow the steps above for service module first, and in ui/angular/src/app/service/service.service.ts
// public BACKEND_SERVER = ""; public BACKEND_SERVER = 'http://localhost:8080';
Making the change, you can start service module locally, and test your UI module using local service.
After that, you can run local server using following command:
cd ui/angular ../angular/node_modules/.bin/ng serve --port 8080
If you only wanna develop the measure module, you can ignore both of service or UI module. You can test your measure JAR built in the docker container, using the existed spark environment.
For debug purpose, you'd better install hadoop, spark, hive locally, so you can test your program more quickly.
Note: If you run Hadoop in a pseudo-distributed mode on MacOS you have to update hdfs-site.xml, namely, comment out the parameters dfs.namenode.servicerpc-address and dfs.namenode.rpc-address, otherwise you can get the error like: “java.net.ConnectException: Call From mycomputer/127.0.1.1 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused”
Firstly, in the griffin directory, build your packages at once.
mvn clean install
docker exec -it <griffin docker container id> bash cd ~/service ps -ef | grep service.jar kill -9 <pid of service.jar>
service/target/service-<version>.jar
, copy it into your docker container.docker cp service-<version>.jar <griffin docker container id>:/root/service/service.jar
cd ~/service nohup java -jar service.jar > service.log &
Now you can follow the service log by tail -f service.log
.
measure/target/measure-<version>.jar
, copy it into your docker container.docker cp measure-<version>.jar <griffin docker container id>:/root/measure/griffin-measure.jar
docker exec -it <griffin docker container id> bash hadoop fs -rm /griffin/griffin-measure.jar hadoop fs -put /root/measure/griffin-measure.jar /griffin/griffin-measure.jar
Now the Apache Griffin service will submit jobs by using this new griffin-measure.jar.
For end2end test, you will need to build a new Apache Griffin docker image, for more convenient test.
git clone https://github.com/bhlx3lyx7/griffin-docker.git
cp service-<version>.jar <path to>/griffin-docker/griffin_spark2/prep/service/service.jar cp measure-<version>.jar <path to>/griffin-docker/griffin_spark2/prep/measure/griffin-measure.jar
cd <path to>/griffin-docker/griffin_spark2 docker build -t <image name>[:<image version>] .
griffin: image: <image name>[:<image version>]
docker-compose -f <docker-compose file> up -d