Apache Griffin Deployment Guide

For Griffin users, please follow the instructions below to deploy Griffin in your environment. Note that there are some dependencies that should be installed firstly.


You need to install following items

  • JDK (1.8 or later versions).
  • PostgreSQL(version 10.4) or MySQL(version 8.0.11).
  • npm (version 6.0.0+).
  • Hadoop (2.6.0 or later), you can get some help here.
  • Spark (version 2.2.1), if you want to install Pseudo Distributed/Single Node Cluster, you can get some help here.
  • Hive (version 2.2.0), you can get some help here. You need to make sure that your spark cluster could access your HiveContext.
  • Livy, you can get some help here. Griffin need to schedule spark jobs by server, we use livy to submit our jobs. For some issues of Livy for HiveContext, we need to download 3 files or get them from Spark lib $SPARK_HOME/lib/, and put them into HDFS.
  • ElasticSearch (5.0 or later versions). ElasticSearch works as a metrics collector, Griffin produces metrics into it, and our default UI gets metrics from it, you can use them by your own way as well.



Create database ‘quartz’ in PostgreSQL

createdb -O <username> quartz

Init quartz tables in PostgreSQL using Init_quartz_postgres.sql

psql -p <password> -h <host address> -U <username> -f Init_quartz_postgres.sql quartz


Create database ‘quartz’ in MySQL

mysql -u <username> -e "create database quartz" -p

Init quartz tables in MySQL using Init_quartz_mysql_innodb.sql.sql

mysql -u <username> -p quartz < Init_quartz_mysql_innodb.sql.sql

You should also modify some configurations of Griffin for your environment.

  • service/src/main/resources/application.properties

    # griffin server port (default 8080)
    server.port = 8080
    # jpa
    spring.datasource.url = jdbc:postgresql://<your IP>:5432/quartz?autoReconnect=true&useSSL=false
    spring.datasource.username = <user name>
    spring.datasource.password = <password>
    spring.datasource.driverClassName = org.postgresql.Driver
    spring.jpa.show-sql = true
    # hive metastore
    hive.metastore.uris = thrift://<your IP>:9083
    hive.metastore.dbname = <hive database name>    # default is "default"
    # external properties directory location, ignore it if not required
    external.config.location =
    	# login strategy, default is "default"
    	login.strategy = <default or ldap>
    	# ldap properties, ignore them if ldap is not enabled
    	ldap.url = ldap://hostname:port
    	ldap.email = @example.com
    	ldap.searchBase = DC=org,DC=example
    	ldap.searchPattern = (sAMAccountName={0})
    	# hdfs, ignore it if you do not need predicate job
    	fs.defaultFS = hdfs://<hdfs-default-name>
    	# elasticsearch
    	elasticsearch.host = <your IP>
    	elasticsearch.port = <your elasticsearch rest port>
    	# authentication properties, uncomment if basic authentication is enabled
    	# elasticsearch.user = user
    	# elasticsearch.password = password
    	# livy
    	# Port Livy: 8998 Livy2:8999
    	# yarn url
  • service/src/main/resources/sparkProperties.json

    	  "file": "hdfs:///<griffin measure path>/griffin-measure.jar",
    	  "className": "org.apache.griffin.measure.Application",
    	  "name": "griffin",
    	  "queue": "default",
    	  "numExecutors": 3,
    	  "executorCores": 1,
    	  "driverMemory": "1g",
    	  "executorMemory": "1g",
    	  "conf": {
    		"spark.yarn.dist.files": "hdfs:///<path to>/hive-site.xml"
    	  "files": [
    • <griffin measure path> is the location where you should put the jar file of measure module.

Build and Run

Build the whole project and deploy. (NPM should be installed)

mvn clean install

Put jar file of measure module into <griffin measure path> in HDFS

cp measure/target/measure-<version>-incubating-SNAPSHOT.jar measure/target/griffin-measure.jar
hdfs dfs -put measure/target/griffin-measure.jar <griffin measure path>/

After all environment services startup, we can start our server.

java -jar service/target/service.jar

After a few seconds, we can visit our default UI of Griffin (by default the port of spring boot is 8080).

http://<your IP>:8080

You can use UI following the steps here.

Note: The UI does not support all the backend features, to experience the advanced features you can use services directly.