The HugeGraph-Computer
is a distributed graph processing system for HugeGraph (OLAP). It is an implementation of Pregel. It runs on Kubernetes framework.
To run algorithm with HugeGraph-Computer, you need to install 64-bit Java 11 or later versions.
You also need to deploy HugeGraph-Server and Etcd.
There are two ways to get HugeGraph-Computer:
Download the latest version of the HugeGraph-Computer release package:
wget https://github.com/apache/hugegraph-computer/releases/download/v${version}/hugegraph-loader-${version}.tar.gz tar zxvf hugegraph-computer-${version}.tar.gz
Clone the latest version of HugeGraph-Computer source package:
$ git clone https://github.com/apache/hugegraph-computer.git
Compile and generate tar package:
cd hugegraph-computer mvn clean package -DskipTests
You can use
-c
parameter specify the configuration file, more computer config please see:Computer Config Options
cd hugegraph-computer-${version} bin/start-computer.sh -d local -r master
bin/start-computer.sh -d local -r worker
2.5.1 Enable OLAP
index query for server
If OLAP index is not enabled, it needs to enable, more reference: modify-graphs-read-mode
PUT http://localhost:8080/graphs/hugegraph/graph_read_mode "ALL"
2.5.2 Query page_rank
property value:
curl "http://localhost:8080/graphs/hugegraph/graph/vertices?page&limit=3" | gunzip
To run algorithm with HugeGraph-Computer you need to deploy HugeGraph-Server first
# Kubernetes version >= v1.16 kubectl apply -f https://raw.githubusercontent.com/apache/hugegraph-computer/master/computer-k8s-operator/manifest/hugegraph-computer-crd.v1.yaml # Kubernetes version < v1.16 kubectl apply -f https://raw.githubusercontent.com/apache/hugegraph-computer/master/computer-k8s-operator/manifest/hugegraph-computer-crd.v1beta1.yaml
kubectl get crd NAME CREATED AT hugegraphcomputerjobs.hugegraph.apache.org 2021-09-16T08:01:08Z
kubectl apply -f https://raw.githubusercontent.com/apache/hugegraph-computer/master/computer-k8s-operator/manifest/hugegraph-computer-operator.yaml
kubectl get pod -n hugegraph-computer-operator-system NAME READY STATUS RESTARTS AGE hugegraph-computer-operator-controller-manager-58c5545949-jqvzl 1/1 Running 0 15h hugegraph-computer-operator-etcd-28lm67jxk5 1/1 Running 0 15h
More computer crd please see: Computer CRD
More computer config please see: Computer Config Options
cat <<EOF | kubectl apply --filename - apiVersion: hugegraph.apache.org/v1 kind: HugeGraphComputerJob metadata: namespace: hugegraph-computer-system name: &jobName pagerank-sample spec: jobId: *jobName algorithmName: page_rank image: hugegraph/hugegraph-computer:latest # algorithm image url jarFile: /hugegraph/hugegraph-computer/algorithm/builtin-algorithm.jar # algorithm jar path pullPolicy: Always workerCpu: "4" workerMemory: "4Gi" workerInstances: 5 computerConf: job.partitions_count: "20" algorithm.params_class: org.apache.hugegraph.computer.algorithm.centrality.pagerank.PageRankParams hugegraph.url: http://${hugegraph-server-host}:${hugegraph-server-port} # hugegraph server url hugegraph.name: hugegraph # hugegraph graph name EOF
kubectl get hcjob/pagerank-sample -n hugegraph-computer-system NAME JOBID JOBSTATUS pagerank-sample pagerank-sample RUNNING
# Show the master log kubectl logs -l component=pagerank-sample-master -n hugegraph-computer-system # Show the worker log kubectl logs -l component=pagerank-sample-worker -n hugegraph-computer-system # Show diagnostic log of a job # NOTE: diagnostic log exist only when the job fails, and it will only be saved for one hour. kubectl get event --field-selector reason=ComputerJobFailed --field-selector involvedObject.name=pagerank-sample -n hugegraph-computer-system
NOTE: it will only be saved for one hour
kubectl get event --field-selector reason=ComputerJobSucceed --field-selector involvedObject.name=pagerank-sample -n hugegraph-computer-system
If the output to Hugegraph-Server
is consistent with Locally, if output to HDFS
, please check the result file in the directory of /hugegraph-computer/results/{jobId}
directory.
More algorithms please see: Built-In algorithms
TODO
TODO
mvn compile
in advance to generate corresponding classes.