# Integrate into GraphScope

[GraphScope](https://graphscope.io/) is a unified distributed graph
computing platform that provides a one-stop environment for performing
diverse graph operations on a cluster through a user-friendly Python
interface. As an important application case of GraphAr, we have
integrated it into GraphScope.

GraphScope works on a graph G fragmented via a partition strategy picked
by the user and each worker maintains a fragment of G. Given a query, it
posts the same query to all the workers and computes following the BSP
(Bulk Synchronous Parallel) model. More specifically, each worker first
executes processing against its local fragment, to compute partial
answers in parallel. And then each worker may exchange partial results
with other processors via synchronous message passing.

To integrate GraphAr into GraphScope, we implemented
*ArrowFragmentBuilder* and *ArrowFragmentWriter*. *ArrowFragmentBuilder*
establishes the fragments for workers of GraphScope through reading GraphAr
format data in parallel. Conversely, *ArrowFragmentWriter* can take the
GraphScope fragments and save them as GraphAr format files. If you're interested in
knowing more about the implementation, please refer to the [source
code](https://github.com/v6d-io/v6d/commit/0eda2067e45fbb4ac46892398af0edc84fe1c27b).

## Performance Report

### Parameter settings

The time performance of *ArrowFragmentBuilder* and *ArrowFragmentWriter*
in GraphScope is heavily dependent on the partitioning of the graph into
GraphAr format files, that is, the *vertex chunk size* and *edge chunk size*, which
are specified in the vertex information file and in the edge information
file, respectively.

Generally speaking, fewer chunks are created if the file size is large.
On small graphs, this can be disadvantageous as it reduces the degree of
parallelism, prolonging disk I/O time. On the other hand, having too
many small files increases the overhead associated with the file system
and the file parser.

We have conducted micro benchmarks to compare the time performance for
reading/writing GraphAr format files by
*ArrowFragmentBuilder*/*ArrowFragmentWriter*, across different *vertex
chunk size* and *edge chunk size* configurations. The settings we
recommend for *vertex chunk size* and *edge chunk size* are **2^18** and
**2^22**, respectively, which lead to efficient performance in most
cases. These settings can be used as the reference values when
integrating GraphAr into other systems besides GraphScope.

### Time performance results

Here we report the performance results of *ArrowFragmentBuilder*, and
compare it with loading the same graph through the default loading
strategy of GraphScope (through reading the csv files in parallel) . The
execution time reported below includes loading the graph data from the
disk into memory, as well as building GraphScope fragments from such
data. The experiments are conducted on a cluster of 4 AliCloud
ecs.r6.6xlarge instances (24vCPU, 192GB memory), and using
[com-friendster](https://snap.stanford.edu/data/com-Friendster.html) (a
simple graph) and [ldbc-snb-30](https://ldbcouncil.org/benchmarks/snb/)
(a multi-labeled property graph) as datasets.

| Dataset        | Workers | Default Loading | GraphAr Loading |
|----------------|---------|-----------------|-----------------|
| com-friendster | 4       | 282s            |  54s            |
| ldbc-snb-30    | 4       | 196s            |  40s            |