| //// |
| Licensed to the Apache Software Foundation (ASF) under one or more |
| contributor license agreements. See the NOTICE file distributed with |
| this work for additional information regarding copyright ownership. |
| The ASF licenses this file to You under the Apache License, Version 2.0 |
| (the "License"); you may not use this file except in compliance with |
| the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| //// |
| [[development-environment]] |
| = Development Environment |
| |
| TinkerPop is fairly large body of code spread across many modules and covering multiple programming languages. Despite |
| this complexity, it remains relatively straightforward a project to build. This following subsections explain how to |
| configure a development environment for TinkerPop. |
| |
| image:conspiracy.png[] |
| |
| [[system-configuration]] |
| == System Configuration |
| |
| At a minimum, development of TinkerPop requires link:https://openjdk.java.net/projects/jdk8/[Java 8] but it is |
| preferable to use link:https://openjdk.java.net/projects/jdk/11/[Java 11] cross-compiled to Java 8 (the |
| cross-compilation happens automatically as part of the build). Maven (requiring a minimum of |
| link:https://maven.apache.org/download.cgi[Maven 3.5.3+]) is used as the common build system, which even |
| controls the builds of non-JVM link:https://tinkerpop.apache.org/docs/x.y.z/reference/#gremlin-drivers-variants[GLVs] |
| such as `gremlin-python`. Java and Maven are described as a "minimum" for a development environment, because they |
| will only build JVM portions of TinkerPop and many integration tests will not fire with this simple setup. It is |
| possible to get a clean and successful build with this minimum, but it will not be possible to build non-JVM aspects |
| of the project and those will go untested. |
| |
| To gain the ability to execute all aspects of the TinkerPop build system, other environmental configurations must be |
| established. Those prerequisites are defined in the following subsections. |
| |
| As of TinkerPop 3.5.5, environment configuration for Gremlin Language Variants can be optional, as Docker becomes a |
| system requirement to build and test GLVs inside of Maven. Please make sure Docker is installed and running on your system. |
| You will need to install both link:https://docs.docker.com/engine/install/[Docker Engine] and |
| link:https://docs.docker.com/compose/install/[Docker Compose], which are included in |
| link:https://docs.docker.com/desktop/[Docker Desktop]. |
| |
| IMPORTANT: Use Java 11 for documentation generation with `bin/process-docs.sh` and for other build features outside |
| of the basic `mvn clean install` sort of function. |
| |
| IMPORTANT: For those who intend to offer a contribution, building with a minimal configuration may not be sufficient |
| when submitting a pull request. Consider setting up the full environment. |
| |
| TIP: Consider using link:https://sdkman.io/[SDKMAN!] to manage Java and Maven versions and environments. |
| |
| NOTE: For those using Windows, efforts have been made to keep the build OS independent, but, in practice, it is likely |
| that TinkerPop's build system will only allow for a minimum build at best. + |
| + |
| Refer to <<building-on-windows>> section for more details. |
| |
| [[groovy-environment]] |
| === Groovy Environment |
| |
| Groovy is not used in the standard build, but when generating documentation it does require the loading of a Gremlin |
| Console instance. The Gremlin Console is Groovy-based and the documentation bootstrapping loads TinkerPop plugins |
| which requires proper configuration of Graph/Ivy dependency loaders as described in the |
| link:https://tinkerpop.apache.org/docs/x.y.z/reference/#gremlin-applications[Gremlin Applications Section] of the |
| Reference Documentation. |
| |
| The base configuration described in that link may need to be modified if there is a desire to work with the Gremlin |
| Console (for documentation generation or just general testing) in a way that utilizes SNAPSHOT releases in the |
| Apache Snapshots Repository. In that case, the `grapeConfig.xml` will need to include a resolver for that repository |
| and the basic Ivy configuration will look as follows: |
| |
| [source,xml] |
| ---- |
| <ivysettings> |
| <settings defaultResolver="downloadGrapes"/> |
| <resolvers> |
| <chain name="downloadGrapes" returnFirst="true"> |
| <filesystem name="cachedGrapes"> |
| <ivy pattern="${user.home}/.groovy/grapes/[organisation]/[module]/ivy-[revision].xml"/> |
| <artifact pattern="${user.home}/.groovy/grapes/[organisation]/[module]/[type]s/[artifact]-[revision](-[classifier]).[ext]"/> |
| </filesystem> |
| <ibiblio name="localm2" root="${user.home.url}/.m2/repository/" checkmodified="true" changingPattern=".*" changingMatcher="regexp" m2compatible="true"/> |
| <ibiblio name="jcenter" root="https://jcenter.bintray.com/" m2compatible="true"/> |
| <ibiblio name="ibiblio" m2compatible="true"/> |
| <ibiblio name="apache-snapshots" root="http://repository.apache.org/snapshots/" m2compatible="true"/> |
| </chain> |
| </resolvers> |
| </ivysettings> |
| ---- |
| |
| The above configuration is just a modification of the default. Perhaps the most lean common configuration might just |
| be: |
| |
| [source,xml] |
| ---- |
| <ivysettings> |
| <settings defaultResolver="downloadGrapes"/> |
| <resolvers> |
| <chain name="downloadGrapes"> |
| <ibiblio name="local" root="file:${user.home}/.m2/repository/" m2compatible="true"/> |
| <ibiblio name="central" root="https://repo1.maven.org/maven2/" m2compatible="true"/> |
| </chain> |
| </resolvers> |
| </ivysettings> |
| ---- |
| |
| In the above case, the configuration largely relies on the standard Maven builds to create a well cached `.m2` |
| directory. Under typical development circumstances, SNAPSHOT will find themselves deployed there locally and that |
| is all that will be required for Grape to do its work. |
| |
| As a final word, it is important to take note of the order used for these references as Grape will check them in the order |
| they are specified and depending on that order, an artifact other than the one expected may be used which is typically |
| an issue when working with SNAPSHOT dependencies. |
| |
| [[documentation-environment]] |
| === Documentation Environment |
| |
| The documentation generation process is not Maven-based and uses shell scripts to process the project's asciidoc. The |
| scripts should work on Mac and Linux. |
| |
| TIP: We recommend performing documentation generation on Linux. For the scripts to work on Mac, you will need to |
| install GNU versions of the utility programs via `homebrew`, e.g.`grep`, `awk`, `sed`, `findutils`, and `diffutils`. |
| |
| To generate documentation, it is required that link:https://hadoop.apache.org[Hadoop 3.3.x] is running in |
| link:https://hadoop.apache.org/docs/r3.3.1/hadoop-project-dist/hadoop-common/SingleCluster.html#Pseudo-Distributed_Operation[pseudo-distributed] |
| mode. Be sure to set the `HADOOP_GREMLIN_LIBS` environment variable as described in the |
| link:https://tinkerpop.apache.org/docs/x.y.z/reference/#hadoop-gremlin[reference documentation]. It is also important |
| to set the `CLASSPATH` to point at the directory containing the Hadoop configuration files, like `mapred-site.xml`. |
| |
| The `/etc/hadoop/yarn-site.xml` file prefers this configuration over the one provided in the Hadoop documentation |
| referenced above: |
| |
| [source,xml] |
| ---- |
| <configuration> |
| <property> |
| <name>yarn.nodemanager.aux-services</name> |
| <value>mapreduce_shuffle</value> |
| </property> |
| <property> |
| <name>yarn.nodemanager.vmem-check-enabled</name> |
| <value>false</value> |
| </property> |
| <property> |
| <name>yarn.nodemanager.vmem-pmem-ratio</name> |
| <value>4</value> |
| </property> |
| </configuration> |
| ---- |
| |
| The `/etc/hadoop/mapred-site.xml` file prefers the following configuration: |
| |
| [source,xml] |
| ---- |
| <configuration> |
| <property> |
| <name>mapreduce.framework.name</name> |
| <value>yarn</value> |
| </property> |
| <property> |
| <name>mapred.map.tasks</name> |
| <value>4</value> |
| </property> |
| <property> |
| <name>mapred.reduce.tasks</name> |
| <value>4</value> |
| </property> |
| <property> |
| <name>mapreduce.job.counters.limit</name> |
| <value>1000</value> |
| </property> |
| <property> |
| <name>mapreduce.jobtracker.address</name> |
| <value>localhost:9001</value> |
| </property> |
| <property> |
| <name>mapreduce.map.memory.mb</name> |
| <value>2048</value> |
| </property> |
| <property> |
| <name>mapreduce.reduce.memory.mb</name> |
| <value>4096</value> |
| </property> |
| <property> |
| <name>mapreduce.map.java.opts</name> |
| <value>-Xmx2048m</value> |
| </property> |
| <property> |
| <name>mapreduce.reduce.java.opts</name> |
| <value>-Xmx4096m</value> |
| </property> |
| </configuration> |
| ---- |
| |
| Also note that link:http://www.grymoire.com/Unix/Awk.html[awk] version `4.0.1` is required for documentation generation. |
| The link:https://tinkerpop.apache.org/docs/x.y.z/recipes/#olap-spark-yarn[YARN recipe] also uses the `zip` program to |
| create an archive so that needs to be installed, too, if you don't have it already. |
| |
| The Hadoop 3.3.x installation instructions call for installing `pdsh` but installing that seems to cause permission |
| problems when executing `sbin/start-dfs.sh`. Skipping that prerequisite seems to solve the problem. |
| |
| Documentation can be generated locally with: |
| |
| [source,text] |
| bin/process-docs.sh |
| |
| Documentation is generated to the `target/docs` directory. It is also possible to generate documentation locally with |
| Docker. `docker/build.sh -d`. |
| |
| NOTE: The installation of plugins sometimes fails in this step with the error: `Error grabbing grapes - download |
| failed`. It often helps in this case to delete the directories for the dependencies that cannot be downloaded |
| in the `.m2` (`~/.m2/`) and in the `grapes` (`~/.groovy/grapes/`) cache. E.g., if the error is about |
| `asm#asm;3.2!asm.jar`, then remove the `asm/asm` sub directory in both directories. |
| |
| To generate the web site locally, there is no need for any of the above infrastructure. Site generation is a simple |
| shell script: |
| |
| [source,text] |
| bin/generate-home.sh |
| |
| The site will be generated to the `target/site/home` directory. |
| |
| [[python-environment]] |
| === Python Environment |
| |
| As of TinkerPop 3.2.2, the build optionally requires link:https://www.python.org/[Python] to build the `gremlin-python` |
| module. If Python is not installed, TinkerPop will still build with Maven, but native Python tests and |
| Java tests that require Python code will be skipped. Developers should also install link:https://pypi.python.org/pypi/pip[pip] |
| and link:https://virtualenv.pypa.io/en/stable/[virtualenv] (version 15.0.2 - older versions may cause build failures). |
| |
| The build expects Python a `python3` installation which should be 3.5.3 or better. Python also tests kerberos and |
| therefore requires: |
| |
| [source,text] |
| sudo apt install libkrb5-dev krb5-user |
| |
| As of TinkerPop 3.5.5, `gremlin-python` uses Docker for all tests inside of Maven, and Python installation will not be required to |
| run `gremlin-python` through Maven. Please make sure Docker is installed and running on your system. |
| |
| Once the Python environment is established, the full building and testing of `gremlin-python` may commence. It |
| can be done manually from the command line with: |
| |
| [source,text] |
| mvn clean install -Pglv-python |
| |
| which enables the "glv-python" Maven profile or in a more automated fashion simply add a `.glv` file to the root of the |
| `gremlin-python` module which will signify to Maven that the environment is Python-ready. The `.glv` file need not have |
| any contents and is ignored by Git. A standard `mvn clean install` will then build `gremlin-python` in full. |
| |
| The `.glv` file in `gremlin-python` also activates the "console-integration-tests" Maven profile to run gremlin-console |
| integration tests. Alternatively, this profile can be activated manually. Note that unlike `gremlin-python` the tests |
| are actually integration tests and therefore must be actively switched on with `-DskipIntegrationTests=false`: |
| |
| [source,text] |
| mvn clean install -pl gremlin-console -DskipIntegrationTests=false |
| |
| TIP: For those who do not have a full Maven environment, please see <<docker-integration,this section>> for how Docker |
| can be used to help run tests. |
| |
| TIP: Consider use of link:https://realpython.com/intro-to-pyenv/[pyenv] to better manage Python versions and environments. |
| |
| See the <<release-environment,Release Environment>> section for more information on release manager configurations. |
| |
| [[dotnet-environment]] |
| === DotNet Environment |
| |
| The build optionally requires link:https://dotnet.microsoft.com/download[.NET SDK] (>=6.0) to work with the |
| `gremlin-dotnet` module. If .NET SDK is not installed, TinkerPop will still build with Maven, but .NET projects |
| will be skipped. |
| |
| As of TinkerPop 3.5.5, `gremlin-dotnet` uses Docker for running all test projects inside of Maven, and .NET SDK will not |
| be required to run `gremlin-dotnet` tests through Maven. Please make sure Docker is installed and running on your system. |
| |
| `gremlin-dotnet` can be built and tested from the command line with: |
| |
| [source,text] |
| mvn clean install -Pgremlin-dotnet |
| |
| which enables the "gremlin-dotnet" Maven profile or in a more automated fashion simply add a `.glv` file to the `src` |
| and `test` directories of the `gremlin-dotnet` module which will signify to Maven that the environment is .NET-ready. |
| The `.glv` file need not have any contents and is ignored by Git. A standard `mvn clean install` will then build |
| `gremlin-dotnet` in full. |
| |
| In order to pack the Gremlin.Net.Template project, it is also necessary to install link:http://www.mono-project.com/[Mono]. |
| The template can still be built and tested without Mono but packing will be skipped. |
| To pack the template (which will also download the link:https://docs.microsoft.com/en-us/nuget/tools/nuget-exe-cli-reference[NuGet CLI tool]) |
| the `nuget` property has to be set: |
| |
| [source,text] |
| mvn clean install -Dnuget |
| |
| TIP: For those who do not have a full Maven environment, please see <<docker-integration,this section>> for how Docker |
| can be used to help run tests. |
| |
| See the <<release-environment,Release Environment>> section for more information on release manager configurations. |
| |
| [[nodejs-environment]] |
| === JavaScript Environment |
| |
| When building `gremlin-javascript`, mvn command will include a local copy of Node.js runtime and npm inside your project |
| using `com.github.eirslett:frontend-maven-plugin` plugin. This copy of the Node.js runtime will not affect any |
| other existing Node.js runtime instances in your machine. |
| |
| To run the development and build scripts of `gremlint` and its corresponding web page `docs/gremlint`, Node.js and npm |
| have to be installed. When generating or publishing the TinkerPop website, the `docs/gremlint` web page has to be |
| built. Consequently, the scripts `bin/generate-home.sh` and `bin/publish-home.sh` require that Node.js and npm are |
| installed. Version 8.x or newer of npm is required. This is covered in more detail in the <<site,Site>> section. |
| |
| As of TinkerPop 3.5.5, `gremlin-javascript` uses Docker for all tests inside of Maven. Please make sure Docker is |
| installed and running on your system. |
| |
| IMPORTANT: Beware of unexpected or unwanted changes on `package-lock.json` files when committing and merging. Changes |
| should not change SHA1 hashes to SHA512 and the `lockfileVersion` should be "2" for `3.6-dev` and "1" for `3.5-dev`. |
| If the `lockfileVersion` has changed somehow then it likely means that you've used the wrong version of npm as part of |
| the build. Ensure the correct version is in use for the correct branch. |
| |
| IMPORTANT: Beware of unexpected or unwanted changes on `package-lock.json` files when committing and merging. |
| |
| TIP: For those who do not have a full Maven environment, please see <<docker-integration,this section>> for how Docker |
| can be used to help run tests. |
| |
| TIP: Consider using link:https://github.com/nvm-sh/nvm[nvm] to manage node.js versions and environments. |
| |
| See the <<release-environment,Release Environment>> section for more information on release manager configurations. |
| |
| [[go-environment]] |
| === Go Environment |
| |
| The build optionally requires link:https://go.dev/dl/[Go] (>=1.17) to work with the `gremlin-go` module. Creating an |
| empty `.glv` file will enable running of tests inside of Maven. If `.glv` file does not exist, TinkerPop |
| will still build with Maven, but Go projects will be skipped. |
| |
| `gremlin-go` can be built and tested from the command line with: |
| |
| [source,text] |
| mvn clean install -pl gremlin-go |
| |
| Alternatively, after installing Go, `gremlin-go` can be built from the command line with: |
| |
| [source,text] |
| go build |
| |
| Docker allows you to test the driver without installing any dependencies. The following command can be used to run docker: |
| |
| [source,text] |
| docker-compose up --exit-code-from gremlin-go-integration-tests |
| |
| See the <<release-environment,Release Environment>> section for more information on release manager configurations. |
| |
| [[docker-environment]] |
| === Docker Environment |
| |
| The build optionally requires Docker to build Docker images of Gremlin Server and Gremlin Console. The Docker images |
| can be built from the command line with: |
| |
| [source,text] |
| ---- |
| mvn clean install -pl gremlin-server,gremlin-console -DdockerImages |
| ---- |
| |
| which enables the "docker-images" Maven profile. |
| |
| If confronted with "Permission denied" errors on Linux, it may be necessary to do the following: |
| |
| [source,text] |
| ---- |
| sudo groupadd docker |
| sudo usermod -aG docker $USER |
| newgrp docker |
| sudo chmod 666 /var/run/docker.sock |
| ---- |
| |
| As of TinkerPop 3.5.5, a docker image of the Gremlin Server will be built automatically with `mvn clean install`, which |
| is use for GLV tests inside of Docker. To skip building this image, append the `-DskipImageBuild` flag to Maven commands. |
| |
| [[release-environment]] |
| === Release Environment |
| |
| This section is only useful to TinkerPop release managers and describes prerequisites related to deploying an official |
| release of TinkerPop. |
| |
| Maven needs to be configured to deploy maven artifacts. Apache LDAP credentials can be used for this. Release |
| managers should encrypt their Apache LDAP password as described |
| link:https://maven.apache.org/guides/mini/guide-encryption.html[in the Apache Maven docs]. |
| The encrypted password can then be configured in the `settings.xml` as described in the section |
| link:https://infra.apache.org/publishing-maven-artifacts.html["Set up your development environment" of this Apache Infra article]. |
| This configuration will be used by `mvn deploy`. |
| |
| For Python releases, uploading to pypi uses link:https://pypi.python.org/pypi/twine[twine] which is automatically |
| installed by the build process in maven. Twine refers to `HOME/.pypirc` file for configuration on the pypi deploy |
| environments and username and password combinations. The file typically looks like this: |
| |
| [source,text] |
| ---- |
| [distutils] |
| index-servers= |
| pypi |
| pypitest |
| |
| [pypitest] |
| username = <username> |
| password = |
| |
| [pypi] |
| username = <username> |
| password = |
| ---- |
| |
| The approach above assumes basic authentication and a blank password will in `.pyirc` will have Maven prompt for a |
| password. That said, it is preferred that your account be secured with 2FA, in which case, basic authentication will |
| no longer be allowed. Instead, an API token is required. It can be generated under your |
| link:https://pypi.org/manage/account/token/[account settings] and then the username in `.pyirc` is set to `__token__` |
| and the password to the token value. Please see link:https://pypi.org/help/#apitoken[PyPI documentation] for more |
| details. |
| |
| For .NET releases, install link:http://www.mono-project.com/[Mono]. The release process is known to work with 6.12.0, |
| so it is best to probably install that version. Release managers should probably also do an install of |
| link:https://dist.nuget.org/win-x86-commandline/v3.4.4/nuget.exe[nuget 3.4.4] as it will help with environmental setup. |
| To get an environment ready to deploy to NuGet, it is necessary to have a NuGet API key. First, create an account with |
| link:https://www.nuget.org[nuget] and request that a PMC member add your account to the Gremlin.Net and |
| the Gremlin.Net.Template package in nuget so that you can deploy. Next, generate an API key for your account on the |
| nuget website. The API key should be added to `NuGet.Config` with the following: |
| |
| [source,text] |
| ---- |
| mono nuget.exe setApiKey [your-api-key] |
| ---- |
| |
| This should update `~/.config/NuGet/NuGet.Config` a file with an entry containing the encrypted API key. On |
| `mvn deploy`, this file will be referenced on the automated `nuget push`. |
| |
| To deploy JavaScript / TypeScript artifacts on the link:https://www.npmjs.com[npm registry], the release manager must |
| set the authentication information on the ~/.npmrc file. The easiest way to do that is to use the `npm adduser` |
| command. This must be done only once, as the auth token doesn't have an expiration date and it's stored on your file |
| system. If this account is newly created then request that a PMC member add your account to the "gremlin" package on |
| npm. |
| |
| Deploying Docker images to link:https://hub.docker.com/[Docker Hub] requires an account that is a member of the TinkerPop |
| organization. So if you don't already have an account on Docker Hub then create one and request that |
| a PMC member adds your account to the TinkerPop organization. Afterwards, authentication information needs to be added to |
| the `~/.docker/config.json` file. This information can simply be added with the `docker login` command which will ask for |
| credentials. This must be done only once. Finally, `docker push` can be used to push images to Docker Hub which will |
| be done automatically on `mvn deploy` or it can be triggered manually with `mvn docker:push`. |
| |
| [[building-testing]] |
| == Building and Testing |
| |
| The following commands are a mix of Maven flags and shell scripts that handle different build operations |
| |
| * Build project: `mvn clean install` |
| ** Build a specific module (e.g. `gremlin-server`) within the project: `mvn clean install -pl gremlin-server` |
| ** Build without assertions for "iterator leaks" which are enabled by default: `mvn clean install -DtestIteratorLeaks=false` |
| ** Specify specific tests in a TinkerPop Suite to run with the `GREMLIN_TESTS` environment variable, along with the |
| Maven project list argument, e.g.: |
| + |
| ---- |
| export GREMLIN_TESTS='org.apache.tinkerpop.gremlin.process.traversal.step.map.PathTest$Traversals,org.apache.tinkerpop.gremlin.process.traversal.PathTest' |
| mvn -Dmaven.javadoc.skip=true --projects tinkergraph-gremlin test |
| ---- |
| ** Clean the `.groovy/grapes/org.apache.tinkerpop` directory on build: `mvn clean install -DcleanGrapes` |
| ** Turn off "heavy" logging in the "process" tests: `mvn clean install -DargLine="-DmuteTestLogs=true"` |
| ** The test suite for `neo4j-gremlin` is disabled by default - to turn it on: `mvn clean install -DincludeNeo4j` |
| * Generate <<building-testing,test resources>> for `gremlin-io-test`: `mvn clean install -pl :gremlin-io-test -Dio` |
| * Regenerate toy graph data (only necessary given changes to IO classes): `mvn clean install -Dio` from `tinkergraph-gremlin` directory |
| ** If there are changes to the Gryo format, it may be necessary to generate the Grateful Dead dataset from GraphSON (see `IoDataGenerationTest.shouldWriteGratefulDead`) |
| * Start Gremlin Server with Docker using the standard test configuration: `docker/gremlin-server.sh` |
| * Check license headers are present: `mvn apache-rat:check` |
| * Build AsciiDocs (see <<documentation-environment,Documentation Environment>>): `bin/process-docs.sh` |
| ** Build AsciiDocs (but don't evaluate code blocks): `bin/process-docs.sh --dryRun` |
| ** Build AsciiDocs (but don't evaluate code blocks in specific files): `bin/process-docs.sh --dryRun docs/src/reference/the-graph.asciidoc,docs/src/tutorial/getting-started,...` |
| ** Build AsciiDocs (but evaluate code blocks only in specific files): `bin/process-docs.sh --fullRun docs/src/reference/the-graph.asciidoc,docs/src/tutorial/getting-started,...` |
| ** Process a single AsciiDoc file: +pass:[docs/preprocessor/preprocess-file.sh `pwd`/gremlin-console/target/apache-tinkerpop-gremlin-console-*-standalone "" "*" `pwd`/docs/src/xyz.asciidoc]+ |
| * Build JavaDocs/JSDoc: `mvn process-resources -Djavadoc` |
| ** Javadoc to `target/site/apidocs` directory |
| ** JSDoc to the `gremlin-javascript/src/main/javascript/gremlin-javascript/doc/` directory |
| * Specify the seed used for `Random` in tests `mvn clean install -DtestSeed` - useful when a test fails, the seed will be printed in the build output so that the test can run with the same version of random (look for "TestHelper" logger in output) |
| * Check for newer dependencies: `mvn versions:display-dependency-updates` or `mvn versions:display-plugin-updates` |
| * Check the effective `pom.xml`: `mvn -pl gremlin-python -Pglv-python help:effective-pom -Doutput=withProfilePom.xml` |
| * Deploy JavaDocs/AsciiDocs: `bin/publish-docs.sh svn-username` |
| * Integration Tests: `mvn verify -DskipIntegrationTests=false` |
| ** Execute with the `-DincludeNeo4j` option to include transactional tests. |
| ** Execute with the `-DuseEpoll` option to try to use Netty native transport (works on Linux, but will fallback to Java NIO on other OS). |
| * Benchmarks: `mvn verify -DskipBenchmarks=false` |
| ** Reports are generated to the console and to `gremlin-tools/gremlin-benchmark/target/reports/benchmark`. |
| * Test coverage report: `mvn clean install -Dcoverage` - note that the `install` is necessary because report aggregation is bound to that part of the lifecycle. |
| ** Reports are generated to `gremlin-tools/gremlin-coverage/target/site`. |
| * `cd site` |
| ** Generate web site locally: `bin/generate-home.sh` |
| ** Publish web site: `bin/publish-home.sh <username>` |
| |
| [[building-on-windows]] |
| == Building On Windows |
| |
| The following steps must be taken in order to build TinkerPop on Windows: |
| |
| . Install winutils for Hadoop |
| * Download the latest version of link:https://github.com/kontext-tech/winutils[winutils] binaries for Hadoop. The binaries `winutils.exe` and `hadoop.dll` are required. |
| ** Note that these libraries require Microsoft Visual C++ Redistributable 2015-2022 to be installed. We've tested it on Windows 10 and Windows 11 with Microsoft Visual C++ Redistributable 2015-2022 14.32.31326. |
| * Place contents of the bin folder on your local driver in the following folder structure: |
| ** e.g. `hadoop-3.3.1/bin/winutils.exe` |
| * Set `HADOOP_HOME` to point to the `hadoop-3.3.1` folder |
| * Add `%HADOOP_HOME%\bin` to your `PATH` |
| . Run `mvn clean install` from root of tinkerpop |
| . Follow IDE specific steps if applicable: |
| * <<intellij>> |
| |
| You should now be able to work with TinkerPop on Windows. |
| |
| [[docker-integration]] |
| == Docker Integration |
| |
| TinkerPop provides a shell script, that can start several build tasks within a Docker container. The |
| required Docker images will be built automatically if they don't exist yet. Thus the first invocation |
| of the Docker script is expected to take some time. |
| |
| The script can be found under `PROJECT_HOME/docker/build.sh`. The following tasks are currently |
| supported: |
| |
| * run standard test suite |
| * run integration tests |
| * build Java docs |
| * build user docs |
| |
| A list of command line options is provided by `docker/build.sh --help`. The container will install, |
| configure and start all required dependencies, such as Hadoop. |
| |
| By default, this script will run every module in the project. However, if you are planning on working on just a |
| small set of the modules (e.g. the GLVs) then you can use the script options to reduce the modules included by |
| specifically selecting which modules you want. This behavior is currently supported for the non-Java GLVs and |
| gremlin-console. This option will include only the selected modules as well as gremlin-server, gremlin-test, |
| neo4j-gremlin and all their dependencies. This is the minimum set of modules required to build and test the GLVs. |
| |
| [source,bash] |
| .Build and test gremlin-python and gremlin-go with minimal Gremlin Server dependencies |
| ---- |
| ./docker/build.sh --tests --integration-tests --python --golang |
| ---- |
| |
| Options can be passed to Docker by setting the `TINKERPOP_DOCKER_OPTS` environment variable. A speed boost can |
| be gained at the expense of memory by using tmpfs and the special directory `/usr/src/tinkermem`. |
| |
| [source,bash] |
| .Build in-memory |
| ---- |
| TINKERPOP_DOCKER_OPTS="--tmpfs /usr/src/tinkermem:exec,mode=0755,rw,noatime,size=2000m" |
| ---- |
| |
| [source,bash] |
| .Disable IPv6 for Hadoop |
| ---- |
| TINKERPOP_DOCKER_OPTS="--sysctl net.ipv6.conf.all.disable_ipv6=1 --sysctl net.ipv6.conf.default.disable_ipv6=1" |
| ---- |
| |
| A custom maven settings.xml can be supplied, for example, to point to a local proxy. Copy the `settings.xml` to the |
| `PROJECT_HOME/` directory. The Docker script will detect and copy it to the running container. |
| |
| If the container is used to generate the user docs, it will start a web server and show the URL that |
| is used to host the HTML docs. |
| |
| After finishing all tasks, the script will immediately destroy the container. |
| |
| Docker can also be helpful to developers who do not want to run tests from a Maven environment, which may be a bit |
| opaque when dealing with test failures and largely unhelpful for debugging. This situation is typically case for |
| developers doing work on Gremlin Language Variants (e.g. Python). To help alleviate this problem, developers can |
| start a standalone Gremlin Server with its standard test configuration that is used in the standard Maven build. |
| |
| Generally speaking, most developers will want to test their code against the latest build of Gremlin Server in the |
| TinkerPop repository. To do that, first be sure to build a Docker image of the current code: |
| |
| [source,bash] |
| mvn clean install -DskipTests |
| |
| Next, generate the a Docker image for Gremlin Server with: |
| |
| [source,bash] |
| mvn clean install -pl :gremlin-server -DdockerImages -DskipTests |
| |
| IMPORTANT: If changes are made to the repository that need to be reflected in the Gremlin Server Docker image then |
| the old image should be removed and then the above commands re-executed. |
| |
| Finally, start the server with: |
| |
| [source,bash] |
| docker/gremlin-server.sh |
| |
| Starting Gremlin Server this way makes it possible to run Gremlin Language Variant tests without Maven (for example, |
| directly from a debugger) which should greatly reduce development friction for these environments. |
| |
| It is also possible to specify the exact version of Gremlin Server to run with the test configuration. This version |
| should be an existing Docker image version and must be an explicit version that maps to an actual TinkerPop artifact: |
| |
| [source,bash] |
| docker/gremlin-server.sh 3.4.2 |
| |
| To be a bit more clear, the version can not be a Docker tag like "latest" because there is no such TinkerPop artifact |
| that has been published with that version number. |
| |
| [[docker-testing]] |
| == Testing Sub-Modules with Docker |
| |
| Currently the modules gremlin-go, gremlin-javascript, gremlin-dotnet, gremlin-python and gremlin-console can be tested |
| through Docker. |
| |
| Please make sure Docker is installed and running on your system. You will need to install both |
| link:https://docs.docker.com/engine/install/[Docker Engine] and link:https://docs.docker.com/compose/install/[Docker Compose], |
| which are included in link:https://docs.docker.com/desktop/[Docker Desktop]. |
| |
| The following environment variables used by Docker Compose will automatically be set when running through Maven. |
| |
| The docker compose environment variable `GREMLIN_SERVER` specifies the Gremlin server docker image to use, i.e. an |
| image with the tag `tinkerpop/gremlin-server:$GREMLIN_SERVER`, and is a required environment variable. This also |
| requires the specified docker image to exist, either locally or in link:https://hub.docker.com/r/tinkerpop/gremlin-server[Docker Hub]. |
| |
| Running `mvn clean install -pl gremlin-server -DskipTests -DskipIntegrationTests=true -Dci -am` in the main `tinkerpop` |
| directory will automatically build a local SNAPSHOT Gremlin server image. If your OS Platform cannot build a local |
| SNAPSHOT Gremlin server through `maven`, it is recommended to use the latest released server version from |
| link:https://hub.docker.com/r/tinkerpop/gremlin-server[Docker Hub] (do not use `GREMLIN_SERVER=latest`, use actual |
| version number, e.g. `GREMLIN_SERVER=3.5.x` or `GREMLIN_SERVER=3.6.x`). |
| |
| The docker compose environment variable `HOME` specifies the user home directory for mounting volumes during test image |
| set up. This variable is set by default in Unix/Linux, but will need to be set for Windows, for example, run |
| `$env:HOME=$env:USERPROFILE` in PowerShell. |
| |
| There are different ways to launch the test suite and set the `GREMLIN_SERVER` environment variable depending on |
| your Platform: |
| |
| * Run Maven commands, e.g. `mvn clean install` inside of project folder e.g. `tinkerpop/gremlin-go`, or `mvn clean install -pl gremlin-go` |
| inside of `tinkerpop` (platform-agnostic - recommended) |
| * Add `GREMLIN_SERVER=<server-image-version>` and `HOME=<user-home-directory>` to an `.env` file inside project folder and run `docker-compose up --exit-code-from gremlin-go-integration-tests` (Platform-agnostic). |
| * Run `GREMLIN_SERVER=<server-image-version> docker-compose up --exit-code-from gremlin-go-integration-tests` in Unix/Linux. |
| * Run `$env:GREMLIN_SERVER="<server-image-version>";$env:HOME=$env:USERPROFILE;docker-compose up --exit-code-from gremlin-go-integration-tests` in Windows PowerShell. |
| |
| You should see exit code 0 upon successful completion of the test suites. Run `docker-compose down` to remove the |
| service containers (not needed if you executed Maven commands or `run.sh`), or `docker-compose down --rmi all` to |
| remove the service containers while deleting all used images. |
| |
| Note for running docker with MacOS on ARM processors: Docker's performance is extremely poor on ARM Mac's in its |
| default configuration. It is recommended to enable both the "New Virtualization Framework" and "VirtioFS" under |
| Docker Desktop Settings -> Experimental Features. |
| |
| [[intellij]] |
| == Intellij Usage |
| |
| Most core TinkerPop developers are using Intellij for their work so this section helps describe the mechanisms for |
| best working with it as an IDE. |
| |
| === Setup |
| |
| Installation and basic configuration of Intellij is beyond the scope of this writing and it is assumed that the |
| TinkerPop GitHub repository has been cloned and the root of the repository is open in Intellij. From there, we can |
| begin to look at configuration options specifically relevant to TinkerPop itself. |
| |
| TinkerPop has a module called `gremlin-shaded` which contains shaded dependencies for some libraries that are widely |
| used and tend to introduce conflicts. To ensure that Intellij properly interprets this module after importing the |
| Maven `pom.xml` perform the following steps: |
| |
| . Build `gremlin-shaded` from the command line with `mvn clean install`. |
| . Right-click on the `gremlin-shaded` module in the project viewer of Intellij and select "Remove module". If this menu |
| option is not available (as is the case in newer versions of Intellij - first noticed in 13.1.5), then open the "Maven |
| Projects" side panel, right click the `gremlin-shaded` module and select "Ignore Project". |
| . In the "Maven Projects" Tool window and click the tool button for "Reimport All Maven projects" (go to |
| `View | Tool Windows | Maven Projects` on the main menu if this panel is not activated). |
| . At this point it should be possible to compile and run the tests within Intellij, but in the worst case, use |
| `File | Invalidate Caches/Restart` to ensure that indices properly rebuild. |
| |
| Note that it may be necessary to re-execute these steps if the `gremlin-shaded` `pom.xml` is ever updated. |
| |
| You will initially see lots of errors related to the Gremlin lexer/parser. The `gremlin-language` module requires |
| ANTLR processing. While this processing is configured to execute with Maven, it can also be setup to generate parser |
| files within Intellij itself on command: |
| |
| . Install the ANTLR4 Grammar Plugin for Intellij |
| . Right-click on the `Gremlin.g4` file and "Configure ANTLR" |
| . Set "Output directory where all output is generated" to `target/generated-sources/antlr4` |
| . Set "Grammar file encoding" to `utf-8` |
| . Set "Package/namespace for the generated code" to `org.apache.tinkerpop.gremlin.language.grammar` |
| . Set "Language" to `Java` |
| . Set "Case transformation in the Preview window" should be "Leave as-is" |
| . The "generate parse tree listener" should be unchecked and the "generate parse tree visitor" should be checked. |
| |
| With these settings it should be possible to right-click `Gremlin.g4` and "Generate ANTLR Recognizer" which will place |
| the generated code in where specified at `target/generated-sources/antlr4`. Be sure to right-click the `antlr4` |
| directory and "Mark directory as" "Generated Sources Root" which should allow Intellij to recognize it. |
| |
| The `gremlin-groovy` module uses a Java annotation processor to help support DSLs. Annotation processing in Intellij |
| should be set up by the Maven import, but if this is set up incorrectly you will see a cryptic error message when |
| building: `java: Compilation failed: internal java compiler error`. To fix this, search for the Intellij setting |
| `Annotation Processors`. Make sure that annotation processing is enabled for `gremlin-groovy` but disabled for |
| `gremlin-annotations`. This should fix the internal java compile error. Next you may see compile errors complaining |
| that the `jsr223` classes `__`, ``CredentialTraversal`, `CredentialTraversalSource`, and `DefaultCredentialTraversal` |
| cannot be found. Those classes are generated by annotations. To fix these errors, make sure mark the |
| directory `gremlin-groovy/target/generated-sources/annotations` as "Generated Sources Root". |
| |
| Developers working on the `neo4j-gremlin` module should enabled the `include-neo4j` Maven profile in Intellij. |
| This will ensure that tests will properly execute within the IDE. |
| |
| === Debugging |
| |
| It is generally assumed that JVM-based debugging of TinkerPop code in Intellij is a relatively straightforward task |
| for most developers, but it is worth pointing out a few important points related to it and to drill into some specifics |
| for the non-JVM languages. |
| |
| [[debug-java]] |
| ==== Java |
| |
| There are generally no complexities to running the debugger for any JVM-based test in the repository, but the following |
| tips are helpful to know when doing so: |
| |
| * It is not possible to run the tests in `gremlin-test` without an `Graph` implementation. For example, it is not |
| possible to just right-click `org.apache.tinkerpop.gremlin.process.traversal.step.filter.CoinTest` and then select |
| `Debug 'CoinTest'`. Instead, running that test would require opening `tinkergraph-gremlin` and executing the |
| `TinkerGraphProcessStandardTest` (as an example) which runs the full Gremlin test suite to include `CoinTest`. |
| * To run just `CoinTest`, set a `GREMLIN_TESTS` environment variable with the fully qualified path name to the test |
| configuration in Intellij. Note that the fully qualified test name is really |
| `org.apache.tinkerpop.gremlin.process.traversal.step.filter.CoinTest$Traversals`. |
| * Gherkin tests have a similar pattern in that they require a `Graph` implementation to execute them. Therefore, |
| debugging entails going to `tinkergraph-gremlin` and running `TinkerGraphFeatureTest` in the debugger. |
| * It is possible to filter the Gherkin tests by adding a system property to the debug configuration that specifies the |
| tags to use or ignore. For example to just run the `coin()` tests: `-Dcucumber.filter.tags="@StepCoin"` |
| |
| [[debug-python]] |
| ==== Python |
| |
| Debugging Python within this mainly JVM-based project structure requires a bit of configuration. The following steps |
| will help get Intellij prepared for this task: |
| |
| 1. Install the Python plugin from JetBrains which should provide PyCharm like functionality. |
| 1. Right-click the "tinkerpop" top-level module in Intellij's project explorer and "Open Module Settings". |
| 1. Select "Platform Settings | SDKs" and then click the "+" to "Add Python SDK..." |
| 1. Choose a "Virtualenv Environment" and a "New environment". Set the "Location" to |
| "<project-root>/gremlin-python/src/main/python". Select a "Base interpreter" that matches the version required by |
| `gremlin-python` and click "OK". |
| 1. Select "Project Settings | Modules" and then select "gremlin-python" from the listing. Change the "Module SDK" to |
| the newly added Python SDK. |
| 1. Open a terminal to `gremlin-python/scr/main/python` and do `venv/bin/pip3 install -e .` to pull in all of the |
| `gremlin-python` dependencies. |
| |
| At this stage, it should be possible to run unit tests in Python: |
| |
| 1. Pull down the drop down in the toolbar for "Run/Debug Configurations" and select "Edit Configurations..." |
| 1. Click the "+" to "Python Tests | pytest" and browse to a test to supply the "Script path". |
| 1. Go to "Python interpreter" and select "Use SDK of module" and choose "gremlin-python" in the drop-down and click OK. |
| 1. The test should not appear in the "Run/Debug Configurations" and can be executed. |
| |
| NOTE: When the Debug button is pressed, Intellij may display a notification that using the debugger requires some |
| additional downloads - confirm those installations as required. |
| |
| Many of the tests in `gremlin-python` require Gremlin Server. They are effectively integration tests. TinkerPop makes |
| it easy to debug these tests by providing a Docker based test server which is rigged up with all the configurations |
| required for the tests to execute. Start this server with `docker/gremlin-server.sh -n` where the `-n` will enable |
| Neo4j for transaction based tests. |