Merge pull request #1892 from apache/TINKERPOP-2816-3.5
TINKERPOP-2816 for 3.5
diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
index 43cce0b..8bd6826 100644
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@ -30,7 +30,9 @@
* Fixed issue where the `GremlinGroovyScriptEngine` reused the same translator concurrently which lead to incorrect translations.
* Fixed bug where tasks that haven't started running yet time out due to `evaluationTimeout` and never send a response back to the client.
* Set the exact exception in `initializationFailure` on the Java driver instead of the root cause.
+* Improved error message for when `from()` and `to()` are unproductive for `addE()`.
* Added `SparkIOUtil` utility to load graph into Spark RDD.
+* Improved performance of `CloneVertexProgram` by bypassing the shuffle state of `SparkGraphComputer`.
* Changed `JavaTranslator` exception handling so that an `IllegalArgumentException` is used for cases where the method exists but the signature can't be discerned given the arguments supplied.
* Dockerized all test environment for .NET, JavaScript, Python, Go, and Python-based tests for Console, and added Docker as a build requirement.
* Async operations in .NET can now be cancelled. This however does not cancel work that is already happening on the server.
@@ -39,13 +41,15 @@
* Added user agent to web socket handshake in Gremlin.Net driver. Can be controlled by `EnableUserAgentOnConnect` in `ConnectionPoolSettings`. It is enabled by default.
* Added user agent to web socket handshake in go driver. Can be controlled by a new `EnableUserAgentOnConnect` setting. It is enabled by default.
* Added user agent to web socket handshake in python driver. Can be controlled by a new `enable_user_agent_on_connect` setting. It is enabled by default.
+* Added user agent to web socket handshake in javascript driver. Can be controlled by a new `enableUserAgentOnConnect` option. It is enabled by default.
* Added logging in .NET.
* Added `addDefaultXModule` to `GraphSONMapper` as a shortcut for including a version matched GraphSON extension module.
* Modified `GraphSONRecordReader` and `GraphSONRecordWriter` to include the GraphSON extension module by default.
* Bumped `jackson-databind` to 2.14.0 to fix security vulnerability.
-* Bumped to Groovy 2.5.19.
+* Bumped to Groovy 2.5.15.
* Bumped to Netty 4.1.85.
* Bumped `ivy` to 2.5.1 to fix security vulnerability
+* Removed default SSL handshake timeout. The SSL handshake timeout will instead be capped by setting `connectionSetupTimeoutMillis`.
==== Bugs
diff --git a/docs/src/dev/developer/for-committers.asciidoc b/docs/src/dev/developer/for-committers.asciidoc
index af2cbe0..627175d 100644
--- a/docs/src/dev/developer/for-committers.asciidoc
+++ b/docs/src/dev/developer/for-committers.asciidoc
@@ -162,7 +162,8 @@
affect users or providers. This label is important when organizing release notes.
** The "deprecation" label which is assigned to an issue that includes changes to deprecate a portion of the API.
* The "affects/fix version(s)" fields should be appropriately set, where the "fix version" implies the version on
-which that particular issue will completed. This is a field usually only set by committers.
+which that particular issue will completed. This is a field usually only set by committers and should only be set
+when the issue is being closed with a completed disposition (e.g. "Done", "Fixed", etc.).
* The "priority" field can be arbitrarily applied with one exception. The "trivial" option should be reserved for
tasks that are "easy" for a potential new contributor to jump into and do not have significant impact to urgently
required improvements.
diff --git a/docs/src/reference/gremlin-variants.asciidoc b/docs/src/reference/gremlin-variants.asciidoc
index e445417..aec7f11 100644
--- a/docs/src/reference/gremlin-variants.asciidoc
+++ b/docs/src/reference/gremlin-variants.asciidoc
@@ -782,6 +782,34 @@
g.V().hasLabel('person').has('age',gt(30)).order().by('age',desc).toList()
----
+[[gremlin-javascript-configuration]]
+=== Configuration
+The following table describes the various configuration options for the Gremlin-Javascript Driver. They
+can be passed in the constructor of a new `Client` or `DriverRemoteConnection` :
+
+[width="100%",cols="3,3,10,^2",options="header"]
+|=========================================================
+|Key |Type |Description |Default
+|url |String |The resource uri. |None
+|options |Object |The connection options. |{}
+|options.ca |Array |Trusted certificates. |undefined
+|options.cert |String/Array/Buffer |The certificate key. |undefined
+|options.mimeType |String |The mime type to use. |'application/vnd.gremlin-v3.0+json'
+|options.pfx |String/Buffer |The private key, certificate, and CA certs. |undefined
+|options.reader |GraphSONReader/GraphBinaryReader |The reader to use. |select reader according to mimeType
+|options.writer |GraphSONWriter |The writer to use. |select writer according to mimeType
+|options.rejectUnauthorized |Boolean |Determines whether to verify or not the server certificate. |undefined
+|options.traversalSource |String |The traversal source. |'g'
+|options.authenticator |Authenticator |The authentication handler to use. |undefined
+|options.processor |String |The name of the opProcessor to use, leave it undefined or set 'session' when session mode. |undefined
+|options.session |String |The sessionId of Client in session mode. undefined means session-less Client. |undefined
+|options.enableUserAgentOnConnect |Boolean |Determines if a user agent will be sent during connection handshake. |true
+|options.headers |Object |An associative array containing the additional header key/values for the initial request. |undefined
+|options.pingEnabled |Boolean |Setup ping interval. |true
+|options.pingInterval |Number |Ping request interval in ms if ping enabled. |60000
+|options.pongTimeout |Number |Timeout of pong response in ms after sending a ping. |30000
+|=========================================================
+
[[gremlin-javascript-transactions]]
=== Transactions
diff --git a/gremlin-console/src/main/static/NOTICE b/gremlin-console/src/main/static/NOTICE
index 28800e7..c9f1534 100644
--- a/gremlin-console/src/main/static/NOTICE
+++ b/gremlin-console/src/main/static/NOTICE
@@ -18,7 +18,7 @@
Copyright (c) 2008 Alexander Beider & Stephen P. Morse.
------------------------------------------------------------------------
-Apache Groovy 2.5.19 (AL ASF)
+Apache Groovy 2.5.15 (AL ASF)
------------------------------------------------------------------------
This product includes/uses ANTLR (http://www.antlr2.org/)
developed by Terence Parr 1989-2006
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/AddEdgeStep.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/AddEdgeStep.java
index 1bf5d4a..4d3d864 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/AddEdgeStep.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/AddEdgeStep.java
@@ -97,13 +97,30 @@
@Override
protected Edge map(final Traverser.Admin<S> traverser) {
final String edgeLabel = this.parameters.get(traverser, T.label, () -> Edge.DEFAULT_LABEL).get(0);
- final Object theTo = this.parameters.get(traverser, TO, traverser::get).get(0);
+
+ final Object theTo;
+ try {
+ theTo = this.parameters.get(traverser, TO, traverser::get).get(0);
+ } catch (IllegalArgumentException e) { // as thrown by TraversalUtil.apply()
+ throw new IllegalStateException(String.format(
+ "addE(%s) failed because the to() traversal (which should give a Vertex) failed with: %s",
+ edgeLabel, e.getMessage()));
+ }
+
if (!(theTo instanceof Vertex))
throw new IllegalStateException(String.format(
"addE(%s) could not find a Vertex for to() - encountered: %s", edgeLabel,
null == theTo ? "null" : theTo.getClass().getSimpleName()));
- final Object theFrom = this.parameters.get(traverser, FROM, traverser::get).get(0);
+ final Object theFrom;
+ try {
+ theFrom = this.parameters.get(traverser, FROM, traverser::get).get(0);
+ } catch (IllegalArgumentException e) { // as thrown by TraversalUtil.apply()
+ throw new IllegalStateException(String.format(
+ "addE(%s) failed because the from() traversal (which should give a Vertex) failed with: %s",
+ edgeLabel, e.getMessage()));
+ }
+
if (!(theFrom instanceof Vertex))
throw new IllegalStateException(String.format(
"addE(%s) could not find a Vertex for from() - encountered: %s", edgeLabel,
diff --git a/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/Channelizer.java b/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/Channelizer.java
index 67386f6..efe47c5 100644
--- a/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/Channelizer.java
+++ b/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/Channelizer.java
@@ -35,6 +35,7 @@
import io.netty.handler.codec.http.websocketx.WebSocketVersion;
import io.netty.handler.codec.http.websocketx.extensions.compression.WebSocketClientCompressionHandler;
import io.netty.handler.ssl.SslContext;
+import io.netty.handler.ssl.SslHandler;
import io.netty.handler.timeout.IdleStateHandler;
import org.slf4j.Logger;
@@ -86,6 +87,7 @@
protected static final String PIPELINE_GREMLIN_SASL_HANDLER = "gremlin-sasl-handler";
protected static final String PIPELINE_GREMLIN_HANDLER = "gremlin-handler";
+ public static final String PIPELINE_SSL_HANDLER = "gremlin-ssl-handler";
public boolean supportsSsl() {
return cluster.connectionPoolSettings().enableSsl;
@@ -124,7 +126,12 @@
}
if (sslCtx.isPresent()) {
- pipeline.addLast(sslCtx.get().newHandler(socketChannel.alloc(), connection.getUri().getHost(), connection.getUri().getPort()));
+ SslHandler sslHandler = sslCtx.get().newHandler(socketChannel.alloc(), connection.getUri().getHost(), connection.getUri().getPort());
+ // TINKERPOP-2814. Remove the SSL handshake timeout so that handshakes that take longer than 10000ms
+ // (Netty default) but less than connectionSetupTimeoutMillis can succeed. This means the SSL handshake
+ // will instead be capped by connectionSetupTimeoutMillis.
+ sslHandler.setHandshakeTimeoutMillis(0);
+ pipeline.addLast(PIPELINE_SSL_HANDLER, sslHandler);
}
configure(pipeline);
@@ -187,7 +194,7 @@
new WebSocketClientHandler.InterceptedWebSocketClientHandshaker13(
connection.getUri(), WebSocketVersion.V13, null, true,
httpHeaders, maxContentLength, true, false, -1,
- cluster.getHandshakeInterceptor()), cluster.getConnectionSetupTimeout());
+ cluster.getHandshakeInterceptor()), cluster.getConnectionSetupTimeout(), supportsSsl());
final int keepAliveInterval = toIntExact(TimeUnit.SECONDS.convert(
cluster.connectionPoolSettings().keepAliveInterval, TimeUnit.MILLISECONDS));
diff --git a/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/Cluster.java b/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/Cluster.java
index 89f8190..18e0876 100644
--- a/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/Cluster.java
+++ b/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/Cluster.java
@@ -993,9 +993,6 @@
/**
* Sets the duration of time in milliseconds provided for connection setup to complete which includes WebSocket
* handshake and SSL handshake. Beyond this duration an exception would be thrown.
- *
- * Note that this value should be greater that SSL handshake timeout defined in
- * {@link io.netty.handler.ssl.SslHandler} since WebSocket handshake include SSL handshake.
*/
public Builder connectionSetupTimeoutMillis(final long connectionSetupTimeoutMillis) {
this.connectionSetupTimeoutMillis = connectionSetupTimeoutMillis;
diff --git a/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/Settings.java b/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/Settings.java
index f016433..c6550d0 100644
--- a/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/Settings.java
+++ b/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/Settings.java
@@ -407,9 +407,6 @@
* Duration of time in milliseconds provided for connection setup to complete which includes WebSocket
* handshake and SSL handshake. Beyond this duration an exception would be thrown if the handshake is not
* complete by then.
- *
- * Note that this value should be greater that SSL handshake timeout defined in
- * {@link io.netty.handler.ssl.SslHandler} since WebSocket handshake include SSL handshake.
*/
public long connectionSetupTimeoutMillis = Connection.CONNECTION_SETUP_TIMEOUT_MILLIS;
}
diff --git a/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/handler/WebSocketClientHandler.java b/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/handler/WebSocketClientHandler.java
index 21aba24..82ae2fe 100644
--- a/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/handler/WebSocketClientHandler.java
+++ b/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/handler/WebSocketClientHandler.java
@@ -18,6 +18,7 @@
*/
package org.apache.tinkerpop.gremlin.driver.handler;
+import io.netty.channel.Channel;
import io.netty.channel.ChannelFuture;
import io.netty.channel.ChannelHandlerContext;
import io.netty.channel.ChannelPromise;
@@ -28,12 +29,18 @@
import io.netty.handler.codec.http.websocketx.WebSocketClientProtocolHandler;
import io.netty.handler.codec.http.websocketx.WebSocketClientHandshaker;
import io.netty.handler.codec.http.websocketx.WebSocketVersion;
+import io.netty.handler.ssl.SslHandler;
+import io.netty.handler.ssl.SslHandshakeCompletionEvent;
import io.netty.handler.timeout.IdleState;
import io.netty.handler.timeout.IdleStateEvent;
+import io.netty.util.concurrent.Promise;
import java.net.URI;
import java.util.concurrent.TimeoutException;
+import javax.net.ssl.SSLHandshakeException;
+
+import org.apache.tinkerpop.gremlin.driver.Channelizer;
import org.apache.tinkerpop.gremlin.driver.Cluster;
import org.apache.tinkerpop.gremlin.driver.HandshakeInterceptor;
import org.slf4j.Logger;
@@ -48,10 +55,13 @@
private final long connectionSetupTimeoutMillis;
private ChannelPromise handshakeFuture;
+ private boolean sslHandshakeCompleted;
+ private boolean useSsl;
- public WebSocketClientHandler(final WebSocketClientHandshaker handshaker, final long timeoutMillis) {
+ public WebSocketClientHandler(final WebSocketClientHandshaker handshaker, final long timeoutMillis, final boolean useSsl) {
super(handshaker, /*handleCloseFrames*/true, /*dropPongFrames*/true, timeoutMillis);
this.connectionSetupTimeoutMillis = timeoutMillis;
+ this.useSsl = useSsl;
}
public ChannelFuture handshakeFuture() {
@@ -104,10 +114,21 @@
}
} else if (ClientHandshakeStateEvent.HANDSHAKE_TIMEOUT.equals(event)) {
if (!handshakeFuture.isDone()) {
- handshakeFuture.setFailure(
- new TimeoutException(String.format("handshake not completed in stipulated time=[%s]ms",
- connectionSetupTimeoutMillis)));
+ TimeoutException te = new TimeoutException(
+ String.format((useSsl && !sslHandshakeCompleted) ?
+ "SSL handshake not completed in stipulated time=[%s]ms" :
+ "WebSocket handshake not completed in stipulated time=[%s]ms",
+ connectionSetupTimeoutMillis));
+ handshakeFuture.setFailure(te);
+ logger.error(te.getMessage());
}
+
+ if (useSsl && !sslHandshakeCompleted) {
+ SslHandler handler = ((SslHandler) ctx.pipeline().get(Channelizer.AbstractChannelizer.PIPELINE_SSL_HANDLER));
+ ((Promise<Channel>) handler.handshakeFuture()).tryFailure(new SSLHandshakeException("SSL handshake timed out."));
+ }
+ } else if (event instanceof SslHandshakeCompletionEvent) {
+ sslHandshakeCompleted = true;
} else {
super.userEventTriggered(ctx, event);
}
diff --git a/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/simple/WebSocketClient.java b/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/simple/WebSocketClient.java
index 767f5a8..78f4268 100644
--- a/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/simple/WebSocketClient.java
+++ b/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/simple/WebSocketClient.java
@@ -69,7 +69,7 @@
try {
final WebSocketClientHandler wsHandler = new WebSocketClientHandler(WebSocketClientHandshakerFactory.newHandshaker(
- uri, WebSocketVersion.V13, null, true, EmptyHttpHeaders.INSTANCE, 65536), 10000);
+ uri, WebSocketVersion.V13, null, true, EmptyHttpHeaders.INSTANCE, 65536), 10000, false);
final MessageSerializer<GraphBinaryMapper> serializer = new GraphBinaryMessageSerializerV1();
b.channel(NioSocketChannel.class)
.handler(new ChannelInitializer<SocketChannel>() {
diff --git a/gremlin-driver/src/test/java/org/apache/tinkerpop/gremlin/driver/NoOpWebSocketServerHandler.java b/gremlin-driver/src/test/java/org/apache/tinkerpop/gremlin/driver/NoOpWebSocketServerHandler.java
new file mode 100644
index 0000000..8c22c72
--- /dev/null
+++ b/gremlin-driver/src/test/java/org/apache/tinkerpop/gremlin/driver/NoOpWebSocketServerHandler.java
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.tinkerpop.gremlin.driver;
+
+import io.netty.channel.ChannelHandlerContext;
+import io.netty.channel.ChannelInboundHandlerAdapter;
+import io.netty.handler.codec.http.HttpRequest;
+import io.netty.util.ReferenceCountUtil;
+
+/**
+* Handler that will drop requests to the WebSocket path.
+*/
+public class NoOpWebSocketServerHandler extends ChannelInboundHandlerAdapter {
+ private String websocketPath;
+
+ public NoOpWebSocketServerHandler(String websocketPath) {
+ this.websocketPath = websocketPath;
+ }
+
+ @Override
+ public void channelRead(ChannelHandlerContext ctx, Object msg) {
+ if ((msg instanceof HttpRequest) && ((HttpRequest) msg).uri().endsWith(websocketPath)) {
+ ReferenceCountUtil.release(msg);
+ }
+ }
+}
diff --git a/gremlin-driver/src/test/java/org/apache/tinkerpop/gremlin/driver/TestHttpServerInitializer.java b/gremlin-driver/src/test/java/org/apache/tinkerpop/gremlin/driver/TestHttpServerInitializer.java
new file mode 100644
index 0000000..9985b70
--- /dev/null
+++ b/gremlin-driver/src/test/java/org/apache/tinkerpop/gremlin/driver/TestHttpServerInitializer.java
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.tinkerpop.gremlin.driver;
+
+import io.netty.channel.ChannelInitializer;
+import io.netty.channel.ChannelPipeline;
+import io.netty.channel.socket.SocketChannel;
+import io.netty.handler.codec.http.HttpObjectAggregator;
+import io.netty.handler.codec.http.HttpServerCodec;
+
+/**
+ * A base ChannelInitializer that setups the pipeline for HTTP handling. This class should be sub-classed by a handler
+ * that handles the actual data being received.
+ */
+public class TestHttpServerInitializer extends ChannelInitializer<SocketChannel> {
+ protected static final String WEBSOCKET_PATH = "/gremlin";
+
+ @Override
+ public void initChannel(SocketChannel ch) {
+ final ChannelPipeline pipeline = ch.pipeline();
+ pipeline.addLast(new HttpServerCodec());
+ pipeline.addLast(new HttpObjectAggregator(65536));
+ }
+}
diff --git a/gremlin-driver/src/test/java/org/apache/tinkerpop/gremlin/driver/TestWSNoOpInitializer.java b/gremlin-driver/src/test/java/org/apache/tinkerpop/gremlin/driver/TestWSNoOpInitializer.java
new file mode 100644
index 0000000..268144a
--- /dev/null
+++ b/gremlin-driver/src/test/java/org/apache/tinkerpop/gremlin/driver/TestWSNoOpInitializer.java
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.tinkerpop.gremlin.driver;
+
+import io.netty.channel.socket.SocketChannel;
+
+/**
+ * An initializer that adds a handler that will drop WebSocket frames.
+ */
+public class TestWSNoOpInitializer extends TestHttpServerInitializer {
+
+ @Override
+ public void initChannel(SocketChannel ch) {
+ super.initChannel(ch);
+ ch.pipeline().addLast(new NoOpWebSocketServerHandler(WEBSOCKET_PATH));
+ }
+}
diff --git a/gremlin-driver/src/test/java/org/apache/tinkerpop/gremlin/driver/TestWebSocketServerInitializer.java b/gremlin-driver/src/test/java/org/apache/tinkerpop/gremlin/driver/TestWebSocketServerInitializer.java
index 7857ae9..adfa723 100644
--- a/gremlin-driver/src/test/java/org/apache/tinkerpop/gremlin/driver/TestWebSocketServerInitializer.java
+++ b/gremlin-driver/src/test/java/org/apache/tinkerpop/gremlin/driver/TestWebSocketServerInitializer.java
@@ -18,11 +18,8 @@
*/
package org.apache.tinkerpop.gremlin.driver;
-import io.netty.channel.ChannelInitializer;
import io.netty.channel.ChannelPipeline;
import io.netty.channel.socket.SocketChannel;
-import io.netty.handler.codec.http.HttpObjectAggregator;
-import io.netty.handler.codec.http.HttpServerCodec;
import io.netty.handler.codec.http.websocketx.WebSocketServerProtocolHandler;
import io.netty.handler.codec.http.websocketx.extensions.compression.WebSocketServerCompressionHandler;
@@ -30,14 +27,13 @@
* A vanilla WebSocket server Initializer implementation using Netty. This initializer would configure the server for
* WebSocket handshake and decoding incoming WebSocket frames.
*/
-public abstract class TestWebSocketServerInitializer extends ChannelInitializer<SocketChannel> {
- private static final String WEBSOCKET_PATH = "/gremlin";
+public abstract class TestWebSocketServerInitializer extends TestHttpServerInitializer {
@Override
public void initChannel(SocketChannel ch) {
+ super.initChannel(ch);
+
final ChannelPipeline pipeline = ch.pipeline();
- pipeline.addLast(new HttpServerCodec());
- pipeline.addLast(new HttpObjectAggregator(65536));
pipeline.addLast(new WebSocketServerCompressionHandler());
pipeline.addLast(new WebSocketServerProtocolHandler(WEBSOCKET_PATH, null, true));
this.postInit(ch.pipeline());
diff --git a/gremlin-driver/src/test/java/org/apache/tinkerpop/gremlin/driver/WebSocketClientBehaviorIntegrateTest.java b/gremlin-driver/src/test/java/org/apache/tinkerpop/gremlin/driver/WebSocketClientBehaviorIntegrateTest.java
index 2ce10c2..7537721 100644
--- a/gremlin-driver/src/test/java/org/apache/tinkerpop/gremlin/driver/WebSocketClientBehaviorIntegrateTest.java
+++ b/gremlin-driver/src/test/java/org/apache/tinkerpop/gremlin/driver/WebSocketClientBehaviorIntegrateTest.java
@@ -26,6 +26,7 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.log4j.Level;
+import org.apache.tinkerpop.gremlin.driver.exception.NoHostAvailableException;
import org.apache.tinkerpop.gremlin.driver.ser.Serializers;
import org.apache.tinkerpop.gremlin.structure.Vertex;
import org.apache.tinkerpop.gremlin.util.Log4jRecordingAppender;
@@ -41,6 +42,7 @@
import static org.hamcrest.MatcherAssert.assertThat;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
public class WebSocketClientBehaviorIntegrateTest {
@Rule
@@ -67,7 +69,12 @@
rootLogger.addAppender(recordingAppender);
server = new SimpleSocketServer();
- server.start(new TestWSGremlinInitializer());
+ if (name.getMethodName().equals("shouldAttemptHandshakeForLongerThanDefaultNettySslHandshakeTimeout") ||
+ name.getMethodName().equals("shouldPrintCorrectErrorForRegularWebSocketHandshakeTimeout")) {
+ server.start(new TestWSNoOpInitializer());
+ } else {
+ server.start(new TestWSGremlinInitializer());
+ }
}
@After
@@ -309,4 +316,59 @@
.filter(str -> str.contains("Considering new connection on"))
.count());
}
-}
\ No newline at end of file
+
+ /**
+ * (TINKERPOP-2814) Tests to make sure that the SSL handshake is now capped by connectionSetupTimeoutMillis and not
+ * the default Netty SSL handshake timeout of 10,000ms.
+ */
+ @Test
+ public void shouldAttemptHandshakeForLongerThanDefaultNettySslHandshakeTimeout() {
+ final Cluster cluster = Cluster.build("localhost").port(SimpleSocketServer.PORT)
+ .minConnectionPoolSize(1)
+ .maxConnectionPoolSize(1)
+ .connectionSetupTimeoutMillis(20000) // needs to be larger than 10000ms.
+ .enableSsl(true)
+ .create();
+
+ final Client.ClusteredClient client = cluster.connect();
+ final long start = System.currentTimeMillis();
+
+ Exception caught = null;
+ try {
+ client.submit("1");
+ } catch (Exception e) {
+ caught = e;
+ } finally {
+ // Test against 15000ms which should give a big enough buffer to avoid timing issues.
+ assertTrue(System.currentTimeMillis() - start > 15000);
+ assertTrue(caught != null);
+ assertTrue(caught instanceof NoHostAvailableException);
+ assertTrue(recordingAppender.getMessages().stream().anyMatch(str -> str.contains("SSL handshake not completed")));
+ }
+ }
+
+ /**
+ * Tests to make sure that the correct error message is logged when a non-SSL connection attempt times out.
+ */
+ @Test
+ public void shouldPrintCorrectErrorForRegularWebSocketHandshakeTimeout() {
+ final Cluster cluster = Cluster.build("localhost").port(SimpleSocketServer.PORT)
+ .minConnectionPoolSize(1)
+ .maxConnectionPoolSize(1)
+ .connectionSetupTimeoutMillis(100)
+ .create();
+
+ final Client.ClusteredClient client = cluster.connect();
+
+ Exception caught = null;
+ try {
+ client.submit("1");
+ } catch (Exception e) {
+ caught = e;
+ } finally {
+ assertTrue(caught != null);
+ assertTrue(caught instanceof NoHostAvailableException);
+ assertTrue(recordingAppender.getMessages().stream().anyMatch(str -> str.contains("WebSocket handshake not completed")));
+ }
+ }
+}
diff --git a/gremlin-javascript/src/main/javascript/gremlin-javascript/lib/driver/client.js b/gremlin-javascript/src/main/javascript/gremlin-javascript/lib/driver/client.js
index 811b1fe..4f64f7e 100644
--- a/gremlin-javascript/src/main/javascript/gremlin-javascript/lib/driver/client.js
+++ b/gremlin-javascript/src/main/javascript/gremlin-javascript/lib/driver/client.js
@@ -39,8 +39,13 @@
* @param {String} [options.traversalSource] The traversal source. Defaults to: 'g'.
* @param {GraphSONWriter} [options.writer] The writer to use.
* @param {Authenticator} [options.authenticator] The authentication handler to use.
+ * @param {Object} [options.headers] An associative array containing the additional header key/values for the initial request.
+ * @param {Boolean} [options.enableUserAgentOnConnect] Determines if a user agent will be sent during connection handshake. Defaults to: true
* @param {String} [options.processor] The name of the opProcessor to use, leave it undefined or set 'session' when session mode.
* @param {String} [options.session] The sessionId of Client in session mode. Defaults to null means session-less Client.
+ * @param {Boolean} [options.pingEnabled] Setup ping interval. Defaults to: true.
+ * @param {Number} [options.pingInterval] Ping request interval in ms if ping enabled. Defaults to: 60000.
+ * @param {Number} [options.pongTimeout] Timeout of pong response in ms after sending a ping. Defaults to: 30000.
* @constructor
*/
constructor(url, options = {}) {
diff --git a/gremlin-javascript/src/main/javascript/gremlin-javascript/lib/driver/connection.js b/gremlin-javascript/src/main/javascript/gremlin-javascript/lib/driver/connection.js
index cbbfb2e..acaeab4 100644
--- a/gremlin-javascript/src/main/javascript/gremlin-javascript/lib/driver/connection.js
+++ b/gremlin-javascript/src/main/javascript/gremlin-javascript/lib/driver/connection.js
@@ -64,6 +64,7 @@
* @param {GraphSONWriter} [options.writer] The writer to use.
* @param {Authenticator} [options.authenticator] The authentication handler to use.
* @param {Object} [options.headers] An associative array containing the additional header key/values for the initial request.
+ * @param {Boolean} [options.enableUserAgentOnConnect] Determines if a user agent will be sent during connection handshake. Defaults to: true
* @param {Boolean} [options.pingEnabled] Setup ping interval. Defaults to: true.
* @param {Number} [options.pingInterval] Ping request interval in ms if ping enabled. Defaults to: 60000.
* @param {Number} [options.pongTimeout] Timeout of pong response in ms after sending a ping. Defaults to: 30000.
@@ -98,6 +99,7 @@
this.isOpen = false;
this.traversalSource = options.traversalSource || 'g';
this._authenticator = options.authenticator;
+ this._enableUserAgentOnConnect = options.enableUserAgentOnConnect !== false;
this._pingEnabled = this.options.pingEnabled === false ? false : true;
this._pingIntervalDelay = this.options.pingInterval || pingIntervalDelay;
@@ -123,9 +125,16 @@
}
this.emit('log', 'ws open');
+ let headers = this.options.headers;
+ if (this._enableUserAgentOnConnect) {
+ if (!headers) {
+ headers = [];
+ }
+ headers[utils.getUserAgentHeader()] = utils.getUserAgent();
+ }
this._ws = new WebSocket(this.url, {
- headers: this.options.headers,
+ headers: headers,
ca: this.options.ca,
cert: this.options.cert,
pfx: this.options.pfx,
diff --git a/gremlin-javascript/src/main/javascript/gremlin-javascript/lib/driver/driver-remote-connection.js b/gremlin-javascript/src/main/javascript/gremlin-javascript/lib/driver/driver-remote-connection.js
index 05a9345..869d621 100644
--- a/gremlin-javascript/src/main/javascript/gremlin-javascript/lib/driver/driver-remote-connection.js
+++ b/gremlin-javascript/src/main/javascript/gremlin-javascript/lib/driver/driver-remote-connection.js
@@ -48,6 +48,10 @@
* @param {GraphSONWriter} [options.writer] The writer to use.
* @param {Authenticator} [options.authenticator] The authentication handler to use.
* @param {Object} [options.headers] An associative array containing the additional header key/values for the initial request.
+ * @param {Boolean} [options.enableUserAgentOnConnect] Determines if a user agent will be sent during connection handshake. Defaults to: true
+ * @param {Boolean} [options.pingEnabled] Setup ping interval. Defaults to: true.
+ * @param {Number} [options.pingInterval] Ping request interval in ms if ping enabled. Defaults to: 60000.
+ * @param {Number} [options.pongTimeout] Timeout of pong response in ms after sending a ping. Defaults to: 30000.
* @constructor
*/
constructor(url, options = {}) {
diff --git a/gremlin-javascript/src/main/javascript/gremlin-javascript/lib/utils.js b/gremlin-javascript/src/main/javascript/gremlin-javascript/lib/utils.js
index 6dd50ba..b7581d2 100644
--- a/gremlin-javascript/src/main/javascript/gremlin-javascript/lib/utils.js
+++ b/gremlin-javascript/src/main/javascript/gremlin-javascript/lib/utils.js
@@ -24,6 +24,8 @@
'use strict';
const crypto = require('crypto');
+const os = require('os');
+const gremlinVersion = require(__dirname + '/../package.json').version;
exports.toLong = function toLong(value) {
return new Long(value);
@@ -79,3 +81,25 @@
}
exports.ImmutableMap = ImmutableMap;
+
+function generateUserAgent() {
+ const applicationName = (process.env.npm_package_name || 'NotAvailable').replace('_', ' ');
+ const driverVersion = gremlinVersion.replace('_', ' ');
+ const runtimeVersion = process.version.replace(' ', '_');
+ const osName = os.platform().replace(' ', '_');
+ const osVersion = os.release().replace(' ', '_');
+ const cpuArch = process.arch.replace(' ', '_');
+ const userAgent = `${applicationName} Gremlin-Javascript.${driverVersion} ${runtimeVersion} ${osName}.${osVersion} ${cpuArch}`;
+
+ return userAgent;
+}
+
+exports.getUserAgentHeader = function getUserAgentHeader() {
+ return 'User-Agent';
+};
+
+const userAgent = generateUserAgent();
+
+exports.getUserAgent = function getUserAgent() {
+ return userAgent;
+};
diff --git a/gremlin-server/src/main/static/NOTICE b/gremlin-server/src/main/static/NOTICE
index 21ce164..7da4672 100644
--- a/gremlin-server/src/main/static/NOTICE
+++ b/gremlin-server/src/main/static/NOTICE
@@ -5,7 +5,7 @@
The Apache Software Foundation (http://www.apache.org/).
------------------------------------------------------------------------
-Apache Groovy 2.5.19 (AL ASF)
+Apache Groovy 2.5.15 (AL ASF)
------------------------------------------------------------------------
This product includes/uses ANTLR (http://www.antlr2.org/)
developed by Terence Parr 1989-2006
diff --git a/gremlin-tools/gremlin-benchmark/pom.xml b/gremlin-tools/gremlin-benchmark/pom.xml
index 2a05e6a..46a43dd 100644
--- a/gremlin-tools/gremlin-benchmark/pom.xml
+++ b/gremlin-tools/gremlin-benchmark/pom.xml
@@ -27,7 +27,7 @@
<artifactId>gremlin-benchmark</artifactId>
<name>Apache TinkerPop :: Gremlin Benchmark</name>
<properties>
- <jmh.version>1.21</jmh.version>
+ <jmh.version>1.36</jmh.version>
<!-- Skip benchmarks by default because they are time consuming. -->
<skipBenchmarks>true</skipBenchmarks>
<skipTests>${skipBenchmarks}</skipTests>
diff --git a/pom.xml b/pom.xml
index 27767f5..87b8700 100644
--- a/pom.xml
+++ b/pom.xml
@@ -158,7 +158,10 @@
<commons.lang3.version>3.11</commons.lang3.version>
<commons.text.version>1.10.0</commons.text.version>
<exp4j.version>0.4.8</exp4j.version>
- <groovy.version>2.5.19</groovy.version>
+ <!-- performance after 2.5.15 is similar to the poor performance of 3.x and 4.x - we can't upgrade past this
+ version without a accepting a major performance hit. details related to this issue along with links
+ to attempts to solve the problem with the Groovy community can be found on: TINKERPOP-2373 -->
+ <groovy.version>2.5.15</groovy.version>
<hadoop.version>2.7.7</hadoop.version>
<hamcrest.version>2.2</hamcrest.version>
<java.tuples.version>1.2</java.tuples.version>
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
index 9b7a012..62171ae 100644
--- a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
+++ b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
@@ -51,6 +51,7 @@
import org.apache.tinkerpop.gremlin.process.computer.MapReduce;
import org.apache.tinkerpop.gremlin.process.computer.Memory;
import org.apache.tinkerpop.gremlin.process.computer.VertexProgram;
+import org.apache.tinkerpop.gremlin.process.computer.clone.CloneVertexProgram;
import org.apache.tinkerpop.gremlin.process.computer.util.DefaultComputerResult;
import org.apache.tinkerpop.gremlin.process.computer.util.MapMemory;
import org.apache.tinkerpop.gremlin.process.traversal.TraversalStrategies;
@@ -59,6 +60,7 @@
import org.apache.tinkerpop.gremlin.spark.process.computer.traversal.strategy.SparkVertexProgramInterceptor;
import org.apache.tinkerpop.gremlin.spark.process.computer.traversal.strategy.optimization.SparkInterceptorStrategy;
import org.apache.tinkerpop.gremlin.spark.process.computer.traversal.strategy.optimization.SparkSingleIterationStrategy;
+import org.apache.tinkerpop.gremlin.spark.process.computer.traversal.strategy.optimization.interceptor.SparkCloneVertexProgramInterceptor;
import org.apache.tinkerpop.gremlin.spark.structure.Spark;
import org.apache.tinkerpop.gremlin.spark.structure.io.InputFormatRDD;
import org.apache.tinkerpop.gremlin.spark.structure.io.InputOutputHelper;
@@ -369,6 +371,12 @@
////////////////////////////////
if (null != this.vertexProgram) {
memory = new SparkMemory(this.vertexProgram, this.mapReducers, sparkContext);
+ // build a shortcut (which reduces the total Spark stages from 3 to 2) for CloneVertexProgram since it does nothing
+ // and this improves the overall performance a lot
+ if (this.vertexProgram.getClass().equals(CloneVertexProgram.class) &&
+ !graphComputerConfiguration.containsKey(Constants.GREMLIN_HADOOP_VERTEX_PROGRAM_INTERCEPTOR)) {
+ graphComputerConfiguration.setProperty(Constants.GREMLIN_HADOOP_VERTEX_PROGRAM_INTERCEPTOR, SparkCloneVertexProgramInterceptor.class.getName());
+ }
/////////////////
// if there is a registered VertexProgramInterceptor, use it to bypass the GraphComputer semantics
if (graphComputerConfiguration.containsKey(Constants.GREMLIN_HADOOP_VERTEX_PROGRAM_INTERCEPTOR)) {
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/traversal/strategy/optimization/interceptor/SparkCloneVertexProgramInterceptor.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/traversal/strategy/optimization/interceptor/SparkCloneVertexProgramInterceptor.java
new file mode 100644
index 0000000..8144032
--- /dev/null
+++ b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/traversal/strategy/optimization/interceptor/SparkCloneVertexProgramInterceptor.java
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.tinkerpop.gremlin.spark.process.computer.traversal.strategy.optimization.interceptor;
+
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
+import org.apache.tinkerpop.gremlin.process.computer.VertexProgram;
+import org.apache.tinkerpop.gremlin.spark.process.computer.SparkMemory;
+import org.apache.tinkerpop.gremlin.spark.process.computer.traversal.strategy.SparkVertexProgramInterceptor;
+
+public class SparkCloneVertexProgramInterceptor implements SparkVertexProgramInterceptor<VertexProgram> {
+ @Override
+ public JavaPairRDD<Object, VertexWritable> apply(VertexProgram vertexProgram, JavaPairRDD<Object, VertexWritable> graph, SparkMemory memory) {
+ // bypass the VertexProgram since CloneVertexProgram does nothing in its execute method
+ return graph;
+ }
+}
diff --git a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/traversal/strategy/optimization/interceptor/SparkCloneVertexProgramInterceptorTest.java b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/traversal/strategy/optimization/interceptor/SparkCloneVertexProgramInterceptorTest.java
new file mode 100644
index 0000000..cd57d42
--- /dev/null
+++ b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/traversal/strategy/optimization/interceptor/SparkCloneVertexProgramInterceptorTest.java
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.tinkerpop.gremlin.spark.process.computer.traversal.strategy.optimization.interceptor;
+
+import org.apache.commons.configuration2.Configuration;
+import org.apache.hadoop.mapreduce.lib.output.NullOutputFormat;
+import org.apache.tinkerpop.gremlin.TestHelper;
+import org.apache.tinkerpop.gremlin.hadoop.Constants;
+import org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoInputFormat;
+import org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoOutputFormat;
+import org.apache.tinkerpop.gremlin.process.computer.clone.CloneVertexProgram;
+import org.apache.tinkerpop.gremlin.spark.AbstractSparkTest;
+import org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer;
+import org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializerIntegrateTest;
+import org.apache.tinkerpop.gremlin.structure.Graph;
+import org.apache.tinkerpop.gremlin.structure.io.IoCore;
+import org.apache.tinkerpop.gremlin.structure.util.GraphFactory;
+import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph;
+import org.junit.Test;
+
+import java.util.UUID;
+
+import static org.junit.Assert.*;
+
+/**
+ * This class verify the default exporting behavior which requires the CloneVertexProgram's execute() is empty,
+ * then the SparkCloneVertexProgramInterceptor will bypass the VertexProgram.
+ */
+public class SparkCloneVertexProgramInterceptorTest extends AbstractSparkTest {
+ @Test
+ public void shouldExportCorrectGraph() throws Exception {
+ // Build the random graph
+ final TinkerGraph randomGraph = TinkerGraph.open();
+ final int totalVertices = 200000;
+ TestHelper.createRandomGraph(randomGraph, totalVertices, 100);
+ final String inputLocation = TestHelper.makeTestDataFile(GryoSerializerIntegrateTest.class,
+ UUID.randomUUID().toString(),
+ "random-graph.kryo");
+ randomGraph.io(IoCore.gryo()).writeGraph(inputLocation);
+ randomGraph.clear();
+ randomGraph.close();
+
+ // Serialize the graph to disk by CloneVertexProgram + SparkGraphComputer
+ final String outputLocation = TestHelper.makeTestDataDirectory(GryoSerializerIntegrateTest.class, UUID.randomUUID().toString());
+ Configuration configuration = getBaseConfiguration();
+ configuration.clearProperty(Constants.SPARK_SERIALIZER); // ensure proper default to GryoSerializer
+ configuration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, inputLocation);
+ configuration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_READER, GryoInputFormat.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_WRITER, GryoOutputFormat.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, outputLocation);
+ configuration.setProperty(Constants.GREMLIN_SPARK_PERSIST_CONTEXT, false);
+ Graph graph = GraphFactory.open(configuration);
+ graph.compute(SparkGraphComputer.class).program(CloneVertexProgram.build().create()).submit().get();
+
+ // Read the total Vertex/Edge count for golden reference through the original graph
+ configuration = getBaseConfiguration();
+ configuration.clearProperty(Constants.SPARK_SERIALIZER); // ensure proper default to GryoSerializer
+ configuration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, inputLocation);
+ configuration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_READER, GryoInputFormat.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_WRITER, NullOutputFormat.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_SPARK_PERSIST_CONTEXT, false);
+ graph = GraphFactory.open(configuration);
+ long totalVRef = graph.traversal().withComputer(SparkGraphComputer.class).V().count().next().longValue();
+ long totalERef = graph.traversal().withComputer(SparkGraphComputer.class).E().count().next().longValue();
+
+ assertEquals(totalVRef, totalVertices);
+ // Read the total Vertex/Edge count from the exported graph
+ configuration = getBaseConfiguration();
+ configuration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, outputLocation);
+ configuration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_READER, GryoInputFormat.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_WRITER, NullOutputFormat.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_STORAGE_LEVEL, "MEMORY_ONLY"); // this should be ignored as you can't change the persistence level once created
+ configuration.setProperty(Constants.GREMLIN_SPARK_PERSIST_STORAGE_LEVEL, "MEMORY_AND_DISK");
+ configuration.setProperty(Constants.GREMLIN_SPARK_PERSIST_CONTEXT, true);
+ graph = GraphFactory.open(configuration);
+ long totalV = graph.traversal().withComputer(SparkGraphComputer.class).V().count().next().longValue();
+ long totalE = graph.traversal().withComputer(SparkGraphComputer.class).E().count().next().longValue();
+ assertEquals(totalV, totalVRef);
+ assertEquals(totalE, totalERef);
+ }
+}
