layout: global title: Setting up Spark Connect license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Spark Connect ships with every Spark release since 3.4 and supports PySpark and Scala applications. For what Spark Connect is and how it differs from classic Spark, see the Spark Connect Overview.
First, download Spark from the Download Apache Spark page. Choose the latest release in the release drop down at the top of the page. Then choose your package type, typically “Pre-built for Apache Hadoop 3.5 and later”, and click the link to download.
Now extract the Spark package you just downloaded on your computer, for example:
tar -xvf spark-{{site.SPARK_VERSION_SHORT}}-bin-hadoop3.tgz
In a terminal window, go to the spark folder in the location where you extracted Spark before and run the start-connect-server.sh script to start Spark server with Spark Connect. If you already have Spark installed and SPARK_HOME defined, you can use that too.
./sbin/start-connect-server.sh # alternately "$SPARK_HOME/sbin/start-connect-server.sh"
Alternatively, ./bin/spark-connect-shell starts an interactive Scala shell with the Connect server hosted inside the shell process itself.
The script accepts the same options as spark-submit, so --master, --jars, --packages and --conf work as they do for any other Spark application. Everything that the server reads at startup, such as spark.sql.extensions, spark.jars, spark.jars.packages and the JARs that catalogs load, is set here. Clients can only change runtime configuration of their own session; that includes defining a catalog (spark.sql.catalog.<name>) whose JAR is already on the server.
By default the server listens on port 15002 on all interfaces and runs as a background daemon (./sbin/stop-connect-server.sh stops it). Pass --wait as the first argument to keep it in the foreground instead, for example as a container entrypoint or under a process supervisor. With --wait, the script sets SPARK_NO_DAEMONIZE=1, writes no PID file and sends the logs to standard output. spark.connect.grpc.binding.address restricts the listening interface and spark.connect.grpc.binding.port changes the port:
./sbin/start-connect-server.sh --wait \ --conf spark.connect.grpc.binding.address=127.0.0.1 \ --conf spark.connect.grpc.binding.port=15002
Spark does not retry another port when the configured one is taken (spark.connect.grpc.port.maxRetries defaults to 0), so a port conflict fails the server with a java.net.BindException instead of silently moving it. A daemonized server logs to logs/ under the Spark folder (or SPARK_LOG_DIR), and the Spark UI of the server, at port 4040 by default, has a Connect tab listing sessions and requests.
If you do not use one of the mechanisms outlined here, your Spark session will work just like before, without leveraging Spark Connect.
If you set the SPARK_REMOTE environment variable on the client machine where your Spark client application is running and create a new Spark Session as in the following example, the session will be a Spark Connect session. With this approach, there is no code change needed to start using Spark Connect.
In a terminal window, set the SPARK_REMOTE environment variable to point to the local Spark server you started previously on your computer:
export SPARK_REMOTE="sc://localhost"
And start the Spark shell as usual:
./bin/pyspark
The PySpark shell is now connected to Spark using Spark Connect as indicated in the welcome message:
Client connected to the Spark Connect server at localhost
You can also specify that you want to use Spark Connect explicitly when you create a Spark session.
For example, you can launch the PySpark shell with Spark Connect as illustrated here.
To launch the PySpark shell with Spark Connect, simply include the remote parameter and specify the location of your Spark server. We are using localhost in this example to connect to the local Spark server we started previously:
./bin/pyspark --remote "sc://localhost"
And you will notice that the PySpark shell welcome message tells you that you have connected to Spark using Spark Connect:
Client connected to the Spark Connect server at localhost
You can also check the Spark session type. If it includes .connect. you are using Spark Connect as shown in this example:
SparkSession available as 'spark'. >>> type(spark) <class 'pyspark.sql.connect.session.SparkSession'>
Now you can run PySpark code in the shell to see Spark Connect in action:
>>> columns = ["id", "name"] >>> data = [(1,"Sarah"), (2,"Maria")] >>> df = spark.createDataFrame(data).toDF(*columns) >>> df.show() +---+-----+ | id| name| +---+-----+ | 1|Sarah| | 2|Maria| +---+-----+
./bin/spark-shell --remote "sc://localhost"
A greeting message will appear when the REPL successfully initializes:
Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version {{site.SPARK_VERSION}} /_/ Type in expressions to have them evaluated. Spark session available as 'spark'.
By default, the REPL will attempt to connect to a local Spark Server. Run the following Scala code in the shell to see Spark Connect in action:
@ spark.range(10).count res0: Long = 10L
By default, the shells and the remote() builder attempt to connect to a local Spark Server on port 15002. The connection, however, may be configured in several ways as described in this configuration reference. A connection string has the form sc://host:port/;param1=value;param2=value: the scheme is always sc://, the path component must stay empty because gRPC uses the HTTP/2 path for its own method names, and parameters such as token, use_ssl, user_id and user_agent follow a /; separator. Setting token turns on TLS for any host except localhost; to use TLS with localhost, set use_ssl=true as well.
In the Python client, a parameter value cannot contain a literal /, because its parser reads everything from the first / as the path:
SparkSession.builder.remote("sc://localhost:15002/;user_agent=myapp/1").getOrCreate() # [INVALID_CONNECT_URL] Invalid URL for Spark Connect: # The path component '/;user_agent=myapp/1' must be empty.
Write the / as %2F instead: user_agent=myapp%2F1 is accepted and read as myapp/1.
The SPARK_REMOTE environment variable can be set on the client machine to customize the client-server connection that is initialized at REPL startup.
export SPARK_REMOTE="sc://example.com:443/;token=ABCDEFG" ./bin/spark-shell
The connection may also be programmatically created using SparkSession#builder as in this example:
When you develop or test locally with
from pyspark.sql import SparkSession spark = SparkSession.builder.remote("local[*]").getOrCreate()
PySpark boots a fresh in-process Spark Connect server that lives only as long as that Python process. Every python script.py invocation (or every new test worker process) therefore re-pays the one-time startup cost -- JVM warmup, SparkContext construction, and Connect server boot -- which can take a few seconds and makes a quick edit/run loop feel slow.
The in-process server binds port 15002 by default and the client dials sc://localhost, so if another Spark Connect server is already listening on that port (a container, or a server left running from an earlier session) the start fails with java.net.BindException: Failed to bind to address 0.0.0.0/0.0.0.0:15002. Moving the port with spark.connect.grpc.binding.port does not help here, because the client would still dial 15002 and reach the other server; stop the other server or connect to it by URL instead.
To amortize that cost across runs, start one persistent local Spark Connect server and point every run at it:
# Start once; it stays up across runs. (--master is optional; it defaults to local[*].) $SPARK_HOME/sbin/start-connect-server.sh --master "local[*]" # Every run reconnects instead of booting a new server. python -c 'from pyspark.sql import SparkSession; SparkSession.builder.remote("sc://localhost:15002").getOrCreate()' # Stop it when you are done. $SPARK_HOME/sbin/stop-connect-server.sh
Alternatively, on POSIX systems PySpark can manage this persistent server for you. With SPARK_LOCAL_CONNECT_REUSE=1 set (or spark.local.connect.reuse=true on the builder), SparkSession.builder.remote("local[*]").getOrCreate() starts a persistent server through sbin/start-connect-server.sh on the first run and reconnects to it on later runs, so scripts keep the plain local[*] URL:
export SPARK_LOCAL_CONNECT_REUSE=1 # The first run starts the server; later runs reconnect to it. python -c 'from pyspark.sql import SparkSession; SparkSession.builder.remote("local[*]").getOrCreate()' # Stop the managed server when you are done. python -m pyspark.sql.connect.local_server --stop
The managed server is an ordinary spark-daemon.sh daemon, but it runs with a per-user pid directory and ident string so it cannot collide with a server you started by hand -- which also means a plain sbin/stop-connect-server.sh does not find it. The --stop command signals the recorded server and cleans up the discovery file; killing the server's pid directly also works, and the next run notices the dead server and starts a fresh one.
This managed-server workflow is experimental. The --stop command and the discovery file location and format may change in a future release, for example if local server management is folded into a unified spark connect CLI.
The connection details (host, port, auth token, pid, Spark version) are recorded in a discovery file in a private per-user directory; set SPARK_LOCAL_CONNECT_DISCOVERY to override its location. A run reconnects only to a server whose Spark version matches. After upgrading Spark, the server from the previous version cannot be reused and the next run fails with an error asking you to stop it; run the --stop command above and rerun to start a fresh server.
Each run connects as its own Connect session, so session-local state -- temp views, runtime SQL configurations, and session artifacts -- is fresh on every run and never leaks between runs. State backed by the shared SparkContext (the persistent catalog/warehouse, global temp views, and cached datasets) is shared across runs, so namespace per-run databases or clear that state yourself if your runs must be fully isolated.
First, install PySpark with pip install pyspark-client=={{site.SPARK_VERSION_SHORT}} or if building a packaged PySpark application/library, add it your setup.py file as:
install_requires=[ 'pyspark-client=={{site.SPARK_VERSION_SHORT}}' ]
When writing your own code, include the remote function with a reference to your Spark server when you create a Spark session, as in this example:
from pyspark.sql import SparkSession spark = SparkSession.builder.remote("sc://localhost").getOrCreate()
For illustration purposes, we’ll create a simple Spark Connect application, SimpleApp.py:
"""SimpleApp.py""" from pyspark.sql import SparkSession logFile = "YOUR_SPARK_HOME/README.md" # a path the server can read spark = SparkSession.builder.remote("sc://localhost").appName("SimpleApp").getOrCreate() logData = spark.read.text(logFile).cache() numAs = logData.filter(logData.value.contains('a')).count() numBs = logData.filter(logData.value.contains('b')).count() print("Lines with a: %i, lines with b: %i" % (numAs, numBs)) spark.stop()
This program just counts the number of lines containing ‘a’ and the number containing ‘b’ in a text file. Note that you’ll need to replace YOUR_SPARK_HOME with the location where Spark is installed.
We can run this application with the regular Python interpreter as follows:
# Use the Python interpreter to run your application $ python SimpleApp.py ... Lines with a: 72, lines with b: 39
Depend on spark-connect-client-jvm only, and start the JVM with --add-opens=java.base/java.nio=ALL-UNNAMED.
When writing your own code, include the remote function with a reference to your Spark server when you create a Spark session, as in this example:
import org.apache.spark.sql.SparkSession val spark = SparkSession.builder().remote("sc://localhost").getOrCreate()
Note: Operations that reference User Defined Code such as UDFs, filter, map, etc require a ClassFinder to be registered to pickup and upload any required classfiles. Also, any JAR dependencies must be uploaded to the server using SparkSession#AddArtifact.
Example:
import org.apache.spark.sql.connect.client.REPLClassDirMonitor // Register a ClassFinder to monitor and upload the classfiles from the build output. val classFinder = new REPLClassDirMonitor(<ABSOLUTE_PATH_TO_BUILD_OUTPUT_DIR>) spark.registerClassFinder(classFinder) // Upload JAR dependencies spark.addArtifact(<ABSOLUTE_PATH_JAR_DEP>)
Here, ABSOLUTE_PATH_TO_BUILD_OUTPUT_DIR is the output directory where the build system writes classfiles into and ABSOLUTE_PATH_JAR_DEP is the location of the JAR on the local file system.
The REPLClassDirMonitor is a provided implementation of ClassFinder that monitors a specific directory but one may implement their own class extending ClassFinder for customized search and monitoring.
For extending Spark Connect with custom server-side functionality, see Extending Spark with Spark Server Libraries.
There are three ways to get a Spark Connect session, depending on where the server runs:
sc:// connection string, given as spark.remote, --remote or the remote() argument, connects to a server that is already running: a shared cluster's Spark Connect endpoint, or the persistent local server during development.local[...] or local-cluster[...] value of spark.remote or --remote starts a local Spark Connect server for the application and connects to it. Use it for local development and tests.--master ... together with spark.api.mode=connect starts a Spark Connect server in the application's driver, running on that master, and runs the application against it. It accepts the same master URLs as Spark Classic, such as spark://, so an existing Spark Classic application switches to Spark Connect without code changes.Depending on the value of spark.api.mode, the application runs in either Spark Classic or Spark Connect mode. Set it in the session builder:
from pyspark.sql import SparkSession SparkSession.builder.config("spark.api.mode", "connect").master("...").getOrCreate()
or when submitting a Scala or PySpark application:
spark-submit --master "..." --conf spark.api.mode=connect
Spark Connect has no user-facing authentication of its own. Its gRPC HTTP/2 interface allows for the use of authenticating proxies, which makes it possible to secure Spark Connect without having to implement authentication logic in Spark directly.
When several services share a single hostname behind a Kubernetes Ingress (or another reverse proxy), it is tempting to give each service a URL path prefix (for example sc://host/sparkConnect) and route on that path. This does not work for gRPC: the gRPC method name is the HTTP/2 :path (/spark.connect.SparkConnectService/ExecutePlan), so a connection string cannot carry a separate routing path. Prepending a prefix to the :path produces an unknown method and the server responds with UNIMPLEMENTED, unless the proxy is configured to strip the prefix back off before forwarding, a two-sided contract that is not part of gRPC's design.
The routing dimension that is free is the HTTP/2 :authority (the virtual host). Set it to a routing tag with the grpc.default_authority channel option, and route on it at the proxy (for example, an Ingress host: rule). The client still dials the shared hostname (default_authority only overrides the :authority header used for routing, not the address it connects to), and the gRPC method :path is never touched, so no path rewrite is needed. This is the approach the gRPC maintainers recommend for this scenario (grpc/grpc#14900).
The example below uses the Python client, which exposes gRPC channel options directly:
from pyspark.sql.connect.session import SparkSession from pyspark.sql.connect.client import DefaultChannelBuilder cb = DefaultChannelBuilder("sc://example.com:443") cb.setChannelOption("grpc.default_authority", "sparkconnect") # routing tag spark = SparkSession.builder.channelBuilder(cb).getOrCreate()
A corresponding Kubernetes Ingress routes on that tag and keeps the service at path / (no subpath, no rewrite). Note the routing tag must be a lowercase RFC 1123 name, since a Kubernetes Ingress host: requires one:
spec: rules: - host: sparkconnect # matches grpc.default_authority http: paths: - path: / pathType: Prefix backend: service: name: spark-connect-server port: { number: 15002 }
Over TLS, gRPC uses the :authority as the certificate-verification name by default, so overriding it with a routing tag would fail verification (the tag is not in the server certificate's SAN). Keep the two names separate: set grpc.ssl_target_name_override to the real server hostname (used for certificate verification and SNI) and grpc.default_authority to the routing tag.
cb = DefaultChannelBuilder("sc://example.com:443/;use_ssl=true") cb.setChannelOption("grpc.ssl_target_name_override", "example.com") # certificate verification / SNI cb.setChannelOption("grpc.default_authority", "sparkconnect") # routing tag spark = SparkSession.builder.channelBuilder(cb).getOrCreate()
use_ssl=true verifies the server certificate against the system‘s trusted CA store. If your gateway’s certificate is issued by a CA the client does not trust by default (a self-signed or internal CA), make that CA trusted on the client side (for example, via GRPC_DEFAULT_SSL_ROOTS_FILE_PATH) rather than through the connection string. Configuring the proxy's own TLS (which certificate it presents for which hostname) is part of your ingress setup and is out of scope here.
Two notes on grpc.ssl_target_name_override: gRPC documents it as testing-oriented because its typical misuse is to mask a certificate name mismatch (verifying against a name the server does not actually present, which defeats hostname verification). Here it is set to the real, verified hostname, so the certificate is still checked correctly; it only prevents the routing tag from being used as the verification name. If you prefer to avoid the option entirely, issue the server certificate with the routing tag included in its SAN; then grpc.default_authority alone is sufficient and no override is needed.